Charge AI training crawlers for access (@profullstack/x402-gateway)

Training crawlers (GPTBot, ClaudeBot, CCBot, meta-externalagent, Bytespider,
Applebot-Extended) get 402 Payment Required with an x402 offer, or the sales
page at /crawl, and a paid pass opens the site for a day. People, search
engines and retrieval crawlers pass through untouched. robots.txt is now
generated from the same lists.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YafYxayh7Gqe5MWNNQMev2
This commit is contained in:
Anthony Ettinger 2026-09-05 21:30:50 +00:00
parent 80a36269bb
commit 071acf6fff
7 changed files with 61 additions and 31 deletions

View file

@ -16,6 +16,7 @@
"@logicsrc/openprd": "file:../../packages/openprd",
"@profullstack/autoblog": "github:profullstack/autoblog#75e54af",
"@profullstack/stack": "^0.1.3",
"@profullstack/x402-gateway": "^0.1.0",
"@supabase/supabase-js": "^2.105.4",
"marked": "^18.0.5",
"next": "16.2.6",