bbs.profullstack/logicsrc
1
0
Fork 0
mirror of https://github.com/profullstack/logicsrc.git synced 2026-09-10 19:26:00 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

Charge AI training crawlers for access (x402 gateway) (#141)

Browse source 
* Charge AI training crawlers for access (@profullstack/x402-gateway)

Training crawlers (GPTBot, ClaudeBot, CCBot, meta-externalagent, Bytespider,
Applebot-Extended) get 402 Payment Required with an x402 offer, or the sales
page at /crawl, and a paid pass opens the site for a day. People, search
engines and retrieval crawlers pass through untouched. robots.txt is now
generated from the same lists.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YafYxayh7Gqe5MWNNQMev2

* Type the middleware as returning Response | NextResponse

The crawl gateway answers with a plain Fetch Response.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YafYxayh7Gqe5MWNNQMev2

* Contract test awaits the now-async proxy

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YafYxayh7Gqe5MWNNQMev2

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
Anthony Ettinger 2026-09-05 14:49:39 -07:00 • committed by GitHub
parent 80a36269bb
commit 93af4770ac
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
8 changed files with 65 additions and 35 deletions
Download patch file Download diff file Expand all files Collapse all files

27
apps/logicsrc-web/src/lib/crawl-gateway.ts Normal file
View file

@ -0,0 +1,27 @@
import { createGateway } from "@profullstack/x402-gateway";
import { x402Proxy } from "@profullstack/x402-gateway/next";
/**
* Sells crawl access to AI training crawlers (GPTBot, ClaudeBot, CCBot,
* meta-externalagent, Bytespider, Applebot-Extended, ...) by the day over
* x402, settled by CoinPay in USDC. People, Googlebot and the retrieval
* crawlers behind AI search pass through untouched.
*
* Runs inside the middleware, so nothing here may import Node-only modules.
* The env is read through a non-literal key on purpose: Next inlines
* `process.env.NAME` at build time, and these are runtime secrets. Without
* COINPAY_X402_KEY and CRAWL_PAY_TO the gateway still answers training
* crawlers with 402, just with an empty offer.
*/
const env = (name: string) => process.env[name];
export const gateway = createGateway({
siteUrl: env("SITE_URL") || env("NEXT_PUBLIC_SITE_URL") || "https://logicsrc.com",
siteName: "LogicSRC",
coinpay: { apiKey: env("COINPAY_X402_KEY") },
payTo: env("CRAWL_PAY_TO"),
contact: "mailto:support@logicsrc.com",
});
/** Resolves to a Response for a refused crawler, or undefined to carry on. */
export const gate = x402Proxy(gateway);