bbs.profullstack/logicsrc
1
0
Fork 0
mirror of https://github.com/profullstack/logicsrc.git synced 2026-10-03 13:17:07 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

Charge AI training crawlers for access (x402 gateway) (#141)

Browse source 
* Charge AI training crawlers for access (@profullstack/x402-gateway)

Training crawlers (GPTBot, ClaudeBot, CCBot, meta-externalagent, Bytespider,
Applebot-Extended) get 402 Payment Required with an x402 offer, or the sales
page at /crawl, and a paid pass opens the site for a day. People, search
engines and retrieval crawlers pass through untouched. robots.txt is now
generated from the same lists.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YafYxayh7Gqe5MWNNQMev2

* Type the middleware as returning Response | NextResponse

The crawl gateway answers with a plain Fetch Response.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YafYxayh7Gqe5MWNNQMev2

* Contract test awaits the now-async proxy

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YafYxayh7Gqe5MWNNQMev2

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
Anthony Ettinger 2026-09-05 14:49:39 -07:00 • committed by GitHub
parent 80a36269bb
commit 93af4770ac
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
8 changed files with 65 additions and 35 deletions
Download patch file Download diff file Expand all files Collapse all files

9
apps/logicsrc-web/src/proxy.ts
View file

@ -1,3 +1,4 @@
import { gate } from "@/lib/crawl-gateway";
import { NextResponse } from "next/server";
import type { NextRequest } from "next/server";
@ -6,7 +7,13 @@ import type { NextRequest } from "next/server";
// This is the Next 16 "proxy" (formerly middleware) entrypoint.
const ALLOWED_APEX = process.env.PUBLIC_DOMAIN || "logicsrc.com";
export function proxy(request: NextRequest): NextResponse {
export async function proxy(request: NextRequest): Promise<Response | NextResponse> {
// Crawl gateway first: AI training crawlers get 402 Payment Required (or the
// sales page at /crawl) unless they present a paid pass. People, Googlebot
// and retrieval crawlers fall through to everything below.
const answer = await gate(request);
if (answer) return answer;
const host = request.headers.get("host") ?? "";
if (host === `www.${ALLOWED_APEX}`) {
const { pathname, search } = request.nextUrl;