logicsrc/apps/logicsrc-web/src/proxy.ts
Anthony Ettinger 93af4770ac
Charge AI training crawlers for access (x402 gateway) (#141)
* Charge AI training crawlers for access (@profullstack/x402-gateway)

Training crawlers (GPTBot, ClaudeBot, CCBot, meta-externalagent, Bytespider,
Applebot-Extended) get 402 Payment Required with an x402 offer, or the sales
page at /crawl, and a paid pass opens the site for a day. People, search
engines and retrieval crawlers pass through untouched. robots.txt is now
generated from the same lists.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YafYxayh7Gqe5MWNNQMev2

* Type the middleware as returning Response | NextResponse

The crawl gateway answers with a plain Fetch Response.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YafYxayh7Gqe5MWNNQMev2

* Contract test awaits the now-async proxy

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YafYxayh7Gqe5MWNNQMev2

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-05 14:49:39 -07:00

28 lines
1.2 KiB
TypeScript

import { gate } from "@/lib/crawl-gateway";
import { NextResponse } from "next/server";
import type { NextRequest } from "next/server";
// Canonical host: 301 www.* to the bare apex domain over https, preserving
// path + query (e.g. https://www.logicsrc.com/foo -> https://logicsrc.com/foo).
// This is the Next 16 "proxy" (formerly middleware) entrypoint.
const ALLOWED_APEX = process.env.PUBLIC_DOMAIN || "logicsrc.com";
export async function proxy(request: NextRequest): Promise<Response | NextResponse> {
// Crawl gateway first: AI training crawlers get 402 Payment Required (or the
// sales page at /crawl) unless they present a paid pass. People, Googlebot
// and retrieval crawlers fall through to everything below.
const answer = await gate(request);
if (answer) return answer;
const host = request.headers.get("host") ?? "";
if (host === `www.${ALLOWED_APEX}`) {
const { pathname, search } = request.nextUrl;
return NextResponse.redirect(`https://${ALLOWED_APEX}${pathname}${search}`, 301);
}
return NextResponse.next();
}
export const config = {
// Run on everything except Next's static assets and the favicon.
matcher: ["/((?!_next/static|_next/image|favicon.ico).*)"]
};