bbs.profullstack/logicsrc
1
0
Fork 0
mirror of https://github.com/profullstack/logicsrc.git synced 2026-09-10 19:26:00 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

Charge AI training crawlers for access (x402 gateway) (#141)

Browse source 
* Charge AI training crawlers for access (@profullstack/x402-gateway)

Training crawlers (GPTBot, ClaudeBot, CCBot, meta-externalagent, Bytespider,
Applebot-Extended) get 402 Payment Required with an x402 offer, or the sales
page at /crawl, and a paid pass opens the site for a day. People, search
engines and retrieval crawlers pass through untouched. robots.txt is now
generated from the same lists.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YafYxayh7Gqe5MWNNQMev2

* Type the middleware as returning Response | NextResponse

The crawl gateway answers with a plain Fetch Response.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YafYxayh7Gqe5MWNNQMev2

* Contract test awaits the now-async proxy

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YafYxayh7Gqe5MWNNQMev2

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
Anthony Ettinger 2026-09-05 14:49:39 -07:00 • committed by GitHub
parent 80a36269bb
commit 93af4770ac
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
8 changed files with 65 additions and 35 deletions
Download patch file Download diff file Expand all files Collapse all files

8
apps/logicsrc-web/contract/logicsrc-web.contract.test.ts
View file

@ -60,21 +60,21 @@ function jsonResponse(body: unknown, status = 200): Response {
}
describe("www canonical redirect (proxy.ts)", () => {
it("301s www to the apex host over https, preserving path + query", () => {
it("301s www to the apex host over https, preserving path + query", async () => {
const request = new NextRequest("https://www.logicsrc.com/openspec?ref=email", {
headers: { host: "www.logicsrc.com" }
});
const response = proxy(request);
const response = await proxy(request);
expect(response.status).toBe(301);
expect(response.headers.get("location")).toBe("https://logicsrc.com/openspec?ref=email");
});
it("passes through apex requests untouched", () => {
it("passes through apex requests untouched", async () => {
const request = new NextRequest("https://logicsrc.com/hire-us", {
headers: { host: "logicsrc.com" }
});
const response = proxy(request);
const response = await proxy(request);
// NextResponse.next() yields a non-redirect response.
expect(response.status).toBe(200);