From dde596b276167b1f6e4e3c868b3ee81c9db66176 Mon Sep 17 00:00:00 2001 From: Anthony Ettinger Date: Sun, 4 Oct 2026 09:17:06 -0700 Subject: [PATCH] OpenErrand reference runner: @logicsrc/openerrand and logicsrc errand (#228) * OpenErrand 0.1: an errand on a website with no API, with the human steps kept human docs/openerrand.md mints OpenErrand: one JSON file per errand (register an account, download a transcript) naming the site, the inputs with a sensitivity class and ordered sources (document, vault, prompt, generate, derive, candidate, literal), field rules matched by id then label, page and wait steps, five human gates a runner never performs (declare, identity-proofing, code, mail, captcha), outcomes, the never-retried shared secret, vault and download outputs, hand-off cards that may name only public inputs, the publisher index at /.well-known/openerrand.json, and thirteen runner rules. The worked example is the MyFTB business registration that cli-tools `ftb` performs (profullstack/cli-tools#125), with no personal data. - @logicsrc/schemas: openerrand + openerrand-index schemas and fixtures - @logicsrc/validators: semantic checks (references, templates, no personal or secret input on a card) and tests that validate the spec's own examples - logicsrc-web: registry entry (process family), /openerrand landing page, the example and the index served as static files, contract tests Co-Authored-By: Claude Opus 5.5 * OpenErrand: hand-off cards stay on the surface that owns the data Anthony's ruling: tax and finance data never touches a social or promotion tool, and nothing is sent to a CPA or preparer. - Hand-off cards are delivered only on the surface that owns the errand's data (for a tax or finance errand, the principal's finance app through its CLI, PWA, MCP server or API, such as CoinPay, or the runner's terminal), never a social, promotion or third-party posting service, and never to anyone but the principal. A card for an errand with personal or secret inputs does not leave that surface. Runner rule 9 says the same. - The run record and the sample run name the card by an opaque id (pin-letter/7f3k2q) instead of a mynaposter.com URL; the myna mention is gone. - `principal: represented` no longer cites a preparer with a power of attorney. - The FTB card's last step no longer suggests sending the PIN to someone else. Co-Authored-By: Claude Opus 5.5 * OpenErrand: user-agent rule, captcha solver policy, reference runner note Anthony's answers on #227 ("go with your recommendations"): - Rule 11: a runner may run headless with a normal desktop browser user agent (dropping HeadlessChrome) and nothing more: no fingerprint spoofing beyond the UA string, no stealth plugins, no solving or evading a bot challenge. A challenge the browser completes itself is a wait step; any other is a captcha gate. - Captcha solvers: new site.sector and captcha step `solver` (forbidden by default | allowed). Never allowed on government, tax, financial, healthcare or identity-provider sites, nor on any errand with a declare or identity-proofing step or a secret input; elsewhere only when the file says so, with every use logged. The validator rejects `allowed` in the forbidden set or without a stated sector; six new tests. The FTB example states sector "tax". - Reference runner: @logicsrc/openerrand / `logicsrc errand run`, marked in progress; ftb stays the runner the example was taken from. - Name stays OpenErrand; family stays Agents and process. Co-Authored-By: Claude Opus 5.5 * OpenErrand reference runner: @logicsrc/openerrand and logicsrc errand Ship the runner docs/openerrand.md promised. `logicsrc errand run ` reads an OpenErrand 0.1 file, validates it with @logicsrc/validators, and drives headless Chrome through it under the spec's thirteen rules; `errand validate` shows what a file will ask of you and `errand status` shows the last run of each errand, its card and any lockout. The engine is generalised from cli-tools `ftb` (PR #125) with no dependency on cli-tools: the CDP client and Chrome finder from wcag.ts, the page reader, native-setter fill and forward-button picker from ftb-run.ts, and the rule matcher, throttle and outcome logic from ftb.ts, all now driven by the file. - Inputs: document (an extractor hook; the one shipped runs a local command that reads JSON requests and prints records), vault (teams or OpenCreds), prompt (no echo for secrets), generate, derive, candidate, literal. `--input name=value` wins. Shared-secret candidates are ranked as the spec says, one is submitted, and a rejection lists the others for --candidate. - Rules: id before label, step rules first, choices before text, an id match final, an unmatched required field stops the run naming it. - Gates: declare only with --declare after the values are shown; identity proofing never touched (URL only) and handed over or stopped on; code from the terminal or a code file, used once, a wrong code waits for the next; mail ends the run waiting with the card; captcha is the person's, and a CaptchaSolver interface is called only where the spec permits (no solver is bundled); wait steps are polled, never solved. - Throttle: 2 runs per errand and account in 30 minutes, 4 a day, 2 minutes between runs on a site, lockouts from metadata.lockout or a default, held per site and account, never lifted by --force. - Outputs: credentials written before success to a teams vault by pull, merge, push (metadata.vault or --vault), else a 0600 file said aloud; downloads type-checked and never overwritten with different bytes; cards only in the local run record. - The user agent is Chrome's own with HeadlessChrome replaced, given at launch: a CDP override did not reach a navigation the page's own script started, which is exactly the proof-of-work interstitial case. Tests: 85 in the package (rule engine, inputs, gates, throttle, outcomes, captcha gating, vaults, outputs, commands) including an integration test that runs the published FTB example unchanged in real headless Chrome against a local HTTPS fake site (Chrome maps webapp.ftb.ca.gov to it and every other host to NOTFOUND; all data fictional), and 2 in the CLI. CLI 0.6.0 -> 0.7.0; @logicsrc/schemas and @logicsrc/validators 0.3.0 -> 0.4.0 (the OpenErrand schemas, and the vocabularies now exported for runners); PRD 0009; the spec's Reference runner section and the landing page say it ships. Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: Claude Opus 5.5 --- README.md | 2 + apps/logicsrc-web/src/app/openerrand/page.tsx | 5 +- bun.lock | 22 +- docs/cli.md | 1 + docs/openerrand.md | 18 +- package.json | 4 +- packages/cli/package.json | 3 +- packages/cli/src/errand.test.ts | 42 ++ packages/cli/src/errand.ts | 35 ++ packages/cli/src/index.ts | 2 + packages/logicsrc-mcp/src/standards.test.ts | 2 +- packages/openerrand/README.md | 117 ++++ packages/openerrand/package.json | 49 ++ packages/openerrand/src/browser.ts | 277 ++++++++++ packages/openerrand/src/captcha.ts | 27 + packages/openerrand/src/commands.test.ts | 94 ++++ packages/openerrand/src/commands.ts | 270 +++++++++ packages/openerrand/src/driver.ts | 285 ++++++++++ packages/openerrand/src/extract.ts | 76 +++ packages/openerrand/src/fake-site.ts | 231 ++++++++ packages/openerrand/src/index.ts | 38 ++ packages/openerrand/src/inputs.test.ts | 174 ++++++ packages/openerrand/src/inputs.ts | 435 +++++++++++++++ packages/openerrand/src/integration.test.ts | 160 ++++++ packages/openerrand/src/load.ts | 88 +++ packages/openerrand/src/outputs.test.ts | 110 ++++ packages/openerrand/src/outputs.ts | 122 ++++ packages/openerrand/src/pages.test.ts | 133 +++++ packages/openerrand/src/pages.ts | 103 ++++ packages/openerrand/src/prompt.ts | 63 +++ packages/openerrand/src/rules.test.ts | 156 ++++++ packages/openerrand/src/rules.ts | 149 +++++ packages/openerrand/src/run.test.ts | 420 ++++++++++++++ packages/openerrand/src/run.ts | 523 ++++++++++++++++++ packages/openerrand/src/store.ts | 163 ++++++ packages/openerrand/src/testing.ts | 135 +++++ packages/openerrand/src/throttle.ts | 103 ++++ packages/openerrand/src/types.ts | 210 +++++++ packages/openerrand/src/util.ts | 104 ++++ packages/openerrand/src/vault.ts | 191 +++++++ packages/openerrand/tsconfig.json | 9 + packages/schemas/package.json | 2 +- packages/validators/package.json | 4 +- packages/validators/src/index.ts | 5 + prd/0009-openerrand-reference-runner.md | 114 ++++ prd/README.md | 1 + 46 files changed, 5263 insertions(+), 14 deletions(-) create mode 100644 packages/cli/src/errand.test.ts create mode 100644 packages/cli/src/errand.ts create mode 100644 packages/openerrand/README.md create mode 100644 packages/openerrand/package.json create mode 100644 packages/openerrand/src/browser.ts create mode 100644 packages/openerrand/src/captcha.ts create mode 100644 packages/openerrand/src/commands.test.ts create mode 100644 packages/openerrand/src/commands.ts create mode 100644 packages/openerrand/src/driver.ts create mode 100644 packages/openerrand/src/extract.ts create mode 100644 packages/openerrand/src/fake-site.ts create mode 100644 packages/openerrand/src/index.ts create mode 100644 packages/openerrand/src/inputs.test.ts create mode 100644 packages/openerrand/src/inputs.ts create mode 100644 packages/openerrand/src/integration.test.ts create mode 100644 packages/openerrand/src/load.ts create mode 100644 packages/openerrand/src/outputs.test.ts create mode 100644 packages/openerrand/src/outputs.ts create mode 100644 packages/openerrand/src/pages.test.ts create mode 100644 packages/openerrand/src/pages.ts create mode 100644 packages/openerrand/src/prompt.ts create mode 100644 packages/openerrand/src/rules.test.ts create mode 100644 packages/openerrand/src/rules.ts create mode 100644 packages/openerrand/src/run.test.ts create mode 100644 packages/openerrand/src/run.ts create mode 100644 packages/openerrand/src/store.ts create mode 100644 packages/openerrand/src/testing.ts create mode 100644 packages/openerrand/src/throttle.ts create mode 100644 packages/openerrand/src/types.ts create mode 100644 packages/openerrand/src/util.ts create mode 100644 packages/openerrand/src/vault.ts create mode 100644 packages/openerrand/tsconfig.json create mode 100644 prd/0009-openerrand-reference-runner.md diff --git a/README.md b/README.md index 1370870..c99a8f2 100644 --- a/README.md +++ b/README.md @@ -18,6 +18,7 @@ packages/ openontology OpenOntology reference engine (entities, claims, queries, change sets) openprd OpenPRD reference implementation (numbered PRDs, lifecycle, task bridge) openfleet OpenFleet reference implementation (record, ledger, sysop verbs, Claude Code hooks) + openerrand OpenErrand reference runner (errand files in headless Chrome, human gates kept human) logicsrc-mcp @profullstack/logicsrc-mcp standards MCP server sdk SDK contract types and helpers tui terminal UI @@ -167,6 +168,7 @@ logicsrc prd … # OpenPRD logicsrc ontology … # OpenOntology logicsrc context … # OpenContext (also `opencontext`) logicsrc fleet … # OpenFleet: open, cap, tree, stop, log, hooks install +logicsrc errand … # OpenErrand: run, validate, status (an errand on a site with no API, gates kept human) logicsrc openmcp … # OpenMCP: relays, find, call, add, probe, serve (also `openmcp`) logicsrc openspec … # import, export, change; any other word is OpenSpec.dev's own CLI (init, list, validate, archive, show) logicsrc mcp # the LogicSRC MCP server over stdio (also `logicsrc-mcp`) diff --git a/apps/logicsrc-web/src/app/openerrand/page.tsx b/apps/logicsrc-web/src/app/openerrand/page.tsx index 7a51782..c6cc7ee 100644 --- a/apps/logicsrc-web/src/app/openerrand/page.tsx +++ b/apps/logicsrc-web/src/app/openerrand/page.tsx @@ -55,8 +55,9 @@ export default function OpenErrandPage(): ReactNode { values that are secret, the steps that belong to a person, and the point where it stops.

- Status: 0.1. A generic runner, logicsrc errand run from{" "} - @logicsrc/openerrand, is in progress. The worked example + Status: 0.1. The reference runner is logicsrc errand run, from{" "} + @logicsrc/openerrand: it reads an errand file and drives + headless Chrome through it under the rules below. The worked example transcribes the rule table of ftb in cli-tools, which registers and activates MyFTB accounts at the California Franchise Tax Board. An excerpt:

diff --git a/bun.lock b/bun.lock index c964981..6bf5367 100644 --- a/bun.lock +++ b/bun.lock @@ -144,7 +144,7 @@ }, "packages/cli": { "name": "@logicsrc/cli", - "version": "0.6.0", + "version": "0.7.0", "bin": { "logicsrc": "dist/index.js", }, @@ -153,6 +153,7 @@ "@logicsrc/account-core": "file:../account-core", "@logicsrc/opencontext": "file:../opencontext", "@logicsrc/opencreds": "file:../opencreds", + "@logicsrc/openerrand": "file:../openerrand", "@logicsrc/openfleet": "file:../openfleet", "@logicsrc/openmcp": "^0.3.1", "@logicsrc/openontology": "file:../openontology", @@ -222,6 +223,17 @@ "vitest": "^4.0.8", }, }, + "packages/openerrand": { + "name": "@logicsrc/openerrand", + "version": "0.1.0", + "dependencies": { + "@logicsrc/validators": "^0.4.0", + "commander": "^14.0.2", + }, + "devDependencies": { + "vitest": "^4.0.8", + }, + }, "packages/openfleet": { "name": "@logicsrc/openfleet", "version": "0.1.0", @@ -267,7 +279,7 @@ }, "packages/schemas": { "name": "@logicsrc/schemas", - "version": "0.3.0", + "version": "0.4.0", }, "packages/sdk": { "name": "@logicsrc/sdk", @@ -290,12 +302,12 @@ }, "packages/validators": { "name": "@logicsrc/validators", - "version": "0.3.0", + "version": "0.4.0", "bin": { "logicsrc-validate": "dist/cli.js", }, "dependencies": { - "@logicsrc/schemas": "^0.3.0", + "@logicsrc/schemas": "^0.4.0", "ajv": "^8.17.1", "ajv-formats": "^3.0.1", "yaml": "^2.8.1", @@ -675,6 +687,8 @@ "@logicsrc/opencreds": ["@logicsrc/opencreds@workspace:packages/opencreds"], + "@logicsrc/openerrand": ["@logicsrc/openerrand@workspace:packages/openerrand"], + "@logicsrc/openfleet": ["@logicsrc/openfleet@workspace:packages/openfleet"], "@logicsrc/openmcp": ["@logicsrc/openmcp@0.3.1", "", { "dependencies": { "@hono/node-server": "^2.1.1", "hono": "^4.13.7" }, "bin": { "openmcp": "bin/openmcp.mjs" } }, "sha512-uqfEFyA+XNquADqtmhhl71cobuRck8+IUlhBpsCBGdlOMhCxjoLx+VagjsNES3tmmO+sEjnvW7zEHA7DtZ/oHA=="], diff --git a/docs/cli.md b/docs/cli.md index 7b021e7..020ee20 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -36,6 +36,7 @@ social email credentials fleet +errand openspec plugins tui diff --git a/docs/openerrand.md b/docs/openerrand.md index cc9530d..ef430ce 100644 --- a/docs/openerrand.md +++ b/docs/openerrand.md @@ -641,7 +641,23 @@ npx @logicsrc/validators openerrand ftb-register-business.json ## Reference runner -In progress: a generic runner, the `@logicsrc/openerrand` package in the LogicSRC repository, run as `logicsrc errand run `, reads an errand file and drives headless Chrome through it under the rules above. Until it ships, `ftb` in cli-tools is the runner the worked example was taken from; it has the same rule table compiled in rather than reading the file. +`logicsrc errand run `, from the [`@logicsrc/openerrand`](https://github.com/profullstack/logicsrc/tree/master/packages/openerrand) package, reads an errand file, validates it with `@logicsrc/validators`, and drives headless Chrome through it under the rules above. `logicsrc errand validate ` shows what a file will ask of you; `logicsrc errand status` shows the last run of each errand, its hand-off card and any lockout. + +``` +logicsrc errand run ftb-register-business.json --extractor "python3 extract.py ~/taxes" --dry-run +logicsrc errand run ftb-register-business.json --extractor "python3 extract.py ~/taxes" --declare --vault teams:profullstack/ftb/prod +``` + +What the runner adds that the file does not say: + +- **Documents** come from an extractor the principal names with `--extractor`: a local command that reads the requested forms and fields as JSON on stdin and prints records (`form`, `field`, `value`, `year`, `label`, `file`, `page`). The runner ships no extractor of its own. +- **The vault** is `--vault`, or a string in the file's `metadata.vault`: `teams://` (read with `logicsrc teams pull`, written by pull, merge, push), `opencreds` (read-only), or `file:`. With none, credentials go to a 0600 file under `~/.local/share/logicsrc/errand/credentials/` and the runner says so. +- **A code** is typed at the prompt, or written to `~/.local/share/logicsrc/errand/codes/.code` by whoever holds the phone. A wrong code waits for the next one. +- **The throttle**: 2 runs of an errand per account in 30 minutes, 4 a day, 2 minutes between any two runs on one site, and nothing at all during a recorded lockout. A lockout is read from `metadata.lockout.text` (a pattern) and lasts `metadata.lockout.duration`, or a default pattern and 35 minutes. `--force` lifts the caps and never a lockout. +- **One Chrome profile per site** is kept between runs, so a bot check the browser has passed stays passed. The user agent drops `HeadlessChrome` and nothing more. +- **A captcha solver** is an interface a program embedding the runner may pass; none is bundled, and the runner calls one only where the [captcha rules](#captcha) permit it. + +`ftb` in [cli-tools](https://github.com/profullstack/cli-tools/pull/125) is the runner the worked example was taken from; it has the same rule table compiled in rather than reading the file. ## Not diff --git a/package.json b/package.json index cfb1ad7..e38b656 100644 --- a/package.json +++ b/package.json @@ -12,14 +12,14 @@ "apps/*" ], "scripts": { - "build": "bun run --cwd packages/schemas build && bun run --cwd packages/validators build && bun run --cwd packages/sdk build && bun run --cwd packages/agentad build && bun run --cwd packages/ans build && bun run --cwd packages/plugin-core build && bun run --cwd packages/agentstack build && bun run --cwd packages/agentswarm build && bun run --cwd packages/account-core build && bun run --cwd plugins/coinpay build && bun run --cwd plugins/ugig build && bun run --cwd plugins/sh1pt build && bun run --cwd plugins/c0mpute build && bun run --cwd plugins/feed-discovery build && bun run --cwd plugins/social-accounts build && bun run --cwd plugins/email-accounts build && bun run --cwd plugins/agentbbs build && bun run --cwd plugins/agentgit build && bun run --cwd plugins/agentmail build && bun run --cwd plugins/credential-sharing build && bun run --cwd packages/openontology build && bun run --cwd packages/openprd build && bun run --cwd packages/opencontext build && bun run --cwd packages/opencreds build && bun run --cwd packages/tui build && bun run --cwd packages/openfleet build && bun run --cwd packages/cli build && bun run --cwd packages/logicsrc-mcp build && bun run --cwd apps/commandboard-api build && bun run --cwd apps/commandboard-web build && bun run --cwd apps/logicsrc-web build", + "build": "bun run --cwd packages/schemas build && bun run --cwd packages/validators build && bun run --cwd packages/sdk build && bun run --cwd packages/agentad build && bun run --cwd packages/ans build && bun run --cwd packages/plugin-core build && bun run --cwd packages/agentstack build && bun run --cwd packages/agentswarm build && bun run --cwd packages/account-core build && bun run --cwd plugins/coinpay build && bun run --cwd plugins/ugig build && bun run --cwd plugins/sh1pt build && bun run --cwd plugins/c0mpute build && bun run --cwd plugins/feed-discovery build && bun run --cwd plugins/social-accounts build && bun run --cwd plugins/email-accounts build && bun run --cwd plugins/agentbbs build && bun run --cwd plugins/agentgit build && bun run --cwd plugins/agentmail build && bun run --cwd plugins/credential-sharing build && bun run --cwd packages/openontology build && bun run --cwd packages/openprd build && bun run --cwd packages/opencontext build && bun run --cwd packages/opencreds build && bun run --cwd packages/tui build && bun run --cwd packages/openfleet build && bun run --cwd packages/openerrand build && bun run --cwd packages/cli build && bun run --cwd packages/logicsrc-mcp build && bun run --cwd apps/commandboard-api build && bun run --cwd apps/commandboard-web build && bun run --cwd apps/logicsrc-web build", "start": "bun run --cwd apps/logicsrc-web start", "test": "bun run --filter '*' test", "check": "bun run build && bun run test", "schemas:validate": "bun run --cwd packages/validators validate:fixtures", "test:contract": "bun run --cwd apps/commandboard-api test:contract && bun run --cwd apps/logicsrc-web test:contract", "test:e2e": "bun run --cwd apps/commandboard-web test:e2e && bun run --cwd apps/logicsrc-web test:e2e", - "build:cli": "bun run --cwd packages/schemas build && bun run --cwd packages/validators build && bun run --cwd packages/plugin-core build && bun run --cwd packages/account-core build && bun run --cwd plugins/coinpay build && bun run --cwd plugins/ugig build && bun run --cwd plugins/feed-discovery build && bun run --cwd plugins/social-accounts build && bun run --cwd plugins/email-accounts build && bun run --cwd plugins/agentbbs build && bun run --cwd plugins/credential-sharing build && bun run --cwd packages/openontology build && bun run --cwd packages/openprd build && bun run --cwd packages/opencontext build && bun run --cwd packages/opencreds build && bun run --cwd packages/tui build && bun run --cwd packages/logicsrc-mcp build && bun run --cwd packages/openfleet build && bun run --cwd packages/cli build" + "build:cli": "bun run --cwd packages/schemas build && bun run --cwd packages/validators build && bun run --cwd packages/plugin-core build && bun run --cwd packages/account-core build && bun run --cwd plugins/coinpay build && bun run --cwd plugins/ugig build && bun run --cwd plugins/feed-discovery build && bun run --cwd plugins/social-accounts build && bun run --cwd plugins/email-accounts build && bun run --cwd plugins/agentbbs build && bun run --cwd plugins/credential-sharing build && bun run --cwd packages/openontology build && bun run --cwd packages/openprd build && bun run --cwd packages/opencontext build && bun run --cwd packages/opencreds build && bun run --cwd packages/tui build && bun run --cwd packages/logicsrc-mcp build && bun run --cwd packages/openfleet build && bun run --cwd packages/openerrand build && bun run --cwd packages/cli build" }, "devDependencies": { "@types/node": "^24.10.1", diff --git a/packages/cli/package.json b/packages/cli/package.json index a0c9050..801fb5f 100644 --- a/packages/cli/package.json +++ b/packages/cli/package.json @@ -1,6 +1,6 @@ { "name": "@logicsrc/cli", - "version": "0.6.0", + "version": "0.7.0", "description": "LogicSRC CLI: every LogicSRC standard and tool as one command.", "type": "module", "main": "./dist/index.js", @@ -18,6 +18,7 @@ "@logicsrc/account-core": "file:../account-core", "@logicsrc/opencontext": "file:../opencontext", "@logicsrc/opencreds": "file:../opencreds", + "@logicsrc/openerrand": "file:../openerrand", "@logicsrc/openfleet": "file:../openfleet", "@logicsrc/openmcp": "^0.3.1", "@logicsrc/openontology": "file:../openontology", diff --git a/packages/cli/src/errand.test.ts b/packages/cli/src/errand.test.ts new file mode 100644 index 0000000..a276b68 --- /dev/null +++ b/packages/cli/src/errand.test.ts @@ -0,0 +1,42 @@ +import { mkdtempSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { fileURLToPath } from "node:url"; +import { Command } from "commander"; +import { afterEach, describe, expect, it } from "vitest"; +import { registerErrandCommands } from "./errand.js"; + +/** A program shaped like the real one: positional options on, no process.exit. */ +function program(): Command { + const p = new Command(); + p.name("logicsrc").enablePositionalOptions().exitOverride(); + return p; +} + +const example = fileURLToPath(new URL("../../schemas/fixtures/openerrand/ftb-register-business.json", import.meta.url)); + +describe("logicsrc errand", () => { + const homes: string[] = []; + afterEach(() => { + for (const home of homes.splice(0)) rmSync(home, { recursive: true, force: true }); + process.exitCode = 0; + }); + + it("mounts run, validate and status", () => { + const p = program(); + registerErrandCommands(p); + const errand = p.commands.find((c) => c.name() === "errand")!; + expect(errand.commands.map((c) => c.name()).sort()).toEqual(["run", "status", "validate"]); + }); + + it("validates the spec's worked example through the umbrella", async () => { + const out: string[] = []; + const home = mkdtempSync(join(tmpdir(), "logicsrc-errand-cli-")); + homes.push(home); + const p = program(); + registerErrandCommands(p, { out: (l) => out.push(l), say: () => undefined, env: { LOGICSRC_ERRAND_HOME: home } }); + await p.parseAsync(["node", "logicsrc", "errand", "validate", example]); + expect(process.exitCode).toBe(0); + expect(out.at(-1)).toMatch(/^valid OpenErrand 0\.1/); + }); +}); diff --git a/packages/cli/src/errand.ts b/packages/cli/src/errand.ts new file mode 100644 index 0000000..4f76b74 --- /dev/null +++ b/packages/cli/src/errand.ts @@ -0,0 +1,35 @@ +import { spawnSync } from "node:child_process"; +import type { Command } from "commander"; +import { registerErrandCommands as registerRunner, type Deps, type LogicsrcExec } from "@logicsrc/openerrand/commands"; + +/** + * `logicsrc errand …` + * + * The runner lives in `@logicsrc/openerrand`, the reference implementation of + * OpenErrand; this file only mounts it. A team vault named by an errand + * (`teams://`) is read and written through this same CLI's + * `teams pull` / `teams push`, re-entered as a child process so the runner + * never needs the CLI's session code and the plaintext lives for one call. + * + * `deps` is injectable so the umbrella test drives the group without a + * terminal, a browser or a vault. + */ +export function registerErrandCommands(program: Command, deps: Partial = {}): void { + const errand = program + .command("errand") + .description( + "OpenErrand: run an errand file on a website with no API, in headless Chrome, stopping at every step " + + "that belongs to a person (declarations, identity proofing, codes, letters, captchas).", + ); + + const self: LogicsrcExec = (args, options) => { + const entry = process.argv[1]; + const result = entry + ? spawnSync(process.execPath, [entry, ...args], { encoding: "utf8", stdio: [options?.inheritStdin ? "inherit" : "ignore", "pipe", "pipe"] }) + : spawnSync("logicsrc", args, { encoding: "utf8", stdio: [options?.inheritStdin ? "inherit" : "ignore", "pipe", "pipe"] }); + if (result.error) return { status: 1, stdout: "", stderr: result.error.message }; + return { status: result.status ?? 1, stdout: result.stdout ?? "", stderr: result.stderr ?? "" }; + }; + + registerRunner(errand, { logicsrc: self, ...deps }); +} diff --git a/packages/cli/src/index.ts b/packages/cli/src/index.ts index 7dd5bea..2fb4853 100644 --- a/packages/cli/src/index.ts +++ b/packages/cli/src/index.ts @@ -41,6 +41,7 @@ import { exportOpenSpecSummary, importOpenSpec, writeOpenSpecChange } from "./op import { registerOpenContextCommands } from "./context.js"; import { registerOpenCredsCommands } from "./creds.js"; import { registerFleetCommands } from "./fleet.js"; +import { registerErrandCommands } from "./errand.js"; import { registerOntologyCommands } from "./ontology.js"; import { registerPrdCommands } from "./prd.js"; import { registerOpenMcpCommands } from "./openmcp.js"; @@ -1176,6 +1177,7 @@ registerOpenCredsCommands(program); registerOntologyCommands(program); registerPrdCommands(program); registerFleetCommands(program); +registerErrandCommands(program); registerOpenMcpCommands(program); registerMcpCommands(program); // Every other word under `logicsrc openspec` is OpenSpec.dev's own CLI. diff --git a/packages/logicsrc-mcp/src/standards.test.ts b/packages/logicsrc-mcp/src/standards.test.ts index e58ab1d..a493cea 100644 --- a/packages/logicsrc-mcp/src/standards.test.ts +++ b/packages/logicsrc-mcp/src/standards.test.ts @@ -203,7 +203,7 @@ describe("MCP: OpenPRD", () => { it("reports the next free id and the allowed lifecycle moves", async () => { const client = await connect(); // Asserted against the live prd/ directory, so this advances with every PRD added. - expect(toolText(await client.callTool({ name: "prd_next_id", arguments: {} }))).toBe("0009"); + expect(toolText(await client.callTool({ name: "prd_next_id", arguments: {} }))).toBe("0010"); const moves = await client.callTool({ name: "prd_next_statuses", arguments: { ref: "0001" } }); const payload = JSON.parse(toolText(moves)) as { status: string; allowedNext: string[] }; diff --git a/packages/openerrand/README.md b/packages/openerrand/README.md new file mode 100644 index 0000000..62d42ef --- /dev/null +++ b/packages/openerrand/README.md @@ -0,0 +1,117 @@ +# @logicsrc/openerrand + +The reference runner for [OpenErrand](https://logicsrc.com/docs/openerrand): +one JSON file that describes an errand on a website with no API, run in +headless Chrome, stopping at every step that belongs to a person. + +It reads an errand file, validates it with `@logicsrc/validators`, resolves +every input before the browser opens, then for each page finds its step, +tests the outcomes, fills the form from the rules (id first, label second, +choices before text), hands every gate to a person, and presses the forward +button, until the site answers or the runner has to stop. + +## Install + +```bash +curl -fsSL https://logicsrc.com/install.sh | sh +logicsrc errand --help +``` + +The commands live in this package and the umbrella CLI mounts them as +`logicsrc errand`. To embed the runner: + +```bash +npm install @logicsrc/openerrand +``` + +## Commands + +```bash +logicsrc errand validate ftb-register-business.json +logicsrc errand run ftb-register-business.json --extractor "python3 extract.py ~/taxes" --dry-run +logicsrc errand run ftb-register-business.json --extractor "python3 extract.py ~/taxes" --declare +logicsrc errand status +``` + +| Flag | What | +| --- | --- | +| `--input name=value` | Give an input's value (repeatable). Wins over every source. | +| `--dry-run` | Fill every page up to the first that holds a declaration or a shared secret, show it with secrets masked and the declaration word for word, and stop before its forward button. Not counted by the throttle. | +| `--declare` | Your consent, for this run, to tick the declarations the errand names. Never read from a file, the environment or a saved default. | +| `--headful` | A visible window, so you can take identity proofing or a captcha yourself. | +| `--chrome PATH` | The Chrome or Chromium binary (default: `CHROME_PATH`, then PATH, then the Puppeteer and Playwright caches). | +| `--vault TARGET` | `teams://`, `opencreds[:]` or `file:`. Default: the file's `metadata.vault`, else a 0600 file. | +| `--extractor CMD` | The local command that reads your documents (see below). | +| `--candidate N` | Which shared-secret candidate to submit; a rejection lists the others. | +| `--account NAME` | Which account at the site, for the throttle. | +| `--force` | Lift the per-window, per-day and spacing caps. Never a lockout. | +| `--yes` | Run a file whose SHA-256 changed since its last run without asking. | +| `--json` | Print the run record on stdout. | + +Exit codes: 0 success, waiting or dry run; 1 rejected; 2 invalid file or +usage; 3 stopped (a gate, an unmatched field, a loop, off-site, a timeout); +4 refused by the throttle. + +## Inputs + +| Source | Served by | +| --- | --- | +| `document` | `--extractor`: a command on this machine. It gets `{ "errand", "documents": [{ "input", "form", "field", "match" }] }` on stdin and prints `[{ "form", "field", "value", "year", "label", "file", "page" }]`. The runner applies `match`, `years`, `pick` and `transform`, and keeps the values in memory only. No extractor is bundled. | +| `vault` | The vault target, read-only here. | +| `prompt` | The terminal; a secret does not echo. With no terminal the next source is tried. | +| `generate` | `crypto.randomInt`, at least one character of each class, a letter first when letters are allowed. | +| `derive`, `candidate`, `literal` | The file and the other inputs. | + +A shared-secret input's document sources become a ranked list of candidates +(sources in order, newest year first within each). A run submits one; a +rejection ends the run and lists the others for you to choose with +`--candidate`. Nothing is ever retried on its own. + +## Gates + +| Gate | What the runner does | +| --- | --- | +| `declare` | Stops on the page and prints the statement word for word, unless you passed `--declare`; then it shows the values on the page (secrets masked) and ticks the box. | +| `identity-proofing` | Never touches the provider's pages (it reads only the URL). Headless, it stops and tells you; with `--headful` at a terminal it waits until the page is back on the site. | +| `code` | The terminal prompt, or the code file `~/.local/share/logicsrc/errand/codes/.code` written by whoever holds the phone. Used once, never logged. A wrong code waits for the next one. | +| `mail` | Ends the run waiting, with the deadline and the hand-off card in the run record. | +| `captcha` | Yours: a visible window with `--headful`, or the run stops. A program may pass a `CaptchaSolver`, and it is called only when the step says `solver: allowed` and the errand is outside the forbidden set (a stated sector that is not government, tax, financial, healthcare or identity-provider; no declaration or identity proofing; no secret input). Each use is logged with the time, URL and service. No solver ships with this package. | +| `wait` | Polled until its match no longer fits, then the run goes on. Never solved or bypassed. | + +## What it keeps + +Under `$LOGICSRC_ERRAND_HOME`, else `~/.local/share/logicsrc/errand` (files +0600, directories 0700): + +| Path | What | +| --- | --- | +| `throttle.json` | Attempts and lockouts. 2 runs of an errand per account in 30 minutes, 4 a day; 2 minutes between runs on one site; nothing during a lockout. A lockout is the file's `metadata.lockout.text` pattern (lasting `metadata.lockout.duration`) or a default pattern (35 minutes), and it holds every errand on that site and account. | +| `approvals.json` | The SHA-256 of each errand file last run. A changed file is shown and not run without `--yes` or a yes at the terminal. | +| `runs/` | One run record per run: outcome, page, candidate (no value), card, deadline. Never an input value. | +| `pages/.jsonl` | The page log: each page's fields (selector, type, label, required, options) and a result page's text. Never a value. | +| `profiles//` | One Chrome profile per site, kept so a passed bot check stays passed. | +| `codes/.code` | Where a code may be written. | +| `credentials/.env` | Credentials when no writable vault is named. | + +Hand-off cards stay in the run record and `errand status`. They are never +posted anywhere, and a card that names a personal or secret input is refused. + +## Browser + +Chrome over the DevTools protocol, no dependency. The user agent is Chrome's +own with `HeadlessChrome` replaced by `Chrome`, given at launch so it reaches +every request; nothing else about the browser is changed. + +## Library + +```ts +import { loadErrand, resolveInputs, runErrand, openCdpDriver, Store, fileVault } from "@logicsrc/openerrand"; +``` + +`runErrand` takes a `Driver` factory, so an embedder can bring its own +browser; the tests drive it with a scripted fake and one integration test runs +the FTB example in real headless Chrome against a local fake site. + +## License + +MIT diff --git a/packages/openerrand/package.json b/packages/openerrand/package.json new file mode 100644 index 0000000..87e03ab --- /dev/null +++ b/packages/openerrand/package.json @@ -0,0 +1,49 @@ +{ + "name": "@logicsrc/openerrand", + "version": "0.1.0", + "description": "Reference runner for the OpenErrand standard: reads an errand file and drives headless Chrome through a website that has no API, stopping at every step that belongs to a person.", + "license": "MIT", + "type": "module", + "main": "./dist/index.js", + "types": "./dist/index.d.ts", + "exports": { + ".": "./dist/index.js", + "./commands": "./dist/commands.js" + }, + "repository": { + "type": "git", + "url": "git+https://github.com/profullstack/logicsrc.git", + "directory": "packages/openerrand" + }, + "homepage": "https://logicsrc.com/docs/openerrand", + "keywords": [ + "logicsrc", + "openerrand", + "errand", + "browser-automation", + "chrome", + "cdp", + "standards", + "cli" + ], + "publishConfig": { + "access": "public" + }, + "files": [ + "dist" + ], + "engines": { + "node": ">=22" + }, + "scripts": { + "build": "tsc -p tsconfig.json", + "test": "vitest run src" + }, + "dependencies": { + "@logicsrc/validators": "^0.4.0", + "commander": "^14.0.2" + }, + "devDependencies": { + "vitest": "^4.0.8" + } +} diff --git a/packages/openerrand/src/browser.ts b/packages/openerrand/src/browser.ts new file mode 100644 index 0000000..7b2c58a --- /dev/null +++ b/packages/openerrand/src/browser.ts @@ -0,0 +1,277 @@ +/** + * Chrome over the DevTools protocol, with no dependency: find a binary, start + * it, and talk to it over one WebSocket in flat session mode. + * + * Ported from cli-tools' wcag.ts (findChrome, Cdp, launchBrowser), which `ftb` + * drives MyFTB with. + */ + +import { type ChildProcess, spawn } from "node:child_process"; +import { accessSync, constants, mkdirSync, mkdtempSync, readdirSync, rmSync } from "node:fs"; +import { homedir, tmpdir } from "node:os"; +import { join } from "node:path"; +import { ErrandError } from "./util.js"; + +const isExecutable = (path: string): boolean => { + try { + accessSync(path, constants.X_OK); + return true; + } catch { + return false; + } +}; + +const listDir = (path: string): string[] => { + try { + return readdirSync(path); + } catch { + return []; + } +}; + +/** `linux-152.0.7977.42` before `linux-131.0.6778.204`: newest first by version. */ +const byVersionDesc = (a: string, b: string): number => { + const parse = (name: string): number[] => (/(\d+(?:\.\d+)*)/.exec(name)?.[1] ?? "0").split(".").map(Number); + const left = parse(a); + const right = parse(b); + for (let i = 0; i < Math.max(left.length, right.length); i += 1) { + const d = (right[i] ?? 0) - (left[i] ?? 0); + if (d !== 0) return d; + } + return 0; +}; + +/** Every place a Chrome might be: CHROME_PATH, PATH names, /opt, Puppeteer and Playwright caches, macOS bundles. */ +export function chromeCandidates(env: NodeJS.ProcessEnv = process.env, home: string = homedir()): string[] { + const out: string[] = []; + if (env.CHROME_PATH) out.push(env.CHROME_PATH); + for (const name of ["google-chrome", "google-chrome-stable", "chromium", "chromium-browser", "chrome"]) { + for (const dir of (env.PATH ?? "").split(":").filter(Boolean)) out.push(join(dir, name)); + } + out.push("/opt/google/chrome/chrome"); + // Full Chrome before the headless shell: a person may need the window (--headful). + const puppeteer = join(home, ".cache", "puppeteer"); + for (const flavour of ["chrome", "chrome-headless-shell"]) { + for (const build of listDir(join(puppeteer, flavour)).sort(byVersionDesc)) { + const dir = join(puppeteer, flavour, build); + const inner = listDir(dir).find((entry) => entry.startsWith(flavour)); + if (inner) out.push(join(dir, inner, flavour)); + } + } + const playwright = join(home, ".cache", "ms-playwright"); + for (const build of listDir(playwright).sort(byVersionDesc)) { + if (build.startsWith("chromium-")) out.push(join(playwright, build, "chrome-linux", "chrome")); + } + out.push("/Applications/Google Chrome.app/Contents/MacOS/Google Chrome"); + out.push("/Applications/Chromium.app/Contents/MacOS/Chromium"); + return out; +} + +export function findChrome(env: NodeJS.ProcessEnv = process.env, home: string = homedir(), executable: (path: string) => boolean = isExecutable): string | null { + return chromeCandidates(env, home).find(executable) ?? null; +} + +export const NO_CHROME = `no Chrome found. Pass --chrome PATH or set CHROME_PATH, install one +(apt install chromium, brew install --cask google-chrome), or let Puppeteer fetch one +(npx puppeteer browsers install chrome).`; + +interface CdpMessage { + id?: number; + method?: string; + params?: Record; + sessionId?: string; + result?: Record; + error?: { message: string }; +} + +export interface CdpEvent { + method: string; + params: Record; + sessionId?: string; +} + +/** Requests with ids and events by name, over one socket for the browser and every page. */ +export class Cdp { + private nextId = 1; + private readonly pending = new Map) => void; reject: (error: Error) => void }>(); + private readonly listeners = new Set<(message: CdpMessage) => void>(); + private readonly socket: WebSocket; + + constructor(socket: WebSocket) { + this.socket = socket; + socket.addEventListener("message", (event) => { + const message = JSON.parse(String(event.data)) as CdpMessage; + if (message.id !== undefined && this.pending.has(message.id)) { + const { resolve, reject } = this.pending.get(message.id)!; + this.pending.delete(message.id); + if (message.error) reject(new ErrandError(message.error.message)); + else resolve(message.result ?? {}); + return; + } + for (const listener of this.listeners) listener(message); + }); + socket.addEventListener("close", () => { + for (const { reject } of this.pending.values()) reject(new ErrandError("browser closed")); + this.pending.clear(); + }); + } + + static async connect(url: string): Promise { + const socket = new WebSocket(url); + await new Promise((resolve, reject) => { + socket.addEventListener("open", () => resolve(), { once: true }); + socket.addEventListener("error", () => reject(new ErrandError(`could not connect to ${url}`)), { once: true }); + }); + return new Cdp(socket); + } + + send(method: string, params: Record = {}, sessionId?: string): Promise> { + const id = this.nextId; + this.nextId += 1; + return new Promise((resolve, reject) => { + this.pending.set(id, { resolve, reject }); + this.socket.send(JSON.stringify(sessionId ? { id, method, params, sessionId } : { id, method, params })); + }); + } + + on(listener: (event: CdpEvent) => void): () => void { + const wrapped = (message: CdpMessage): void => { + if (message.method) listener({ method: message.method, params: message.params ?? {}, ...(message.sessionId ? { sessionId: message.sessionId } : {}) }); + }; + this.listeners.add(wrapped); + return () => this.listeners.delete(wrapped); + } + + waitFor(method: string, sessionId: string, timeoutMs: number): Promise { + return new Promise((resolve, reject) => { + const timer = setTimeout(() => { + this.listeners.delete(listener); + reject(new ErrandError(`timed out after ${Math.round(timeoutMs / 1000)}s waiting for ${method}`)); + }, timeoutMs); + const listener = (message: CdpMessage): void => { + if (message.method === method && message.sessionId === sessionId) { + clearTimeout(timer); + this.listeners.delete(listener); + resolve(); + } + }; + this.listeners.add(listener); + }); + } + + close(): void { + this.socket.close(); + } +} + +export interface Browser { + cdp: Cdp; + path: string; + /** Settles when Chrome ends, including a person closing its window. */ + exited: Promise; + close(): Promise; +} + +export interface LaunchOptions { + chrome?: string; + env?: NodeJS.ProcessEnv; + sandbox?: boolean; + timeoutMs?: number; + /** A profile directory to keep between runs; without one a throwaway profile is removed on close. */ + profile?: string; + /** false opens a window a person can use; the default is headless. */ + headless?: boolean; + /** Extra Chrome switches. The tests use this to point a hostname at a local fake site. */ + args?: string[]; +} + +export async function launchBrowser(options: LaunchOptions = {}): Promise { + const env = options.env ?? process.env; + const path = options.chrome ?? findChrome(env); + if (!path) throw new ErrandError(NO_CHROME); + if (!isExecutable(path)) throw new ErrandError(`${path} is not an executable`); + + const keep = options.profile !== undefined; + if (keep) mkdirSync(options.profile!, { recursive: true, mode: 0o700 }); + const profile = options.profile ?? mkdtempSync(join(tmpdir(), "logicsrc-errand-chrome-")); + const sandbox = options.sandbox ?? !(env.CHROME_NO_SANDBOX || process.getuid?.() === 0); + const args = [ + ...(options.headless === false ? [] : ["--headless=new"]), + "--remote-debugging-port=0", + `--user-data-dir=${profile}`, + "--no-first-run", + "--no-default-browser-check", + "--disable-gpu", + "--disable-extensions", + "--disable-background-networking", + "--window-size=1280,900", + ...(sandbox ? [] : ["--no-sandbox"]), + ...(options.args ?? []), + "about:blank", + ]; + + const child: ChildProcess = spawn(path, args, { env, stdio: ["ignore", "ignore", "pipe"] }); + const exited = new Promise((resolve) => child.once("exit", () => resolve())); + const cleanup = (): void => { + if (keep) return; + try { + rmSync(profile, { recursive: true, force: true }); + } catch { + // Still held open; the next run's tmpdir sweep takes it. + } + }; + + const url = await new Promise((resolve, reject) => { + let stderr = ""; + const timeoutMs = options.timeoutMs ?? 20_000; + const timer = setTimeout(() => reject(new ErrandError(`${path} did not start within ${timeoutMs / 1000}s\n${stderr}`)), timeoutMs); + child.stderr?.on("data", (chunk: Buffer) => { + stderr += chunk.toString(); + const match = /DevTools listening on (ws:\/\/\S+)/.exec(stderr); + if (match?.[1]) { + clearTimeout(timer); + resolve(match[1]); + } + }); + child.on("exit", (code) => { + clearTimeout(timer); + reject(new ErrandError(`${path} exited with ${code ?? "a signal"} before it was ready\n${stderr.trim()}`)); + }); + child.on("error", (error) => { + clearTimeout(timer); + reject(new ErrandError(`${path}: ${error.message}`)); + }); + }).catch((error: Error) => { + child.kill(); + cleanup(); + throw error; + }); + + const cdp = await Cdp.connect(url); + return { + cdp, + path, + exited, + async close() { + try { + await Promise.race([cdp.send("Browser.close"), new Promise((resolve) => setTimeout(resolve, 2000))]); + } catch { + // Already gone. + } + // Cookies reach a kept profile on a clean exit: give Chrome one before the kill. + if (keep) await Promise.race([exited, new Promise((resolve) => setTimeout(resolve, 10_000))]); + cdp.close(); + child.kill(); + cleanup(); + }, + }; +} + +/** + * Rule 11: a runner may present a normal desktop user agent by dropping + * `HeadlessChrome` from the string. That is all this does: no other header, + * no navigator patching, no fingerprint change. + */ +export function plainUserAgent(userAgent: string): string | null { + return userAgent.includes("HeadlessChrome") ? userAgent.replace("HeadlessChrome", "Chrome") : null; +} diff --git a/packages/openerrand/src/captcha.ts b/packages/openerrand/src/captcha.ts new file mode 100644 index 0000000..388039b --- /dev/null +++ b/packages/openerrand/src/captcha.ts @@ -0,0 +1,27 @@ +/** + * The captcha solver hook. The runner ships no solver and never will: by + * default a captcha is handed to the person in a visible window, or the run + * stops. A program embedding the runner may pass a {@link CaptchaSolver}, and + * the runner calls it only when {@link solverPermitted} says the errand allows + * one (the step says `solver: allowed`, the site's stated sector is outside + * government, tax, financial, healthcare and identity-provider, and the errand + * has no declare or identity-proofing step and no secret input). Every use is + * logged with the time, the page URL and the service, never the image or the + * answer. + */ + +export { solverPermitted } from "./pages.js"; + +export interface CaptchaContext { + /** The page the captcha is on. */ + url: string; + /** Evaluate script in the page, for a solver that must place its token. */ + evaluate(expression: string): Promise; +} + +export interface CaptchaSolver { + /** The service's name, written to the log on every use. */ + readonly service: string; + /** Resolve true when the captcha is cleared. */ + solve(context: CaptchaContext): Promise; +} diff --git a/packages/openerrand/src/commands.test.ts b/packages/openerrand/src/commands.test.ts new file mode 100644 index 0000000..69d45b8 --- /dev/null +++ b/packages/openerrand/src/commands.test.ts @@ -0,0 +1,94 @@ +import { readFileSync, writeFileSync } from "node:fs"; +import { join } from "node:path"; +import { Command } from "commander"; +import { afterEach, describe, expect, it } from "vitest"; +import { type Deps, registerErrandCommands } from "./commands.js"; +import { FakeDriver, ftbExamplePath, tempDir } from "./testing.js"; + +function setup(over: Partial = {}) { + const home = tempDir(); + const out: string[] = []; + const say: string[] = []; + const p = new Command(); + p.name("logicsrc").enablePositionalOptions().exitOverride(); + registerErrandCommands(p.command("errand"), { + env: { LOGICSRC_ERRAND_HOME: home, HOME: home }, + out: (l) => out.push(l), + say: (l) => say.push(l), + interactive: false, + now: () => new Date("2026-10-04T12:00:00Z"), + ...over, + }); + return { p, home, out, say }; +} + +afterEach(() => { + process.exitCode = 0; +}); + +describe("logicsrc errand validate", () => { + it("accepts the spec's worked example and shows every gate and input", async () => { + const { p, out } = setup(); + await p.parseAsync(["node", "logicsrc", "errand", "validate", ftbExamplePath()]); + expect(process.exitCode).toBe(0); + const text = out.join("\n"); + expect(text).toContain("declare (declaration): Ticking this box is the representative stating"); + expect(text).toContain("net_income [secret, shared-secret]"); + expect(text).toMatch(/valid OpenErrand 0\.1 {2}sha256 [0-9a-f]{64}/); + }); + + it("rejects a file the validator rejects, with the reason", async () => { + const { p, home, say } = setup(); + const bad = JSON.parse(readFileSync(ftbExamplePath(), "utf8")); + bad.steps.push({ id: "cap", kind: "captcha", match: { selector: ".g-recaptcha" }, solver: "allowed" }); + writeFileSync(join(home, "bad.json"), JSON.stringify(bad)); + await p.parseAsync(["node", "logicsrc", "errand", "validate", join(home, "bad.json")]); + expect(process.exitCode).toBe(2); + expect(say.join("\n")).toMatch(/captcha solver is never allowed on a tax site/); + }); +}); + +describe("logicsrc errand run", () => { + it("refuses an --input the errand does not have, before any browser", async () => { + let opened = false; + const { p, say } = setup({ openDriver: async () => ((opened = true), new FakeDriver({}, () => "")) }); + await p.parseAsync(["node", "logicsrc", "errand", "run", ftbExamplePath(), "--input", "nope=1"]); + expect(process.exitCode).toBe(2); + expect(say.join("\n")).toMatch(/no input named nope/); + expect(opened).toBe(false); + }); + + it("stops before the browser when a required input has no value and nobody is at a terminal", async () => { + let opened = false; + const { p, say } = setup({ openDriver: async () => ((opened = true), new FakeDriver({}, () => "")) }); + await p.parseAsync(["node", "logicsrc", "errand", "run", ftbExamplePath()]); + expect(process.exitCode).toBe(3); + expect(say.join("\n")).toMatch(/no value for Email address/); + expect(opened).toBe(false); + }); + + it("shows a changed file and does not run it without --yes", async () => { + const { p, home, say } = setup({ openDriver: async () => new FakeDriver({}, () => "") }); + const copy = join(home, "errand.json"); + writeFileSync(copy, readFileSync(ftbExamplePath(), "utf8")); + const args = ["node", "logicsrc", "errand", "run", copy, "--dry-run", "--input", "email=jane@example.com", "--input", "phone=5555550100"]; + for (const name of ["first_name=Jane", "last_name=Doe", "street=1234 Maple St", "zip=95814", "corp_id=1234567", "net_income=48210", "tax_year=2025"]) args.push("--input", name); + await p.parseAsync(args); + const edited = JSON.parse(readFileSync(copy, "utf8")); + edited.title = "Register a MyFTB business account (edited)"; + writeFileSync(copy, JSON.stringify(edited)); + say.length = 0; + await p.parseAsync(args); + expect(say.join("\n")).toMatch(/this file changed since its last run/); + expect(say.join("\n")).toContain("Not run."); + expect(process.exitCode).toBe(3); + }); +}); + +describe("logicsrc errand status", () => { + it("says when nothing has run", async () => { + const { p, out } = setup(); + await p.parseAsync(["node", "logicsrc", "errand", "status"]); + expect(out).toEqual(["No errands run yet."]); + }); +}); diff --git a/packages/openerrand/src/commands.ts b/packages/openerrand/src/commands.ts new file mode 100644 index 0000000..a58d510 --- /dev/null +++ b/packages/openerrand/src/commands.ts @@ -0,0 +1,270 @@ +/** + * `logicsrc errand run|validate|status`. + * + * Everything that touches the world (environment, terminal, the logicsrc CLI + * for team vaults, the browser) is injectable, so the commands are tested + * without a terminal and the integration test can point Chrome at a fake site. + * Exit codes go on `process.exitCode`, never `process.exit`. + */ + +import { randomInt } from "node:crypto"; +import type { Command } from "commander"; +import type { CaptchaSolver } from "./captcha.js"; +import { type Driver, openCdpDriver } from "./driver.js"; +import { commandExtractor } from "./extract.js"; +import { type Prompt, resolveInputs } from "./inputs.js"; +import { loadErrand, summarize } from "./load.js"; +import { confirm, terminalPrompt } from "./prompt.js"; +import { runErrand } from "./run.js"; +import { type Env, Store } from "./store.js"; +import { durationMs, ErrandError } from "./util.js"; +import { describeTarget, fileVault, type LogicsrcExec, openVault, parseTarget, pathLogicsrc, type Vault } from "./vault.js"; + +/** 0 done (success, waiting or a dry run), 1 rejected, 2 invalid file or usage, 3 stopped, 4 refused by the throttle. */ +export const EXIT = { OK: 0, REJECTED: 1, INVALID: 2, STOPPED: 3, THROTTLED: 4 } as const; + +export interface Deps { + env: Env; + /** stdout: machine output (--json) and the final line. */ + out: (line: string) => void; + /** stderr: everything a person reads while it runs. */ + say: (line: string) => void; + prompt: Prompt; + confirm: (question: string) => Promise; + interactive: boolean; + logicsrc: LogicsrcExec; + now: () => Date; + random: (max: number) => number; + /** Extra Chrome switches (tests map the site's hostname to a local server). */ + chromeArgs?: string[]; + /** Replace the browser entirely (unit tests). */ + openDriver?: () => Promise; + solver?: CaptchaSolver; + pollMs?: number; + rereadMs?: number; + settleMs?: number; +} + +export function defaultDeps(): Deps { + return { + env: process.env, + out: (line) => process.stdout.write(`${line}\n`), + say: (line) => process.stderr.write(`${line}\n`), + prompt: terminalPrompt, + confirm, + interactive: process.stdin.isTTY === true, + logicsrc: pathLogicsrc, + now: () => new Date(), + random: randomInt, + }; +} + +function collect(value: string, previous: string[]): string[] { + return [...previous, value]; +} + +function parseOverrides(pairs: string[]): Record { + const out: Record = {}; + for (const pair of pairs) { + const at = pair.indexOf("="); + if (at <= 0) throw new ErrandError(`--input ${pair}: expected name=value`); + out[pair.slice(0, at)] = pair.slice(at + 1); + } + return out; +} + +interface RunFlags { + input: string[]; + dryRun?: boolean; + declare?: boolean; + headful?: boolean; + chrome?: string; + vault?: string; + extractor?: string; + candidate?: string; + account?: string; + force?: boolean; + yes?: boolean; + json?: boolean; +} + +export function registerErrandCommands(parent: Command, partial: Partial = {}): void { + const deps: Deps = { ...defaultDeps(), ...partial }; + + parent + .command("run") + .argument("", "an OpenErrand 0.1 JSON file") + .description("Run an errand in headless Chrome, stopping at every step that belongs to a person.") + .option("--input ", "give an input's value (repeatable); wins over every source", collect, []) + .option("--dry-run", "fill every page up to the first that holds a declaration or a shared secret, show it, and stop before its forward button") + .option("--declare", "your consent, for this run, to tick the declarations the errand names after the values are shown") + .option("--headful", "open a visible window, so you can take identity proofing or a captcha yourself") + .option("--chrome ", "the Chrome or Chromium binary (default: CHROME_PATH, then the usual places)") + .option("--vault ", "teams://, opencreds[:] or file: (default: the file's metadata.vault)") + .option("--extractor ", "a command that reads the errand's document requests as JSON on stdin and prints records") + .option("--candidate ", "which shared-secret candidate to submit (1 is the best); a rejection lists the others") + .option("--account ", "which account at the site, for the throttle ledger", "default") + .option("--force", "lift the per-window, per-day and spacing caps (never a lockout)") + .option("--yes", "run a file whose SHA-256 changed since its last run without asking") + .option("--json", "print the run record as JSON on stdout") + .addHelpText( + "after", + ` +Examples: + logicsrc errand validate ftb-register-business.json + logicsrc errand run ftb-register-business.json --extractor "python3 extract.py ~/taxes" --dry-run + logicsrc errand run ftb-register-business.json --extractor "python3 extract.py ~/taxes" --declare + logicsrc errand status + +A one-time code is typed at the prompt, or written to the code file the run +names (~/.local/share/logicsrc/errand/codes/.code) by whoever holds the +phone. Credentials go to the vault the file or --vault names, else a 0600 +file under ~/.local/share/logicsrc/errand/credentials/.`, + ) + .action(async (file: string, flags: RunFlags) => { + process.exitCode = await runCommand(file, flags, deps); + }); + + parent + .command("validate") + .argument("", "an OpenErrand 0.1 JSON file") + .description("Check an errand file against the schema and the spec's rules, and show what it will ask of you.") + .action((file: string) => { + try { + const loaded = loadErrand(file); + for (const line of summarize(loaded.errand, "unverified: a local file")) deps.out(line); + deps.out(`valid OpenErrand 0.1 sha256 ${loaded.sha256}`); + process.exitCode = EXIT.OK; + } catch (error) { + deps.say((error as Error).message); + process.exitCode = EXIT.INVALID; + } + }); + + parent + .command("status") + .description("The last run of each errand, the cards waiting on you, and any lockout.") + .option("--json", "machine-readable") + .action((flags: { json?: boolean }) => { + const store = Store.fromEnv(deps.env); + const runs = store.runs(); + const latest = new Map(); + for (const run of runs) latest.set(run.name, run); + const ledger = store.loadLedger(); + const now = deps.now().getTime(); + const locks = Object.entries(ledger.lockedUntil).filter(([, until]) => Date.parse(until) > now); + if (flags.json) { + deps.out(JSON.stringify({ runs: [...latest.values()], lockouts: Object.fromEntries(locks) }, null, 2)); + process.exitCode = EXIT.OK; + return; + } + if (!latest.size) deps.out("No errands run yet."); + for (const run of latest.values()) { + deps.out(`${run.name} ${run.kind}${run.outcome !== run.kind ? ` (${run.outcome})` : ""} ${run.at.slice(0, 16).replace("T", " ")}${run.reason ? ` ${run.reason}` : ""}`); + if (run.card && !run.card.done) { + deps.out(` card ${run.card.id}: ${run.card.title}${run.card.expires_on ? ` (before ${run.card.expires_on})` : ""}`); + for (const [i, s] of run.card.steps.entries()) deps.out(` ${i + 1}. ${s}`); + if (run.card.open) deps.out(` open ${run.card.open}`); + if (run.card.command) deps.out(` ${run.card.command}`); + } + } + for (const [key, until] of locks) deps.out(`locked: ${key.replace("|", " account ")} until ${until}`); + process.exitCode = EXIT.OK; + }); +} + +export async function runCommand(file: string, flags: RunFlags, deps: Deps): Promise { + const store = Store.fromEnv(deps.env); + try { + const { errand, sha256 } = loadErrand(file); + + // Rule 1: show it before running it, and show a change to a file run before. + for (const line of summarize(errand, "unverified: a local file, not fetched from a publisher's index")) deps.say(line); + const approved = store.approvals()[errand.name]; + if (approved && approved.sha256 !== sha256) { + deps.say(` this file changed since its last run (${approved.sha256.slice(0, 12)} on ${approved.at.slice(0, 10)}, now ${sha256.slice(0, 12)})`); + if (!flags.yes && !(deps.interactive && (await deps.confirm("Run the changed file?")))) { + deps.say("Not run. Read the change, then rerun with --yes."); + return EXIT.STOPPED; + } + } + + const metaVault = (errand.metadata as { vault?: unknown } | undefined)?.vault; + const targetText = flags.vault ?? (typeof metaVault === "string" ? metaVault : undefined); + const vault: Vault | null = targetText ? openVault(parseTarget(targetText), deps.logicsrc) : null; + const fallbackVault = fileVault(store.credentialsFile(errand.name)); + const reader = vault ?? fallbackVault; + + const candidate = flags.candidate !== undefined ? Number(flags.candidate) : undefined; + if (candidate !== undefined && (!Number.isInteger(candidate) || candidate < 1)) throw new ErrandError("--candidate is 1, 2, 3, ..."); + const inputs = await resolveInputs(errand, { + overrides: parseOverrides(flags.input ?? []), + vault: reader, + extractor: flags.extractor ? commandExtractor(flags.extractor, { errand: errand.name }) : null, + prompt: deps.interactive ? deps.prompt : async () => null, + random: deps.random, + now: deps.now(), + ...(candidate !== undefined ? { candidate } : {}), + }); + + deps.say("Values for this run (secrets masked):"); + for (const name of Object.keys(errand.inputs ?? {})) { + const resolved = inputs.values.get(name); + const qa = inputs.qa.get(name); + const from = qa ? (Object.keys(qa.answers).length ? "vault" : "generated as questions are chosen") : resolved ? `${resolved.from}${resolved.origin ? ` ${resolved.origin}` : ""}` : "none"; + deps.say(` ${name} = ${qa ? "••••" : inputs.display(name)} (${from})`); + } + if (inputs.chosen && inputs.candidates.length > 1) deps.say(` shared secret: candidate ${inputs.candidates.indexOf(inputs.chosen) + 1} of ${inputs.candidates.length} (${inputs.chosen.year ?? ""} ${inputs.chosen.form} ${inputs.chosen.field}, ${inputs.chosen.source})`); + deps.say(` credentials go to ${vault?.write ? vault.describe() : `${fallbackVault.describe()} (0600)`}`); + + store.approve(errand.name, sha256, file, deps.now()); + const origin = errand.site.origins[0]!; + const runId = `${deps.now().toISOString().replace(/[:.]/g, "-")}-${errand.name}`; + const { record } = await runErrand({ + errand, + source: file, + sha256, + inputs, + store, + declare: flags.declare === true, + dryRun: flags.dryRun === true, + headful: flags.headful === true, + interactive: deps.interactive, + prompt: deps.prompt, + say: deps.say, + now: deps.now, + random: deps.random, + vault, + fallbackVault, + account: flags.account ?? "default", + force: flags.force === true, + ...(deps.solver ? { solver: deps.solver } : {}), + ...(deps.pollMs !== undefined ? { pollMs: deps.pollMs } : {}), + ...(deps.rereadMs !== undefined ? { rereadMs: deps.rereadMs } : {}), + openDriver: + deps.openDriver ?? + (() => + openCdpDriver({ + errand, + headless: flags.headful !== true, + profile: store.profile(origin), + pageTimeoutMs: durationMs(errand.limits?.page_timeout, 30_000), + downloadDir: store.downloadsDir(runId), + ...(flags.chrome ? { chrome: flags.chrome } : {}), + ...(deps.chromeArgs ? { args: deps.chromeArgs } : {}), + ...(deps.settleMs !== undefined ? { settleMs: deps.settleMs } : {}), + })), + }); + + if (flags.json) deps.out(JSON.stringify(record, null, 2)); + else deps.out(`${record.name}: ${record.kind}${record.outcome !== record.kind ? ` (${record.outcome})` : ""}${record.reason ? `: ${record.reason}` : ""}`); + if (record.kind === "rejected") return EXIT.REJECTED; + if (record.kind === "stopped") return record.reason?.startsWith("throttle") ? EXIT.THROTTLED : EXIT.STOPPED; + return EXIT.OK; + } catch (error) { + deps.say((error as Error).message); + return error instanceof ErrandError && /not a valid OpenErrand|is not JSON|cannot read|^--input|^--candidate|^vault target/.test((error as Error).message) ? EXIT.INVALID : EXIT.STOPPED; + } +} + +export { describeTarget, type LogicsrcExec }; diff --git a/packages/openerrand/src/driver.ts b/packages/openerrand/src/driver.ts new file mode 100644 index 0000000..6bea055 --- /dev/null +++ b/packages/openerrand/src/driver.ts @@ -0,0 +1,285 @@ +/** + * The browser as the run loop sees it. {@link Driver} is an interface so the + * loop (gates, outcomes, loop detection) is tested against a scripted fake; + * {@link openCdpDriver} is the real one, on Chrome over CDP. + */ + +import { existsSync } from "node:fs"; +import { join } from "node:path"; +import { type Browser, launchBrowser, plainUserAgent } from "./browser.js"; +import type { Errand, Field, Page } from "./types.js"; +import { ErrandError, sleep } from "./util.js"; + +export type FillAction = { kind: "text" | "select"; value: string } | { kind: "check" }; + +export interface DownloadedFile { + url: string; + filename: string; + path: string; +} + +export interface Driver { + goto(url: string): Promise; + /** The current URL, read from the browser without running script on the page. */ + url(): Promise; + read(): Promise; + /** Which of these CSS selectors find an element on the current page. */ + selectorHits(selectors: string[]): Promise>; + fill(field: Field, action: FillAction): Promise; + /** Press the page's forward button. Returns its label, or `?a|b` with the buttons seen when none fits. */ + submit(): Promise; + /** Wait for the page that follows a submit to load. */ + settle(): Promise; + /** Files the site handed over during this run, completed. */ + downloads(): DownloadedFile[]; + /** Run script in the page. Used only by a captcha solver the errand allows; never on an identity provider's page. */ + evaluate(expression: string): Promise; + close(): Promise; +} + +const DEFAULT_LABELS = "^(submit|continue|next|send|verify|confirm|sign ?in|log ?in)$"; +const DEFAULT_NEVER = "^(back|cancel|previous)$"; +const DEFAULT_IGNORE = "#timer, .modal"; + +/** Every visible, enabled control, with the label a person would read for it. Values are never read. */ +export function readPageScript(ignore: string): string { + return `(() => { + const IGNORE = ${JSON.stringify(ignore)}; + const visible = (el) => !!(el.offsetWidth || el.offsetHeight || el.getClientRects().length) && getComputedStyle(el).visibility !== 'hidden'; + const clean = (s) => (s || '').replace(/\\*\\s*Required Field/gi, '').replace(/\\s+/g, ' ').trim(); + const ignored = (el) => { try { return !!el.closest(IGNORE); } catch { return false; } }; + const forLabel = (el) => el.id ? document.querySelector('label[for="' + CSS.escape(el.id) + '"]') : null; + const labelOf = (el) => { + const byFor = forLabel(el); + const wrap = el.closest('label'); + const group = el.closest('fieldset, .form-group, .row'); + const legend = group && group.querySelector('legend, .col-form-label'); + return clean((byFor && byFor.innerText) || el.getAttribute('aria-label') || (wrap && wrap.innerText) || (legend && legend.innerText) || el.placeholder || ''); + }; + const selectorOf = (el) => { + if (el.id) return '#' + CSS.escape(el.id); + if (el.name) return el.tagName.toLowerCase() + '[name="' + CSS.escape(el.name) + '"]' + (el.type === 'radio' || el.type === 'checkbox' ? '[value="' + CSS.escape(el.value) + '"]' : ''); + return null; + }; + const controls = [...document.querySelectorAll('input, select, textarea')] + .filter((el) => !['hidden', 'submit', 'button', 'image', 'reset', 'file'].includes(el.type) && !el.disabled && visible(el) && !ignored(el)); + let lastSelect = null; + const fields = []; + for (const el of controls) { + if (el.tagName === 'SELECT') lastSelect = el; + const selector = selectorOf(el); + if (!selector) continue; + const choice = el.type === 'radio' || el.type === 'checkbox'; + const own = clean(((el.closest('label') || forLabel(el) || {}).innerText) || ''); + const group = el.closest('fieldset, .form-group'); + const label = choice ? (own || labelOf(el)) : labelOf(el); + const printed = group ? clean([...group.querySelectorAll('p, span, div')].map((n) => n.children.length ? '' : n.innerText).join(' ')) : ''; + fields.push({ + selector, + id: el.id || '', + name: el.name || '', + type: el.type, + label, + required: el.required || el.getAttribute('aria-required') === 'true', + options: el.tagName === 'SELECT' ? [...el.options].map((o) => ({ value: o.value, text: o.text.trim() })) : undefined, + question: el.tagName !== 'SELECT' && !choice ? (lastSelect && lastSelect.selectedIndex > 0 ? lastSelect.options[lastSelect.selectedIndex].text.trim() : (printed || undefined)) : undefined, + groupChecked: el.type === 'radio' && el.name ? !!document.querySelector('input[type=radio][name="' + CSS.escape(el.name) + '"]:checked') : undefined, + }); + } + const errors = [...document.querySelectorAll('.alert-danger, .validation-summary-errors li, .field-validation-error, .error-message, .invalid-feedback, [role=alert]')] + .filter((n) => visible(n) && !ignored(n)).map((n) => clean(n.innerText)).filter(Boolean); + return { url: location.href, title: document.title, text: clean(document.body ? document.body.innerText : '').slice(0, 6000), errors: [...new Set(errors)], fields }; +})()`; +} + +/** Set a value with the element's native setter, then fire input, change and blur so the page's own validation sees it. */ +export function fillScript(selector: string, action: FillAction): string { + const target = `document.querySelector(${JSON.stringify(selector)})`; + if (action.kind === "check") { + return `(() => { const el = ${target}; if (el && !el.checked) el.click(); return !!el && el.checked; })()`; + } + return `(() => { + const el = ${target}; if (!el) return false; + const proto = el.tagName === 'SELECT' ? HTMLSelectElement.prototype : el.tagName === 'TEXTAREA' ? HTMLTextAreaElement.prototype : HTMLInputElement.prototype; + Object.getOwnPropertyDescriptor(proto, 'value').set.call(el, ${JSON.stringify(action.value)}); + for (const type of ['input', 'change', 'blur']) el.dispatchEvent(new Event(type, { bubbles: true })); + return el.value === ${JSON.stringify(action.value)}; + })()`; +} + +/** + * The page's own forward button: a label from `submit.labels`, else the only + * other button; never one matching `submit.never` or inside `submit.ignore`. + */ +export function submitScript(submit: Errand["submit"]): string { + const labels = submit?.labels ?? DEFAULT_LABELS; + const never = submit?.never ?? DEFAULT_NEVER; + const ignore = submit?.ignore ?? DEFAULT_IGNORE; + return `(() => { + const LABELS = new RegExp(${JSON.stringify(labels)}, 'i'); + const NEVER = new RegExp(${JSON.stringify(never)}, 'i'); + const IGNORE = ${JSON.stringify(ignore)}; + const visible = (el) => !!(el.offsetWidth || el.offsetHeight || el.getClientRects().length); + const ignored = (el) => { try { return !!el.closest(IGNORE); } catch { return false; } }; + const text = (b) => (b.innerText || b.value || '').replace(/\\s+/g, ' ').trim(); + const buttons = [...document.querySelectorAll('button, input[type=submit], input[type=button]')] + .filter((b) => visible(b) && !b.disabled && !ignored(b) && text(b) && !NEVER.test(text(b))); + const pick = buttons.find((b) => LABELS.test(text(b))) || (buttons.length === 1 ? buttons[0] : null); + if (!pick) return '?' + buttons.map(text).join('|'); + pick.click(); + return text(pick); +})()`; +} + +export interface CdpDriverOptions { + errand: Errand; + chrome?: string; + headless: boolean; + profile?: string; + pageTimeoutMs: number; + /** Where downloads land before they are filed. */ + downloadDir?: string; + args?: string[]; + /** Pause after each load so a page's own scripts finish (default 1500 ms). */ + settleMs?: number; +} + +/** The plain user agent of each Chrome binary, learned once per process. */ +const plainAgents = new Map(); + +export async function openCdpDriver(options: CdpDriverOptions): Promise { + const launch = (extra: string[]): Promise => + launchBrowser({ + ...(options.chrome ? { chrome: options.chrome } : {}), + ...(options.profile ? { profile: options.profile } : {}), + headless: options.headless, + timeoutMs: 30_000, + args: [...(options.args ?? []), ...extra], + }); + // Rule 11. A CDP user-agent override does not reach every request (a + // navigation the page's own script starts still says HeadlessChrome), so + // the plain string is given to Chrome at launch: the browser's own user + // agent with "HeadlessChrome" replaced by "Chrome", and nothing else. + const key = options.chrome ?? "default"; + let browser: Browser; + if (options.headless && plainAgents.has(key)) { + const plain = plainAgents.get(key); + browser = await launch(plain ? [`--user-agent=${plain}`] : []); + } else { + browser = await launch([]); + if (options.headless) { + const { userAgent } = (await browser.cdp.send("Browser.getVersion")) as { userAgent: string }; + const plain = plainUserAgent(userAgent); + plainAgents.set(key, plain); + if (plain) { + await browser.close(); + browser = await launch([`--user-agent=${plain}`]); + } + } + } + const { cdp } = browser; + const settleMs = options.settleMs ?? 1_500; + try { + const { targetInfos } = (await cdp.send("Target.getTargets")) as { targetInfos: Array<{ targetId: string; type: string }> }; + let targetId = targetInfos.find((t) => t.type === "page")?.targetId; + if (!targetId) ({ targetId } = (await cdp.send("Target.createTarget", { url: "about:blank" })) as { targetId: string }); + const { sessionId } = (await cdp.send("Target.attachToTarget", { targetId, flatten: true })) as { sessionId: string }; + await cdp.send("Page.enable", {}, sessionId); + + const downloads: DownloadedFile[] = []; + if (options.downloadDir) { + await cdp.send("Browser.setDownloadBehavior", { behavior: "allowAndName", downloadPath: options.downloadDir, eventsEnabled: true }); + const begun = new Map(); + cdp.on((event) => { + if (event.method === "Browser.downloadWillBegin") { + begun.set(String(event.params.guid), { url: String(event.params.url), filename: String(event.params.suggestedFilename) }); + } else if (event.method === "Browser.downloadProgress" && event.params.state === "completed") { + const guid = String(event.params.guid); + const meta = begun.get(guid); + const path = join(options.downloadDir!, guid); + if (meta && existsSync(path)) downloads.push({ ...meta, path }); + } + }); + } + + const evaluate = async (expression: string): Promise => { + try { + const { result, exceptionDetails } = (await cdp.send("Runtime.evaluate", { expression, returnByValue: true, awaitPromise: true }, sessionId)) as { + result: { value?: unknown }; + exceptionDetails?: unknown; + }; + return exceptionDetails ? null : ((result.value ?? null) as T | null); + } catch { + return null; + } + }; + const ignore = options.errand.submit?.ignore ?? DEFAULT_IGNORE; + let pending: Promise | null = null; + + return { + async goto(url) { + const loaded = cdp.waitFor("Page.loadEventFired", sessionId, options.pageTimeoutMs).catch(() => undefined); + const result = (await cdp.send("Page.navigate", { url }, sessionId)) as { errorText?: string }; + if (result.errorText) throw new ErrandError(`could not open ${url}: ${result.errorText}`); + await loaded; + await sleep(settleMs); + }, + async url() { + const { targetInfo } = (await cdp.send("Target.getTargetInfo", { targetId })) as { targetInfo: { url: string } }; + return targetInfo.url; + }, + async read() { + const page = await evaluate(readPageScript(ignore)); + if (!page) throw new ErrandError("could not read the page"); + return page; + }, + async selectorHits(selectors) { + if (!selectors.length) return {}; + const hits = await evaluate>( + `(() => { const out = {}; for (const s of ${JSON.stringify(selectors)}) { try { out[s] = !!document.querySelector(s); } catch { out[s] = false; } } return out; })()`, + ); + return hits ?? {}; + }, + async fill(field, action) { + return (await evaluate(fillScript(field.selector, action))) === true; + }, + async submit() { + const loaded = cdp.waitFor("Page.loadEventFired", sessionId, options.pageTimeoutMs).catch(() => undefined); + const pressed = (await evaluate(submitScript(options.errand.submit))) ?? "?"; + if (!pressed.startsWith("?")) pending = loaded; + else void loaded; + return pressed; + }, + async settle() { + await Promise.race([pending ?? Promise.resolve(), sleep(options.pageTimeoutMs)]); + pending = null; + await sleep(settleMs); + }, + downloads: () => [...downloads], + evaluate: (expression) => evaluate(expression), + close: () => browser.close(), + }; + } catch (error) { + await browser.close(); + throw error; + } +} + +/** A download is what its media type says, checked by its first bytes. Unknown types are refused. */ +export function looksLike(type: string, head: Buffer): boolean { + const starts = (bytes: number[]): boolean => bytes.every((b, i) => head[i] === b); + if (type === "application/pdf") return starts([0x25, 0x50, 0x44, 0x46]); + if (type === "image/png") return starts([0x89, 0x50, 0x4e, 0x47]); + if (type === "image/jpeg") return starts([0xff, 0xd8, 0xff]); + if (type === "application/zip") return starts([0x50, 0x4b, 0x03, 0x04]); + if (type === "application/json") { + try { + JSON.parse(head.toString("utf8")); + return true; + } catch { + return false; + } + } + if (type.startsWith("text/")) return !head.includes(0); + return false; +} diff --git a/packages/openerrand/src/extract.ts b/packages/openerrand/src/extract.ts new file mode 100644 index 0000000..5515124 --- /dev/null +++ b/packages/openerrand/src/extract.ts @@ -0,0 +1,76 @@ +/** + * The one document extractor the runner ships: a command the principal names + * (`--extractor "python3 extract.py ~/taxes"`), run on this machine, that + * reads a JSON request on stdin and prints JSON records on stdout. + * + * Request: `{ "errand": "", "documents": [{ "input", "form", "field", "match"? }] }` + * Reply: `[{ "form", "field", "value", "year"?, "label"?, "file"?, "page"? }]` + * (or `{ "records": [...] }`). + * + * The extractor reads the documents; the runner only ever sees the values it + * prints, keeps them in memory, and records the file and page each came from + * for the person to see. Nothing about a document is sent anywhere else. + */ + +import { spawn } from "node:child_process"; +import type { DocumentExtractor, DocumentRecord, DocumentRequest } from "./inputs.js"; +import { ErrandError } from "./util.js"; + +function isRecord(row: unknown): row is DocumentRecord { + if (!row || typeof row !== "object") return false; + const r = row as Record; + return ( + typeof r.form === "string" && + typeof r.field === "string" && + (typeof r.value === "string" || typeof r.value === "number") && + (r.year === undefined || Number.isInteger(r.year)) && + (r.label === undefined || typeof r.label === "string") && + (r.file === undefined || typeof r.file === "string") && + (r.page === undefined || Number.isInteger(r.page)) + ); +} + +export function parseRecords(stdout: string): DocumentRecord[] { + let parsed: unknown; + try { + parsed = JSON.parse(stdout); + } catch { + throw new ErrandError("the extractor printed something that is not JSON"); + } + const rows = Array.isArray(parsed) ? parsed : (parsed as { records?: unknown })?.records; + if (!Array.isArray(rows)) throw new ErrandError("the extractor did not print a list of records"); + return rows.filter(isRecord); +} + +export function commandExtractor(command: string, options: { errand: string; timeoutMs?: number; cwd?: string } = { errand: "" }): DocumentExtractor { + return { + extract(requests: DocumentRequest[]): Promise { + return new Promise((resolve, reject) => { + const child = spawn("/bin/sh", ["-c", command], { stdio: ["pipe", "pipe", "pipe"], ...(options.cwd ? { cwd: options.cwd } : {}) }); + let stdout = ""; + let stderr = ""; + const timer = setTimeout(() => { + child.kill(); + reject(new ErrandError(`the extractor did not finish in ${(options.timeoutMs ?? 120_000) / 1000}s`)); + }, options.timeoutMs ?? 120_000); + child.stdout.on("data", (chunk: Buffer) => (stdout += chunk.toString())); + child.stderr.on("data", (chunk: Buffer) => (stderr += chunk.toString())); + child.on("error", (error) => { + clearTimeout(timer); + reject(new ErrandError(`could not run the extractor: ${error.message}`)); + }); + child.on("close", (code) => { + clearTimeout(timer); + // stderr is the extractor's own; it is not echoed, since an extractor may print what it read. + if (code !== 0) return reject(new ErrandError(`the extractor exited with ${code}${stderr ? ` (${stderr.trim().split("\n").length} lines on stderr, not shown)` : ""}`)); + try { + resolve(parseRecords(stdout)); + } catch (error) { + reject(error); + } + }); + child.stdin.end(JSON.stringify({ errand: options.errand, documents: requests })); + }); + }, + }; +} diff --git a/packages/openerrand/src/fake-site.ts b/packages/openerrand/src/fake-site.ts new file mode 100644 index 0000000..0b07c0b --- /dev/null +++ b/packages/openerrand/src/fake-site.ts @@ -0,0 +1,231 @@ +/** + * A tiny fake MyFTB for the integration test: Terms, Profile, a + * proof-of-work interstitial that clears itself, Business, Phone, Code and + * Confirmation, served over HTTPS with a throwaway self-signed certificate. + * Chrome reaches it as https://webapp.ftb.ca.gov through + * `--host-resolver-rules`, with every other hostname mapped to NOTFOUND, so + * the test runs the published example file unchanged and cannot reach the + * real site. Every value it checks is fictional (Jane Doe, corporation + * 1234567). + * + * Not part of the build. + */ + +import { execFileSync } from "node:child_process"; +import { mkdtempSync, readFileSync, rmSync } from "node:fs"; +import { createServer, type Server } from "node:https"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; + +export const FAKE = { + firstName: "Jane", + lastName: "Doe", + zip: "95814", + addressNumbers: "1234", + corpId: "1234567", + netIncome: "48210", + year: "2025", + phone: "5555550100", + code: "482913", +} as const; + +export interface FakeSite { + port: number; + /** Every request: method, path and user agent. */ + requests: Array<{ method: string; path: string; userAgent: string }>; + /** Called on GET /Code and on a wrong code, so the test can relay the next code. */ + onCode: (attempt: number) => void; + posted: Record>; + close(): Promise; +} + +const page = (title: string, body: string, error = ""): string => `${title} +
Franchise Tax Board e-Services
+

${title.replace(/^.*\| /, "")}

+${error ? `` : ""} +${body} +
`; + +const form = (action: string, inner: string, buttons = ' '): string => + `
${inner}
${buttons}
`; + +const input = (id: string, label: string, type = "text", required = true): string => + `
`; + +const select = (id: string, label: string, options: string[], required = true): string => + `
`; + +const QUESTIONS = ["What was the name of your first pet?", "What city were you born in?", "What was your first car?", "What was the name of your elementary school?"]; + +const PAGES = { + terms: () => + page( + "Registration | Terms", + form( + "/MyFTBAccess/Registration/NewAccount", + `

Terms of use.

+
+
`, + ), + ), + profile: (error = "") => + page( + "Registration | Profile", + form( + "/MyFTBAccess/Registration/Profile", + [ + input("FstName", "First Name"), + input("MInitial", "Middle Initial", "text", false), + input("LstName", "Last Name"), + input("UserName", "User Name"), + input("ReUserName", "Confirm User Name"), + input("Email", "Email Address", "email"), + input("ReEmail", "Confirm Email Address", "email"), + input("Password", "Password", "password"), + input("RePassword", "Confirm Password", "password"), + ...[1, 2, 3].flatMap((n) => [select(`SecQ${n}`, `Security Question ${n}`, QUESTIONS), input(`SecA${n}`, `Answer ${n}`)]), + ].join("\n"), + ), + error, + ), + challenge: () => `Challenge Validation +
Checking your browser…
+ +`, + business: (error = "") => + page( + "Registration | Business", + form( + "/MyFTBAccess/Registration/Business", + `
I am registering as + +
+${select("CoType", "Type of company", ["Corporation", "Partnership", "Limited Liability Company"])} +${select("FormType", "Form type", ["100", "100S", "100W", "109"])} +${select("TaxYear", "Tax year", ["2025", "2024", "2023"])} +${input("CorpNo", "California corporation number")} +${input("Zip", "ZIP code", "tel")} +${input("AddrNum", "The numbers in your mailing address", "tel")} +${input("NetInc", "Net income for tax purposes", "tel")} +
`, + ), + error, + ), + phone: () => + page( + "Registration | Phone", + form( + "/MyFTBAccess/Registration/Phone", + `${input("PhoneNumber", "Phone number", "tel")} + + +`, + ), + ), + code: (error = "") => + page("Registration | Verify", form("/MyFTBAccess/Registration/Code", input("VerificationCode", "Enter the verification code", "text"), ''), error), + confirmation: () => + page( + "Registration Confirmation", + "

Registration confirmation: your MyFTB account was successfully created.

We will mail you a PIN at the address we have on file. It expires 21 days from today.

", + ), +}; + +function readBody(req: import("node:http").IncomingMessage): Promise> { + return new Promise((resolve) => { + let body = ""; + req.on("data", (chunk: Buffer) => (body += chunk.toString())); + req.on("end", () => resolve(Object.fromEntries(new URLSearchParams(body)))); + }); +} + +/** A self-signed certificate for webapp.ftb.ca.gov, made with openssl in a temp dir. */ +export function selfSigned(): { key: Buffer; cert: Buffer } { + const dir = mkdtempSync(join(tmpdir(), "openerrand-cert-")); + execFileSync("openssl", ["req", "-x509", "-newkey", "rsa:2048", "-nodes", "-keyout", join(dir, "key.pem"), "-out", join(dir, "cert.pem"), "-days", "1", "-subj", "/CN=webapp.ftb.ca.gov"], { stdio: "ignore" }); + try { + return { key: readFileSync(join(dir, "key.pem")), cert: readFileSync(join(dir, "cert.pem")) }; + } finally { + rmSync(dir, { recursive: true, force: true }); + } +} + +export async function startFakeSite(): Promise { + const requests: FakeSite["requests"] = []; + const posted: FakeSite["posted"] = {}; + let codeAttempts = 0; + const site: Partial = { requests, posted, onCode: () => undefined }; + + const send = (res: import("node:http").ServerResponse, html: string, status = 200): void => { + res.writeHead(status, { "content-type": "text/html; charset=utf-8", "cache-control": "no-store" }); + res.end(html); + }; + const go = (res: import("node:http").ServerResponse, to: string): void => { + res.writeHead(303, { location: to }); + res.end(); + }; + + const server: Server = createServer(selfSigned(), async (req, res) => { + const path = (req.url ?? "/").split("?")[0]!; + requests.push({ method: req.method ?? "GET", path, userAgent: String(req.headers["user-agent"] ?? "") }); + const base = "/MyFTBAccess/Registration"; + if (req.method === "GET") { + if (path === `${base}/NewAccount`) return send(res, PAGES.terms()); + if (path === `${base}/Profile`) return send(res, PAGES.profile()); + if (path === `${base}/Challenge`) return send(res, PAGES.challenge()); + if (path === `${base}/Business`) return send(res, PAGES.business()); + if (path === `${base}/Phone`) return send(res, PAGES.phone()); + if (path === `${base}/Code`) { + codeAttempts += 1; + site.onCode!(codeAttempts); + return send(res, PAGES.code()); + } + if (path === `${base}/Confirmation`) return send(res, PAGES.confirmation()); + return send(res, page("Not found", "

Not found

"), 404); + } + const body = await readBody(req); + posted[path] = body; + if (path === `${base}/NewAccount`) return body.ReadTerms && body.AcceptTerms ? go(res, `${base}/Profile`) : send(res, PAGES.terms()); + if (path === `${base}/Profile`) { + const ok = + body.FstName === FAKE.firstName && + body.LstName === FAKE.lastName && + body.UserName && + body.UserName === body.ReUserName && + body.Email === body.ReEmail && + (body.Password ?? "").length >= 15 && + body.Password === body.RePassword && + new Set([body.SecQ1, body.SecQ2, body.SecQ3]).size === 3 && + [body.SecA1, body.SecA2, body.SecA3].every((a) => (a ?? "").length >= 3); + return ok ? go(res, `${base}/Challenge`) : send(res, PAGES.profile("Please correct the errors on this page.")); + } + if (path === `${base}/Business`) { + const ok = + body.Role === "business" && + body.CoType === "Corporation" && + body.FormType === "100S" && + body.TaxYear === FAKE.year && + body.CorpNo === FAKE.corpId && + body.Zip === FAKE.zip && + body.AddrNum === FAKE.addressNumbers && + body.NetInc === FAKE.netIncome && + body.Decl === "true"; + return ok ? go(res, `${base}/Phone`) : send(res, PAGES.business("The information you entered does not match our records.")); + } + if (path === `${base}/Phone`) return body.PhoneNumber === FAKE.phone && body.How === "text" ? go(res, `${base}/Code`) : send(res, PAGES.phone()); + if (path === `${base}/Code`) { + if (body.VerificationCode === FAKE.code) return go(res, `${base}/Confirmation`); + codeAttempts += 1; + site.onCode!(codeAttempts); + return send(res, PAGES.code("The code you entered is incorrect. Enter the newest code we sent.")); + } + return send(res, page("Not found", "

Not found

"), 404); + }); + + await new Promise((resolve) => server.listen(0, "127.0.0.1", resolve)); + const address = server.address(); + if (!address || typeof address === "string") throw new Error("no port"); + site.port = address.port; + site.close = () => new Promise((resolve) => server.close(() => resolve())); + return site as FakeSite; +} diff --git a/packages/openerrand/src/index.ts b/packages/openerrand/src/index.ts new file mode 100644 index 0000000..e0ebcd7 --- /dev/null +++ b/packages/openerrand/src/index.ts @@ -0,0 +1,38 @@ +/** + * @logicsrc/openerrand: the reference runner for OpenErrand 0.1. + * + * `logicsrc errand run ` is the command; this module is the library + * behind it, for a program that wants to embed the runner (its own driver, + * extractor, vault or prompt). + */ + +export * from "./types.js"; +export { ErrandError, durationMs, render, transform, templateNames, MASK } from "./util.js"; +export { + Inputs, + QaSet, + resolveInputs, + generate, + documentValues, + inYears, + type Candidate, + type DocumentExtractor, + type DocumentRecord, + type DocumentRequest, + type Prompt, + type ResolveOptions, + type VaultReader, +} from "./inputs.js"; +export { commandExtractor, parseRecords } from "./extract.js"; +export { act, decide, describeField, isChoice, type Action, type Decision } from "./rules.js"; +export { DEFAULT_LOCKOUT, allowedOrigins, fits, isGate, isLockout, lockoutPattern, outcomeOf, outcomeText, selectorsOf, solverPermitted, stepFor } from "./pages.js"; +export { LIMITS, checkThrottle, emptyLedger, keyFor, lockoutMs, recordAttempt, recordLockout, type Ledger, type ThrottleCheck } from "./throttle.js"; +export { Store, errandHome, type Card, type RunRecord } from "./store.js"; +export { fileVault, opencredsVault, openVault, parseEnv, parseTarget, serializeEnv, teamsVault, type LogicsrcExec, type Vault, type VaultTarget } from "./vault.js"; +export { credentialValues, expiresOn, fileDownloads, renderCard, writeCredentials } from "./outputs.js"; +export { type CaptchaContext, type CaptchaSolver } from "./captcha.js"; +export { fillScript, looksLike, openCdpDriver, readPageScript, submitScript, type Driver, type FillAction } from "./driver.js"; +export { findChrome, launchBrowser, plainUserAgent } from "./browser.js"; +export { loadErrand, sha256, summarize, validateErrand } from "./load.js"; +export { runErrand, type RunDeps, type RunResult } from "./run.js"; +export { EXIT, registerErrandCommands, runCommand, type Deps } from "./commands.js"; diff --git a/packages/openerrand/src/inputs.test.ts b/packages/openerrand/src/inputs.test.ts new file mode 100644 index 0000000..6e1c98a --- /dev/null +++ b/packages/openerrand/src/inputs.test.ts @@ -0,0 +1,174 @@ +import { describe, expect, it } from "vitest"; +import { type DocumentRecord, generate, inYears, resolveInputs, type VaultReader } from "./inputs.js"; +import { ftbExample } from "./testing.js"; +import type { Errand } from "./types.js"; +import { transform } from "./util.js"; + +const NOW = new Date("2026-10-04T12:00:00Z"); + +/** Fictional returns for Jane Doe and her corporation; nothing here is anyone's real data. */ +const RECORDS: DocumentRecord[] = [ + { form: "CA 540", field: "first name", value: "Jane", year: 2025, file: "2025/540.pdf", page: 1 }, + { form: "CA 540", field: "last name", value: "Doe", year: 2025, file: "2025/540.pdf", page: 1 }, + { form: "CA 540", field: "street address", value: "1234 Maple St", year: 2025, file: "2025/540.pdf", page: 1 }, + { form: "CA 540", field: "ZIP code", value: "95814-1234", year: 2025, file: "2025/540.pdf", page: 1 }, + { form: "CA 540", field: "street address", value: "99 Old Rd", year: 2023, file: "2023/540.pdf", page: 1 }, + { form: "CA 100S", field: "California corporation number", value: "1234567", year: 2025, file: "2025/100S.pdf", page: 1 }, + { form: "CA 100S", field: "line 20", label: "Net income for tax purposes", value: "48,210.40", year: 2025, file: "2025/100S.pdf", page: 3 }, + { form: "CA 100S", field: "line 15", label: "Net income (loss) for state purposes", value: "51,377", year: 2025, file: "2025/100S.pdf", page: 3 }, + { form: "CA 100S", field: "line 20", label: "Net income for tax purposes", value: "39,875", year: 2024, file: "2024/100S.pdf", page: 3 }, + // The year in progress and a year too old: neither counts. + { form: "CA 100S", field: "line 20", label: "Net income for tax purposes", value: "1", year: 2026, file: "2026/100S.pdf", page: 3 }, + { form: "CA 100S", field: "line 20", label: "Net income for tax purposes", value: "2", year: 2019, file: "2019/100S.pdf", page: 3 }, +]; + +const extractor = { extract: async () => RECORDS }; +const noVault: VaultReader = { describe: () => "none", get: async () => undefined }; + +function tiny(inputs: Errand["inputs"]): Errand { + return { + type: "logicsrc.openerrand", + version: "0.1", + name: "t", + title: "t", + site: { name: "x", origins: ["https://example.com"], start: ["https://example.com/"] }, + inputs, + steps: [{ id: "form", kind: "page" }], + outcomes: [{ name: "ok", kind: "success", text: "ok" }], + }; +} + +describe("the FTB example's inputs", () => { + it("resolves documents, derives, candidates and generated credentials", async () => { + const inputs = await resolveInputs(ftbExample(), { extractor, vault: noVault, now: NOW, overrides: { email: "jane@example.com", phone: "5555550100" } }); + expect(inputs.get("first_name")).toBe("Jane"); + expect(inputs.get("address_numbers")).toBe("1234"); + expect(inputs.get("zip")).toBe("95814"); + expect(inputs.get("corp_id")).toBe("1234567"); + // Shared secret candidates: line 20 newest first, then line 15; the year in progress and 2019 are out. + expect(inputs.candidates.map((c) => [c.year, c.field, c.value])).toEqual([ + [2025, "line 20", "48210"], + [2024, "line 20", "39875"], + [2025, "line 15", "51377"], + ]); + expect(inputs.get("net_income")).toBe("48210"); + expect(inputs.get("tax_year")).toBe("2025"); + expect(inputs.get("username")).toMatch(/^[a-z][a-z0-9]{14}$/); + expect(inputs.get("password")).toHaveLength(24); + expect(inputs.values.get("username")?.from).toBe("generate"); + }); + + it("--candidate chooses another candidate, and the tax year follows it", async () => { + const inputs = await resolveInputs(ftbExample(), { extractor, now: NOW, candidate: 2, overrides: { email: "jane@example.com", phone: "5555550100" } }); + expect(inputs.get("net_income")).toBe("39875"); + expect(inputs.get("tax_year")).toBe("2024"); + }); + + it("reuses a vault login before generating one", async () => { + const vault: VaultReader = { + describe: () => "test", + get: async (key) => ({ FTB_BUSINESS_USERNAME: "jdoe1", FTB_BUSINESS_SECURITY_ANSWERS: '{"Pet?":"rex"}' })[key], + }; + const inputs = await resolveInputs(ftbExample(), { extractor, vault, now: NOW, overrides: { email: "jane@example.com", phone: "5555550100" } }); + expect(inputs.get("username")).toBe("jdoe1"); + expect(inputs.values.get("username")).toMatchObject({ from: "vault", origin: "FTB_BUSINESS_USERNAME" }); + expect(inputs.values.get("password")?.from).toBe("generate"); + expect(inputs.qa.get("security")?.answers).toEqual({ "Pet?": "rex" }); + }); + + it("masks secrets on the terminal and shows personal values to the principal", async () => { + const inputs = await resolveInputs(ftbExample(), { extractor, now: NOW, overrides: { email: "jane@example.com", phone: "5555550100" } }); + expect(inputs.display("password")).toBe("••••"); + expect(inputs.display("net_income")).toBe("••••"); + expect(inputs.display("first_name")).toBe("Jane"); + }); + + it("stops before the browser when a required input has no value", async () => { + await expect(resolveInputs(ftbExample(), { extractor, now: NOW })).rejects.toThrow(/no value for Email address/); + }); + + it("checks a value against the input's pattern", async () => { + await expect(resolveInputs(ftbExample(), { extractor, now: NOW, overrides: { email: "jane@example.com", phone: "555" } })).rejects.toThrow(/phone does not match/); + }); + + it("refuses an --input for a name the errand does not have", async () => { + await expect(resolveInputs(ftbExample(), { overrides: { nope: "1" } })).rejects.toThrow(/no input named nope/); + }); +}); + +describe("sources", () => { + it("tries sources in order: a prompt with nobody at a terminal falls through to the next", async () => { + const e = tiny({ a: { type: "string", sensitivity: "public", sources: [{ from: "prompt" }, { from: "literal", value: "fixed" }] } }); + expect((await resolveInputs(e, { prompt: async () => null })).get("a")).toBe("fixed"); + expect((await resolveInputs(e, { prompt: async () => "typed" })).get("a")).toBe("typed"); + }); + + it("asks for a secret without echo", async () => { + const asked: boolean[] = []; + const e = tiny({ pw: { type: "string", sensitivity: "secret", sources: [{ from: "prompt", ask: "Password: " }] } }); + await resolveInputs(e, { prompt: async (_q, o) => (asked.push(o.secret), "x") }); + expect(asked).toEqual([true]); + }); + + it("cuts a value to max_length", async () => { + const e = tiny({ n: { type: "string", sensitivity: "personal", max_length: 3, sources: [{ from: "literal", value: "Alexandra" }] } }); + expect((await resolveInputs(e)).get("n")).toBe("Ale"); + }); + + it("picks the oldest when asked", async () => { + const e = tiny({ s: { type: "string", sensitivity: "personal", sources: [{ from: "document", form: "CA 540", field: "street address", pick: "oldest" }] } }); + expect((await resolveInputs(e, { extractor, now: NOW })).get("s")).toBe("99 Old Rd"); + }); + + it("a document match tests the printed label", async () => { + const e = tiny({ s: { type: "integer", sensitivity: "secret", sources: [{ from: "document", form: "100S", field: "line 20", match: "state purposes", transform: "whole" }] } }); + expect((await resolveInputs(e, { extractor, now: NOW })).get("s")).toBeUndefined(); + }); + + it("detects an input that depends on itself through another", async () => { + const e = tiny({ + a: { type: "string", sensitivity: "public", sources: [{ from: "derive", input: "b", transform: "trim" }] }, + b: { type: "string", sensitivity: "public", sources: [{ from: "derive", input: "a", transform: "trim" }] }, + }); + await expect(resolveInputs(e)).rejects.toThrow(/depends on itself/); + }); +}); + +describe("generate", () => { + it("has at least one of each class and only the specials given", () => { + let n = 0; + const random = (max: number) => (n++ * 7) % max; + for (let i = 0; i < 20; i += 1) { + const v = generate({ from: "generate", length: 12, classes: ["lower", "upper", "digit", "special"], special: "!#" }, random); + expect(v).toHaveLength(12); + expect(v).toMatch(/[a-z]/); + expect(v).toMatch(/[A-Z]/); + expect(v).toMatch(/[2-9]/); + expect(v).toMatch(/[!#]/); + expect(v).not.toMatch(/[^a-zA-Z0-9!#]/); + } + }); + + it("starts with a letter when letters are allowed", () => { + for (let i = 0; i < 50; i += 1) expect(generate({ from: "generate", length: 15, classes: ["lower", "digit"] })).toMatch(/^[a-z]/); + }); +}); + +describe("transforms and years", () => { + it("whole keeps whole units with a minus for a loss", () => { + expect(transform("48,210.99", "whole")).toBe("48210"); + expect(transform("(1,234)", "whole")).toBe("-1234"); + expect(transform("-1,234.5", "whole")).toBe("-1234"); + expect(transform("1234 Maple St, Apt 5", "digits")).toBe("12345"); + expect(transform("95814-1234", "first:5")).toBe("95814"); + expect(transform("123456789", "last:4")).toBe("6789"); + }); + + it("counts closed years back and leaves out the year in progress unless asked", () => { + expect(inYears(2025, { back: 5, current: false }, NOW)).toBe(true); + expect(inYears(2021, { back: 5, current: false }, NOW)).toBe(true); + expect(inYears(2020, { back: 5, current: false }, NOW)).toBe(false); + expect(inYears(2026, { back: 5, current: false }, NOW)).toBe(false); + expect(inYears(2026, { back: 5, current: true }, NOW)).toBe(true); + }); +}); diff --git a/packages/openerrand/src/inputs.ts b/packages/openerrand/src/inputs.ts new file mode 100644 index 0000000..1e97f40 --- /dev/null +++ b/packages/openerrand/src/inputs.ts @@ -0,0 +1,435 @@ +/** + * Inputs: every value an errand needs, resolved from its sources in order + * before the browser opens, so a missing value stops the run before anything + * is sent and a dry run can show what each page would receive. + * + * Sources and where they are served from: + * - `document`: a {@link DocumentExtractor} hook. The runner ships one, which + * runs a command the principal names and reads JSON from it; the document + * never leaves the machine because the runner never sees more than the + * values the extractor prints. + * - `vault`: a {@link VaultReader} (logicsrc teams, OpenCreds, or the local + * 0600 state file). Read-only here; writing happens in outputs. + * - `prompt`: asked at the terminal, without echo for a secret. + * - `generate`: the runner's cryptographic generator. + * - `derive`, `candidate`, `literal`: computed from the file and other inputs. + * + * `--input name=value` on the command line is the person typing the value, and + * wins over every source. + */ + +import { randomInt } from "node:crypto"; +import type { Errand, Input, Source } from "./types.js"; +import { ErrandError, MASK, test, transform } from "./util.js"; + +/** One value an extractor found in a document. `file`/`page` say where, for the person; they never reach a site. */ +export interface DocumentRecord { + form: string; + field: string; + value: string | number; + year?: number; + /** The label as printed beside the value, tested against a source's `match`. */ + label?: string; + file?: string; + page?: number; +} + +export interface DocumentRequest { + input: string; + form: string; + field: string; + match?: string; +} + +/** + * Pluggable local extraction. The runner calls it once per run with every + * document source of the errand and keeps what it returns in memory only. + */ +export interface DocumentExtractor { + extract(requests: DocumentRequest[]): Promise; +} + +/** Read-only view of the principal's vault. */ +export interface VaultReader { + /** A human description such as `teams profullstack/ftb/prod`, shown in summaries. */ + describe(): string; + get(key: string): Promise; +} + +export interface PromptOptions { + secret: boolean; +} + +/** null means nobody is at a terminal to answer. */ +export type Prompt = (question: string, options: PromptOptions) => Promise; + +/** One candidate for a shared-secret input, with where it came from. */ +export interface Candidate { + value: string; + year?: number; + form: string; + field: string; + /** `2025/100S.pdf p3`: shown to the person, never sent anywhere. */ + source: string; +} + +export interface Resolved { + value: string; + /** Which source gave it: `flag` for --input. */ + from: Source["from"] | "flag"; + /** The vault key, or the file and page a document value came from. */ + origin?: string; +} + +export interface ResolveOptions { + overrides?: Record; + vault?: VaultReader | null; + extractor?: DocumentExtractor | null; + prompt?: Prompt; + random?: (max: number) => number; + now?: Date; + /** 1-based choice among a shared secret's candidates; the default is the best one. */ + candidate?: number; +} + +const ALPHABET = { + lower: "abcdefghijkmnopqrstuvwxyz", + upper: "ABCDEFGHJKLMNPQRSTUVWXYZ", + digit: "23456789", +} as const; +const DEFAULT_SPECIAL = "!#$*@"; + +/** + * A fresh random value with at least one character of each class. When + * letters are allowed the first character is a letter, because user names on + * most sites may not start with a digit. + */ +export function generate(source: Extract, random: (max: number) => number = randomInt): string { + const sets = source.classes.map((c) => (c === "special" ? source.special ?? DEFAULT_SPECIAL : ALPHABET[c])); + const all = sets.join(""); + const chars = sets.map((set) => set[random(set.length)]!); + while (chars.length < source.length) chars.push(all[random(all.length)]!); + for (let i = chars.length - 1; i > 0; i -= 1) { + const j = random(i + 1); + [chars[i], chars[j]] = [chars[j]!, chars[i]!]; + } + const out = chars.slice(0, source.length); + const letters = (source.classes.includes("lower") ? ALPHABET.lower : "") + (source.classes.includes("upper") ? ALPHABET.upper : ""); + if (letters && !letters.includes(out[0]!)) { + const at = out.findIndex((c) => letters.includes(c)); + if (at > 0) [out[0], out[at]] = [out[at]!, out[0]!]; + } + return out.join(""); +} + +const norm = (s: string): string => s.toLowerCase().replace(/^(form|ca)\s+/g, "").replace(/\s+/g, " ").trim(); + +/** Same form and field, ignoring case, spacing and a leading "Form"/"CA" ("CA 100S" is "100S"). */ +function sameField(record: DocumentRecord, source: Extract): boolean { + return norm(record.form) === norm(source.form) && norm(record.field) === norm(source.field); +} + +/** A closed year within `back`, and the year in progress only when `current` says so. */ +export function inYears(year: number | undefined, years: { back?: number; current?: boolean } | undefined, now: Date): boolean { + if (!years) return true; + if (year === undefined) return false; + const current = now.getFullYear(); + if (year >= current && !years.current) return false; + if (years.back !== undefined && year < current - years.back) return false; + return true; +} + +/** Records that fit one document source, newest year first, transformed. */ +export function documentValues(records: readonly DocumentRecord[], source: Extract, now: Date): Candidate[] { + const out: Candidate[] = []; + for (const record of records) { + if (!sameField(record, source)) continue; + if (source.match && record.label !== undefined && !test(source.match, record.label)) continue; + if (!inYears(record.year, source.years, now)) continue; + const value = transform(String(record.value), source.transform); + if (value === "") continue; + out.push({ + value, + ...(record.year !== undefined ? { year: record.year } : {}), + form: source.form, + field: source.field, + source: [record.file, record.page !== undefined ? `p${record.page}` : undefined].filter(Boolean).join(" ") || "extractor", + }); + } + return out.sort((a, b) => (b.year ?? 0) - (a.year ?? 0)); +} + +function checkValue(name: string, input: Input, value: string): string { + const cut = input.max_length !== undefined && input.type !== "qa-set" ? value.slice(0, input.max_length) : value; + if (input.type === "integer" && !/^-?\d+$/.test(cut)) throw new ErrandError(`${name} must be a whole number`); + if (input.type === "number" && !Number.isFinite(Number(cut))) throw new ErrandError(`${name} must be a number`); + if (input.type === "email" && !/^[^\s@]+@[^\s@]+\.[^\s@]+$/.test(cut)) throw new ErrandError(`${name} must be an email address`); + if (input.type === "boolean" && !/^(true|false)$/i.test(cut)) throw new ErrandError(`${name} must be true or false`); + if (input.pattern && !test(input.pattern, cut)) throw new ErrandError(`${name} does not match ${input.pattern}`); + return cut; +} + +/** + * Security questions and answers. A vault value is the JSON object the runner + * wrote last time; otherwise answers are generated as questions are chosen. + */ +export class QaSet { + readonly answers: Record; + private readonly generator: Extract | undefined; + private readonly random: (max: number) => number; + + constructor(answers: Record, generator: Extract | undefined, random: (max: number) => number) { + this.answers = { ...answers }; + this.generator = generator; + this.random = random; + } + + /** The first offered question not used yet; records an answer for it, unless `peek` asks only which it would be. */ + choose(options: ReadonlyArray<{ value: string; text: string }>, peek = false, slot?: string): { value: string; text: string } | null { + // The same box on a page that comes back keeps the question it chose the first time. + const before = slot ? this.slots.get(slot) : undefined; + if (before) { + const again = options.find((o) => o.text.trim().toLowerCase() === before); + if (again) return again; + } + const picked = this.pick(options, peek); + if (picked && slot && !peek) this.slots.set(slot, picked.text.trim().toLowerCase()); + return picked; + } + + private readonly slots = new Map(); + + private pick(options: ReadonlyArray<{ value: string; text: string }>, peek: boolean): { value: string; text: string } | null { + const used = new Set(Object.keys(this.answers).map((q) => q.toLowerCase())); + // A stored set must keep its questions: offer the stored ones first. + const stored = options.find((o) => o.value !== "" && o.text.trim() && Object.keys(this.answers).some((q) => q.toLowerCase() === o.text.trim().toLowerCase()) && !this.chosen.has(o.text.trim().toLowerCase())); + if (stored) { + if (!peek) this.chosen.add(stored.text.trim().toLowerCase()); + return stored; + } + if (!this.generator) return null; + const fresh = options.find((o) => o.value !== "" && o.text.trim() && !used.has(o.text.trim().toLowerCase())); + if (!fresh) return null; + if (peek) return fresh; + this.answers[fresh.text.trim()] = generate(this.generator, this.random); + this.chosen.add(fresh.text.trim().toLowerCase()); + return fresh; + } + + private readonly chosen = new Set(); + + /** The answer for the question a page shows beside a box, or by its position ("Answer 2"). */ + answerFor(context: string, index: number | undefined): string | null { + const lower = context.toLowerCase(); + const known = Object.entries(this.answers).find(([question]) => lower.includes(question.toLowerCase())); + if (known) return known[1]; + if (index !== undefined && index > 0) return Object.values(this.answers)[index - 1] ?? null; + return null; + } + + toJSON(): string { + return JSON.stringify(this.answers); + } +} + +export class Inputs { + readonly errand: Errand; + readonly values = new Map(); + readonly qa = new Map(); + /** Every candidate of the shared-secret input, best first, and the one this run submits. */ + candidates: Candidate[] = []; + chosen: Candidate | null = null; + sharedSecret: string | null = null; + + constructor(errand: Errand) { + this.errand = errand; + } + + input(name: string): Input | undefined { + return this.errand.inputs?.[name]; + } + + /** The raw value, for the site's own field and the vault. */ + get(name: string): string | undefined { + const qa = this.qa.get(name); + if (qa) return qa.toJSON(); + return this.values.get(name)?.value; + } + + /** What the terminal may show: secrets masked, everything else as is. */ + display(name: string): string { + const input = this.input(name); + const value = this.get(name); + if (value === undefined) return "(none)"; + return input?.sensitivity === "secret" ? MASK : value; + } + + sensitivity(name: string): Input["sensitivity"] | undefined { + return this.input(name)?.sensitivity; + } + + /** Inputs whose value came from `generate`: the credentials a success writes to the vault. */ + generated(): string[] { + return [...this.values.entries()].filter(([, v]) => v.from === "generate").map(([k]) => k); + } +} + +/** Resolve every input of an errand. Throws {@link ErrandError} for a required input nothing fills. */ +export async function resolveInputs(errand: Errand, options: ResolveOptions = {}): Promise { + const inputs = new Inputs(errand); + const defs = errand.inputs ?? {}; + const random = options.random ?? randomInt; + const now = options.now ?? new Date(); + const overrides = options.overrides ?? {}; + + for (const name of Object.keys(overrides)) { + if (!(name in defs)) throw new ErrandError(`--input ${name}: this errand has no input named ${name}`); + } + + // One extractor call for the whole errand, only when a document source is not already overridden. + const requests: DocumentRequest[] = []; + for (const [name, input] of Object.entries(defs)) { + if (name in overrides) continue; + for (const source of input.sources) { + if (source.from === "document") requests.push({ input: name, form: source.form, field: source.field, ...(source.match ? { match: source.match } : {}) }); + } + } + const records = requests.length && options.extractor ? await options.extractor.extract(requests) : []; + + const resolving = new Set(); + + async function resolve(name: string): Promise { + if (inputs.values.has(name) || inputs.qa.has(name)) return; + const input = defs[name]; + if (!input) throw new ErrandError(`no input named ${name}`); + if (resolving.has(name)) throw new ErrandError(`input ${name} depends on itself`); + resolving.add(name); + try { + if (input.type === "qa-set") return await resolveQa(name, input); + if (name in overrides) { + inputs.values.set(name, { value: checkValue(name, input, overrides[name]!), from: "flag" }); + if (input.role === "shared-secret") { + inputs.sharedSecret = name; + inputs.chosen = { value: overrides[name]!, form: "", field: "", source: "--input" }; + } + return; + } + if (input.role === "shared-secret") return await resolveSharedSecret(name, input); + for (const source of input.sources) { + const got = await fromSource(name, input, source); + if (got) { + inputs.values.set(name, { ...got, value: checkValue(name, input, got.value) }); + return; + } + } + if (input.required) throw new ErrandError(`no value for ${input.label ?? name} (${name}); give it with --input ${name}=...`); + } finally { + resolving.delete(name); + } + } + + async function fromSource(name: string, input: Input, source: Source): Promise & { value: string } | null> { + switch (source.from) { + case "literal": + return { value: String(source.value), from: "literal" }; + case "vault": { + const value = await options.vault?.get(source.key); + return value === undefined || value === "" ? null : { value, from: "vault", origin: source.key }; + } + case "prompt": { + if (!options.prompt) return null; + const answer = await options.prompt(source.ask ?? `${input.label ?? name}: `, { secret: input.sensitivity === "secret" }); + return answer === null || answer === "" ? null : { value: answer, from: "prompt" }; + } + case "generate": + return { value: generate(source, random), from: "generate" }; + case "derive": { + await resolve(source.input); + const base = inputs.get(source.input); + if (base === undefined) return null; + const value = transform(base, source.transform); + return value === "" ? null : { value, from: "derive" }; + } + case "candidate": { + await resolve(source.input); + const c = inputs.chosen; + if (!c) return null; + const part = source.part === "year" ? c.year : c[source.part]; + return part === undefined || part === "" ? null : { value: String(part), from: "candidate" }; + } + case "document": { + const found = documentValues(records, source, now); + const pickOldest = source.pick === "oldest"; + const hit = pickOldest ? found[found.length - 1] : found[0]; + return hit ? { value: hit.value, from: "document", origin: hit.source } : null; + } + } + } + + /** Candidates from document sources in order (newest first within each); other sources give one candidate. */ + async function resolveSharedSecret(name: string, input: Input): Promise { + inputs.sharedSecret = name; + const seen = new Set(); + const candidates: Candidate[] = []; + for (const source of input.sources) { + if (source.from === "document") { + for (const c of documentValues(records, source, now)) { + const key = `${c.year}:${c.form}:${c.field}:${c.value}`; + if (seen.has(key)) continue; + seen.add(key); + candidates.push(c); + } + } else { + const got = await fromSource(name, input, source); + if (got) candidates.push({ value: got.value, form: "", field: "", source: got.origin ?? got.from }); + } + } + inputs.candidates = candidates; + const index = (options.candidate ?? 1) - 1; + if (index < 0 || index >= Math.max(candidates.length, 1)) throw new ErrandError(`--candidate ${options.candidate}: there are ${candidates.length} candidates`); + const chosen = candidates[index]; + if (!chosen) { + if (input.required !== false) throw new ErrandError(`no candidate for ${input.label ?? name} (${name}): the extractor found none, or give one with --input ${name}=...`); + return; + } + inputs.chosen = chosen; + inputs.values.set(name, { value: checkValue(name, input, chosen.value), from: "document", origin: chosen.source }); + } + + async function resolveQa(name: string, input: Input): Promise { + let answers: Record = {}; + let generator: Extract | undefined; + const raw = overrides[name]; + if (raw !== undefined) answers = parseQa(name, raw); + else { + for (const source of input.sources) { + if (source.from === "vault") { + const value = await options.vault?.get(source.key); + if (value) { + answers = parseQa(name, value); + break; + } + } else if (source.from === "generate") { + generator = source; + break; + } + } + } + inputs.qa.set(name, new QaSet(answers, generator, random)); + } + + for (const name of Object.keys(defs)) await resolve(name); + return inputs; +} + +function parseQa(name: string, raw: string): Record { + try { + const parsed = JSON.parse(raw) as unknown; + if (parsed && typeof parsed === "object" && !Array.isArray(parsed) && Object.values(parsed).every((v) => typeof v === "string")) { + return parsed as Record; + } + } catch { + // fall through + } + throw new ErrandError(`${name} is a qa-set: its value is a JSON object of question to answer`); +} diff --git a/packages/openerrand/src/integration.test.ts b/packages/openerrand/src/integration.test.ts new file mode 100644 index 0000000..9ade22a --- /dev/null +++ b/packages/openerrand/src/integration.test.ts @@ -0,0 +1,160 @@ +/** + * The FTB worked example, run unchanged by `logicsrc errand run` in real + * headless Chrome against a local fake site. Chrome resolves + * webapp.ftb.ca.gov to the fake server and every other hostname to nothing, + * so the real site is unreachable from this test. All data is fictional. + */ + +import { execFileSync } from "node:child_process"; +import { readFileSync, writeFileSync } from "node:fs"; +import { join } from "node:path"; +import { Command } from "commander"; +import { afterAll, beforeAll, describe, expect, it } from "vitest"; +import { findChrome } from "./browser.js"; +import { type Deps, registerErrandCommands } from "./commands.js"; +import { FAKE, type FakeSite, startFakeSite } from "./fake-site.js"; +import type { RunRecord } from "./store.js"; +import { ftbExamplePath, tempDir } from "./testing.js"; +import { type LogicsrcExec, parseEnv, serializeEnv } from "./vault.js"; + +const hasOpenssl = (() => { + try { + execFileSync("openssl", ["version"], { stdio: "ignore" }); + return true; + } catch { + return false; + } +})(); +const chrome = findChrome(); +const enabled = Boolean(chrome && hasOpenssl && !process.env.OPENERRAND_SKIP_BROWSER); + +describe.skipIf(!enabled)("logicsrc errand run, the FTB example, real Chrome, fake site", () => { + let site: FakeSite; + let home: string; + let extractor: string; + let remote: Record; + const out: string[] = []; + const say: string[] = []; + + beforeAll(async () => { + site = await startFakeSite(); + if (!Number.isInteger(site.port)) throw new Error("fake site has no port"); + home = tempDir("openerrand-it-"); + // The extractor the person would point at their returns: here it prints fictional records. + extractor = join(home, "extract.mjs"); + const records = [ + { form: "CA 540", field: "first name", value: FAKE.firstName, year: 2025, file: "2025/540.pdf", page: 1 }, + { form: "CA 540", field: "last name", value: FAKE.lastName, year: 2025, file: "2025/540.pdf", page: 1 }, + { form: "CA 540", field: "street address", value: "1234 Maple St", year: 2025, file: "2025/540.pdf", page: 1 }, + { form: "CA 540", field: "ZIP code", value: "95814-0001", year: 2025, file: "2025/540.pdf", page: 1 }, + { form: "CA 100S", field: "California corporation number", value: FAKE.corpId, year: 2025, file: "2025/100S.pdf", page: 1 }, + { form: "CA 100S", field: "line 20", label: "Net income for tax purposes", value: "48,210", year: 2025, file: "2025/100S.pdf", page: 3 }, + { form: "CA 100S", field: "line 20", label: "Net income for tax purposes", value: "39,875", year: 2024, file: "2024/100S.pdf", page: 3 }, + ]; + writeFileSync(extractor, `let input = ""; process.stdin.on("data", (c) => (input += c)); process.stdin.on("end", () => { JSON.parse(input); process.stdout.write(${JSON.stringify(JSON.stringify(records))}); });`); + remote = { OTHER_TEAM_KEY: "keep me" }; + }); + + afterAll(async () => { + await site?.close(); + }); + + function program(): Command { + const fakeTeams: LogicsrcExec = (args) => { + // logicsrc teams pull/push against an in-memory vault: the pull-merge-push is what is under test. + const file = args[args.indexOf("--env") + 1]!; + if (args[0] === "teams" && args[1] === "pull") writeFileSync(file, serializeEnv(remote)); + else if (args[0] === "teams" && args[1] === "push") remote = parseEnv(readFileSync(file, "utf8")); + else return { status: 1, stdout: "", stderr: "unexpected" }; + return { status: 0, stdout: "", stderr: "" }; + }; + const deps: Partial = { + env: { ...process.env, LOGICSRC_ERRAND_HOME: home }, + out: (line) => out.push(line), + say: (line) => { + say.push(line); + // Whoever holds the phone writes the code to the file the runner names. + if (/WAITING for the sms code/.test(line)) relayNext(); + }, + interactive: false, + logicsrc: fakeTeams, + chromeArgs: [ + `--host-resolver-rules=MAP webapp.ftb.ca.gov:443 127.0.0.1:${site.port}, MAP * ~NOTFOUND`, + "--ignore-certificate-errors", + // CI runners often lack the user namespaces Chrome's sandbox needs. + ...(process.env.CI ? ["--no-sandbox"] : []), + ], + pollMs: 250, + rereadMs: 300, + settleMs: 300, + }; + const p = new Command(); + p.name("logicsrc").enablePositionalOptions().exitOverride(); + registerErrandCommands(p.command("errand"), deps); + return p; + } + + const codes = ["000000", FAKE.code]; + function relayNext(): void { + const code = codes.shift(); + if (code) setTimeout(() => writeFileSync(join(home, "codes", "ftb-register-business.code"), `${code}\n`, { mode: 0o600 }), 300); + } + + const lastRecord = (): RunRecord => JSON.parse(out.filter((l) => l.startsWith("{")).at(-1)!) as RunRecord; + const args = (...extra: string[]) => ["node", "logicsrc", "errand", "run", ftbExamplePath(), "--extractor", `${process.execPath} ${extractor}`, "--input", "email=jane@example.com", "--input", `phone=${FAKE.phone}`, "--vault", "teams:test/ftb/prod", "--json", ...extra]; + + it("a dry run stops before the business page's forward button", async () => { + await program().parseAsync(args("--dry-run")); + expect(process.exitCode).toBe(0); + expect(lastRecord()).toMatchObject({ kind: "dry-run", page: "https://webapp.ftb.ca.gov/MyFTBAccess/Registration/Business" }); + expect(site.posted["/MyFTBAccess/Registration/Business"]).toBeUndefined(); + // The challenge interstitial was waited out, not solved. + expect(site.requests.some((r) => r.path.endsWith("/Challenge"))).toBe(true); + expect(say.join("\n")).toContain("declaration (not ticked)"); + }, 120_000); + + it("a run with --declare registers, relays the code after a wrong one, pushes the login into the team vault, and keeps the card local", async () => { + site.requests.length = 0; + await program().parseAsync(args("--declare")); + const record = lastRecord(); + expect(record, say.join("\n")).toMatchObject({ outcome: "registered", kind: "success", vault: "teams test/ftb/prod" }); + expect(process.exitCode).toBe(0); + expect(record.card?.id).toMatch(/^pin-letter\//); + expect(record.waiting?.what).toBe("MyFTB PIN letter"); + + // What the site received: the shared secret once, the declaration ticked, the code on the second try. + expect(site.posted["/MyFTBAccess/Registration/Business"]).toMatchObject({ NetInc: FAKE.netIncome, TaxYear: "2025", Decl: "true", Role: "business", Zip: FAKE.zip, AddrNum: "1234" }); + expect(site.requests.filter((r) => r.method === "POST" && r.path.endsWith("/Business"))).toHaveLength(1); + expect(site.requests.filter((r) => r.method === "POST" && r.path.endsWith("/Code"))).toHaveLength(2); + // Rule 11: the user agent says Chrome, not HeadlessChrome. + expect(site.requests.filter((r) => r.userAgent.includes("HeadlessChrome")).map((r) => `${r.method} ${r.path}`)).toEqual([]); + + // The team vault: merged, nothing dropped, the generated login in it. + const profile = site.posted["/MyFTBAccess/Registration/Profile"]!; + expect(remote).toMatchObject({ OTHER_TEAM_KEY: "keep me", FTB_BUSINESS_USERNAME: profile.UserName, FTB_BUSINESS_PASSWORD: profile.Password, FTB_BUSINESS_EMAIL: "jane@example.com" }); + expect(Object.keys(JSON.parse(remote.FTB_BUSINESS_SECURITY_ANSWERS!))).toHaveLength(3); + + // Rule 8: values never reach the page log or the run record; secrets never reach the terminal. + const log = readFileSync(join(home, "pages", "ftb-register-business.jsonl"), "utf8"); + for (const value of [profile.Password!, FAKE.netIncome, FAKE.code, "jane@example.com", FAKE.phone]) { + expect(log).not.toContain(value); + expect(JSON.stringify(record)).not.toContain(value); + } + expect(say.join("\n")).not.toContain(profile.Password!); + expect(say.join("\n")).not.toContain(FAKE.netIncome); + + // `errand status` shows the card from the local run record. + out.length = 0; + await program().parseAsync(["node", "logicsrc", "errand", "status"]); + expect(out.join("\n")).toMatch(/ftb-register-business {2}success \(registered\)/); + expect(out.join("\n")).toContain("ftb activate business --pin "); + }, 180_000); + + it("a third run inside the window is refused by the throttle before Chrome starts", async () => { + const before = site.requests.length; + await program().parseAsync(args("--declare")); + expect(process.exitCode).toBe(4); + expect(site.requests.length).toBe(before); + process.exitCode = 0; + }, 30_000); +}); diff --git a/packages/openerrand/src/load.ts b/packages/openerrand/src/load.ts new file mode 100644 index 0000000..69bb005 --- /dev/null +++ b/packages/openerrand/src/load.ts @@ -0,0 +1,88 @@ +/** + * Loading an errand file: parse, validate with @logicsrc/validators (the same + * schema and semantic checks `logicsrc-validate openerrand` runs), hash, and + * describe it for the person before anything runs (rule 1). + */ + +import { createHash } from "node:crypto"; +import { readFileSync } from "node:fs"; +import { validate } from "@logicsrc/validators"; +import { isGate } from "./pages.js"; +import type { Errand, Source } from "./types.js"; +import { ErrandError } from "./util.js"; + +export interface Loaded { + errand: Errand; + sha256: string; + file: string; +} + +export function sha256(text: string): string { + return createHash("sha256").update(text).digest("hex"); +} + +/** Validation errors as one line each: `/steps/2/solver: a captcha solver is never allowed on a tax site`. */ +export function validateErrand(data: unknown): string[] { + const result = validate("openerrand", data); + if (result.ok) return []; + return result.errors.map((e) => `${e.instancePath || "/"}: ${e.message ?? e.keyword}`); +} + +export function loadErrand(file: string): Loaded { + let text: string; + try { + text = readFileSync(file, "utf8"); + } catch (error) { + throw new ErrandError(`cannot read ${file}: ${(error as Error).message}`); + } + let data: unknown; + try { + data = JSON.parse(text); + } catch (error) { + throw new ErrandError(`${file} is not JSON: ${(error as Error).message}`); + } + const errors = validateErrand(data); + if (errors.length) throw new ErrandError(`${file} is not a valid OpenErrand 0.1 file:\n${errors.map((e) => ` ${e}`).join("\n")}`); + return { errand: data as Errand, sha256: sha256(text), file }; +} + +function describeSource(source: Source): string { + switch (source.from) { + case "document": + return `document ${source.form} ${source.field}`; + case "vault": + return `vault ${source.key}`; + case "prompt": + return "asked at the terminal"; + case "generate": + return `generated (${source.length} chars)`; + case "derive": + return `from ${source.input} (${source.transform})`; + case "candidate": + return `the ${source.part} of the ${source.input} candidate`; + case "literal": + return "fixed in the file"; + } +} + +/** + * What a person reads before the first run: title, publisher, verification, + * every gate with its why, every input with its sensitivity and sources. + */ +export function summarize(errand: Errand, verification: string): string[] { + const lines = [`${errand.title} (${errand.name})`, ` site: ${errand.site.name} ${errand.site.origins.join(" ")}${errand.site.sector ? ` [${errand.site.sector}]` : ""}`]; + lines.push(` publisher: ${errand.publisher ?? "not stated"} ${verification}`); + const gates = errand.steps.filter(isGate); + if (gates.length) { + lines.push(" steps that are yours:"); + for (const gate of gates) lines.push(` ${gate.kind} (${gate.id}): ${"why" in gate && gate.why ? gate.why : ("what" in gate ? gate.what : "")}`); + } + const inputs = Object.entries(errand.inputs ?? {}); + if (inputs.length) { + lines.push(" inputs:"); + for (const [name, input] of inputs) { + lines.push(` ${name} [${input.sensitivity}${input.role ? `, ${input.role}` : ""}]: ${input.sources.map(describeSource).join(", then ")}`); + } + } + return lines; +} diff --git a/packages/openerrand/src/outputs.test.ts b/packages/openerrand/src/outputs.test.ts new file mode 100644 index 0000000..a16ab77 --- /dev/null +++ b/packages/openerrand/src/outputs.test.ts @@ -0,0 +1,110 @@ +import { readFileSync, statSync, writeFileSync } from "node:fs"; +import { join } from "node:path"; +import { describe, expect, it } from "vitest"; +import { resolveInputs } from "./inputs.js"; +import { credentialValues, fileDownloads, renderCard, writeCredentials } from "./outputs.js"; +import { ftbExample, tempDir } from "./testing.js"; +import type { Errand } from "./types.js"; +import { fileVault, parseEnv, parseTarget, serializeEnv, teamsVault, type LogicsrcExec, type Vault } from "./vault.js"; + +const OVERRIDES = { email: "jane@example.com", phone: "5555550100", first_name: "Jane", last_name: "Doe", street: "1234 Maple St", zip: "95814", corp_id: "1234567", net_income: "48210", tax_year: "2025" }; + +describe("hand-off cards", () => { + it("renders built-ins and keeps the card to public values", async () => { + const e = ftbExample(); + const inputs = await resolveInputs(e, { overrides: OVERRIDES }); + const card = renderCard(e, "pin-letter", inputs, { expires_on: "2026-10-25" }, () => 0); + expect(card.id).toMatch(/^pin-letter\/[a-z2-9]{6}$/); + expect(card.steps[1]).toBe("Activate before 2026-10-25: the PIN expires 21 days after registration."); + expect(JSON.stringify(card)).not.toMatch(/jane|5555550100|48210|Maple/i); + }); + + it("refuses a card that names a personal or secret input, whatever the validator said", async () => { + const e: Errand = { ...ftbExample(), handoffs: { "pin-letter": { title: "PIN for {{first_name}}", steps: ["x"] } } }; + const inputs = await resolveInputs(e, { overrides: OVERRIDES }); + expect(() => renderCard(e, "pin-letter", inputs, {}, () => 0)).toThrow(/personal/); + }); +}); + +describe("credentials", () => { + it("renders the keys only for the outcome named in when", async () => { + const e = ftbExample(); + const inputs = await resolveInputs(e, { overrides: OVERRIDES }); + const values = credentialValues(e, e.outcomes.find((o) => o.name === "registered")!, inputs); + expect(Object.keys(values).sort()).toEqual(["FTB_BUSINESS_EMAIL", "FTB_BUSINESS_PASSWORD", "FTB_BUSINESS_SECURITY_ANSWERS", "FTB_BUSINESS_USERNAME"]); + expect(values.FTB_BUSINESS_PASSWORD).toBe(inputs.get("password")); + expect(credentialValues(e, e.outcomes.find((o) => o.name === "rejected")!, inputs)).toEqual({}); + }); + + it("falls back to the 0600 file when the vault is read-only or the write fails, and says so", async () => { + const dir = tempDir(); + const fallback = fileVault(join(dir, "creds.env")); + const readOnly: Vault = { describe: () => "OpenCreds", get: async () => undefined }; + const r1 = await writeCredentials({ A: "1" }, readOnly, fallback); + expect(r1.warning).toMatch(/read-only/); + expect(statSync(join(dir, "creds.env")).mode & 0o777).toBe(0o600); + const broken: Vault = { ...readOnly, write: async () => Promise.reject(new Error("offline")) }; + const r2 = await writeCredentials({ B: "2" }, broken, fallback); + expect(r2.warning).toMatch(/offline/); + expect(parseEnv(readFileSync(join(dir, "creds.env"), "utf8"))).toEqual({ A: "1", B: "2" }); + }); +}); + +describe("vaults", () => { + it("round-trips values with quotes, specials and JSON", () => { + const values = { P: "a#b$c*d@e!f", Q: '{"Pet?":"rex"}', R: "plain", S: "back\\slash" }; + expect(parseEnv(serializeEnv(values))).toEqual(values); + }); + + it("parses targets", () => { + expect(parseTarget("teams:profullstack/ftb/prod")).toEqual({ kind: "teams", team: "profullstack", project: "ftb", env: "prod" }); + expect(parseTarget("opencreds")).toEqual({ kind: "opencreds" }); + expect(() => parseTarget("s3://x")).toThrow(/expected/); + }); + + it("a teams write pulls, merges and pushes, never dropping a key already there", async () => { + let remote: Record = { OTHER_KEY: "keep me", FTB_BUSINESS_EMAIL: "old@example.com" }; + const calls: string[] = []; + const exec: LogicsrcExec = (args) => { + calls.push(args.slice(0, 5).join(" ")); + const file = args[args.indexOf("--env") + 1]!; + if (args[1] === "pull") writeFileSync(file, serializeEnv(remote)); + if (args[1] === "push") remote = parseEnv(readFileSync(file, "utf8")); + return { status: 0, stdout: "", stderr: "" }; + }; + const vault = teamsVault({ kind: "teams", team: "profullstack", project: "ftb", env: "prod" }, exec); + expect(await vault.get("OTHER_KEY")).toBe("keep me"); + await vault.write!({ FTB_BUSINESS_EMAIL: "jane@example.com", FTB_BUSINESS_PASSWORD: "x#y" }); + expect(remote).toEqual({ OTHER_KEY: "keep me", FTB_BUSINESS_EMAIL: "jane@example.com", FTB_BUSINESS_PASSWORD: "x#y" }); + expect(calls).toEqual(["teams pull profullstack ftb prod", "teams pull profullstack ftb prod", "teams push profullstack ftb prod"]); + }); + + it("a failed push is an error the caller turns into the file fallback", async () => { + const exec: LogicsrcExec = (args) => (args[1] === "push" ? { status: 1, stdout: "", stderr: "not logged in" } : { status: 0, stdout: "", stderr: "" }); + const vault = teamsVault({ kind: "teams", team: "t", project: "p", env: "e" }, exec); + await expect(vault.write!({ A: "1" })).rejects.toThrow(/not logged in/); + }); +}); + +describe("downloads", () => { + it("checks the type, files it, and never overwrites different bytes", async () => { + const e: Errand = { + ...ftbExample(), + outputs: { downloads: [{ match: { type: "application/pdf", url: "transcript" }, to: "{{dir}}/{{tax_year}}/transcript.pdf", when: "registered" }] }, + inputs: { ...ftbExample().inputs, dir: { type: "string", sensitivity: "public", sources: [{ from: "literal", value: tempDir() }] } }, + }; + const inputs = await resolveInputs(e, { overrides: OVERRIDES }); + const dl = tempDir(); + writeFileSync(join(dl, "a"), "%PDF-1.7 fake"); + const outcome = e.outcomes.find((o) => o.name === "registered")!; + const files = [{ url: "https://webapp.ftb.ca.gov/transcript?id=1", filename: "t.pdf", path: join(dl, "a") }]; + const [to] = fileDownloads(e, outcome, inputs, files); + expect(to).toMatch(/2025\/transcript\.pdf$/); + // Same bytes again: fine. Different bytes: refused. + expect(fileDownloads(e, outcome, inputs, files)).toEqual([to]); + writeFileSync(join(dl, "b"), "%PDF-1.7 different"); + expect(() => fileDownloads(e, outcome, inputs, [{ ...files[0]!, path: join(dl, "b") }])).toThrow(/different bytes/); + writeFileSync(join(dl, "c"), "not a pdf"); + expect(() => fileDownloads(e, outcome, inputs, [{ ...files[0]!, path: join(dl, "c") }])).toThrow(/is not application\/pdf/); + }); +}); diff --git a/packages/openerrand/src/outputs.ts b/packages/openerrand/src/outputs.ts new file mode 100644 index 0000000..61997b1 --- /dev/null +++ b/packages/openerrand/src/outputs.ts @@ -0,0 +1,122 @@ +/** + * What a run keeps: credentials (written before success is reported), files + * the site handed over, and hand-off cards (kept in the local run record + * only, never posted anywhere). + */ + +import { copyFileSync, existsSync, readFileSync } from "node:fs"; +import { homedir } from "node:os"; +import { dirname } from "node:path"; +import { HANDOFF_BUILTINS } from "@logicsrc/validators"; +import { type DownloadedFile, looksLike } from "./driver.js"; +import type { Inputs } from "./inputs.js"; +import { type Card, ensureDir } from "./store.js"; +import type { Errand, Outcome } from "./types.js"; +import { ErrandError, re, render, shortId, templateNames } from "./util.js"; +import type { Vault } from "./vault.js"; + +/** + * Render a hand-off card. Only `{{expires_on}}`, `{{errand.title}}`, + * `{{site.name}}` and public inputs may appear; anything else and the card is + * refused, whatever the validator said about the file. + */ +export function renderCard(errand: Errand, id: string, inputs: Inputs, extra: { expires_on?: string }, random: (max: number) => number): Card { + const card = errand.handoffs?.[id]; + if (!card) throw new ErrandError(`no hand-off card named ${id}`); + const lookup = (name: string): string | undefined => { + if (name === "expires_on") return extra.expires_on ?? "(no deadline)"; + if (name === "errand.title") return errand.title; + if (name === "site.name") return errand.site.name; + return inputs.get(name); + }; + const texts = [card.title, ...card.steps, ...(card.command !== undefined ? [card.command] : [])]; + for (const text of texts) { + for (const name of templateNames(text)) { + if ((HANDOFF_BUILTINS as readonly string[]).includes(name)) continue; + const sensitivity = inputs.sensitivity(name); + if (sensitivity !== "public") throw new ErrandError(`card ${id} names {{${name}}}, which is ${sensitivity ?? "not an input"}; a card carries built-ins and public inputs only`); + } + } + return { + id: `${id}/${shortId(random)}`, + title: render(card.title, lookup), + ...(card.open ? { open: card.open } : {}), + steps: card.steps.map((s) => render(s, lookup)), + ...(card.command !== undefined ? { command: render(card.command, lookup) } : {}), + ...(extra.expires_on ? { expires_on: extra.expires_on } : {}), + }; +} + +/** The credential keys for this outcome, rendered from the file's templates. Empty when the outcome is not `when`. */ +export function credentialValues(errand: Errand, outcome: Outcome, inputs: Inputs): Record { + const vault = errand.outputs?.vault; + if (!vault || (vault.when !== undefined && vault.when !== outcome.name)) return {}; + const out: Record = {}; + for (const [key, template] of Object.entries(vault.keys)) { + out[key] = render(template, (name) => inputs.get(name)); + } + return out; +} + +/** + * Write credentials to the vault, falling back to the 0600 state file when + * there is no writable vault or the write fails. Returns where they went and + * any warning, so the caller says so. + */ +export async function writeCredentials(values: Record, vault: Vault | null, fallback: Vault): Promise<{ where: string; warning?: string }> { + if (!Object.keys(values).length) return { where: "" }; + if (vault?.write) { + try { + await vault.write(values); + return { where: vault.describe() }; + } catch (error) { + await fallback.write!(values); + return { where: fallback.describe(), warning: `vault write failed (${(error as Error).message}); the login is only in ${fallback.describe()}` }; + } + } + await fallback.write!(values); + return { + where: fallback.describe(), + warning: vault ? `${vault.describe()} is read-only; the login is in ${fallback.describe()} (0600)` : `no vault named; the login is in ${fallback.describe()} (0600)`, + }; +} + +const expand = (path: string): string => path.replace(/^~(?=\/|$)/, homedir()); + +/** + * File what the site handed over. Each entry matches by URL, file name and + * media type (checked against the file's first bytes); the target path is + * rendered from public and personal inputs, never secrets, and an existing + * file with different bytes is never overwritten. + */ +export function fileDownloads(errand: Errand, outcome: Outcome, inputs: Inputs, downloaded: readonly DownloadedFile[]): string[] { + const filed: string[] = []; + for (const entry of errand.outputs?.downloads ?? []) { + if (entry.when !== undefined && entry.when !== outcome.name) continue; + const file = downloaded.find( + (d) => (entry.match.url === undefined || re(entry.match.url).test(d.url)) && (entry.match.filename === undefined || re(entry.match.filename).test(d.filename)), + ); + if (!file) continue; + const bytes = readFileSync(file.path); + if (entry.match.type && !looksLike(entry.match.type, bytes)) throw new ErrandError(`${file.filename} is not ${entry.match.type}; left in ${file.path}`); + const to = expand( + render(entry.to, (name) => { + if (inputs.sensitivity(name) === "secret") throw new ErrandError(`download path ${entry.to} names a secret input`); + return inputs.get(name); + }), + ); + if (existsSync(to)) { + if (!readFileSync(to).equals(bytes)) throw new ErrandError(`${to} exists with different bytes; the new file is left in ${file.path}`); + } else { + ensureDir(dirname(to)); + copyFileSync(file.path, to); + } + filed.push(to); + } + return filed; +} + +/** Run date plus an ISO 8601 duration, as YYYY-MM-DD. */ +export function expiresOn(now: Date, ms: number): string { + return new Date(now.getTime() + ms).toISOString().slice(0, 10); +} diff --git a/packages/openerrand/src/pages.test.ts b/packages/openerrand/src/pages.test.ts new file mode 100644 index 0000000..f4fe218 --- /dev/null +++ b/packages/openerrand/src/pages.test.ts @@ -0,0 +1,133 @@ +import { describe, expect, it } from "vitest"; +import { validateErrand } from "./load.js"; +import { isLockout, outcomeOf, solverPermitted, stepFor } from "./pages.js"; +import { ftbExample } from "./testing.js"; +import { checkThrottle, emptyLedger, keyFor, LIMITS, lockoutMs, recordAttempt, recordLockout } from "./throttle.js"; +import type { CaptchaStep, Errand } from "./types.js"; + +const page = (url: string, title = "", text = "", errors: string[] = []) => ({ url, title, text, errors, fields: [] }); + +describe("which step a page is", () => { + const e = ftbExample(); + it("a wait step whose title and selector both fit", () => { + expect(stepFor(e, page("https://webapp.ftb.ca.gov/x", "Challenge Validation"), { "#sec-cpt-if": true })?.id).toBe("bot-check"); + // All given parts must fit: the title alone is not enough. + expect(stepFor(e, page("https://webapp.ftb.ca.gov/x", "Challenge Validation"), { "#sec-cpt-if": false })?.id).toBe("form"); + }); + + it("falls back to the page step with no match", () => { + expect(stepFor(e, page("https://webapp.ftb.ca.gov/MyFTBAccess/Registration/NewAccount", "Registration"), {})?.id).toBe("form"); + }); + + it("an identity provider's origin is its gate", () => { + const withId: Errand = { ...e, steps: [...e.steps, { id: "id-me", kind: "identity-proofing", provider: "ID.me", origins: ["https://api.id.me"], why: "yours" }] }; + expect(stepFor(withId, page("https://api.id.me/en/session"), {})?.id).toBe("id-me"); + }); +}); + +describe("outcomes", () => { + const e = ftbExample(); + it("tests rejected outcomes first, against the errors when there are any", () => { + const p = page("https://webapp.ftb.ca.gov/c", "Confirmation", "Registration confirmation", ["The information does not match our records."]); + expect(outcomeOf(e, p)?.name).toBe("rejected"); + }); + + it("reads the page text when there are no errors", () => { + expect(outcomeOf(e, page("https://webapp.ftb.ca.gov/c", "x", "Registration Confirmation. We will mail you a PIN."))?.name).toBe("registered"); + expect(outcomeOf(e, page("https://webapp.ftb.ca.gov/c", "x", "Please enter your name"))).toBeNull(); + }); + + it("needs both text and url when both are given", () => { + const both: Errand = { ...e, outcomes: [{ name: "done", kind: "success", text: "done", url: "/finished$" }] }; + expect(outcomeOf(both, page("https://webapp.ftb.ca.gov/finished", "", "done"))?.name).toBe("done"); + expect(outcomeOf(both, page("https://webapp.ftb.ca.gov/other", "", "done"))).toBeNull(); + }); + + it("knows a lockout by the file's signature or the default", () => { + expect(isLockout(e, "Your account has been locked for 30 minutes")).toBe(true); + expect(isLockout(e, "You have exceeded the allowed number of attempts")).toBe(true); + expect(isLockout(e, "Welcome")).toBe(false); + const own: Errand = { ...e, metadata: { lockout: { text: "come back tomorrow", duration: "PT24H" } } }; + expect(isLockout(own, "Please come back tomorrow")).toBe(true); + expect(isLockout(own, "account locked")).toBe(false); + expect(lockoutMs(own)).toBe(24 * 3_600_000); + expect(lockoutMs(e)).toBe(LIMITS.lockoutMs); + }); +}); + +describe("captcha solver gating", () => { + const captcha: CaptchaStep = { id: "cap", kind: "captcha", match: { selector: ".g-recaptcha" }, solver: "allowed" }; + const commercial = (over: Partial = {}): Errand => ({ + type: "logicsrc.openerrand", + version: "0.1", + name: "newsletter", + title: "Sign up", + site: { name: "Shop", sector: "commercial", origins: ["https://shop.example"], start: ["https://shop.example/"] }, + inputs: { email: { type: "email", sensitivity: "personal", sources: [{ from: "prompt" }] } }, + steps: [{ id: "form", kind: "page" }, captcha], + outcomes: [{ name: "ok", kind: "success", text: "thanks" }], + ...over, + }); + + it("allows a solver only on a commercial or other site with nothing sensitive, as the file says", () => { + expect(validateErrand(commercial())).toEqual([]); + expect(solverPermitted(commercial(), captcha)).toBe(true); + expect(solverPermitted(commercial(), { ...captcha, solver: "forbidden" })).toBe(false); + expect(solverPermitted(commercial(), { ...captcha, solver: undefined })).toBe(false); + }); + + for (const sector of ["government", "tax", "financial", "healthcare", "identity-provider"] as const) { + it(`never on a ${sector} site, even when the file says allowed (and the validator rejects the file)`, () => { + const e = commercial({ site: { name: "x", sector, origins: ["https://x.example"], start: ["https://x.example/"] } }); + expect(solverPermitted(e, captcha)).toBe(false); + expect(validateErrand(e).join(" ")).toMatch(/captcha solver is never allowed/); + }); + } + + it("never without a stated sector, with a secret input, or with a declaration", () => { + expect(solverPermitted(commercial({ site: { name: "x", origins: ["https://x.example"], start: ["https://x.example/"] } }), captcha)).toBe(false); + expect(solverPermitted(commercial({ inputs: { pw: { type: "string", sensitivity: "secret", sources: [{ from: "prompt" }] } } }), captcha)).toBe(false); + expect(solverPermitted(commercial({ steps: [{ id: "form", kind: "page" }, captcha, { id: "d", kind: "declare", statement: "x", why: "y" }] }), captcha)).toBe(false); + }); +}); + +describe("throttle", () => { + const e = ftbExample(); + const key = keyFor(e); + const t0 = new Date("2026-10-04T10:00:00Z"); + const at = (min: number) => new Date(t0.getTime() + min * 60_000); + + it("spaces runs on one site 2 minutes apart", () => { + const l = recordAttempt(emptyLedger(), key, t0); + expect(checkThrottle(l, key, at(1))).toMatchObject({ ok: false, lockout: false }); + expect(checkThrottle(l, { ...key, account: "other", errand: "ftb-activate-business" }, at(1)).ok).toBe(false); + expect(checkThrottle(l, key, at(2)).ok).toBe(true); + }); + + it("allows 2 runs in 30 minutes and 4 a day per errand and account", () => { + let l = recordAttempt(emptyLedger(), key, t0); + l = recordAttempt(l, key, at(3)); + expect(checkThrottle(l, key, at(10))).toMatchObject({ ok: false, reason: expect.stringMatching(/2 runs/) }); + expect(checkThrottle(l, { ...key, account: "personal" }, at(10)).ok).toBe(true); + l = recordAttempt(l, key, at(40)); + l = recordAttempt(l, key, at(80)); + expect(checkThrottle(l, key, at(200))).toMatchObject({ ok: false, reason: expect.stringMatching(/4 runs/) }); + expect(checkThrottle(l, key, at(24 * 60 + 1)).ok).toBe(true); + }); + + it("--force lifts the caps and never a lockout", () => { + let l = recordAttempt(emptyLedger(), key, t0); + expect(checkThrottle(l, key, at(1), true).ok).toBe(true); + l = recordLockout(l, key, at(1)); + expect(checkThrottle(l, key, at(10), true)).toMatchObject({ ok: false, lockout: true }); + // The lock is per site and account: another errand on the same account is held too. + expect(checkThrottle(l, { ...key, errand: "ftb-activate-business" }, at(10), true)).toMatchObject({ ok: false, lockout: true }); + expect(checkThrottle(l, key, at(1 + 35), true).ok).toBe(true); + }); + + it("a shorter later lock never shortens a longer one", () => { + let l = recordLockout(emptyLedger(), key, t0, 24 * 3_600_000); + l = recordLockout(l, key, at(5)); + expect(checkThrottle(l, key, at(60), true).ok).toBe(false); + }); +}); diff --git a/packages/openerrand/src/pages.ts b/packages/openerrand/src/pages.ts new file mode 100644 index 0000000..5850ca0 --- /dev/null +++ b/packages/openerrand/src/pages.ts @@ -0,0 +1,103 @@ +/** + * Reading a page against the errand: which step it is, and whether it is an + * outcome. Pure; the driver supplies the page and which CSS selectors hit. + */ + +import { NO_SOLVER_SECTORS } from "@logicsrc/validators"; +import type { CaptchaStep, Errand, GateStep, IdentityStep, Match, Outcome, Page, Step } from "./types.js"; +import { originOf, re } from "./util.js"; + +/** Every CSS selector the errand's matches name, so the driver can test them in one evaluation. */ +export function selectorsOf(errand: Errand): string[] { + const out = new Set(); + for (const step of errand.steps) { + const match = (step as { match?: Match }).match; + if (match?.selector) out.add(match.selector); + } + return [...out]; +} + +/** All the given parts of a match must fit. */ +export function fits(match: Match, page: Pick, hits: Record): boolean { + if (match.url !== undefined && !re(match.url).test(page.url)) return false; + if (match.title !== undefined && !re(match.title).test(page.title)) return false; + if (match.text !== undefined && !re(match.text).test(page.text)) return false; + if (match.selector !== undefined && !hits[match.selector]) return false; + return true; +} + +/** + * The step for a page: the first step whose match fits, an identity-proofing + * gate whose provider origin the page is on, else the first page step with no + * match (the fallback). null when nothing fits. + */ +export function stepFor(errand: Errand, page: Pick, hits: Record): Step | null { + const origin = originOf(page.url); + for (const step of errand.steps) { + if (step.kind === "identity-proofing" && step.origins.includes(origin)) return step; + const match = (step as { match?: Match }).match; + if (match && (step.kind === "page" || step.kind === "wait" || step.kind === "captcha" || step.kind === "identity-proofing") && fits(match, page, hits)) return step; + } + return errand.steps.find((s) => s.kind === "page" && !s.match) ?? null; +} + +export function isGate(step: Step): step is GateStep { + return step.kind === "declare" || step.kind === "identity-proofing" || step.kind === "code" || step.kind === "mail" || step.kind === "captcha"; +} + +export function stepById(errand: Errand, id: string): Step | undefined { + return errand.steps.find((s) => s.id === id); +} + +/** The text an outcome is tested against: the page's error messages, or its text when it shows none. */ +export function outcomeText(page: Pick): string { + return page.errors.length ? page.errors.join(" ") : page.text; +} + +/** Rejected outcomes first, then the others in file order. Both `text` and `url` must fit when both are given. */ +export function outcomeOf(errand: Errand, page: Pick): Outcome | null { + const text = outcomeText(page); + const ordered = [...errand.outcomes.filter((o) => o.kind === "rejected"), ...errand.outcomes.filter((o) => o.kind !== "rejected")]; + for (const outcome of ordered) { + if (outcome.text !== undefined && !re(outcome.text).test(text)) continue; + if (outcome.url !== undefined && !re(outcome.url).test(page.url)) continue; + return outcome; + } + return null; +} + +/** Origins a run may be on: the site's, plus every identity provider's (only while that gate is handed over). */ +export function allowedOrigins(errand: Errand): Set { + const out = new Set(errand.site.origins); + for (const step of errand.steps) if (step.kind === "identity-proofing") for (const o of (step as IdentityStep).origins) out.add(o); + return out; +} + +/** + * Whether a captcha solver may be used for this step: the step says `allowed` + * and the errand is outside the forbidden set (a stated sector that is not + * government, tax, financial, healthcare or identity-provider; no declare or + * identity-proofing step; no secret input). The validator rejects a file that + * breaks this, and the runner checks again so a hand-edited file that skipped + * validation still cannot reach a solver. + */ +export function solverPermitted(errand: Errand, step: CaptchaStep): boolean { + if (step.solver !== "allowed") return false; + const sector = errand.site.sector; + if (!sector || (NO_SOLVER_SECTORS as readonly string[]).includes(sector)) return false; + if (errand.steps.some((s) => s.kind === "declare" || s.kind === "identity-proofing")) return false; + if (Object.values(errand.inputs ?? {}).some((i) => i.sensitivity === "secret")) return false; + return true; +} + +/** The text that marks a lockout: `metadata.lockout.text` when the file gives one, else a default that covers the common wordings. */ +export const DEFAULT_LOCKOUT = "account (is |has been )?(temporarily )?locked|locked out|too many (failed |unsuccessful )?attempts|exceeded the (allowed |maximum )?number of attempts"; + +export function lockoutPattern(errand: Errand): string { + const lockout = (errand.metadata as { lockout?: { text?: unknown } } | undefined)?.lockout; + return typeof lockout?.text === "string" && lockout.text ? lockout.text : DEFAULT_LOCKOUT; +} + +export function isLockout(errand: Errand, text: string): boolean { + return re(lockoutPattern(errand)).test(text); +} diff --git a/packages/openerrand/src/prompt.ts b/packages/openerrand/src/prompt.ts new file mode 100644 index 0000000..107a63b --- /dev/null +++ b/packages/openerrand/src/prompt.ts @@ -0,0 +1,63 @@ +/** + * Terminal prompts. Without a terminal they answer null: the runner then + * moves to the next source, or stops and says what it needed, rather than + * read a secret from a pipe it cannot see the other end of. + */ + +import type { Prompt } from "./inputs.js"; + +function readSecret(label: string): Promise { + process.stderr.write(label); + process.stdin.setRawMode(true); + process.stdin.resume(); + return new Promise((resolve) => { + let value = ""; + const onData = (chunk: Buffer): void => { + for (const byte of chunk) { + if (byte === 0x0d || byte === 0x0a || byte === 0x04) return finish(); + if (byte === 0x03) { + process.stdin.setRawMode(false); + process.stderr.write("\n"); + process.exit(130); + } + if (byte === 0x7f || byte === 0x08) { + value = value.slice(0, -1); + continue; + } + value += String.fromCharCode(byte); + } + }; + const finish = (): void => { + process.stdin.off("data", onData); + process.stdin.setRawMode(false); + process.stdin.pause(); + process.stderr.write("\n"); + resolve(value.trim()); + }; + process.stdin.on("data", onData); + }); +} + +function readLine(label: string): Promise { + process.stderr.write(label); + return new Promise((resolve) => { + process.stdin.setEncoding("utf8"); + process.stdin.resume(); + process.stdin.once("data", (chunk) => { + process.stdin.pause(); + resolve(String(chunk).trim()); + }); + }); +} + +/** Secrets do not echo. */ +export const terminalPrompt: Prompt = async (question, { secret }) => { + if (!process.stdin.isTTY) return null; + return secret ? readSecret(question) : readLine(question); +}; + +export async function confirm(question: string): Promise { + if (!process.stdin.isTTY) return false; + const answer = await readLine(`${question} [y/N] `); + return /^y(es)?$/i.test(answer); +} diff --git a/packages/openerrand/src/rules.test.ts b/packages/openerrand/src/rules.test.ts new file mode 100644 index 0000000..9d68074 --- /dev/null +++ b/packages/openerrand/src/rules.test.ts @@ -0,0 +1,156 @@ +import { describe, expect, it } from "vitest"; +import { resolveInputs } from "./inputs.js"; +import { decide } from "./rules.js"; +import { field, ftbExample } from "./testing.js"; +import type { Errand, Rule } from "./types.js"; + +function errand(inputs: Errand["inputs"], rules: Rule[] = []): Errand { + return { + type: "logicsrc.openerrand", + version: "0.1", + name: "t", + title: "t", + site: { name: "Example", origins: ["https://example.com"], start: ["https://example.com/"] }, + inputs, + rules, + steps: [{ id: "form", kind: "page" }], + outcomes: [{ name: "ok", kind: "success", text: "done" }], + }; +} + +async function ftbInputs() { + const e = ftbExample(); + return { + errand: e, + inputs: await resolveInputs(e, { + overrides: { + email: "jane@example.com", + phone: "5555550100", + first_name: "Jane", + last_name: "Doe", + street: "1234 Maple St", + zip: "95814", + corp_id: "1234567", + net_income: "48210", + tax_year: "2025", + }, + random: () => 0, + }), + }; +} + +describe("matching: id first, label second", () => { + it("tries every id rule before any label rule, whatever the order in the file", async () => { + const e = errand({ name: { type: "string", sensitivity: "personal", sources: [{ from: "literal", value: "Jane" }] } }, [ + { name: "by label", label: "name", do: { text: "label-{{name}}" } }, + { name: "by id", id: "^FstName$", do: { text: "{{name}}" } }, + ]); + const inputs = await resolveInputs(e); + const d = decide(e.rules!, field({ id: "FstName", label: "First name" }), inputs); + expect(d?.rule.name).toBe("by id"); + expect(d?.action).toMatchObject({ kind: "text", value: "Jane" }); + }); + + it("an id match is final even when it skips", async () => { + const { errand: e, inputs } = await ftbInputs(); + const d = decide(e.rules!, field({ id: "MInitial", label: "Middle initial name" }), inputs); + expect(d?.rule.name).toBe("middle initial"); + expect(d?.action).toEqual({ kind: "skip" }); + }); + + it("an id match that cannot act leaves the field unmatched rather than trying labels", async () => { + const e = errand({ name: { type: "string", sensitivity: "personal", sources: [{ from: "prompt" }] } }, [ + { name: "by id", id: "^FstName$", do: { text: "{{name}}" } }, + { name: "by label", label: "first", do: { text: "x" } }, + ]); + const inputs = await resolveInputs(e); + const d = decide(e.rules!, field({ id: "FstName", label: "First name" }), inputs); + expect(d?.rule.name).toBe("by id"); + expect(d?.action).toBeNull(); + expect(d?.reason).toMatch(/no value/); + }); + + it("a label match that cannot act lets the next rule try", async () => { + const { errand: e, inputs } = await ftbInputs(); + const radio = field({ id: "RoleInd", name: "Role", type: "radio", label: "Individual" }); + // The role rule only ticks the business representative radio. + expect(decide(e.rules!, radio, inputs)).toBeNull(); + const business = field({ id: "RoleBus", name: "Role", type: "radio", label: "Business Representative" }); + expect(decide(e.rules!, business, inputs)?.action).toEqual({ kind: "check" }); + }); + + it("types limit a rule; a rule without types fits any", async () => { + const { errand: e, inputs } = await ftbInputs(); + const select = field({ id: "Yr", type: "select-one", label: "Tax year", options: [{ value: "", text: "Select" }, { value: "2024", text: "2024" }, { value: "2025", text: "2025" }] }); + expect(decide(e.rules!, select, inputs)?.action).toMatchObject({ kind: "select", value: "2025" }); + const box = field({ id: "Yr2", type: "tel", label: "Tax year" }); + expect(decide(e.rules!, box, inputs)?.action).toMatchObject({ kind: "text", value: "2025" }); + expect(decide(e.rules!, field({ id: "Yr3", type: "checkbox", label: "Tax year" }), inputs)).toBeNull(); + }); + + it("a page step's own rules come first", async () => { + const e = errand({ a: { type: "string", sensitivity: "public", sources: [{ from: "literal", value: "A" }] } }, [{ name: "errand", label: "code", do: { text: "errand" } }]); + const inputs = await resolveInputs(e); + const rules: Rule[] = [{ name: "step", label: "code", do: { text: "{{a}}" } }, ...e.rules!]; + expect(decide(rules, field({ id: "c", label: "Code" }), inputs)?.rule.name).toBe("step"); + }); + + it("flags secrets and the shared secret so they are masked and a dry run stops there", async () => { + const { errand: e, inputs } = await ftbInputs(); + const d = decide(e.rules!, field({ id: "NetInc", type: "text", label: "Net income for tax purposes" }), inputs); + expect(d?.action).toMatchObject({ kind: "text", value: "48210", secret: true, sharedSecret: true }); + const zip = decide(e.rules!, field({ id: "Zip", type: "text", label: "ZIP code" }), inputs); + expect(zip?.action).toMatchObject({ value: "95814", secret: false, sharedSecret: false }); + }); +}); + +describe("actions", () => { + it("split spreads a value over boxes by the digit at the end of the id", async () => { + const e = errand({ ssn: { type: "string", sensitivity: "secret", sources: [{ from: "literal", value: "123456789" }] } }, [ + { name: "ssn", label: "social security", do: { text: "{{ssn}}", split: [3, 2, 4] } }, + ]); + const inputs = await resolveInputs(e); + const parts = [1, 2, 3].map((n) => decide(e.rules!, field({ id: `Ssn${n}`, label: "Social security number" }), inputs)?.action); + expect(parts.map((p) => (p as { value: string }).value)).toEqual(["123", "45", "6789"]); + }); + + it("select escapes the input inside a pattern and tries patterns in order", async () => { + const e = errand({ v: { type: "string", sensitivity: "public", sources: [{ from: "literal", value: "a.b" }] } }, [ + { name: "pick", label: "pick", types: ["select-one"], do: { select: ["^{{v}}$", "fallback"] } }, + ]); + const inputs = await resolveInputs(e); + const opts = [{ value: "1", text: "aXb" }, { value: "2", text: "a.b" }, { value: "3", text: "fallback" }]; + expect(decide(e.rules!, field({ id: "p", type: "select-one", label: "Pick", options: opts }), inputs)?.action).toMatchObject({ value: "2" }); + const noExact = opts.filter((o) => o.value !== "2"); + expect(decide(e.rules!, field({ id: "p", type: "select-one", label: "Pick", options: noExact }), inputs)?.action).toMatchObject({ value: "3" }); + }); + + it("choose picks a different unused question each time and answer finds its answer", async () => { + const { errand: e, inputs } = await ftbInputs(); + const options = [{ value: "", text: "Select a question" }, { value: "q1", text: "Your first pet?" }, { value: "q2", text: "Your first car?" }, { value: "q3", text: "Your first school?" }]; + const picks = [1, 2, 3].map((n) => decide(e.rules!, field({ id: `SecQ${n}`, type: "select-one", label: `Security question ${n}`, options }), inputs)?.action); + expect(picks.map((p) => (p as { value: string }).value)).toEqual(["q1", "q2", "q3"]); + const qa = inputs.qa.get("security")!; + expect(Object.keys(qa.answers)).toEqual(["Your first pet?", "Your first car?", "Your first school?"]); + const answer = decide(e.rules!, field({ id: "SecA2", type: "text", label: "Answer 2", question: "Your first car?" }), inputs); + expect(answer?.action).toMatchObject({ kind: "text", value: qa.answers["Your first car?"], secret: true }); + }); + + it("peeking uses nothing up, and a box that comes back keeps its question", async () => { + const { errand: e, inputs } = await ftbInputs(); + const options = [{ value: "", text: "Select" }, { value: "q1", text: "Pet?" }, { value: "q2", text: "Car?" }]; + const q1 = field({ id: "SecQ1", type: "select-one", label: "Security question 1", options }); + expect(decide(e.rules!, q1, inputs, true)?.action).toMatchObject({ value: "q1" }); + expect(decide(e.rules!, q1, inputs, true)?.action).toMatchObject({ value: "q1" }); + expect(inputs.qa.get("security")!.answers).toEqual({}); + expect(decide(e.rules!, q1, inputs)?.action).toMatchObject({ value: "q1" }); + expect(decide(e.rules!, q1, inputs)?.action).toMatchObject({ value: "q1" }); + expect(decide(e.rules!, field({ id: "SecQ2", type: "select-one", label: "Security question 2", options }), inputs)?.action).toMatchObject({ value: "q2" }); + }); + + it("a gate action names its step", async () => { + const { errand: e, inputs } = await ftbInputs(); + expect(decide(e.rules!, field({ id: "Decl", type: "checkbox", label: "I declare under penalty of perjury" }), inputs)?.action).toEqual({ kind: "gate", step: "declaration" }); + expect(decide(e.rules!, field({ id: "Code", type: "text", label: "Enter the verification code" }), inputs)?.action).toEqual({ kind: "gate", step: "text-code" }); + }); +}); diff --git a/packages/openerrand/src/rules.ts b/packages/openerrand/src/rules.ts new file mode 100644 index 0000000..1544133 --- /dev/null +++ b/packages/openerrand/src/rules.ts @@ -0,0 +1,149 @@ +/** + * The field rules: what to do with each control on a page. Pure, so every + * matching rule of the spec is a unit test. + * + * 1. Id first, label second: every rule with an `id` is tried before any + * rule's `label`. + * 2. The first fitting rule decides. An id match is final even when it skips + * or cannot act; a label match that cannot act lets the next rule try. + * 3. A page step's own rules come before the errand's (the caller passes them + * concatenated in that order). + * 4. Choices before text is the driver's job: it calls this for selects, + * radios and checkboxes, reads the page again, then calls it for the rest. + * 5. An unmatched required field stops the run: the driver's job, given null. + */ + +import type { Inputs } from "./inputs.js"; +import type { Field, Rule } from "./types.js"; +import { escapeRegExp, re, templateNames } from "./util.js"; + +export type Action = + | { kind: "text"; value: string; secret: boolean; sharedSecret: boolean } + | { kind: "select"; value: string; text: string; secret: boolean; sharedSecret: boolean } + | { kind: "check" } + | { kind: "skip" } + | { kind: "gate"; step: string }; + +export interface Decision { + rule: Rule; + /** null when an id rule matched but could not act: the field is then unmatched, and the reason says why. */ + action: Action | null; + reason?: string; +} + +const CHOICE_TYPES = new Set(["select-one", "radio", "checkbox"]); + +export function isChoice(field: Field): boolean { + return CHOICE_TYPES.has(field.type); +} + +/** The digit at the end of a field's id: `Ssn2` is part 2. */ +function partIndex(field: Field): number | undefined { + const m = /(\d)\s*$/.exec(field.id) ?? /(\d)\s*$/.exec(field.name); + return m ? Number(m[1]) : undefined; +} + +function templateFlags(template: string, inputs: Inputs): { secret: boolean; sharedSecret: boolean } { + const names = templateNames(template); + return { + secret: names.some((n) => inputs.sensitivity(n) === "secret"), + sharedSecret: names.some((n) => n === inputs.sharedSecret), + }; +} + +function fill(template: string, inputs: Inputs): string | null { + let missing = false; + const value = template.replace(/\{\{\s*([a-z][a-z0-9_.]*)\s*\}\}/g, (_, name: string) => { + const v = inputs.get(name); + if (v === undefined) missing = true; + return v ?? ""; + }); + return missing ? null : value; +} + +/** What one rule does to one field, or why it cannot. */ +export function act(rule: Rule, field: Field, inputs: Inputs, peek = false): { action: Action | null; reason?: string } { + const how = rule.do; + if ("skip" in how) return { action: { kind: "skip" } }; + if ("gate" in how) return { action: { kind: "gate", step: how.gate } }; + if ("check" in how) { + if (how.check === true) return { action: { kind: "check" } }; + return re(how.check.label).test(field.label) ? { action: { kind: "check" } } : { action: null, reason: "its label is not the one to tick" }; + } + if ("text" in how) { + const value = fill(how.text, inputs); + if (value === null) return { action: null, reason: `${how.text} has no value` }; + const flags = templateFlags(how.text, inputs); + if (how.split) { + // A value spread over boxes 3, 2 and 4 wide: the box's own digit says which part. + const part = partIndex(field); + if (part !== undefined && part >= 1 && part <= how.split.length) { + const start = how.split.slice(0, part - 1).reduce((a, b) => a + b, 0); + return { action: { kind: "text", value: value.slice(start, start + how.split[part - 1]!), ...flags } }; + } + } + return { action: { kind: "text", value, ...flags } }; + } + if ("select" in how) { + for (const pattern of how.select) { + let missing = false; + const source = pattern.replace(/\{\{\s*([a-z][a-z0-9_.]*)\s*\}\}/g, (_, name: string) => { + const v = inputs.get(name); + if (v === undefined) missing = true; + return escapeRegExp(v ?? ""); + }); + if (missing) continue; + const compiled = new RegExp(source, "i"); + const option = field.options?.find((o) => o.value !== "" && (compiled.test(o.text) || compiled.test(o.value))); + if (option) { + const flags = how.select.reduce((acc, p) => { + const f = templateFlags(p, inputs); + return { secret: acc.secret || f.secret, sharedSecret: acc.sharedSecret || f.sharedSecret }; + }, { secret: false, sharedSecret: false }); + return { action: { kind: "select", value: option.value, text: option.text, ...flags } }; + } + } + return { action: null, reason: "no option matches" }; + } + if ("choose" in how) { + const qa = inputs.qa.get(how.choose); + const option = qa?.choose(field.options ?? [], peek, field.selector); + return option + ? { action: { kind: "select", value: option.value, text: option.text, secret: false, sharedSecret: false } } + : { action: null, reason: "no unused question to choose" }; + } + if ("answer" in how) { + const qa = inputs.qa.get(how.answer); + const labelIndex = /answer\s*(\d)/i.exec(field.label)?.[1]; + const answer = qa?.answerFor(`${field.question ?? ""} ${field.label}`, partIndex(field) ?? (labelIndex ? Number(labelIndex) : undefined)); + return answer ? { action: { kind: "text", value: answer, secret: true, sharedSecret: false } } : { action: null, reason: "no answer for this question" }; + } + return { action: null, reason: "unknown action" }; +} + +/** + * The first rule whose pattern and type fit, and what it does. null when no + * rule fits at all. `peek` decides without recording anything (a security + * question is not used up), for a caller that only wants to know. + */ +export function decide(rules: readonly Rule[], field: Field, inputs: Inputs, peek = false): Decision | null { + const haystack = `${field.id} ${field.name} ${field.label}`; + for (const rule of rules) { + if (!rule.id || (rule.types && !rule.types.includes(field.type as never))) continue; + if (!re(rule.id).test(field.id)) continue; + const { action, reason } = act(rule, field, inputs, peek); + return { rule, action, ...(reason ? { reason } : {}) }; + } + for (const rule of rules) { + if (!rule.label || (rule.types && !rule.types.includes(field.type as never))) continue; + if (!re(rule.label).test(haystack)) continue; + const { action } = act(rule, field, inputs, peek); + if (action) return { rule, action }; + } + return null; +} + +/** How a field is named in logs and errors: its label, type and whether it is required. Never its value. */ +export function describeField(field: Field): string { + return `${field.label || field.name || field.id || field.selector} (${field.type}${field.required ? ", required" : ""})`; +} diff --git a/packages/openerrand/src/run.test.ts b/packages/openerrand/src/run.test.ts new file mode 100644 index 0000000..5002460 --- /dev/null +++ b/packages/openerrand/src/run.test.ts @@ -0,0 +1,420 @@ +import { readFileSync, writeFileSync } from "node:fs"; +import { describe, expect, it } from "vitest"; +import type { CaptchaSolver } from "./captcha.js"; +import { type DocumentRecord, resolveInputs } from "./inputs.js"; +import { runErrand, type RunDeps } from "./run.js"; +import { type Store } from "./store.js"; +import { FakeDriver, type FakePage, field, ftbExample, tempStore } from "./testing.js"; +import { recordLockout, keyFor } from "./throttle.js"; +import type { Errand } from "./types.js"; +import { fileVault, parseEnv } from "./vault.js"; + +const O = "https://webapp.ftb.ca.gov"; +const NOW = new Date("2026-10-04T17:20:11Z"); + +/** Fictional returns: Jane Doe, 1234 Maple St, Sacramento 95814, corporation 1234567. */ +const RECORDS: DocumentRecord[] = [ + { form: "CA 540", field: "first name", value: "Jane", year: 2025, file: "2025/540.pdf", page: 1 }, + { form: "CA 540", field: "last name", value: "Doe", year: 2025, file: "2025/540.pdf", page: 1 }, + { form: "CA 540", field: "street address", value: "1234 Maple St", year: 2025, file: "2025/540.pdf", page: 1 }, + { form: "CA 540", field: "ZIP code", value: "95814", year: 2025, file: "2025/540.pdf", page: 1 }, + { form: "CA 100S", field: "California corporation number", value: "1234567", year: 2025, file: "2025/100S.pdf", page: 1 }, + { form: "CA 100S", field: "line 20", label: "Net income for tax purposes", value: "48210", year: 2025, file: "2025/100S.pdf", page: 3 }, + { form: "CA 100S", field: "line 20", label: "Net income for tax purposes", value: "39875", year: 2024, file: "2024/100S.pdf", page: 3 }, +]; + +interface SiteOptions { + expectedIncome?: string; + code?: string; + lockout?: boolean; + /** Where Terms goes instead of Profile. */ + termsTo?: string; + extraBusinessField?: boolean; + challenge?: number; +} + +/** A scripted MyFTB: Terms, Profile, (challenge), Business, Phone, Code, Confirmation. */ +function site(options: SiteOptions = {}): FakeDriver { + const questions = [{ value: "", text: "Select" }, { value: "1", text: "First pet?" }, { value: "2", text: "First car?" }, { value: "3", text: "First school?" }]; + let challengeReads = 0; + const pages: Record FakePage)> = { + [`${O}/MyFTBAccess/Registration/NewAccount`]: { + title: "Registration | Terms", + fields: [field({ id: "ReadTerms", type: "checkbox", label: "I have read the terms", required: true }), field({ id: "AcceptTerms", type: "checkbox", label: "I accept", required: true })], + }, + [`${O}/Profile`]: { + title: "Registration | Profile", + fields: [ + field({ id: "FstName", label: "First Name", required: true }), + field({ id: "MInitial", label: "Middle Initial" }), + field({ id: "LstName", label: "Last Name", required: true }), + field({ id: "UserName", label: "User Name", required: true }), + field({ id: "ReUserName", label: "Confirm User Name", required: true }), + field({ id: "Email", type: "email", label: "Email", required: true }), + field({ id: "ReEmail", type: "email", label: "Confirm Email", required: true }), + field({ id: "Password", type: "password", label: "Password", required: true }), + field({ id: "RePassword", type: "password", label: "Confirm Password", required: true }), + field({ id: "SecQ1", type: "select-one", label: "Security Question 1", options: questions, required: true }), + field({ id: "SecA1", type: "text", label: "Answer 1", required: true, question: "First pet?" }), + ], + }, + [`${O}/Challenge`]: (d) => { + challengeReads += 1; + if (challengeReads > (options.challenge ?? 0)) { + // The page's own script finished its proof of work and moved on. + d.current = `${O}/Business`; + return d.page(); + } + return { title: "Challenge Validation", hits: ["#sec-cpt-if"], fields: [] }; + }, + [`${O}/Business`]: (d) => ({ + title: "Registration | Business", + errors: d.current.endsWith("?err") ? ["The information you entered does not match our records."] : d.current.endsWith("?locked") ? ["Your account has been locked."] : [], + fields: [ + field({ id: "RoleInd", name: "Role", type: "radio", label: "Individual" }), + field({ id: "RoleBus", name: "Role", type: "radio", label: "Business Representative", required: true }), + field({ id: "CoType", type: "select-one", label: "Type of Company", options: [{ value: "", text: "Select" }, { value: "C", text: "Corporation" }, { value: "P", text: "Partnership" }] }), + field({ id: "FormType", type: "select-one", label: "Form type", options: [{ value: "", text: "Select" }, { value: "100", text: "100" }, { value: "100S", text: "100S" }] }), + field({ id: "Year", type: "select-one", label: "Tax year", options: [{ value: "", text: "Select" }, { value: "2025", text: "2025" }, { value: "2024", text: "2024" }] }), + field({ id: "CorpNo", label: "California corporation number", required: true }), + field({ id: "Zip", label: "ZIP Code", required: true }), + field({ id: "AddrNum", label: "Numbers in your mailing address", required: true }), + field({ id: "NetInc", label: "Net income for tax purposes", required: true }), + field({ id: "Decl", type: "checkbox", label: "I declare under penalty of perjury that the information is true", required: true }), + ...(options.extraBusinessField ? [field({ id: "Mystery", label: "Favourite colour", required: true })] : []), + ], + }), + [`${O}/Phone`]: { title: "Registration | Phone", fields: [field({ id: "Ph", type: "tel", label: "Phone Number", required: true }), field({ id: "Txt", name: "How", type: "radio", label: "Send me a text message" })] }, + [`${O}/Code`]: (d) => ({ + title: "Registration | Code", + errors: d.current.endsWith("?wrong") ? ["The code you entered is incorrect."] : [], + fields: [field({ id: "VerificationCode", label: "Enter the verification code", required: true })], + }), + [`${O}/Confirmation`]: { title: "Registration Confirmation", text: "Registration Confirmation. Your account was successfully created. We will mail you a PIN.", fields: [] }, + [`${O}/Locked`]: { title: "Locked", text: "Your account has been locked. Try again in 30 minutes.", fields: [] }, + }; + for (const url of [`${O}/Business?err`, `${O}/Business?locked`]) pages[url] = pages[`${O}/Business`]!; + pages[`${O}/Code?wrong`] = pages[`${O}/Code`]!; + return new FakeDriver(pages, (url, values) => { + const path = url.replace(O, "").replace(/\?.*$/, ""); + if (path === "/MyFTBAccess/Registration/NewAccount") return options.termsTo ?? `${O}/Profile`; + if (path === "/Profile") return options.challenge !== undefined ? `${O}/Challenge` : `${O}/Business`; + if (path === "/Business") { + if (options.lockout) return `${O}/Locked`; + if (values["#Decl"] !== true) return `${O}/Business?err`; + return values["#NetInc"] === (options.expectedIncome ?? "48210") ? `${O}/Phone` : `${O}/Business?err`; + } + if (path === "/Phone") return `${O}/Code`; + if (path === "/Code") return values["#VerificationCode"] === (options.code ?? "482913") ? `${O}/Confirmation` : `${O}/Code?wrong`; + return `${O}/Nowhere`; + }); +} + +async function deps(store: Store, driver: FakeDriver, over: Partial & { errand?: Errand; candidate?: number } = {}): Promise { + const errand = over.errand ?? ftbExample(); + const inputs = await resolveInputs(errand, { + extractor: { extract: async () => RECORDS }, + overrides: Object.fromEntries(Object.entries({ email: "jane@example.com", phone: "5555550100" }).filter(([k]) => k in (errand.inputs ?? {}))), + now: NOW, + random: () => 3, + ...(over.candidate ? { candidate: over.candidate } : {}), + }); + const lines: string[] = []; + return { + errand, + source: "ftb-register-business.json", + sha256: "0".repeat(64), + inputs, + store, + openDriver: async () => driver, + declare: false, + dryRun: false, + headful: false, + interactive: false, + prompt: async () => null, + say: (line) => lines.push(line), + now: () => NOW, + random: () => 3, + vault: null, + fallbackVault: fileVault(store.credentialsFile(errand.name)), + pollMs: 5, + rereadMs: 0, + lines, + ...over, + }; +} + +/** Write the code file when the runner says it is waiting: `codes` in order, one per wait. */ +function relay(store: Store, d: { say: (line: string) => void; lines: string[] }, codes: string[]): void { + const say = d.say; + d.say = (line) => { + say(line); + if (/WAITING for the sms code/.test(line)) { + const code = codes.shift(); + if (code) setTimeout(() => writeFileSync(store.codeFile("ftb-register-business"), `${code}\n`), 20); + } + }; +} + +describe("a full run", () => { + it("registers with --declare, takes the code from the file after a wrong one, writes the login, and ends on the PIN-letter card", async () => { + const store = tempStore(); + const driver = site(); + const d = await deps(store, driver, { declare: true }); + relay(store, d, ["000000", "482913"]); + const { record } = await runErrand(d); + + expect(record).toMatchObject({ outcome: "registered", kind: "success", page: `${O}/Confirmation`, handoff: expect.stringMatching(/^pin-letter\//) }); + expect(record.waiting).toEqual({ step: "pin-letter", what: "MyFTB PIN letter", expires_on: "2026-10-25", resume: "ftb-activate-business" }); + expect(record.card?.steps[1]).toContain("2026-10-25"); + expect(record.candidate).toEqual({ year: 2025, form: "CA 100S", field: "line 20", source: "2025/100S.pdf p3" }); + // Credentials written (to the 0600 fallback here) before success. + const saved = parseEnv(readFileSync(store.credentialsFile("ftb-register-business"), "utf8")); + expect(saved.FTB_BUSINESS_EMAIL).toBe("jane@example.com"); + expect(saved.FTB_BUSINESS_PASSWORD).toBe(d.inputs.get("password")); + expect(JSON.parse(saved.FTB_BUSINESS_SECURITY_ANSWERS!)).toHaveProperty(["First pet?"]); + // The business page was submitted exactly once, with the declaration ticked after the values. + expect(driver.submits.filter((s) => s.url.startsWith(`${O}/Business`))).toHaveLength(1); + const lines = d.lines.join("\n"); + expect(lines.indexOf("net income = ••••")).toBeLessThan(lines.indexOf("declaration ticked on your --declare")); + expect(lines).toContain("the site said: The code you entered is incorrect."); + // Rule 8: no value in the page log or the run record. + const log = readFileSync(store.pageLog("ftb-register-business"), "utf8"); + const record_ = JSON.stringify(record); + for (const secret of [d.inputs.get("password")!, "48210", "482913", "jane@example.com", "Maple", "5555550100"]) { + expect(log).not.toContain(secret); + expect(record_).not.toContain(secret); + } + expect(lines).not.toContain(d.inputs.get("password")!); + expect(lines).not.toContain("48210"); + expect(store.loadLedger().attempts).toHaveLength(1); + expect(driver.closed).toBe(true); + }); + + it("waits out a challenge the page clears by itself", async () => { + const store = tempStore(); + const d = await deps(store, site({ challenge: 2 }), { declare: true }); + relay(store, d, ["482913"]); + const errand = ftbExample(); + errand.steps[0] = { ...errand.steps[0]!, poll: "PT0S" } as Errand["steps"][number]; + const { record } = await runErrand({ ...d, errand }); + expect(record.reason).toBeUndefined(); + expect(record.kind).toBe("success"); + expect(d.lines.join("\n")).toContain("proof of work"); + }); +}); + +describe("the declare gate", () => { + it("without --declare stops on the page, prints the statement word for word, and submits nothing there", async () => { + const store = tempStore(); + const driver = site(); + const d = await deps(store, driver); + const { record } = await runErrand(d); + expect(record).toMatchObject({ kind: "stopped", reason: expect.stringMatching(/^gate: declaration needs --declare/) }); + expect(driver.submits.map((s) => s.url)).not.toContain(`${O}/Business`); + expect(driver.fills.find((f) => f.selector === "#Decl")).toBeUndefined(); + expect(d.lines).toContain(' "I declare under penalty of perjury that the information is true"'); + }); +}); + +describe("the dry run", () => { + it("fills up to the shared-secret page, shows it masked, and stops before its forward button without counting an attempt", async () => { + const store = tempStore(); + const driver = site(); + const { record } = await runErrand(await deps(store, driver, { dryRun: true })); + expect(record).toMatchObject({ kind: "dry-run", page: `${O}/Business` }); + expect(driver.submits.map((s) => s.url)).toEqual([`${O}/MyFTBAccess/Registration/NewAccount`, `${O}/Profile`]); + expect(driver.fills.find((f) => f.selector === "#Decl")).toBeUndefined(); + expect(store.loadLedger().attempts).toHaveLength(0); + }); +}); + +describe("a rejected shared secret", () => { + it("ends the run, retries nothing, and lists the other candidates for the person to choose", async () => { + const store = tempStore(); + const driver = site({ expectedIncome: "39875" }); + const d = await deps(store, driver, { declare: true }); + const { record } = await runErrand(d); + expect(record).toMatchObject({ outcome: "rejected", kind: "rejected" }); + expect(driver.submits.filter((s) => s.url.startsWith(`${O}/Business`))).toHaveLength(1); + expect(record.others).toEqual([{ index: 2, year: 2024, form: "CA 100S", field: "line 20", source: "2024/100S.pdf p3" }]); + expect(d.lines).toContain("Nothing was retried."); + expect(JSON.stringify(record)).not.toContain("39875"); + }); + + it("the next run, with --candidate 2, submits the person's choice", async () => { + const store = tempStore(); + const driver = site({ expectedIncome: "39875" }); + const d = await deps(store, driver, { declare: true, candidate: 2 }); + relay(store, d, ["482913"]); + expect((await runErrand(d)).record.kind).toBe("success"); + }); +}); + +describe("lockouts and the throttle", () => { + it("records a lockout page and refuses the next run even with --force", async () => { + const store = tempStore(); + const { record } = await runErrand(await deps(store, site({ lockout: true }), { declare: true })); + expect(record).toMatchObject({ kind: "rejected" }); + expect(Object.keys(store.loadLedger().lockedUntil)).toEqual(["https://webapp.ftb.ca.gov|default"]); + const later = new Date(NOW.getTime() + 10 * 60_000); + const driver = site(); + const again = await runErrand(await deps(store, driver, { declare: true, force: true, now: () => later })); + expect(again.record).toMatchObject({ kind: "stopped", reason: expect.stringMatching(/^throttle: the site locked/) }); + expect(driver.visits).toEqual([]); + }); + + it("refuses a third run inside 30 minutes, and --force lifts that cap", async () => { + const store = tempStore(); + for (const min of [0, 3]) await runErrand(await deps(store, site(), { now: () => new Date(NOW.getTime() + min * 60_000) })); + const at = () => new Date(NOW.getTime() + 6 * 60_000); + expect((await runErrand(await deps(store, site(), { now: at }))).record.reason).toMatch(/^throttle: 2 runs/); + expect((await runErrand(await deps(store, site(), { now: at, force: true }))).record.reason).toMatch(/^gate/); + }); + + it("a lockout recorded by another errand on the same site and account holds this one", async () => { + const store = tempStore(); + const other = { ...ftbExample(), name: "ftb-activate-business" }; + store.saveLedger(recordLockout(store.loadLedger(), keyFor(other), NOW)); + expect((await runErrand(await deps(store, site(), { force: true }))).record.reason).toMatch(/locked/); + }); +}); + +describe("stops", () => { + it("an unmatched required field stops the run and names it", async () => { + const store = tempStore(); + const { record } = await runErrand(await deps(store, site({ extraBusinessField: true }), { declare: true })); + expect(record.reason).toMatch(/^unmatched: no rule fills Favourite colour \(text, required\) \(id "Mystery"/); + }); + + it("a loop stops the run", async () => { + const store = tempStore(); + const { record } = await runErrand(await deps(store, site({ termsTo: `${O}/MyFTBAccess/Registration/NewAccount` }))); + expect(record.reason).toMatch(/^loop: the same page came back 3 times/); + }); + + it("leaving the site stops the run", async () => { + const store = tempStore(); + const { record } = await runErrand(await deps(store, site({ termsTo: "https://evil.example/" }))); + expect(record.reason).toMatch(/^off-site/); + }); + + it("a validation error on a later page ends the run as rejected", async () => { + const store = tempStore(); + const driver = site(); + // Profile submits; the business page then shows an error because nothing was declared on a page that has no declare gate. + const errand = ftbExample(); + errand.outcomes = errand.outcomes.filter((o) => o.kind !== "rejected"); + const d = await deps(store, driver, { errand, declare: true, candidate: 2 }); + const { record } = await runErrand(d); + expect(record).toMatchObject({ kind: "rejected", outcome: "page-errors" }); + }); +}); + +describe("identity proofing", () => { + const errand = (): Errand => ({ + ...ftbExample(), + steps: [...ftbExample().steps, { id: "id-me", kind: "identity-proofing", provider: "ID.me", origins: ["https://api.id.me"], handoff: "pin-letter", why: "ID.me may ask for a selfie. That is yours to do." }], + }); + + it("headless, it stops and tells the person, without reading the provider's page", async () => { + const store = tempStore(); + const driver = site({ termsTo: "https://api.id.me/session" }); + driver.pages["https://api.id.me/session"] = { title: "ID.me", fields: [field({ id: "selfie", label: "Take a selfie", required: true })] }; + const readsBefore = () => driver.reads; + const d = await deps(store, driver, { errand: errand() }); + const { record } = await runErrand(d); + expect(record.reason).toMatch(/^gate: identity proofing with ID.me is yours/); + expect(d.lines.join("\n")).toContain("ID.me: ID.me may ask for a selfie"); + expect(driver.fills.filter((f) => f.url.startsWith("https://api.id.me"))).toEqual([]); + expect(readsBefore()).toBe(2); // Terms read twice (choices, then text); never the provider's page. + }); + + it("headful at a terminal, it waits untouched until the site is back", async () => { + const store = tempStore(); + const driver = site({ termsTo: "https://api.id.me/session" }); + driver.pages["https://api.id.me/session"] = { title: "ID.me", fields: [] }; + setTimeout(() => (driver.current = `${O}/Profile`), 30); + const d = await deps(store, driver, { errand: errand(), headful: true, interactive: true }); + const { record } = await runErrand(d); + expect(record.reason).toMatch(/^gate: declaration/); + expect(driver.fills.filter((f) => f.url.startsWith("https://api.id.me"))).toEqual([]); + }); +}); + +describe("captcha", () => { + const commercial = (sector: "commercial" | "tax" = "commercial"): Errand => ({ + type: "logicsrc.openerrand", + version: "0.1", + name: "newsletter", + title: "Sign up", + site: { name: "Shop", sector, origins: ["https://shop.example"], start: ["https://shop.example/"] }, + inputs: { email: { type: "email", sensitivity: "personal", sources: [{ from: "literal", value: "jane@example.com" }] } }, + rules: [{ name: "email", label: "email", do: { text: "{{email}}" } }], + steps: [{ id: "cap", kind: "captcha", match: { selector: ".g-recaptcha" }, solver: "allowed", why: "a test for people" }, { id: "form", kind: "page" }], + outcomes: [{ name: "ok", kind: "success", text: "thanks" }], + }); + const shop = (): FakeDriver => { + let solved = false; + const driver = new FakeDriver( + { + "https://shop.example/": () => (solved ? { title: "Sign up", fields: [field({ id: "e", label: "Email", required: true })] } : { title: "Check", hits: [".g-recaptcha"], fields: [] }), + "https://shop.example/thanks": { title: "Thanks", text: "thanks", fields: [] }, + }, + () => "https://shop.example/thanks", + ); + (driver as unknown as { solve: () => void }).solve = () => (solved = true); + return driver; + }; + + it("headless with no solver: hands it to the person and stops", async () => { + const store = tempStore(); + const d = await deps(store, shop(), { errand: commercial() }); + expect((await runErrand(d)).record.reason).toMatch(/^gate: a captcha is the person's/); + }); + + it("uses a given solver only where the errand allows one, and logs the use without the answer", async () => { + const store = tempStore(); + const driver = shop(); + const used: string[] = []; + const solver: CaptchaSolver = { service: "test-solver", solve: async (c) => (used.push(c.url), (driver as unknown as { solve: () => void }).solve(), true) }; + const d = await deps(store, driver, { errand: commercial(), solver }); + expect((await runErrand(d)).record.kind).toBe("success"); + expect(used).toEqual(["https://shop.example/"]); + expect(readFileSync(store.pageLog("newsletter"), "utf8")).toContain('"captcha":{"url":"https://shop.example/","service":"test-solver"}'); + }); + + it("never calls a solver on a tax site, even when one is given and the file says allowed", async () => { + const store = tempStore(); + const used: string[] = []; + const solver: CaptchaSolver = { service: "test-solver", solve: async (c) => (used.push(c.url), true) }; + const d = await deps(store, shop(), { errand: commercial("tax"), solver }); + expect((await runErrand(d)).record.reason).toMatch(/^gate: a captcha/); + expect(used).toEqual([]); + }); +}); + +describe("resuming", () => { + it("the errand a mail gate named marks that card done when it succeeds", async () => { + const store = tempStore(); + store.saveRun({ errand: "x", name: "ftb-register-business", title: "t", sha256: "0", outcome: "registered", kind: "success", at: "2026-10-01T00:00:00.000Z", log: "", waiting: { step: "pin-letter", what: "PIN", resume: "ftb-activate-business" }, card: { id: "pin-letter/abc234", title: "PIN", steps: ["x"] } }); + const activate: Errand = { ...ftbExample(), name: "ftb-activate-business", inputs: {}, rules: [], steps: [{ id: "form", kind: "page" }], outcomes: [{ name: "activated", kind: "success", text: "activated" }], outputs: {}, handoffs: {} }; + const driver = new FakeDriver({ [`${O}/MyFTBAccess/Registration/NewAccount`]: { title: "Done", text: "Your account is activated.", fields: [] } }, () => ""); + const { record } = await runErrand(await deps(store, driver, { errand: activate })); + expect(record.kind).toBe("success"); + expect(store.runs().find((r) => r.name === "ftb-register-business")?.card?.done).toBe(true); + }); +}); + +describe("the mail gate", () => { + it("a rule that hands a field to a mail gate ends the run waiting, with the card", async () => { + const store = tempStore(); + const errand = ftbExample(); + errand.rules = [{ name: "pin", label: "pin", do: { gate: "pin-letter" } }, ...errand.rules!]; + const driver = site({ termsTo: `${O}/Pin` }); + driver.pages[`${O}/Pin`] = { title: "Activate", fields: [field({ id: "Pin", label: "PIN from the letter", required: true })] }; + const { record } = await runErrand(await deps(store, driver, { errand })); + expect(record).toMatchObject({ kind: "waiting", outcome: "pin-letter", waiting: { expires_on: "2026-10-25" } }); + expect(record.card?.command).toBe("ftb activate business --pin "); + }); +}); diff --git a/packages/openerrand/src/run.ts b/packages/openerrand/src/run.ts new file mode 100644 index 0000000..b6cc426 --- /dev/null +++ b/packages/openerrand/src/run.ts @@ -0,0 +1,523 @@ +/** + * The run loop: open the start page, then for each page find its step, test + * the outcomes, fill it from the rules, hand every gate to a person, and press + * the forward button, until the site answers or the runner has to stop. + * + * Everything that touches the world comes in through {@link RunDeps}, so the + * loop is tested against a scripted fake browser and only the integration + * test drives a real Chrome. + */ + +import { randomInt } from "node:crypto"; +import { existsSync, readFileSync, rmSync } from "node:fs"; +import type { CaptchaSolver } from "./captcha.js"; +import type { Driver } from "./driver.js"; +import type { Inputs, Prompt } from "./inputs.js"; +import { credentialValues, expiresOn, fileDownloads, renderCard, writeCredentials } from "./outputs.js"; +import { allowedOrigins, fits, isLockout, outcomeOf, selectorsOf, solverPermitted, stepById, stepFor } from "./pages.js"; +import { type Action, decide, describeField, isChoice } from "./rules.js"; +import type { Card, RunRecord, Store } from "./store.js"; +import { checkThrottle, keyFor, lockoutMs, recordAttempt, recordLockout } from "./throttle.js"; +import type { CaptchaStep, CodeStep, DeclareStep, Errand, Field, IdentityStep, MailStep, Outcome, Page, StopReason, WaitStep } from "./types.js"; +import { durationMs, ErrandError, MASK, originOf, re, sleep } from "./util.js"; +import type { Vault } from "./vault.js"; + +export interface RunDeps { + errand: Errand; + /** Where the file came from, and its SHA-256: both go in the run record. */ + source: string; + sha256: string; + inputs: Inputs; + store: Store; + openDriver: () => Promise; + /** The principal's consent to tick declaration boxes for this run (`--declare`). Never from a file or the environment. */ + declare: boolean; + dryRun: boolean; + /** A visible window a person can use. */ + headful: boolean; + /** A person is at a terminal. */ + interactive: boolean; + prompt: Prompt; + /** One line for the person, on the terminal (stderr). */ + say: (line: string) => void; + now?: () => Date; + random?: (max: number) => number; + vault: Vault | null; + fallbackVault: Vault; + solver?: CaptchaSolver; + account?: string; + force?: boolean; + /** Test knobs: how often files and pages are polled, and the pause between the choice and text passes. */ + pollMs?: number; + rereadMs?: number; +} + +export interface RunResult { + record: RunRecord; + file: string; +} + +class Stop extends Error { + constructor( + readonly reason: StopReason, + message: string, + readonly page?: string, + ) { + super(message); + } +} + +/** Run one errand. Never throws for anything the site or the person does; the record says what happened. */ +export async function runErrand(deps: RunDeps): Promise { + const { errand, inputs, store, say } = deps; + const now = deps.now ?? (() => new Date()); + const random = deps.random ?? randomInt; + const pollMs = deps.pollMs ?? 2_000; + const rereadMs = deps.rereadMs ?? 1_000; + const log = store.pageLog(errand.name); + const startedAt = now(); + const base = { + errand: errand.id ?? deps.source, + name: errand.name, + title: errand.title, + sha256: deps.sha256, + at: startedAt.toISOString(), + log, + ...(inputs.chosen ? { candidate: { ...(inputs.chosen.year !== undefined ? { year: inputs.chosen.year } : {}), form: inputs.chosen.form, field: inputs.chosen.field, source: inputs.chosen.source } } : {}), + }; + const finish = (record: Omit & Partial): RunResult => { + const full = { ...base, ...record } as RunRecord; + return { record: full, file: store.saveRun(full) }; + }; + + // The throttle: a dry run creates nothing a site keeps, so it is not an attempt. + const key = keyFor(errand, deps.account); + if (!deps.dryRun) { + const ledger = store.loadLedger(); + const check = checkThrottle(ledger, key, startedAt, deps.force); + if (!check.ok) { + say(`Not running: ${check.reason}. Next allowed at ${check.until.toISOString()}.`); + return finish({ outcome: "stopped", kind: "stopped", reason: `throttle: ${check.reason}; next at ${check.until.toISOString()}` }); + } + store.saveLedger(recordAttempt(ledger, key, startedAt)); + } + + const codeFile = store.codeFile(errand.name); + rmSync(codeFile, { force: true }); + + const limits = { pages: errand.limits?.pages ?? 15, samePage: errand.limits?.same_page ?? 2, pageMs: durationMs(errand.limits?.page_timeout, 30_000) }; + const selectors = selectorsOf(errand); + const allowed = allowedOrigins(errand); + const pageRules = (stepRules: Errand["rules"]) => [...(stepRules ?? []), ...(errand.rules ?? [])]; + + const logPage = (page: Page, extra: Record = {}): void => { + // Fields without values; the text only of a page with nothing to fill, which is a result page. + const fields = page.fields.map(({ selector, type, label, required, options }) => ({ selector, type, label, required, ...(options ? { options: options.slice(0, 15) } : {}) })); + store.appendPageLog(errand.name, { + at: now().toISOString(), + url: page.url, + title: page.title, + errors: page.errors, + fields, + ...(page.fields.length ? {} : { text: page.text.slice(0, 2000) }), + ...extra, + }); + }; + + let driver: Driver | null = null; + try { + driver = await deps.openDriver(); + let opened = false; + for (const url of errand.site.start) { + try { + await driver.goto(url); + opened = true; + break; + } catch (error) { + say(` could not open ${url}: ${(error as Error).message}`); + } + } + if (!opened) throw new Stop("error", "none of the start URLs opened"); + + let submitted = 0; + let lastUrl = ""; + let comebacks = 0; + let codeSubmitted = false; + let afterWait = false; + let refilled = false; + + for (;;) { + // The URL first, read without running script: an identity provider's page is never touched. + const url = await driver.url(); + const origin = originOf(url); + if (!errand.site.origins.includes(origin)) { + const identity = errand.steps.find((s): s is IdentityStep => s.kind === "identity-proofing" && s.origins.includes(origin)); + if (identity) { + await identityGate(identity, url); + continue; + } + throw new Stop("off-site", `the browser left the site for ${origin || url}`, url); + } + + let page = await readSettled(driver); + const hits = await driver.selectorHits(selectors); + const step = stepFor(errand, page, hits); + + if (step?.kind === "wait") { + await waitStep(step); + afterWait = true; + continue; + } + if (step?.kind === "captcha") { + await captchaGate(step, page.url); + continue; + } + if (step?.kind === "identity-proofing") { + await identityGate(step, page.url); + continue; + } + + logPage(page); + const codeFields = page.fields.filter((f) => { + const d = decide(pageRules(step?.kind === "page" ? step.rules : undefined), f, inputs, true); + return d?.action?.kind === "gate" && stepById(errand, d.action.step)?.kind === "code"; + }); + // A wrong or late code leaves the code page up with an error: wait for the next code, do not end the run. + const codeRetry = codeSubmitted && page.errors.length > 0 && codeFields.length > 0; + + const outcome = outcomeOf(errand, page); + if (outcome?.kind === "rejected") return rejected(outcome, page); + if (isLockout(errand, lockoutText(page))) return rejected({ name: "locked", kind: "rejected" }, page); + if (outcome) return await succeeded(outcome, page); + + if (page.errors.length && submitted > 0 && (errand.retry?.page_errors ?? "rejected") === "rejected" && !codeRetry) { + if (afterWait && !refilled) { + // The interstitial replayed the form without its values; the site never saw them. Fill it once more. + refilled = true; + say(" (the page came back from the bot check without its values; filling it again, once)"); + } else { + return rejected({ name: "page-errors", kind: "rejected" }, page); + } + } + afterWait = false; + if (codeRetry) say(` the site said: ${page.errors[0]}`); + + comebacks = page.url === lastUrl ? comebacks + 1 : 0; + if (comebacks >= limits.samePage && !codeRetry) throw new Stop("loop", `the same page came back ${comebacks + 1} times: ${page.url} (fields logged to ${log})`, page.url); + lastUrl = page.url; + if (submitted >= limits.pages) throw new Stop("pages", `more than ${limits.pages} pages (fields logged to ${log})`, page.url); + if (!step || step.kind !== "page") throw new Stop("unmatched", `no page step fits ${page.url}`, page.url); + + say(` ${page.title || "(untitled)"} ${page.url}`); + if (step.say) say(` ${step.say}`); + const rules = pageRules(step.rules); + const filled: string[] = []; + const missing: Field[] = []; + const declares: Array<{ field: Field; gate: DeclareStep }> = []; + const codes: Array<{ field: Field; gate: CodeStep }> = []; + let sharedSecretHere = false; + + for (const pass of ["choices", "text"] as const) { + if (pass === "text") { + // A choice can rewrite the page around it; let that land, then read again. + await sleep(rereadMs); + page = await readSettled(driver); + } + for (const field of page.fields) { + if ((pass === "choices") !== isChoice(field)) continue; + const decision = decide(rules, field, inputs); + if (!decision || !decision.action) { + // A radio group with one already chosen is answered; a required box nothing fills stops the run. + if (field.required && !(field.type === "radio" && field.groupChecked)) missing.push(field); + continue; + } + const { rule, action } = decision; + if (action.kind === "skip") continue; + if (action.kind === "gate") { + const gate = stepById(errand, action.step); + if (gate?.kind === "declare") declares.push({ field, gate }); + else if (gate?.kind === "code") codes.push({ field, gate }); + else if (gate?.kind === "mail") return waitingOnMail(gate, page); + else if (gate?.kind === "identity-proofing") await identityGate(gate, page.url); + else if (gate?.kind === "captcha") await captchaGate(gate, page.url); + continue; + } + if ((action.kind === "text" || action.kind === "select") && action.sharedSecret) sharedSecretHere = true; + if (!(await driver.fill(field, action.kind === "check" ? { kind: "check" } : { kind: action.kind, value: action.value }))) { + throw new Stop("error", `could not fill ${describeField(field)} on ${page.url}`, page.url); + } + filled.push(describeAction(rule.name, field, action)); + } + } + + for (const field of missing) { + if (step.unmatched === "ask" && deps.interactive) { + const answer = await deps.prompt(`${field.label || field.name}${field.options ? ` [${field.options.map((o) => o.text).join(" | ")}]` : ""}: `, { secret: field.type === "password" }); + if (answer === null) throw new Stop("unmatched", unmatchedMessage(field, page), page.url); + const option = field.options?.find((o) => o.text === answer || o.value === answer); + await driver.fill(field, field.type === "checkbox" || field.type === "radio" ? { kind: "check" } : { kind: field.options ? "select" : "text", value: option?.value ?? answer }); + filled.push(`${field.label || field.name} = (typed)`); + continue; + } + throw new Stop("unmatched", unmatchedMessage(field, page), page.url); + } + + for (const line of filled) say(` ${line}`); + + // Rule 12: a dry run stops on the first page holding a declaration or a shared secret, before its forward button. + if (deps.dryRun && (declares.length || sharedSecretHere)) { + for (const { field, gate } of declares) { + say(` declaration (not ticked): "${field.label}"`); + say(` ${gate.why}`); + } + say("Dry run: stopped before the forward button of the first page that sends a declaration or a shared secret."); + return finish({ outcome: "dry-run", kind: "dry-run", page: page.url }); + } + + for (const { field, gate } of declares) { + if (!deps.declare) { + say(""); + say(`This page asks the principal to state:`); + say(` "${field.label}"`); + say(` ${gate.why}`); + say("The values above are what would be attested. Nothing was submitted. Rerun with --declare once you have checked them."); + throw new Stop("gate", `declaration needs --declare: "${field.label}"`, page.url); + } + if (!(await driver.fill(field, { kind: "check" }))) throw new Stop("error", `could not tick ${describeField(field)}`, page.url); + say(` declaration ticked on your --declare: "${field.label}"`); + } + + let filledCode = false; + for (const { field, gate } of codes) { + const code = await waitForCode(gate); + if (!(await driver.fill(field, { kind: "text", value: code }))) throw new Stop("error", `could not fill ${describeField(field)}`, page.url); + say(` code = ${MASK}`); + filledCode = true; + } + + const pressed = await driver.submit(); + if (pressed.startsWith("?")) throw new Stop("no-forward", `no forward button on ${page.url}; buttons seen: ${pressed.slice(1) || "none"}`, page.url); + say(` -> ${pressed}`); + submitted += 1; + codeSubmitted = filledCode; + await driver.settle(); + } + } catch (error) { + if (error instanceof Stop) { + say(`Stopped (${error.reason}): ${error.message}`); + return finish({ outcome: "stopped", kind: "stopped", reason: `${error.reason}: ${error.message}`, ...(error.page ? { page: error.page } : {}) }); + } + const message = (error as Error).message; + say(`Stopped (error): ${message}`); + return finish({ outcome: "stopped", kind: "stopped", reason: `error: ${message}` }); + } finally { + await driver?.close(); + } + + // ------------------------------------------------------------------------- + + /** A page caught mid-navigation cannot be read; give it a moment, twice, before calling it an error. */ + async function readSettled(d: Driver): Promise { + for (let attempt = 0; ; attempt += 1) { + try { + return await d.read(); + } catch (error) { + if (attempt >= 2) throw error; + await sleep(Math.max(pollMs, 500)); + } + } + } + + function unmatchedMessage(field: Field, page: Page): string { + return `no rule fills ${describeField(field)} (id "${field.id}", name "${field.name}") on ${page.url}; add a rule, or set "unmatched": "ask" and run at a terminal (fields logged to ${log})`; + } + + function rejected(outcome: Outcome, page: Page): RunResult { + logPage(page, { outcome: outcome.name }); + if (!deps.dryRun && isLockout(errand, `${page.title} ${lockoutText(page)}`)) { + store.saveLedger(recordLockout(store.loadLedger(), key, now(), lockoutMs(errand))); + say(`The site locked the account. Nothing will run against it for ${Math.round(lockoutMs(errand) / 60_000)} minutes, --force or not.`); + } + say(`Rejected: ${page.errors.join(" ") || page.text.slice(0, 300)}`); + const others = inputs.candidates + .map((c, i) => ({ index: i + 1, ...(c.year !== undefined ? { year: c.year } : {}), form: c.form, field: c.field, source: c.source })) + .filter((c) => !(inputs.chosen && inputs.candidates[c.index - 1] === inputs.chosen)); + if (inputs.sharedSecret) { + // Rule 5: one shared secret per run, never retried. The person chooses the next one. + say("Nothing was retried."); + if (others.length) { + say("Other candidates (choose one for the next run with --candidate N):"); + for (const c of others) say(` ${c.index}. ${c.year ?? ""} ${c.form} ${c.field} from ${c.source}`.replace(/\s+/g, " ")); + } + } + return finish({ outcome: outcome.name, kind: "rejected", page: page.url, ...(inputs.sharedSecret && others.length ? { others } : {}) }); + } + + async function succeeded(outcome: Outcome, page: Page): Promise { + logPage(page, { outcome: outcome.name }); + // Rule 10: credentials are written before success is reported. + const values = credentialValues(errand, outcome, inputs); + const saved = await writeCredentials(values, deps.vault, deps.fallbackVault); + const downloads = fileDownloads(errand, outcome, inputs, driver?.downloads() ?? []); + const then = outcome.then ? stepById(errand, outcome.then) : undefined; + let card: Card | undefined; + let waiting: RunRecord["waiting"]; + if (then?.kind === "mail") { + const expires = then.expires ? expiresOn(now(), durationMs(then.expires, 0)) : undefined; + card = then.handoff ? renderCard(errand, then.handoff, inputs, expires ? { expires_on: expires } : {}, random) : undefined; + waiting = { step: then.id, what: then.what, ...(expires ? { expires_on: expires } : {}), ...(then.resume ? { resume: then.resume } : {}) }; + } + const kind = outcome.kind === "waiting" ? "waiting" : "success"; + // The errand a mail gate said to resume with has succeeded: its card is done. + if (kind === "success") { + for (const earlier of store.runs()) { + if (earlier.waiting?.resume === errand.name && earlier.card && !earlier.card.done) { + store.saveRun({ ...earlier, card: { ...earlier.card, done: true } }); + say(` card ${earlier.card.id} is done`); + } + } + } + say(`${kind === "waiting" ? "Waiting" : "Done"}: ${outcome.name} (${errand.title}).`); + if (saved.where) say(` credentials saved to ${saved.where}`); + if (saved.warning) say(` ${saved.warning}`); + for (const d of downloads) say(` filed ${d}`); + if (waiting) say(` ${waiting.what}${waiting.expires_on ? `: act before ${waiting.expires_on}` : ""}`); + if (card) { + say(` card ${card.id} (logicsrc errand status shows it; it is not sent anywhere)`); + for (const [i, s] of card.steps.entries()) say(` ${i + 1}. ${s}`); + if (card.command) say(` ${card.command}`); + } + return finish({ + outcome: outcome.name, + kind, + page: page.url, + ...(saved.where ? { vault: saved.where } : {}), + ...(downloads.length ? { downloads } : {}), + ...(card ? { handoff: card.id, card } : {}), + ...(waiting ? { waiting } : {}), + }); + } + + function waitingOnMail(gate: MailStep, page: Page): RunResult { + const expires = gate.expires ? expiresOn(now(), durationMs(gate.expires, 0)) : undefined; + const card = gate.handoff ? renderCard(errand, gate.handoff, inputs, expires ? { expires_on: expires } : {}, random) : undefined; + say(`Waiting: ${gate.what}${expires ? `, act before ${expires}` : ""}.`); + if (card) say(` card ${card.id} (logicsrc errand status shows it)`); + return finish({ + outcome: gate.id, + kind: "waiting", + page: page.url, + waiting: { step: gate.id, what: gate.what, ...(expires ? { expires_on: expires } : {}), ...(gate.resume ? { resume: gate.resume } : {}) }, + ...(card ? { handoff: card.id, card } : {}), + }); + } + + /** An interstitial the page clears by itself: poll until its match no longer fits. Never solved, never bypassed. */ + async function waitStep(step: WaitStep): Promise { + say(` ${step.say ?? `waiting for ${step.id} to clear`}`); + const until = now().getTime() + durationMs(step.timeout, 90_000); + const poll = durationMs(step.poll, 3_000); + while (now().getTime() < until) { + await sleep(Math.min(poll, pollMs * 5)); + const url = await driver!.url(); + if (!errand.site.origins.includes(originOf(url))) return; + // The page may be navigating away as it is read; that is the interstitial clearing, so poll again. + const page = await driver!.read().catch(() => null); + if (!page) continue; + if (!fits(step.match, page, await driver!.selectorHits(selectors))) { + await sleep(Math.min(1_500, pollMs)); + return; + } + } + throw new Stop("timeout", `${step.id} did not clear in ${step.timeout}`); + } + + /** Identity proofing is the person's: hand them the window, or stop. Nothing on the provider's pages is touched. */ + async function identityGate(step: IdentityStep, url: string): Promise { + say(""); + say(`${step.provider}: ${step.why}`); + if (!(deps.headful && deps.interactive)) { + const card = step.handoff ? renderCard(errand, step.handoff, inputs, {}, random) : undefined; + if (card) say(` card ${card.id} (logicsrc errand status shows it)`); + throw new Stop("gate", `identity proofing with ${step.provider} is yours to do; rerun with --headful at a terminal to do it in the window`, url); + } + say(` The window is yours. The runner waits, without touching the page, until it is back on ${errand.site.origins.join(" or ")}.`); + const until = now().getTime() + durationMs(step.timeout, 30 * 60_000); + while (now().getTime() < until) { + await sleep(pollMs); + const current = await driver!.url(); + if (errand.site.origins.includes(originOf(current))) return; + if (!allowed.has(originOf(current))) throw new Stop("off-site", `the browser left for ${originOf(current) || current}`, current); + } + throw new Stop("timeout", `identity proofing did not return to the site within ${step.timeout ?? "PT30M"}`, url); + } + + /** A captcha is the person's unless the errand allows a solver where the rules permit one, and one was given. */ + async function captchaGate(step: CaptchaStep, url: string): Promise { + if (deps.solver && solverPermitted(errand, step)) { + store.appendPageLog(errand.name, { at: now().toISOString(), captcha: { url, service: deps.solver.service } }); + say(` captcha: using ${deps.solver.service}, as this errand allows (logged)`); + if (!(await deps.solver.solve({ url, evaluate: (e) => driver!.evaluate(e) }))) throw new Stop("gate", `${deps.solver.service} did not clear the captcha`, url); + await driver!.settle(); + return; + } + say(""); + say(`A captcha: ${step.why ?? "a test meant for a person."}`); + if (!(deps.headful && deps.interactive)) throw new Stop("gate", "a captcha is the person's to answer; rerun with --headful at a terminal", url); + say(" Answer it in the window; the runner waits until it is gone."); + const until = now().getTime() + durationMs(step.timeout, 10 * 60_000); + while (now().getTime() < until) { + await sleep(pollMs); + const current = await driver!.url(); + if (!errand.site.origins.includes(originOf(current))) return; + const page = await driver!.read(); + if (!fits(step.match, page, await driver!.selectorHits(selectors))) return; + } + throw new Stop("timeout", "the captcha was not answered in time", url); + } + + /** + * A one-time code through a declared relay: typed at the terminal, or + * written to the code file by whoever holds the phone. Used once, never + * logged or shown. A code that does not fit the pattern is ignored and the + * wait goes on. + */ + async function waitForCode(step: CodeStep): Promise { + const pattern = step.pattern ?? "^\\w{4,10}$"; + const timeoutMs = durationMs(step.timeout, 15 * 60_000); + if (step.why) say(` ${step.why}`); + if (deps.interactive && step.relay.includes("terminal")) { + for (;;) { + const code = await deps.prompt(`Code sent by ${step.channel}: `, { secret: false }); + if (code === null) break; + if (re(pattern).test(code.trim())) return code.trim(); + say(" that does not look like the code; try again"); + } + } + if (!step.relay.includes("file")) throw new Stop("gate", `the code arrives by ${step.channel} and this errand relays it only through ${step.relay.join(", ")}; nobody is at a terminal`); + say(` WAITING for the ${step.channel} code: write it to ${codeFile} (${Math.round(timeoutMs / 60_000)} minutes)`); + const until = now().getTime() + timeoutMs; + while (now().getTime() < until) { + await sleep(pollMs); + if (!existsSync(codeFile)) continue; + const code = readFileSync(codeFile, "utf8").trim(); + rmSync(codeFile, { force: true }); + if (re(pattern).test(code)) return code; + } + throw new Stop("timeout", `no code arrived in ${step.timeout}`); + } +} + +/** Where a lockout is read: the errors a page shows, or the text of a result page with nothing to fill (never a form's own warnings). */ +function lockoutText(page: Page): string { + return page.errors.length ? page.errors.join(" ") : page.fields.length ? "" : page.text; +} + +function describeAction(name: string, field: Field, action: Action): string { + if (action.kind === "text") return `${name} = ${action.secret ? MASK : action.value}`; + if (action.kind === "select") return `${name} = ${action.secret ? MASK : action.text}`; + if (action.kind === "check") return `${name}${field.type === "radio" ? ` = ${field.label}` : ""}`; + return name; +} + +export { ErrandError }; diff --git a/packages/openerrand/src/store.ts b/packages/openerrand/src/store.ts new file mode 100644 index 0000000..0d8da5b --- /dev/null +++ b/packages/openerrand/src/store.ts @@ -0,0 +1,163 @@ +/** + * Everything the runner keeps on disk, under one directory only the principal + * can read: `$LOGICSRC_ERRAND_HOME`, else `$XDG_DATA_HOME/logicsrc/errand`, + * else `~/.local/share/logicsrc/errand`. + * + * | Path | What | + * | --- | --- | + * | `throttle.json` | The attempt ledger and lockouts. | + * | `approvals.json` | The SHA-256 of each errand file last run, by errand name. | + * | `runs/-.json` | One run record per run. No input values, ever. | + * | `pages/.jsonl` | The page log: fields (never values) and result-page text. | + * | `profiles//` | One Chrome profile per site, kept between runs. | + * | `codes/.code` | Where a one-time code may be written by whoever holds the phone. | + * | `credentials/.env` | Credentials, when the errand names no vault (0600). | + * + * Files are 0600 and directories 0700. + */ + +import { appendFileSync, existsSync, mkdirSync, readFileSync, readdirSync, renameSync, writeFileSync } from "node:fs"; +import { homedir } from "node:os"; +import { dirname, join } from "node:path"; +import { emptyLedger, type Ledger } from "./throttle.js"; + +export type Env = Record; + +export function errandHome(env: Env = process.env): string { + if (env.LOGICSRC_ERRAND_HOME) return env.LOGICSRC_ERRAND_HOME; + const data = env.XDG_DATA_HOME || join(env.HOME || homedir(), ".local", "share"); + return join(data, "logicsrc", "errand"); +} + +export function ensureDir(path: string): string { + mkdirSync(path, { recursive: true, mode: 0o700 }); + return path; +} + +/** Write via a temp file and rename, so a crash never leaves half a ledger. */ +export function writePrivate(path: string, text: string): void { + ensureDir(dirname(path)); + const tmp = `${path}.${process.pid}.tmp`; + writeFileSync(tmp, text, { mode: 0o600 }); + renameSync(tmp, path); +} + +function readJson(path: string, fallback: T): T { + try { + return JSON.parse(readFileSync(path, "utf8")) as T; + } catch { + return fallback; + } +} + +export class Store { + readonly home: string; + + constructor(home: string) { + this.home = home; + } + + static fromEnv(env: Env = process.env): Store { + return new Store(errandHome(env)); + } + + path(...parts: string[]): string { + return join(this.home, ...parts); + } + + loadLedger(): Ledger { + const ledger = readJson(this.path("throttle.json"), emptyLedger()); + return { attempts: ledger.attempts ?? [], lockedUntil: ledger.lockedUntil ?? {} }; + } + + saveLedger(ledger: Ledger): void { + writePrivate(this.path("throttle.json"), `${JSON.stringify(ledger, null, 2)}\n`); + } + + approvals(): Record { + return readJson(this.path("approvals.json"), {}); + } + + approve(name: string, sha256: string, file: string, at: Date): void { + const all = this.approvals(); + all[name] = { sha256, file, at: at.toISOString() }; + writePrivate(this.path("approvals.json"), `${JSON.stringify(all, null, 2)}\n`); + } + + pageLog(name: string): string { + return this.path("pages", `${name}.jsonl`); + } + + appendPageLog(name: string, line: unknown): void { + const file = this.pageLog(name); + ensureDir(dirname(file)); + appendFileSync(file, `${JSON.stringify(line)}\n`, { mode: 0o600 }); + } + + profile(origin: string): string { + const host = origin.replace(/^https?:\/\//, "").replace(/[^a-z0-9.-]/gi, "_"); + return ensureDir(this.path("profiles", host)); + } + + codeFile(name: string): string { + ensureDir(this.path("codes")); + return this.path("codes", `${name}.code`); + } + + credentialsFile(name: string): string { + return this.path("credentials", `${name}.env`); + } + + downloadsDir(runId: string): string { + return ensureDir(this.path("downloads", runId)); + } + + saveRun(record: RunRecord): string { + const file = this.path("runs", `${record.at.replace(/[:.]/g, "-")}-${record.name}.json`); + writePrivate(file, `${JSON.stringify(record, null, 2)}\n`); + return file; + } + + runs(): RunRecord[] { + const dir = this.path("runs"); + if (!existsSync(dir)) return []; + return readdirSync(dir) + .filter((f) => f.endsWith(".json")) + .sort() + .map((f) => readJson(join(dir, f), null)) + .filter((r): r is RunRecord => r !== null); + } +} + +/** A hand-off card as kept in the run record: built-ins and public inputs only. */ +export interface Card { + id: string; + title: string; + open?: string; + steps: string[]; + command?: string; + expires_on?: string; + done?: boolean; +} + +/** The record an agent can read after a run. It never holds an input value. */ +export interface RunRecord { + errand: string; + name: string; + title: string; + sha256: string; + outcome: string; + kind: "success" | "rejected" | "waiting" | "stopped" | "dry-run"; + reason?: string; + at: string; + page?: string; + candidate?: { year?: number; form: string; field: string; source: string }; + /** The other candidates for a person to choose from after a rejection (no values). */ + others?: Array<{ index: number; year?: number; form: string; field: string; source: string }>; + handoff?: string; + card?: Card; + waiting?: { step: string; what: string; expires_on?: string; resume?: string }; + vault?: string; + downloads?: string[]; + log: string; +} diff --git a/packages/openerrand/src/testing.ts b/packages/openerrand/src/testing.ts new file mode 100644 index 0000000..39a1cec --- /dev/null +++ b/packages/openerrand/src/testing.ts @@ -0,0 +1,135 @@ +/** + * Test helpers: a scripted browser and fixtures. Not part of the build. + */ + +import { mkdtempSync, readFileSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { dirname, join } from "node:path"; +import { fileURLToPath } from "node:url"; +import { afterAll } from "vitest"; +import type { DownloadedFile, Driver, FillAction } from "./driver.js"; +import { Store } from "./store.js"; +import type { Errand, Field, Page } from "./types.js"; + +const here = dirname(fileURLToPath(import.meta.url)); + +/** The worked example from the spec, as published in @logicsrc/schemas' fixtures. */ +export function ftbExample(): Errand { + return JSON.parse(readFileSync(join(here, "..", "..", "schemas", "fixtures", "openerrand", "ftb-register-business.json"), "utf8")) as Errand; +} + +export function ftbExamplePath(): string { + return join(here, "..", "..", "schemas", "fixtures", "openerrand", "ftb-register-business.json"); +} + +const temps: string[] = []; +// Registered in each test file that imports this module (vitest isolates modules per file). +afterAll(() => { + for (const dir of temps.splice(0)) rmSync(dir, { recursive: true, force: true }); +}); + +/** A temp dir removed after the test file finishes. */ +export function tempDir(prefix = "openerrand-test-"): string { + const dir = mkdtempSync(join(tmpdir(), prefix)); + temps.push(dir); + return dir; +} + +export function tempStore(): Store { + return new Store(tempDir()); +} + +export function field(partial: Partial & { id: string }): Field { + return { selector: `#${partial.id}`, name: partial.id, type: "text", label: "", required: false, ...partial }; +} + +export interface FakePage { + title?: string; + text?: string; + errors?: string[]; + fields?: Field[]; + /** CSS selectors that find something on this page. */ + hits?: string[]; + buttons?: string[]; +} + +/** + * A browser over a table of pages. `submit` calls `next(url, values)` to say + * where the site goes; `values` maps selector to what was filled (true for a + * ticked box). Every fill and submit is recorded so a test can assert that + * nothing was sent. + */ +export class FakeDriver implements Driver { + current = ""; + values: Record = {}; + readonly fills: Array<{ url: string; selector: string; value: string | boolean }> = []; + readonly submits: Array<{ url: string; values: Record }> = []; + readonly visits: string[] = []; + reads = 0; + closed = false; + files: DownloadedFile[] = []; + + constructor( + readonly pages: Record FakePage)>, + readonly next: (url: string, values: Record, driver: FakeDriver) => string, + ) {} + + page(): FakePage { + const p = this.pages[this.current]; + if (!p) return { title: "Not found", text: "404", fields: [] }; + return typeof p === "function" ? p(this) : p; + } + + async goto(url: string): Promise { + this.current = url; + this.values = {}; + this.visits.push(url); + } + + async url(): Promise { + return this.current; + } + + async read(): Promise { + this.reads += 1; + const p = this.page(); + return { url: this.current, title: p.title ?? "", text: p.text ?? "", errors: p.errors ?? [], fields: p.fields ?? [] }; + } + + async selectorHits(selectors: string[]): Promise> { + const hits = this.page().hits ?? []; + return Object.fromEntries(selectors.map((s) => [s, hits.includes(s)])); + } + + async fill(f: Field, action: FillAction): Promise { + const value = action.kind === "check" ? true : action.value; + this.values[f.selector] = value; + this.fills.push({ url: this.current, selector: f.selector, value }); + return true; + } + + async submit(): Promise { + const buttons = this.page().buttons ?? ["Continue"]; + if (!buttons.length) return "?"; + this.submits.push({ url: this.current, values: { ...this.values } }); + const to = this.next(this.current, { ...this.values }, this); + this.current = to; + this.values = {}; + this.visits.push(to); + return buttons[0]!; + } + + async settle(): Promise {} + + downloads(): DownloadedFile[] { + return this.files; + } + + async evaluate(): Promise { + return null; + } + + async close(): Promise { + this.closed = true; + } +} diff --git a/packages/openerrand/src/throttle.ts b/packages/openerrand/src/throttle.ts new file mode 100644 index 0000000..77b9b34 --- /dev/null +++ b/packages/openerrand/src/throttle.ts @@ -0,0 +1,103 @@ +/** + * The attempt ledger. Sites escalate: a tax board restarts its 30-minute lock + * on any attempt made inside it, others ban an account or an address for + * longer each time. A retry loop turns one lockout into an open-ended one, so + * every run that sends something is recorded before it starts, and a run is + * refused while a recorded lockout lasts. + * + * - Window and day caps are per errand and account: 2 runs in 30 minutes, 4 a + * day. + * - Spacing is per site, whatever the errand or account: 2 minutes between + * runs, because the site sees one browser and one phone. + * - A lockout is per site and account, so an activation errand respects the + * lock a registration errand ran into on the same account. Its length is + * `metadata.lockout.duration` when the file gives one, else 35 minutes. + * - `--force` lifts the caps and the spacing. It never lifts a lockout. + */ + +import type { Errand } from "./types.js"; +import { durationMs } from "./util.js"; + +export interface Attempt { + site: string; + account: string; + errand: string; + at: string; +} + +export interface Ledger { + attempts: Attempt[]; + /** `|` to the time before which nothing may be sent. */ + lockedUntil: Record; +} + +export const LIMITS = { + windowMs: 30 * 60_000, + perWindow: 2, + perDay: 4, + spacingMs: 2 * 60_000, + /** A 30-minute lock and 5 more, so a skewed clock cannot restart it. */ + lockoutMs: 35 * 60_000, +} as const; + +const DAY = 86_400_000; + +export function emptyLedger(): Ledger { + return { attempts: [], lockedUntil: {} }; +} + +export interface ThrottleKey { + site: string; + account: string; + errand: string; +} + +export function keyFor(errand: Errand, account = "default"): ThrottleKey { + return { site: errand.site.origins[0]!, account, errand: errand.name }; +} + +const lockKey = (k: ThrottleKey): string => `${k.site}|${k.account}`; + +export type ThrottleCheck = { ok: true } | { ok: false; until: Date; reason: string; lockout: boolean }; + +export function checkThrottle(ledger: Ledger, key: ThrottleKey, now: Date, force = false): ThrottleCheck { + const t = now.getTime(); + const locked = ledger.lockedUntil[lockKey(key)]; + if (locked && Date.parse(locked) > t) { + return { ok: false, lockout: true, until: new Date(locked), reason: `the site locked the ${key.account} account; any attempt before then can restart the lock, and --force does not lift it` }; + } + if (force) return { ok: true }; + const today = ledger.attempts.map((a) => ({ ...a, ms: Date.parse(a.at) })).filter((a) => t - a.ms < DAY); + const onSite = today.filter((a) => a.site === key.site); + const last = Math.max(0, ...onSite.map((a) => a.ms)); + if (last && t - last < LIMITS.spacingMs) { + return { ok: false, lockout: false, until: new Date(last + LIMITS.spacingMs), reason: "runs on one site are spaced 2 minutes apart" }; + } + const mine = onSite.filter((a) => a.account === key.account && a.errand === key.errand).sort((a, b) => a.ms - b.ms); + const recent = mine.filter((a) => t - a.ms < LIMITS.windowMs); + if (recent.length >= LIMITS.perWindow) { + return { ok: false, lockout: false, until: new Date(recent[0]!.ms + LIMITS.windowMs), reason: `${LIMITS.perWindow} runs of ${key.errand} in 30 minutes` }; + } + if (mine.length >= LIMITS.perDay) { + return { ok: false, lockout: false, until: new Date(mine[0]!.ms + DAY), reason: `${LIMITS.perDay} runs of ${key.errand} today` }; + } + return { ok: true }; +} + +export function recordAttempt(ledger: Ledger, key: ThrottleKey, now: Date): Ledger { + const dayAgo = now.getTime() - DAY; + return { ...ledger, attempts: [...ledger.attempts.filter((a) => Date.parse(a.at) > dayAgo), { ...key, at: now.toISOString() }] }; +} + +export function lockoutMs(errand: Errand): number { + const duration = (errand.metadata as { lockout?: { duration?: unknown } } | undefined)?.lockout?.duration; + return typeof duration === "string" ? durationMs(duration, LIMITS.lockoutMs) : LIMITS.lockoutMs; +} + +export function recordLockout(ledger: Ledger, key: ThrottleKey, now: Date, ms: number = LIMITS.lockoutMs): Ledger { + const until = new Date(now.getTime() + ms).toISOString(); + const current = ledger.lockedUntil[lockKey(key)]; + // A later lock never shortens an earlier, longer one. + const next = current && current > until ? current : until; + return { ...ledger, lockedUntil: { ...ledger.lockedUntil, [lockKey(key)]: next } }; +} diff --git a/packages/openerrand/src/types.ts b/packages/openerrand/src/types.ts new file mode 100644 index 0000000..7a2c4b7 --- /dev/null +++ b/packages/openerrand/src/types.ts @@ -0,0 +1,210 @@ +/** + * The errand file as the runner reads it. These mirror + * `logicsrc-openerrand.schema.json` in @logicsrc/schemas; the runner never + * trusts a file it has not first passed through @logicsrc/validators, so the + * types describe a document that is already known to be well formed. + */ + +export type Sensitivity = "public" | "personal" | "secret"; +export type InputType = "string" | "integer" | "number" | "email" | "date" | "boolean" | "qa-set"; +export type Role = "shared-secret" | "credential" | "identifier"; +export type Sector = "government" | "tax" | "financial" | "healthcare" | "identity-provider" | "commercial" | "other"; +export type FieldType = "text" | "password" | "email" | "tel" | "number" | "date" | "textarea" | "select-one" | "radio" | "checkbox"; + +export type Source = + | { from: "prompt"; ask?: string } + | { from: "vault"; key: string } + | { + from: "document"; + form: string; + field: string; + match?: string; + pick?: "newest" | "oldest" | "each"; + years?: { back?: number; current?: boolean }; + transform?: string; + } + | { from: "derive"; input: string; transform: string } + | { from: "candidate"; input: string; part: "year" | "form" | "field" | "source" } + | { from: "generate"; length: number; classes: Array<"lower" | "upper" | "digit" | "special">; special?: string } + | { from: "literal"; value: string | number | boolean }; + +export interface Input { + label?: string; + type: InputType; + sensitivity: Sensitivity; + role?: Role; + required?: boolean; + pattern?: string; + max_length?: number; + count?: number; + sources: Source[]; +} + +export type RuleAction = + | { text: string; split?: number[] } + | { select: string[] } + | { check: true | { label: string } } + | { skip: true } + | { gate: string } + | { choose: string } + | { answer: string }; + +export interface Rule { + name: string; + id?: string; + label?: string; + types?: FieldType[]; + do: RuleAction; +} + +export interface Match { + url?: string; + title?: string; + text?: string; + selector?: string; +} + +export interface PageStep { + id: string; + kind: "page"; + match?: Match; + rules?: Rule[]; + unmatched?: "stop" | "ask"; + say?: string; +} +export interface WaitStep { + id: string; + kind: "wait"; + match: Match; + timeout: string; + poll?: string; + say?: string; +} +export interface DeclareStep { + id: string; + kind: "declare"; + statement: string; + why: string; + say?: string; +} +export interface IdentityStep { + id: string; + kind: "identity-proofing"; + provider: string; + origins: string[]; + match?: Match; + timeout?: string; + handoff?: string; + why: string; + say?: string; +} +export interface CodeStep { + id: string; + kind: "code"; + channel: "sms" | "email" | "voice" | "app"; + relay: Array<"terminal" | "file" | "page">; + pattern?: string; + timeout: string; + why?: string; + say?: string; +} +export interface MailStep { + id: string; + kind: "mail"; + what: string; + arrives?: string; + expires?: string; + resume?: string; + input?: string; + handoff?: string; + why?: string; + say?: string; +} +export interface CaptchaStep { + id: string; + kind: "captcha"; + match: Match; + solver?: "forbidden" | "allowed"; + timeout?: string; + why?: string; + say?: string; +} + +export type Step = PageStep | WaitStep | DeclareStep | IdentityStep | CodeStep | MailStep | CaptchaStep; +export type GateStep = DeclareStep | IdentityStep | CodeStep | MailStep | CaptchaStep; + +export interface Outcome { + name: string; + kind: "success" | "rejected" | "waiting"; + text?: string; + url?: string; + then?: string; +} + +export interface Handoff { + title: string; + open?: string; + steps: string[]; + command?: string; +} + +export interface Download { + match: { url?: string; filename?: string; type?: string }; + to: string; + when?: string; +} + +export interface Errand { + type: "logicsrc.openerrand"; + version: "0.1"; + id?: string; + name: string; + title: string; + description?: string; + publisher?: string; + updated?: string; + reference?: string; + principal?: "self" | "represented"; + site: { name: string; sector?: Sector; origins: string[]; start: string[]; terms?: string }; + limits?: { pages?: number; page_timeout?: string; same_page?: number }; + inputs?: Record; + rules?: Rule[]; + steps: Step[]; + submit?: { labels: string; never?: string; ignore?: string }; + outcomes: Outcome[]; + retry?: { shared_secret?: "never"; page_errors?: "rejected" | "continue" }; + outputs?: { + vault?: { when?: string; keys: Record }; + downloads?: Download[]; + }; + handoffs?: Record; + metadata?: Record; +} + +/** One form control as the runner reads it off a page. Never carries a value. */ +export interface Field { + /** A CSS selector that finds exactly this control. */ + selector: string; + id: string; + name: string; + type: string; + label: string; + required: boolean; + options?: Array<{ value: string; text: string }>; + /** For a text box: the chosen option of the nearest select before it, or the question printed beside it. */ + question?: string; + /** For a radio: whether any radio of its group is already checked. */ + groupChecked?: boolean; +} + +/** What the runner reads from a page. `text` is the visible body text, cut short. */ +export interface Page { + url: string; + title: string; + text: string; + errors: string[]; + fields: Field[]; +} + +/** The outcome the runner adds of its own, with why it stopped. */ +export type StopReason = "unmatched" | "loop" | "pages" | "timeout" | "gate" | "off-site" | "no-forward" | "throttle" | "input" | "error"; diff --git a/packages/openerrand/src/util.ts b/packages/openerrand/src/util.ts new file mode 100644 index 0000000..2e12cb8 --- /dev/null +++ b/packages/openerrand/src/util.ts @@ -0,0 +1,104 @@ +/** Small pure helpers every part of the runner shares. */ + +export class ErrandError extends Error { + constructor(message: string) { + super(message); + this.name = "ErrandError"; + } +} + +const cache = new Map(); + +/** Patterns in an errand file are ECMAScript regular expressions, matched case-insensitively. */ +export function re(pattern: string): RegExp { + let compiled = cache.get(pattern); + if (!compiled) { + compiled = new RegExp(pattern, "i"); + cache.set(pattern, compiled); + } + return compiled; +} + +export function test(pattern: string | undefined, text: string): boolean { + return pattern === undefined ? true : re(pattern).test(text); +} + +export function escapeRegExp(text: string): string { + return text.replace(/[.*+?^${}()|[\]\\]/g, "\\$&"); +} + +/** + * ISO 8601 durations as the file uses them (`PT15M`, `PT90S`, `P21D`, `P1W`). + * Months and years are counted as 30 and 365 days: no errand waits on a + * calendar month to the day. + */ +export function durationMs(duration: string | undefined, fallback: number): number { + if (!duration) return fallback; + const m = /^P(?:(\d+(?:\.\d+)?)Y)?(?:(\d+(?:\.\d+)?)M)?(?:(\d+(?:\.\d+)?)W)?(?:(\d+(?:\.\d+)?)D)?(?:T(?:(\d+(?:\.\d+)?)H)?(?:(\d+(?:\.\d+)?)M)?(?:(\d+(?:\.\d+)?)S)?)?$/.exec(duration); + if (!m) return fallback; + const [, y, mo, w, d, h, mi, s] = m.map((part) => (part === undefined ? 0 : Number(part))); + const day = 86_400_000; + return (y! * 365 + mo! * 30 + w! * 7 + d!) * day + h! * 3_600_000 + mi! * 60_000 + s! * 1000; +} + +const TEMPLATE = /\{\{\s*([a-z][a-z0-9_.]*)\s*\}\}/g; + +/** Every `{{name}}` in a template. */ +export function templateNames(text: string): string[] { + return [...text.matchAll(TEMPLATE)].map((m) => m[1]!); +} + +/** Replace each `{{name}}` with `lookup(name)`; a name with no value throws, so nothing is sent half-filled. */ +export function render(text: string, lookup: (name: string) => string | undefined): string { + return text.replace(TEMPLATE, (_, name: string) => { + const value = lookup(name); + if (value === undefined) throw new ErrandError(`{{${name}}} has no value`); + return value; + }); +} + +/** The transforms the spec names: digits, whole, upper, lower, trim, first:N, last:N. */ +export function transform(value: string, name: string | undefined): string { + if (!name) return value; + if (name === "digits") return (value.match(/\d+/g) ?? []).join(""); + if (name === "whole") { + // Whole units, a leading minus for a loss, no separators: "(1,234.56)" and "-1,234.56" are both -1234. + const negative = /^\s*\(.*\)\s*$/.test(value) || /^\s*-/.test(value); + const digits = value.replace(/[^0-9.]/g, ""); + const whole = digits === "" ? "" : String(Math.trunc(Number(digits))); + return whole === "" ? "" : negative && whole !== "0" ? `-${whole}` : whole; + } + if (name === "upper") return value.toUpperCase(); + if (name === "lower") return value.toLowerCase(); + if (name === "trim") return value.trim(); + const first = /^first:(\d+)$/.exec(name); + if (first) return value.slice(0, Number(first[1])); + const last = /^last:(\d+)$/.exec(name); + if (last) return value.slice(-Number(last[1])); + throw new ErrandError(`unknown transform ${name}`); +} + +/** Secrets are shown as four dots, whatever their length, so the length leaks nothing either. */ +export const MASK = "••••"; + +/** The https origin of a URL, or "" when it has none. */ +export function originOf(url: string): string { + try { + const u = new URL(url); + return u.origin === "null" ? "" : u.origin; + } catch { + return ""; + } +} + +export function sleep(ms: number): Promise { + return new Promise((resolve) => setTimeout(resolve, ms)); +} + +/** A short opaque id for a card or a run: lowercase letters and digits. */ +export function shortId(random: (max: number) => number, length = 6): string { + const alphabet = "abcdefghijkmnpqrstuvwxyz23456789"; + let out = ""; + while (out.length < length) out += alphabet[random(alphabet.length)]; + return out; +} diff --git a/packages/openerrand/src/vault.ts b/packages/openerrand/src/vault.ts new file mode 100644 index 0000000..1d877c5 --- /dev/null +++ b/packages/openerrand/src/vault.ts @@ -0,0 +1,191 @@ +/** + * Vaults the runner reads `vault` sources from and writes `outputs.vault` to. + * + * A target is named by a string, from `--vault` or the file's + * `metadata.vault` (the schema has no slot for it, and `metadata` is where a + * publisher puts what a runner needs): + * + * | Target | Reads | Writes | + * | --- | --- | --- | + * | `teams://` | `logicsrc teams pull` | pull, merge, `logicsrc teams push` | + * | `opencreds` | `logicsrc vault get --field --reveal` | never (read-only) | + * | `file:` | a 0600 dotenv file | the same file, 0600 | + * + * With no target the runner uses `file:` on its own credentials file and says + * so, as the spec asks: a login that exists only in a terminal's scrollback is + * a login lost. + * + * The teams push replaces the vault's contents with the file it is given, so + * a write always pulls first and merges, and the plaintext lives for one call + * in a 0700 temporary directory removed in a `finally`. + */ + +import { spawnSync } from "node:child_process"; +import { existsSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs"; +import { homedir, tmpdir } from "node:os"; +import { join } from "node:path"; +import type { VaultReader } from "./inputs.js"; +import { writePrivate } from "./store.js"; +import { ErrandError } from "./util.js"; + +export interface Vault extends VaultReader { + /** Merge these keys in. Absent for a read-only vault. */ + write?: (values: Record) => Promise; +} + +// --------------------------------------------------------------------------- +// dotenv, the same dialect as logicsrc's env provider, so a file this writes +// pushes and pulls back unchanged. +// --------------------------------------------------------------------------- + +const QUOTED = /^(['"])(.*)\1$/s; + +export function parseEnv(body: string): Record { + const out: Record = {}; + for (const raw of body.split(/\r?\n/)) { + const line = raw.trim(); + if (!line || line.startsWith("#")) continue; + const bare = line.startsWith("export ") ? line.slice(7) : line; + const eq = bare.indexOf("="); + if (eq <= 0) continue; + const key = bare.slice(0, eq).trim(); + let value = bare.slice(eq + 1).trim(); + const quoted = QUOTED.exec(value); + if (quoted) { + value = quoted[2]!; + if (quoted[1] === '"') value = value.replace(/\\n/g, "\n").replace(/\\"/g, '"').replace(/\\\\/g, "\\"); + } + out[key] = value; + } + return out; +} + +export function serializeEnv(values: Record): string { + const quote = (v: string): string => (/[\s#'"=\\]|^$/.test(v) || v.includes("\n") ? `"${v.replace(/\\/g, "\\\\").replace(/"/g, '\\"').replace(/\n/g, "\\n")}"` : v); + return `${Object.keys(values) + .sort() + .map((k) => `${k}=${quote(values[k]!)}`) + .join("\n")}\n`; +} + +// --------------------------------------------------------------------------- + +export type VaultTarget = + | { kind: "teams"; team: string; project: string; env: string } + | { kind: "opencreds"; field?: string } + | { kind: "file"; path: string }; + +export function parseTarget(text: string): VaultTarget { + const teams = /^teams:([^/\s]+)\/([^/\s]+)\/([^/\s]+)$/.exec(text); + if (teams) return { kind: "teams", team: teams[1]!, project: teams[2]!, env: teams[3]! }; + const creds = /^opencreds(?::(.+))?$/.exec(text); + if (creds) return creds[1] ? { kind: "opencreds", field: creds[1] } : { kind: "opencreds" }; + const file = /^file:(.+)$/.exec(text); + if (file) return { kind: "file", path: file[1]!.replace(/^~(?=\/|$)/, homedir()) }; + throw new ErrandError(`vault target ${text}: expected teams://, opencreds[:] or file:`); +} + +export function describeTarget(target: VaultTarget): string { + if (target.kind === "teams") return `teams ${target.team}/${target.project}/${target.env}`; + if (target.kind === "opencreds") return "OpenCreds vault (read-only)"; + return target.path; +} + +/** Runs the logicsrc CLI. The umbrella passes one that re-enters itself; standalone it is `logicsrc` on PATH. */ +export type LogicsrcExec = (args: string[], options?: { inheritStdin?: boolean }) => { status: number; stdout: string; stderr: string }; + +export const pathLogicsrc: LogicsrcExec = (args, options) => { + const result = spawnSync("logicsrc", args, { encoding: "utf8", stdio: [options?.inheritStdin ? "inherit" : "ignore", "pipe", "pipe"] }); + if (result.error) { + const missing = (result.error as NodeJS.ErrnoException).code === "ENOENT"; + return { status: 127, stdout: "", stderr: missing ? "logicsrc is not installed (https://logicsrc.com)" : result.error.message }; + } + return { status: result.status ?? 1, stdout: result.stdout ?? "", stderr: result.stderr ?? "" }; +}; + +const lastLines = (text: string): string => text.trim().split("\n").slice(-2).join(" "); + +export function fileVault(path: string): Vault { + return { + describe: () => path, + async get(key) { + return existsSync(path) ? parseEnv(readFileSync(path, "utf8"))[key] : undefined; + }, + async write(values) { + const current = existsSync(path) ? parseEnv(readFileSync(path, "utf8")) : {}; + writePrivate(path, serializeEnv({ ...current, ...values })); + }, + }; +} + +export function teamsVault(target: Extract, exec: LogicsrcExec = pathLogicsrc): Vault { + let cache: Record | null = null; + + function pull(): Record { + const dir = mkdtempSync(join(tmpdir(), "logicsrc-errand-")); + try { + const file = join(dir, "vault.env"); + const result = exec(["teams", "pull", target.team, target.project, target.env, "--env", file, "--format", "json"]); + if (result.status !== 0) throw new ErrandError(`logicsrc teams pull failed: ${lastLines(result.stderr || result.stdout)}`); + return existsSync(file) ? parseEnv(readFileSync(file, "utf8")) : {}; + } finally { + rmSync(dir, { recursive: true, force: true }); + } + } + + return { + describe: () => describeTarget(target), + async get(key) { + if (!cache) { + try { + cache = pull(); + } catch { + // A vault that does not exist yet has nothing to reuse; the next source (usually generate) runs. + cache = {}; + } + } + return cache[key]; + }, + async write(values) { + let current: Record = {}; + try { + current = pull(); + } catch { + // A vault that does not exist yet is created by the push. + } + const dir = mkdtempSync(join(tmpdir(), "logicsrc-errand-")); + try { + const file = join(dir, "vault.env"); + writeFileSync(file, serializeEnv({ ...current, ...values }), { mode: 0o600 }); + const result = exec(["teams", "push", target.team, target.project, target.env, "--env", file, "--format", "json"]); + if (result.status !== 0) throw new ErrandError(`logicsrc teams push failed: ${lastLines(result.stderr || result.stdout)}`); + cache = { ...current, ...values }; + } finally { + rmSync(dir, { recursive: true, force: true }); + } + }, + }; +} + +/** + * The personal OpenCreds vault, read-only: an item named by the key, the + * field `login.password` unless the target names another. Unlocking may ask + * for the master password, so stdin is the terminal's. + */ +export function opencredsVault(target: Extract, exec: LogicsrcExec = pathLogicsrc): Vault { + return { + describe: () => describeTarget(target), + async get(key) { + const result = exec(["vault", "get", key, "--field", target.field ?? "login.password", "--reveal"], { inheritStdin: true }); + if (result.status !== 0) return undefined; + const value = result.stdout.replace(/\n$/, ""); + return value === "" ? undefined : value; + }, + }; +} + +export function openVault(target: VaultTarget, exec?: LogicsrcExec): Vault { + if (target.kind === "teams") return teamsVault(target, exec); + if (target.kind === "opencreds") return opencredsVault(target, exec); + return fileVault(target.path); +} diff --git a/packages/openerrand/tsconfig.json b/packages/openerrand/tsconfig.json new file mode 100644 index 0000000..b863bf5 --- /dev/null +++ b/packages/openerrand/tsconfig.json @@ -0,0 +1,9 @@ +{ + "extends": "../../tsconfig.base.json", + "compilerOptions": { + "rootDir": "src", + "outDir": "dist" + }, + "include": ["src/**/*.ts"], + "exclude": ["src/**/*.test.ts", "src/testing.ts", "src/fake-site.ts"] +} diff --git a/packages/schemas/package.json b/packages/schemas/package.json index 93f6215..7a09f07 100644 --- a/packages/schemas/package.json +++ b/packages/schemas/package.json @@ -1,6 +1,6 @@ { "name": "@logicsrc/schemas", - "version": "0.3.0", + "version": "0.4.0", "description": "LogicSRC JSON schemas for tasks, agents, runs, events, plugins, the AgentAd ad standard, the OpenOntology knowledge contracts, the OpenContext context plane, the OpenCreds credential vault, OpenRental membership and rental offers, and the OpenWall messaging proposal. Includes the OpenABTest draft experiment and event contracts.", "license": "MIT", "type": "module", diff --git a/packages/validators/package.json b/packages/validators/package.json index 0d46a6b..f6da568 100644 --- a/packages/validators/package.json +++ b/packages/validators/package.json @@ -1,6 +1,6 @@ { "name": "@logicsrc/validators", - "version": "0.3.0", + "version": "0.4.0", "description": "LogicSRC schema validation helpers.", "type": "module", "main": "./dist/index.js", @@ -14,7 +14,7 @@ "validate:fixtures": "node dist/cli.js task ../schemas/fixtures/task.yaml && node dist/cli.js agent ../schemas/fixtures/agent.yaml && node dist/cli.js agentad-ad ../schemas/fixtures/agentad-ad.yaml && node dist/cli.js agentad-placement ../schemas/fixtures/agentad-placement.yaml && node dist/cli.js repo ../schemas/fixtures/repo.yaml && node dist/cli.js pull-request ../schemas/fixtures/pull-request.yaml && node dist/cli.js openontology-manifest ../schemas/fixtures/openontology/valid/manifest.json && node dist/cli.js openontology-claim ../schemas/fixtures/openontology/valid/claim-relationship.json && node dist/cli.js openontology-changeset ../schemas/fixtures/openontology/valid/changeset.json && node dist/cli.js opencontext-manifest ../schemas/fixtures/opencontext/valid/manifest.json && node dist/cli.js opencontext-object ../schemas/fixtures/opencontext/valid/object-policy.json && node dist/cli.js opencontext-bundle ../schemas/fixtures/opencontext/valid/bundle.json && node dist/cli.js opencontext-decision ../schemas/fixtures/opencontext/valid/decision.json && node dist/cli.js opencontext-role ../schemas/fixtures/opencontext/valid/role.json && node dist/cli.js opencontext-provenance ../schemas/fixtures/opencontext/valid/provenance.json && node dist/cli.js opencontext-diagnostic ../schemas/fixtures/opencontext/valid/diagnostic.json && node dist/cli.js opencontext-audit-event ../schemas/fixtures/opencontext/valid/audit-event.json && node dist/cli.js openrental ../schemas/fixtures/openrental/mixed.json && node dist/cli.js openerrand ../schemas/fixtures/openerrand/ftb-register-business.json && node dist/cli.js openerrand-index ../schemas/fixtures/openerrand/index.json && node dist/cli.js openwall-message ../schemas/fixtures/openwall/broadcast.json && node dist/cli.js openwall-message ../schemas/fixtures/openwall/direct.json && node dist/cli.js openwall-message ../schemas/fixtures/openwall/announcement.json && node dist/cli.js openwall-receipt ../schemas/fixtures/openwall/receipt-accepted.json && node dist/cli.js openwall-receipt ../schemas/fixtures/openwall/receipt-retrying.json && node dist/cli.js openabtest-manifest ../schemas/fixtures/openabtest/chovy-manifest.json && node dist/cli.js openabtest-event ../schemas/fixtures/openabtest/reconciliation.json" }, "dependencies": { - "@logicsrc/schemas": "^0.3.0", + "@logicsrc/schemas": "^0.4.0", "ajv": "^8.17.1", "ajv-formats": "^3.0.1", "yaml": "^2.8.1" diff --git a/packages/validators/src/index.ts b/packages/validators/src/index.ts index eb895d7..af72f24 100644 --- a/packages/validators/src/index.ts +++ b/packages/validators/src/index.ts @@ -95,3 +95,8 @@ export function assertSchemaKind(value: string): SchemaKind { } export { schemas, type SchemaKind }; + +// OpenErrand's fixed vocabularies, for runners that must apply the same rules +// the validator does (a captcha solver only outside these sectors, nothing but +// these names on a hand-off card). +export { GATE_KINDS, NO_SOLVER_SECTORS, HANDOFF_BUILTINS } from "./openerrand.js"; diff --git a/prd/0009-openerrand-reference-runner.md b/prd/0009-openerrand-reference-runner.md new file mode 100644 index 0000000..69d625c --- /dev/null +++ b/prd/0009-openerrand-reference-runner.md @@ -0,0 +1,114 @@ +--- +openprd: "0.3" +id: "0009" +title: "Ship the OpenErrand reference runner" +status: Draft +authors: + - anthony@profullstack.com +created: 2026-10-04 +updated: 2026-10-04 +repo: profullstack/logicsrc +discussion: +implementation: packages/openerrand +tags: + - openerrand + - errand + - browser-automation + - human-gates + - cli +supersedes: +superseded-by: +--- + +## Problem + +OpenErrand 0.1 (docs/openerrand.md) describes an errand on a website with no +API as one JSON file, and its worked example was transcribed from `ftb` in +cli-tools, which has the same rule table compiled in. Until a runner reads the +file, the file is a description and not a program: nobody can run the example, +write a second errand, or check that a runner keeps the thirteen rules. + +## Goals + +- `logicsrc errand run ` runs any valid OpenErrand 0.1 file in headless + Chrome and stops exactly where the spec says a person is needed. +- The FTB example runs unchanged. +- The throttle that kept `ftb` from snowballing a lockout is part of every + errand, not of one tool. + +## Non-Goals + +- No document extractor beyond a `command` hook; reading a tax form is the + principal's tool's job. +- No captcha solver, ever. Only the interface, gated by the spec. +- No run against a real government site in tests. + +## Users + +- A person with a form to fill on a site with no API, who wants to read what + will be sent before it is sent. +- An agent running an errand for its human, which must hand back at every gate. +- A publisher writing an errand file who needs the page log to fix a rule. + +## Requirements + +- R1 [P0] `@logicsrc/openerrand` 0.1.0: load and validate with + `@logicsrc/validators`, resolve inputs (document via an extractor hook, + vault, prompt masked for secrets, generate, derive, candidate, literal), + rules matched id first then label with step rules first and choices before + text, outcomes rejected first, and the stopped reasons the spec names. +- R2 [P0] Gates: `declare` only with `--declare` after the values are shown; + `identity-proofing` never touched (URL only), stop or hand the window over; + `code` from the terminal or a code file, used once, a wrong code waits for + the next; `mail` ends the run waiting with the card; `captcha` is the + person's, and a solver interface is called only where the spec permits; + `wait` polled, never solved. +- R3 [P0] One shared secret per run, never retried; a rejection lists the + other candidates and `--candidate` chooses the next. +- R4 [P0] Throttle ledger: 2 runs per errand and account in 30 minutes, 4 a + day, 2 minutes between runs on a site, lockout from the file's + `metadata.lockout` or a default, held per site and account, never lifted by + `--force`. +- R5 [P0] Credentials written before success: a logicsrc teams vault by + pull, merge, push; OpenCreds read-only; else a 0600 file, said aloud. +- R6 [P0] Page log of fields only and result-page text; run record with no + values; cards kept in the run record and `errand status`, never posted. +- R7 [P1] Rule 1 (show before run, SHA-256 change shown), rule 11 (the user + agent drops `HeadlessChrome`, nothing more), rule 12 (dry run), rule 13 + (loop and page limits). +- R8 [P1] `logicsrc errand run|validate|status` in CLI 0.7.0; docs and the + landing page say the runner ships. +- R9 [P1] Unit tests for the rule engine, gates, throttle, outcomes, inputs + and captcha gating; one integration test of the FTB example in real + headless Chrome against a local fake site with fictional data. + +## UX Notes + +A run prints the errand summary, the values it will use (secrets as `••••`), +each page's filled fields, and one final line on stdout (or the run record +with `--json`). A stop names the field, label, type and page, and where the +page log is. + +## Tech Stack + +TypeScript, NodeNext, commander 14, vitest 4, Chrome over CDP with no +dependency (ported from cli-tools' wcag.ts). Node 22+ for the global +WebSocket. + +## Monetization + +None. It is the reference implementation of an open standard. + +## Success Metrics + +- The FTB example runs end to end against the fake site, and `ftb` can later + be reduced to the errand files plus its PDF extractor. +- A second errand can be written and run without changing the runner. + +## Risks & Open Questions + +- The runner has not been run against the real MyFTB; the first real run + should be a `--dry-run`. +- Candidate order follows the spec's text (sources in order, newest year + within each); the worked example's `ftb secrets` listing orders by year + first. One of the two should change. diff --git a/prd/README.md b/prd/README.md index 2cf5adf..6164a7f 100644 --- a/prd/README.md +++ b/prd/README.md @@ -19,3 +19,4 @@ Status lives in each file's front-matter and is the source of truth: | [0006](./0006-add-pay2seed-spec.md) | Add pay2seed, paid2seed, pay2stream and paid2stream to the OpenSwarm family | Draft | openswarm, pay2seed, paid2seed, pay2stream, paid2stream, iplive, hls, ipfile, ippay, ipdb, bittorrent, torlink, bittorrented, c0mpute | | [0007](./0007-add-tech-stack-and-monetization-to-openprd.md) | Add Tech Stack and Monetization sections to OpenPRD | Draft | openprd, standards, monetization | | [0008](./0008-openfleet-reference-implementation.md) | Ship the OpenFleet reference implementation | Draft | openfleet, agents, fleet, swarm, claude-code, moshcode, cli | +| [0009](./0009-openerrand-reference-runner.md) | Ship the OpenErrand reference runner | Draft | openerrand, errand, browser-automation, human-gates, cli |