mirror of
https://github.com/profullstack/logicsrc.git
synced 2026-10-06 14:38:15 +00:00
OpenErrand reference runner: @logicsrc/openerrand and logicsrc errand (#228)
* OpenErrand 0.1: an errand on a website with no API, with the human steps kept human docs/openerrand.md mints OpenErrand: one JSON file per errand (register an account, download a transcript) naming the site, the inputs with a sensitivity class and ordered sources (document, vault, prompt, generate, derive, candidate, literal), field rules matched by id then label, page and wait steps, five human gates a runner never performs (declare, identity-proofing, code, mail, captcha), outcomes, the never-retried shared secret, vault and download outputs, hand-off cards that may name only public inputs, the publisher index at /.well-known/openerrand.json, and thirteen runner rules. The worked example is the MyFTB business registration that cli-tools `ftb` performs (profullstack/cli-tools#125), with no personal data. - @logicsrc/schemas: openerrand + openerrand-index schemas and fixtures - @logicsrc/validators: semantic checks (references, templates, no personal or secret input on a card) and tests that validate the spec's own examples - logicsrc-web: registry entry (process family), /openerrand landing page, the example and the index served as static files, contract tests Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * OpenErrand: hand-off cards stay on the surface that owns the data Anthony's ruling: tax and finance data never touches a social or promotion tool, and nothing is sent to a CPA or preparer. - Hand-off cards are delivered only on the surface that owns the errand's data (for a tax or finance errand, the principal's finance app through its CLI, PWA, MCP server or API, such as CoinPay, or the runner's terminal), never a social, promotion or third-party posting service, and never to anyone but the principal. A card for an errand with personal or secret inputs does not leave that surface. Runner rule 9 says the same. - The run record and the sample run name the card by an opaque id (pin-letter/7f3k2q) instead of a mynaposter.com URL; the myna mention is gone. - `principal: represented` no longer cites a preparer with a power of attorney. - The FTB card's last step no longer suggests sending the PIN to someone else. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * OpenErrand: user-agent rule, captcha solver policy, reference runner note Anthony's answers on #227 ("go with your recommendations"): - Rule 11: a runner may run headless with a normal desktop browser user agent (dropping HeadlessChrome) and nothing more: no fingerprint spoofing beyond the UA string, no stealth plugins, no solving or evading a bot challenge. A challenge the browser completes itself is a wait step; any other is a captcha gate. - Captcha solvers: new site.sector and captcha step `solver` (forbidden by default | allowed). Never allowed on government, tax, financial, healthcare or identity-provider sites, nor on any errand with a declare or identity-proofing step or a secret input; elsewhere only when the file says so, with every use logged. The validator rejects `allowed` in the forbidden set or without a stated sector; six new tests. The FTB example states sector "tax". - Reference runner: @logicsrc/openerrand / `logicsrc errand run`, marked in progress; ftb stays the runner the example was taken from. - Name stays OpenErrand; family stays Agents and process. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * OpenErrand reference runner: @logicsrc/openerrand and logicsrc errand Ship the runner docs/openerrand.md promised. `logicsrc errand run <file>` reads an OpenErrand 0.1 file, validates it with @logicsrc/validators, and drives headless Chrome through it under the spec's thirteen rules; `errand validate` shows what a file will ask of you and `errand status` shows the last run of each errand, its card and any lockout. The engine is generalised from cli-tools `ftb` (PR #125) with no dependency on cli-tools: the CDP client and Chrome finder from wcag.ts, the page reader, native-setter fill and forward-button picker from ftb-run.ts, and the rule matcher, throttle and outcome logic from ftb.ts, all now driven by the file. - Inputs: document (an extractor hook; the one shipped runs a local command that reads JSON requests and prints records), vault (teams or OpenCreds), prompt (no echo for secrets), generate, derive, candidate, literal. `--input name=value` wins. Shared-secret candidates are ranked as the spec says, one is submitted, and a rejection lists the others for --candidate. - Rules: id before label, step rules first, choices before text, an id match final, an unmatched required field stops the run naming it. - Gates: declare only with --declare after the values are shown; identity proofing never touched (URL only) and handed over or stopped on; code from the terminal or a code file, used once, a wrong code waits for the next; mail ends the run waiting with the card; captcha is the person's, and a CaptchaSolver interface is called only where the spec permits (no solver is bundled); wait steps are polled, never solved. - Throttle: 2 runs per errand and account in 30 minutes, 4 a day, 2 minutes between runs on a site, lockouts from metadata.lockout or a default, held per site and account, never lifted by --force. - Outputs: credentials written before success to a teams vault by pull, merge, push (metadata.vault or --vault), else a 0600 file said aloud; downloads type-checked and never overwritten with different bytes; cards only in the local run record. - The user agent is Chrome's own with HeadlessChrome replaced, given at launch: a CDP override did not reach a navigation the page's own script started, which is exactly the proof-of-work interstitial case. Tests: 85 in the package (rule engine, inputs, gates, throttle, outcomes, captcha gating, vaults, outputs, commands) including an integration test that runs the published FTB example unchanged in real headless Chrome against a local HTTPS fake site (Chrome maps webapp.ftb.ca.gov to it and every other host to NOTFOUND; all data fictional), and 2 in the CLI. CLI 0.6.0 -> 0.7.0; @logicsrc/schemas and @logicsrc/validators 0.3.0 -> 0.4.0 (the OpenErrand schemas, and the vocabularies now exported for runners); PRD 0009; the spec's Reference runner section and the landing page say it ships. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
GitHub
parent
1d69dc3804
commit
dde596b276
46 changed files with 5263 additions and 14 deletions
117
packages/openerrand/README.md
Normal file
117
packages/openerrand/README.md
Normal file
|
|
@ -0,0 +1,117 @@
|
|||
# @logicsrc/openerrand
|
||||
|
||||
The reference runner for [OpenErrand](https://logicsrc.com/docs/openerrand):
|
||||
one JSON file that describes an errand on a website with no API, run in
|
||||
headless Chrome, stopping at every step that belongs to a person.
|
||||
|
||||
It reads an errand file, validates it with `@logicsrc/validators`, resolves
|
||||
every input before the browser opens, then for each page finds its step,
|
||||
tests the outcomes, fills the form from the rules (id first, label second,
|
||||
choices before text), hands every gate to a person, and presses the forward
|
||||
button, until the site answers or the runner has to stop.
|
||||
|
||||
## Install
|
||||
|
||||
```bash
|
||||
curl -fsSL https://logicsrc.com/install.sh | sh
|
||||
logicsrc errand --help
|
||||
```
|
||||
|
||||
The commands live in this package and the umbrella CLI mounts them as
|
||||
`logicsrc errand`. To embed the runner:
|
||||
|
||||
```bash
|
||||
npm install @logicsrc/openerrand
|
||||
```
|
||||
|
||||
## Commands
|
||||
|
||||
```bash
|
||||
logicsrc errand validate ftb-register-business.json
|
||||
logicsrc errand run ftb-register-business.json --extractor "python3 extract.py ~/taxes" --dry-run
|
||||
logicsrc errand run ftb-register-business.json --extractor "python3 extract.py ~/taxes" --declare
|
||||
logicsrc errand status
|
||||
```
|
||||
|
||||
| Flag | What |
|
||||
| --- | --- |
|
||||
| `--input name=value` | Give an input's value (repeatable). Wins over every source. |
|
||||
| `--dry-run` | Fill every page up to the first that holds a declaration or a shared secret, show it with secrets masked and the declaration word for word, and stop before its forward button. Not counted by the throttle. |
|
||||
| `--declare` | Your consent, for this run, to tick the declarations the errand names. Never read from a file, the environment or a saved default. |
|
||||
| `--headful` | A visible window, so you can take identity proofing or a captcha yourself. |
|
||||
| `--chrome PATH` | The Chrome or Chromium binary (default: `CHROME_PATH`, then PATH, then the Puppeteer and Playwright caches). |
|
||||
| `--vault TARGET` | `teams:<team>/<project>/<env>`, `opencreds[:<field>]` or `file:<path>`. Default: the file's `metadata.vault`, else a 0600 file. |
|
||||
| `--extractor CMD` | The local command that reads your documents (see below). |
|
||||
| `--candidate N` | Which shared-secret candidate to submit; a rejection lists the others. |
|
||||
| `--account NAME` | Which account at the site, for the throttle. |
|
||||
| `--force` | Lift the per-window, per-day and spacing caps. Never a lockout. |
|
||||
| `--yes` | Run a file whose SHA-256 changed since its last run without asking. |
|
||||
| `--json` | Print the run record on stdout. |
|
||||
|
||||
Exit codes: 0 success, waiting or dry run; 1 rejected; 2 invalid file or
|
||||
usage; 3 stopped (a gate, an unmatched field, a loop, off-site, a timeout);
|
||||
4 refused by the throttle.
|
||||
|
||||
## Inputs
|
||||
|
||||
| Source | Served by |
|
||||
| --- | --- |
|
||||
| `document` | `--extractor`: a command on this machine. It gets `{ "errand", "documents": [{ "input", "form", "field", "match" }] }` on stdin and prints `[{ "form", "field", "value", "year", "label", "file", "page" }]`. The runner applies `match`, `years`, `pick` and `transform`, and keeps the values in memory only. No extractor is bundled. |
|
||||
| `vault` | The vault target, read-only here. |
|
||||
| `prompt` | The terminal; a secret does not echo. With no terminal the next source is tried. |
|
||||
| `generate` | `crypto.randomInt`, at least one character of each class, a letter first when letters are allowed. |
|
||||
| `derive`, `candidate`, `literal` | The file and the other inputs. |
|
||||
|
||||
A shared-secret input's document sources become a ranked list of candidates
|
||||
(sources in order, newest year first within each). A run submits one; a
|
||||
rejection ends the run and lists the others for you to choose with
|
||||
`--candidate`. Nothing is ever retried on its own.
|
||||
|
||||
## Gates
|
||||
|
||||
| Gate | What the runner does |
|
||||
| --- | --- |
|
||||
| `declare` | Stops on the page and prints the statement word for word, unless you passed `--declare`; then it shows the values on the page (secrets masked) and ticks the box. |
|
||||
| `identity-proofing` | Never touches the provider's pages (it reads only the URL). Headless, it stops and tells you; with `--headful` at a terminal it waits until the page is back on the site. |
|
||||
| `code` | The terminal prompt, or the code file `~/.local/share/logicsrc/errand/codes/<name>.code` written by whoever holds the phone. Used once, never logged. A wrong code waits for the next one. |
|
||||
| `mail` | Ends the run waiting, with the deadline and the hand-off card in the run record. |
|
||||
| `captcha` | Yours: a visible window with `--headful`, or the run stops. A program may pass a `CaptchaSolver`, and it is called only when the step says `solver: allowed` and the errand is outside the forbidden set (a stated sector that is not government, tax, financial, healthcare or identity-provider; no declaration or identity proofing; no secret input). Each use is logged with the time, URL and service. No solver ships with this package. |
|
||||
| `wait` | Polled until its match no longer fits, then the run goes on. Never solved or bypassed. |
|
||||
|
||||
## What it keeps
|
||||
|
||||
Under `$LOGICSRC_ERRAND_HOME`, else `~/.local/share/logicsrc/errand` (files
|
||||
0600, directories 0700):
|
||||
|
||||
| Path | What |
|
||||
| --- | --- |
|
||||
| `throttle.json` | Attempts and lockouts. 2 runs of an errand per account in 30 minutes, 4 a day; 2 minutes between runs on one site; nothing during a lockout. A lockout is the file's `metadata.lockout.text` pattern (lasting `metadata.lockout.duration`) or a default pattern (35 minutes), and it holds every errand on that site and account. |
|
||||
| `approvals.json` | The SHA-256 of each errand file last run. A changed file is shown and not run without `--yes` or a yes at the terminal. |
|
||||
| `runs/` | One run record per run: outcome, page, candidate (no value), card, deadline. Never an input value. |
|
||||
| `pages/<name>.jsonl` | The page log: each page's fields (selector, type, label, required, options) and a result page's text. Never a value. |
|
||||
| `profiles/<host>/` | One Chrome profile per site, kept so a passed bot check stays passed. |
|
||||
| `codes/<name>.code` | Where a code may be written. |
|
||||
| `credentials/<name>.env` | Credentials when no writable vault is named. |
|
||||
|
||||
Hand-off cards stay in the run record and `errand status`. They are never
|
||||
posted anywhere, and a card that names a personal or secret input is refused.
|
||||
|
||||
## Browser
|
||||
|
||||
Chrome over the DevTools protocol, no dependency. The user agent is Chrome's
|
||||
own with `HeadlessChrome` replaced by `Chrome`, given at launch so it reaches
|
||||
every request; nothing else about the browser is changed.
|
||||
|
||||
## Library
|
||||
|
||||
```ts
|
||||
import { loadErrand, resolveInputs, runErrand, openCdpDriver, Store, fileVault } from "@logicsrc/openerrand";
|
||||
```
|
||||
|
||||
`runErrand` takes a `Driver` factory, so an embedder can bring its own
|
||||
browser; the tests drive it with a scripted fake and one integration test runs
|
||||
the FTB example in real headless Chrome against a local fake site.
|
||||
|
||||
## License
|
||||
|
||||
MIT
|
||||
49
packages/openerrand/package.json
Normal file
49
packages/openerrand/package.json
Normal file
|
|
@ -0,0 +1,49 @@
|
|||
{
|
||||
"name": "@logicsrc/openerrand",
|
||||
"version": "0.1.0",
|
||||
"description": "Reference runner for the OpenErrand standard: reads an errand file and drives headless Chrome through a website that has no API, stopping at every step that belongs to a person.",
|
||||
"license": "MIT",
|
||||
"type": "module",
|
||||
"main": "./dist/index.js",
|
||||
"types": "./dist/index.d.ts",
|
||||
"exports": {
|
||||
".": "./dist/index.js",
|
||||
"./commands": "./dist/commands.js"
|
||||
},
|
||||
"repository": {
|
||||
"type": "git",
|
||||
"url": "git+https://github.com/profullstack/logicsrc.git",
|
||||
"directory": "packages/openerrand"
|
||||
},
|
||||
"homepage": "https://logicsrc.com/docs/openerrand",
|
||||
"keywords": [
|
||||
"logicsrc",
|
||||
"openerrand",
|
||||
"errand",
|
||||
"browser-automation",
|
||||
"chrome",
|
||||
"cdp",
|
||||
"standards",
|
||||
"cli"
|
||||
],
|
||||
"publishConfig": {
|
||||
"access": "public"
|
||||
},
|
||||
"files": [
|
||||
"dist"
|
||||
],
|
||||
"engines": {
|
||||
"node": ">=22"
|
||||
},
|
||||
"scripts": {
|
||||
"build": "tsc -p tsconfig.json",
|
||||
"test": "vitest run src"
|
||||
},
|
||||
"dependencies": {
|
||||
"@logicsrc/validators": "^0.4.0",
|
||||
"commander": "^14.0.2"
|
||||
},
|
||||
"devDependencies": {
|
||||
"vitest": "^4.0.8"
|
||||
}
|
||||
}
|
||||
277
packages/openerrand/src/browser.ts
Normal file
277
packages/openerrand/src/browser.ts
Normal file
|
|
@ -0,0 +1,277 @@
|
|||
/**
|
||||
* Chrome over the DevTools protocol, with no dependency: find a binary, start
|
||||
* it, and talk to it over one WebSocket in flat session mode.
|
||||
*
|
||||
* Ported from cli-tools' wcag.ts (findChrome, Cdp, launchBrowser), which `ftb`
|
||||
* drives MyFTB with.
|
||||
*/
|
||||
|
||||
import { type ChildProcess, spawn } from "node:child_process";
|
||||
import { accessSync, constants, mkdirSync, mkdtempSync, readdirSync, rmSync } from "node:fs";
|
||||
import { homedir, tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { ErrandError } from "./util.js";
|
||||
|
||||
const isExecutable = (path: string): boolean => {
|
||||
try {
|
||||
accessSync(path, constants.X_OK);
|
||||
return true;
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
};
|
||||
|
||||
const listDir = (path: string): string[] => {
|
||||
try {
|
||||
return readdirSync(path);
|
||||
} catch {
|
||||
return [];
|
||||
}
|
||||
};
|
||||
|
||||
/** `linux-152.0.7977.42` before `linux-131.0.6778.204`: newest first by version. */
|
||||
const byVersionDesc = (a: string, b: string): number => {
|
||||
const parse = (name: string): number[] => (/(\d+(?:\.\d+)*)/.exec(name)?.[1] ?? "0").split(".").map(Number);
|
||||
const left = parse(a);
|
||||
const right = parse(b);
|
||||
for (let i = 0; i < Math.max(left.length, right.length); i += 1) {
|
||||
const d = (right[i] ?? 0) - (left[i] ?? 0);
|
||||
if (d !== 0) return d;
|
||||
}
|
||||
return 0;
|
||||
};
|
||||
|
||||
/** Every place a Chrome might be: CHROME_PATH, PATH names, /opt, Puppeteer and Playwright caches, macOS bundles. */
|
||||
export function chromeCandidates(env: NodeJS.ProcessEnv = process.env, home: string = homedir()): string[] {
|
||||
const out: string[] = [];
|
||||
if (env.CHROME_PATH) out.push(env.CHROME_PATH);
|
||||
for (const name of ["google-chrome", "google-chrome-stable", "chromium", "chromium-browser", "chrome"]) {
|
||||
for (const dir of (env.PATH ?? "").split(":").filter(Boolean)) out.push(join(dir, name));
|
||||
}
|
||||
out.push("/opt/google/chrome/chrome");
|
||||
// Full Chrome before the headless shell: a person may need the window (--headful).
|
||||
const puppeteer = join(home, ".cache", "puppeteer");
|
||||
for (const flavour of ["chrome", "chrome-headless-shell"]) {
|
||||
for (const build of listDir(join(puppeteer, flavour)).sort(byVersionDesc)) {
|
||||
const dir = join(puppeteer, flavour, build);
|
||||
const inner = listDir(dir).find((entry) => entry.startsWith(flavour));
|
||||
if (inner) out.push(join(dir, inner, flavour));
|
||||
}
|
||||
}
|
||||
const playwright = join(home, ".cache", "ms-playwright");
|
||||
for (const build of listDir(playwright).sort(byVersionDesc)) {
|
||||
if (build.startsWith("chromium-")) out.push(join(playwright, build, "chrome-linux", "chrome"));
|
||||
}
|
||||
out.push("/Applications/Google Chrome.app/Contents/MacOS/Google Chrome");
|
||||
out.push("/Applications/Chromium.app/Contents/MacOS/Chromium");
|
||||
return out;
|
||||
}
|
||||
|
||||
export function findChrome(env: NodeJS.ProcessEnv = process.env, home: string = homedir(), executable: (path: string) => boolean = isExecutable): string | null {
|
||||
return chromeCandidates(env, home).find(executable) ?? null;
|
||||
}
|
||||
|
||||
export const NO_CHROME = `no Chrome found. Pass --chrome PATH or set CHROME_PATH, install one
|
||||
(apt install chromium, brew install --cask google-chrome), or let Puppeteer fetch one
|
||||
(npx puppeteer browsers install chrome).`;
|
||||
|
||||
interface CdpMessage {
|
||||
id?: number;
|
||||
method?: string;
|
||||
params?: Record<string, unknown>;
|
||||
sessionId?: string;
|
||||
result?: Record<string, unknown>;
|
||||
error?: { message: string };
|
||||
}
|
||||
|
||||
export interface CdpEvent {
|
||||
method: string;
|
||||
params: Record<string, unknown>;
|
||||
sessionId?: string;
|
||||
}
|
||||
|
||||
/** Requests with ids and events by name, over one socket for the browser and every page. */
|
||||
export class Cdp {
|
||||
private nextId = 1;
|
||||
private readonly pending = new Map<number, { resolve: (value: Record<string, unknown>) => void; reject: (error: Error) => void }>();
|
||||
private readonly listeners = new Set<(message: CdpMessage) => void>();
|
||||
private readonly socket: WebSocket;
|
||||
|
||||
constructor(socket: WebSocket) {
|
||||
this.socket = socket;
|
||||
socket.addEventListener("message", (event) => {
|
||||
const message = JSON.parse(String(event.data)) as CdpMessage;
|
||||
if (message.id !== undefined && this.pending.has(message.id)) {
|
||||
const { resolve, reject } = this.pending.get(message.id)!;
|
||||
this.pending.delete(message.id);
|
||||
if (message.error) reject(new ErrandError(message.error.message));
|
||||
else resolve(message.result ?? {});
|
||||
return;
|
||||
}
|
||||
for (const listener of this.listeners) listener(message);
|
||||
});
|
||||
socket.addEventListener("close", () => {
|
||||
for (const { reject } of this.pending.values()) reject(new ErrandError("browser closed"));
|
||||
this.pending.clear();
|
||||
});
|
||||
}
|
||||
|
||||
static async connect(url: string): Promise<Cdp> {
|
||||
const socket = new WebSocket(url);
|
||||
await new Promise<void>((resolve, reject) => {
|
||||
socket.addEventListener("open", () => resolve(), { once: true });
|
||||
socket.addEventListener("error", () => reject(new ErrandError(`could not connect to ${url}`)), { once: true });
|
||||
});
|
||||
return new Cdp(socket);
|
||||
}
|
||||
|
||||
send(method: string, params: Record<string, unknown> = {}, sessionId?: string): Promise<Record<string, unknown>> {
|
||||
const id = this.nextId;
|
||||
this.nextId += 1;
|
||||
return new Promise((resolve, reject) => {
|
||||
this.pending.set(id, { resolve, reject });
|
||||
this.socket.send(JSON.stringify(sessionId ? { id, method, params, sessionId } : { id, method, params }));
|
||||
});
|
||||
}
|
||||
|
||||
on(listener: (event: CdpEvent) => void): () => void {
|
||||
const wrapped = (message: CdpMessage): void => {
|
||||
if (message.method) listener({ method: message.method, params: message.params ?? {}, ...(message.sessionId ? { sessionId: message.sessionId } : {}) });
|
||||
};
|
||||
this.listeners.add(wrapped);
|
||||
return () => this.listeners.delete(wrapped);
|
||||
}
|
||||
|
||||
waitFor(method: string, sessionId: string, timeoutMs: number): Promise<void> {
|
||||
return new Promise((resolve, reject) => {
|
||||
const timer = setTimeout(() => {
|
||||
this.listeners.delete(listener);
|
||||
reject(new ErrandError(`timed out after ${Math.round(timeoutMs / 1000)}s waiting for ${method}`));
|
||||
}, timeoutMs);
|
||||
const listener = (message: CdpMessage): void => {
|
||||
if (message.method === method && message.sessionId === sessionId) {
|
||||
clearTimeout(timer);
|
||||
this.listeners.delete(listener);
|
||||
resolve();
|
||||
}
|
||||
};
|
||||
this.listeners.add(listener);
|
||||
});
|
||||
}
|
||||
|
||||
close(): void {
|
||||
this.socket.close();
|
||||
}
|
||||
}
|
||||
|
||||
export interface Browser {
|
||||
cdp: Cdp;
|
||||
path: string;
|
||||
/** Settles when Chrome ends, including a person closing its window. */
|
||||
exited: Promise<void>;
|
||||
close(): Promise<void>;
|
||||
}
|
||||
|
||||
export interface LaunchOptions {
|
||||
chrome?: string;
|
||||
env?: NodeJS.ProcessEnv;
|
||||
sandbox?: boolean;
|
||||
timeoutMs?: number;
|
||||
/** A profile directory to keep between runs; without one a throwaway profile is removed on close. */
|
||||
profile?: string;
|
||||
/** false opens a window a person can use; the default is headless. */
|
||||
headless?: boolean;
|
||||
/** Extra Chrome switches. The tests use this to point a hostname at a local fake site. */
|
||||
args?: string[];
|
||||
}
|
||||
|
||||
export async function launchBrowser(options: LaunchOptions = {}): Promise<Browser> {
|
||||
const env = options.env ?? process.env;
|
||||
const path = options.chrome ?? findChrome(env);
|
||||
if (!path) throw new ErrandError(NO_CHROME);
|
||||
if (!isExecutable(path)) throw new ErrandError(`${path} is not an executable`);
|
||||
|
||||
const keep = options.profile !== undefined;
|
||||
if (keep) mkdirSync(options.profile!, { recursive: true, mode: 0o700 });
|
||||
const profile = options.profile ?? mkdtempSync(join(tmpdir(), "logicsrc-errand-chrome-"));
|
||||
const sandbox = options.sandbox ?? !(env.CHROME_NO_SANDBOX || process.getuid?.() === 0);
|
||||
const args = [
|
||||
...(options.headless === false ? [] : ["--headless=new"]),
|
||||
"--remote-debugging-port=0",
|
||||
`--user-data-dir=${profile}`,
|
||||
"--no-first-run",
|
||||
"--no-default-browser-check",
|
||||
"--disable-gpu",
|
||||
"--disable-extensions",
|
||||
"--disable-background-networking",
|
||||
"--window-size=1280,900",
|
||||
...(sandbox ? [] : ["--no-sandbox"]),
|
||||
...(options.args ?? []),
|
||||
"about:blank",
|
||||
];
|
||||
|
||||
const child: ChildProcess = spawn(path, args, { env, stdio: ["ignore", "ignore", "pipe"] });
|
||||
const exited = new Promise<void>((resolve) => child.once("exit", () => resolve()));
|
||||
const cleanup = (): void => {
|
||||
if (keep) return;
|
||||
try {
|
||||
rmSync(profile, { recursive: true, force: true });
|
||||
} catch {
|
||||
// Still held open; the next run's tmpdir sweep takes it.
|
||||
}
|
||||
};
|
||||
|
||||
const url = await new Promise<string>((resolve, reject) => {
|
||||
let stderr = "";
|
||||
const timeoutMs = options.timeoutMs ?? 20_000;
|
||||
const timer = setTimeout(() => reject(new ErrandError(`${path} did not start within ${timeoutMs / 1000}s\n${stderr}`)), timeoutMs);
|
||||
child.stderr?.on("data", (chunk: Buffer) => {
|
||||
stderr += chunk.toString();
|
||||
const match = /DevTools listening on (ws:\/\/\S+)/.exec(stderr);
|
||||
if (match?.[1]) {
|
||||
clearTimeout(timer);
|
||||
resolve(match[1]);
|
||||
}
|
||||
});
|
||||
child.on("exit", (code) => {
|
||||
clearTimeout(timer);
|
||||
reject(new ErrandError(`${path} exited with ${code ?? "a signal"} before it was ready\n${stderr.trim()}`));
|
||||
});
|
||||
child.on("error", (error) => {
|
||||
clearTimeout(timer);
|
||||
reject(new ErrandError(`${path}: ${error.message}`));
|
||||
});
|
||||
}).catch((error: Error) => {
|
||||
child.kill();
|
||||
cleanup();
|
||||
throw error;
|
||||
});
|
||||
|
||||
const cdp = await Cdp.connect(url);
|
||||
return {
|
||||
cdp,
|
||||
path,
|
||||
exited,
|
||||
async close() {
|
||||
try {
|
||||
await Promise.race([cdp.send("Browser.close"), new Promise((resolve) => setTimeout(resolve, 2000))]);
|
||||
} catch {
|
||||
// Already gone.
|
||||
}
|
||||
// Cookies reach a kept profile on a clean exit: give Chrome one before the kill.
|
||||
if (keep) await Promise.race([exited, new Promise((resolve) => setTimeout(resolve, 10_000))]);
|
||||
cdp.close();
|
||||
child.kill();
|
||||
cleanup();
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Rule 11: a runner may present a normal desktop user agent by dropping
|
||||
* `HeadlessChrome` from the string. That is all this does: no other header,
|
||||
* no navigator patching, no fingerprint change.
|
||||
*/
|
||||
export function plainUserAgent(userAgent: string): string | null {
|
||||
return userAgent.includes("HeadlessChrome") ? userAgent.replace("HeadlessChrome", "Chrome") : null;
|
||||
}
|
||||
27
packages/openerrand/src/captcha.ts
Normal file
27
packages/openerrand/src/captcha.ts
Normal file
|
|
@ -0,0 +1,27 @@
|
|||
/**
|
||||
* The captcha solver hook. The runner ships no solver and never will: by
|
||||
* default a captcha is handed to the person in a visible window, or the run
|
||||
* stops. A program embedding the runner may pass a {@link CaptchaSolver}, and
|
||||
* the runner calls it only when {@link solverPermitted} says the errand allows
|
||||
* one (the step says `solver: allowed`, the site's stated sector is outside
|
||||
* government, tax, financial, healthcare and identity-provider, and the errand
|
||||
* has no declare or identity-proofing step and no secret input). Every use is
|
||||
* logged with the time, the page URL and the service, never the image or the
|
||||
* answer.
|
||||
*/
|
||||
|
||||
export { solverPermitted } from "./pages.js";
|
||||
|
||||
export interface CaptchaContext {
|
||||
/** The page the captcha is on. */
|
||||
url: string;
|
||||
/** Evaluate script in the page, for a solver that must place its token. */
|
||||
evaluate(expression: string): Promise<unknown>;
|
||||
}
|
||||
|
||||
export interface CaptchaSolver {
|
||||
/** The service's name, written to the log on every use. */
|
||||
readonly service: string;
|
||||
/** Resolve true when the captcha is cleared. */
|
||||
solve(context: CaptchaContext): Promise<boolean>;
|
||||
}
|
||||
94
packages/openerrand/src/commands.test.ts
Normal file
94
packages/openerrand/src/commands.test.ts
Normal file
|
|
@ -0,0 +1,94 @@
|
|||
import { readFileSync, writeFileSync } from "node:fs";
|
||||
import { join } from "node:path";
|
||||
import { Command } from "commander";
|
||||
import { afterEach, describe, expect, it } from "vitest";
|
||||
import { type Deps, registerErrandCommands } from "./commands.js";
|
||||
import { FakeDriver, ftbExamplePath, tempDir } from "./testing.js";
|
||||
|
||||
function setup(over: Partial<Deps> = {}) {
|
||||
const home = tempDir();
|
||||
const out: string[] = [];
|
||||
const say: string[] = [];
|
||||
const p = new Command();
|
||||
p.name("logicsrc").enablePositionalOptions().exitOverride();
|
||||
registerErrandCommands(p.command("errand"), {
|
||||
env: { LOGICSRC_ERRAND_HOME: home, HOME: home },
|
||||
out: (l) => out.push(l),
|
||||
say: (l) => say.push(l),
|
||||
interactive: false,
|
||||
now: () => new Date("2026-10-04T12:00:00Z"),
|
||||
...over,
|
||||
});
|
||||
return { p, home, out, say };
|
||||
}
|
||||
|
||||
afterEach(() => {
|
||||
process.exitCode = 0;
|
||||
});
|
||||
|
||||
describe("logicsrc errand validate", () => {
|
||||
it("accepts the spec's worked example and shows every gate and input", async () => {
|
||||
const { p, out } = setup();
|
||||
await p.parseAsync(["node", "logicsrc", "errand", "validate", ftbExamplePath()]);
|
||||
expect(process.exitCode).toBe(0);
|
||||
const text = out.join("\n");
|
||||
expect(text).toContain("declare (declaration): Ticking this box is the representative stating");
|
||||
expect(text).toContain("net_income [secret, shared-secret]");
|
||||
expect(text).toMatch(/valid OpenErrand 0\.1 {2}sha256 [0-9a-f]{64}/);
|
||||
});
|
||||
|
||||
it("rejects a file the validator rejects, with the reason", async () => {
|
||||
const { p, home, say } = setup();
|
||||
const bad = JSON.parse(readFileSync(ftbExamplePath(), "utf8"));
|
||||
bad.steps.push({ id: "cap", kind: "captcha", match: { selector: ".g-recaptcha" }, solver: "allowed" });
|
||||
writeFileSync(join(home, "bad.json"), JSON.stringify(bad));
|
||||
await p.parseAsync(["node", "logicsrc", "errand", "validate", join(home, "bad.json")]);
|
||||
expect(process.exitCode).toBe(2);
|
||||
expect(say.join("\n")).toMatch(/captcha solver is never allowed on a tax site/);
|
||||
});
|
||||
});
|
||||
|
||||
describe("logicsrc errand run", () => {
|
||||
it("refuses an --input the errand does not have, before any browser", async () => {
|
||||
let opened = false;
|
||||
const { p, say } = setup({ openDriver: async () => ((opened = true), new FakeDriver({}, () => "")) });
|
||||
await p.parseAsync(["node", "logicsrc", "errand", "run", ftbExamplePath(), "--input", "nope=1"]);
|
||||
expect(process.exitCode).toBe(2);
|
||||
expect(say.join("\n")).toMatch(/no input named nope/);
|
||||
expect(opened).toBe(false);
|
||||
});
|
||||
|
||||
it("stops before the browser when a required input has no value and nobody is at a terminal", async () => {
|
||||
let opened = false;
|
||||
const { p, say } = setup({ openDriver: async () => ((opened = true), new FakeDriver({}, () => "")) });
|
||||
await p.parseAsync(["node", "logicsrc", "errand", "run", ftbExamplePath()]);
|
||||
expect(process.exitCode).toBe(3);
|
||||
expect(say.join("\n")).toMatch(/no value for Email address/);
|
||||
expect(opened).toBe(false);
|
||||
});
|
||||
|
||||
it("shows a changed file and does not run it without --yes", async () => {
|
||||
const { p, home, say } = setup({ openDriver: async () => new FakeDriver({}, () => "") });
|
||||
const copy = join(home, "errand.json");
|
||||
writeFileSync(copy, readFileSync(ftbExamplePath(), "utf8"));
|
||||
const args = ["node", "logicsrc", "errand", "run", copy, "--dry-run", "--input", "email=jane@example.com", "--input", "phone=5555550100"];
|
||||
for (const name of ["first_name=Jane", "last_name=Doe", "street=1234 Maple St", "zip=95814", "corp_id=1234567", "net_income=48210", "tax_year=2025"]) args.push("--input", name);
|
||||
await p.parseAsync(args);
|
||||
const edited = JSON.parse(readFileSync(copy, "utf8"));
|
||||
edited.title = "Register a MyFTB business account (edited)";
|
||||
writeFileSync(copy, JSON.stringify(edited));
|
||||
say.length = 0;
|
||||
await p.parseAsync(args);
|
||||
expect(say.join("\n")).toMatch(/this file changed since its last run/);
|
||||
expect(say.join("\n")).toContain("Not run.");
|
||||
expect(process.exitCode).toBe(3);
|
||||
});
|
||||
});
|
||||
|
||||
describe("logicsrc errand status", () => {
|
||||
it("says when nothing has run", async () => {
|
||||
const { p, out } = setup();
|
||||
await p.parseAsync(["node", "logicsrc", "errand", "status"]);
|
||||
expect(out).toEqual(["No errands run yet."]);
|
||||
});
|
||||
});
|
||||
270
packages/openerrand/src/commands.ts
Normal file
270
packages/openerrand/src/commands.ts
Normal file
|
|
@ -0,0 +1,270 @@
|
|||
/**
|
||||
* `logicsrc errand run|validate|status`.
|
||||
*
|
||||
* Everything that touches the world (environment, terminal, the logicsrc CLI
|
||||
* for team vaults, the browser) is injectable, so the commands are tested
|
||||
* without a terminal and the integration test can point Chrome at a fake site.
|
||||
* Exit codes go on `process.exitCode`, never `process.exit`.
|
||||
*/
|
||||
|
||||
import { randomInt } from "node:crypto";
|
||||
import type { Command } from "commander";
|
||||
import type { CaptchaSolver } from "./captcha.js";
|
||||
import { type Driver, openCdpDriver } from "./driver.js";
|
||||
import { commandExtractor } from "./extract.js";
|
||||
import { type Prompt, resolveInputs } from "./inputs.js";
|
||||
import { loadErrand, summarize } from "./load.js";
|
||||
import { confirm, terminalPrompt } from "./prompt.js";
|
||||
import { runErrand } from "./run.js";
|
||||
import { type Env, Store } from "./store.js";
|
||||
import { durationMs, ErrandError } from "./util.js";
|
||||
import { describeTarget, fileVault, type LogicsrcExec, openVault, parseTarget, pathLogicsrc, type Vault } from "./vault.js";
|
||||
|
||||
/** 0 done (success, waiting or a dry run), 1 rejected, 2 invalid file or usage, 3 stopped, 4 refused by the throttle. */
|
||||
export const EXIT = { OK: 0, REJECTED: 1, INVALID: 2, STOPPED: 3, THROTTLED: 4 } as const;
|
||||
|
||||
export interface Deps {
|
||||
env: Env;
|
||||
/** stdout: machine output (--json) and the final line. */
|
||||
out: (line: string) => void;
|
||||
/** stderr: everything a person reads while it runs. */
|
||||
say: (line: string) => void;
|
||||
prompt: Prompt;
|
||||
confirm: (question: string) => Promise<boolean>;
|
||||
interactive: boolean;
|
||||
logicsrc: LogicsrcExec;
|
||||
now: () => Date;
|
||||
random: (max: number) => number;
|
||||
/** Extra Chrome switches (tests map the site's hostname to a local server). */
|
||||
chromeArgs?: string[];
|
||||
/** Replace the browser entirely (unit tests). */
|
||||
openDriver?: () => Promise<Driver>;
|
||||
solver?: CaptchaSolver;
|
||||
pollMs?: number;
|
||||
rereadMs?: number;
|
||||
settleMs?: number;
|
||||
}
|
||||
|
||||
export function defaultDeps(): Deps {
|
||||
return {
|
||||
env: process.env,
|
||||
out: (line) => process.stdout.write(`${line}\n`),
|
||||
say: (line) => process.stderr.write(`${line}\n`),
|
||||
prompt: terminalPrompt,
|
||||
confirm,
|
||||
interactive: process.stdin.isTTY === true,
|
||||
logicsrc: pathLogicsrc,
|
||||
now: () => new Date(),
|
||||
random: randomInt,
|
||||
};
|
||||
}
|
||||
|
||||
function collect(value: string, previous: string[]): string[] {
|
||||
return [...previous, value];
|
||||
}
|
||||
|
||||
function parseOverrides(pairs: string[]): Record<string, string> {
|
||||
const out: Record<string, string> = {};
|
||||
for (const pair of pairs) {
|
||||
const at = pair.indexOf("=");
|
||||
if (at <= 0) throw new ErrandError(`--input ${pair}: expected name=value`);
|
||||
out[pair.slice(0, at)] = pair.slice(at + 1);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
interface RunFlags {
|
||||
input: string[];
|
||||
dryRun?: boolean;
|
||||
declare?: boolean;
|
||||
headful?: boolean;
|
||||
chrome?: string;
|
||||
vault?: string;
|
||||
extractor?: string;
|
||||
candidate?: string;
|
||||
account?: string;
|
||||
force?: boolean;
|
||||
yes?: boolean;
|
||||
json?: boolean;
|
||||
}
|
||||
|
||||
export function registerErrandCommands(parent: Command, partial: Partial<Deps> = {}): void {
|
||||
const deps: Deps = { ...defaultDeps(), ...partial };
|
||||
|
||||
parent
|
||||
.command("run")
|
||||
.argument("<file>", "an OpenErrand 0.1 JSON file")
|
||||
.description("Run an errand in headless Chrome, stopping at every step that belongs to a person.")
|
||||
.option("--input <name=value>", "give an input's value (repeatable); wins over every source", collect, [])
|
||||
.option("--dry-run", "fill every page up to the first that holds a declaration or a shared secret, show it, and stop before its forward button")
|
||||
.option("--declare", "your consent, for this run, to tick the declarations the errand names after the values are shown")
|
||||
.option("--headful", "open a visible window, so you can take identity proofing or a captcha yourself")
|
||||
.option("--chrome <path>", "the Chrome or Chromium binary (default: CHROME_PATH, then the usual places)")
|
||||
.option("--vault <target>", "teams:<team>/<project>/<env>, opencreds[:<field>] or file:<path> (default: the file's metadata.vault)")
|
||||
.option("--extractor <command>", "a command that reads the errand's document requests as JSON on stdin and prints records")
|
||||
.option("--candidate <n>", "which shared-secret candidate to submit (1 is the best); a rejection lists the others")
|
||||
.option("--account <name>", "which account at the site, for the throttle ledger", "default")
|
||||
.option("--force", "lift the per-window, per-day and spacing caps (never a lockout)")
|
||||
.option("--yes", "run a file whose SHA-256 changed since its last run without asking")
|
||||
.option("--json", "print the run record as JSON on stdout")
|
||||
.addHelpText(
|
||||
"after",
|
||||
`
|
||||
Examples:
|
||||
logicsrc errand validate ftb-register-business.json
|
||||
logicsrc errand run ftb-register-business.json --extractor "python3 extract.py ~/taxes" --dry-run
|
||||
logicsrc errand run ftb-register-business.json --extractor "python3 extract.py ~/taxes" --declare
|
||||
logicsrc errand status
|
||||
|
||||
A one-time code is typed at the prompt, or written to the code file the run
|
||||
names (~/.local/share/logicsrc/errand/codes/<name>.code) by whoever holds the
|
||||
phone. Credentials go to the vault the file or --vault names, else a 0600
|
||||
file under ~/.local/share/logicsrc/errand/credentials/.`,
|
||||
)
|
||||
.action(async (file: string, flags: RunFlags) => {
|
||||
process.exitCode = await runCommand(file, flags, deps);
|
||||
});
|
||||
|
||||
parent
|
||||
.command("validate")
|
||||
.argument("<file>", "an OpenErrand 0.1 JSON file")
|
||||
.description("Check an errand file against the schema and the spec's rules, and show what it will ask of you.")
|
||||
.action((file: string) => {
|
||||
try {
|
||||
const loaded = loadErrand(file);
|
||||
for (const line of summarize(loaded.errand, "unverified: a local file")) deps.out(line);
|
||||
deps.out(`valid OpenErrand 0.1 sha256 ${loaded.sha256}`);
|
||||
process.exitCode = EXIT.OK;
|
||||
} catch (error) {
|
||||
deps.say((error as Error).message);
|
||||
process.exitCode = EXIT.INVALID;
|
||||
}
|
||||
});
|
||||
|
||||
parent
|
||||
.command("status")
|
||||
.description("The last run of each errand, the cards waiting on you, and any lockout.")
|
||||
.option("--json", "machine-readable")
|
||||
.action((flags: { json?: boolean }) => {
|
||||
const store = Store.fromEnv(deps.env);
|
||||
const runs = store.runs();
|
||||
const latest = new Map<string, (typeof runs)[number]>();
|
||||
for (const run of runs) latest.set(run.name, run);
|
||||
const ledger = store.loadLedger();
|
||||
const now = deps.now().getTime();
|
||||
const locks = Object.entries(ledger.lockedUntil).filter(([, until]) => Date.parse(until) > now);
|
||||
if (flags.json) {
|
||||
deps.out(JSON.stringify({ runs: [...latest.values()], lockouts: Object.fromEntries(locks) }, null, 2));
|
||||
process.exitCode = EXIT.OK;
|
||||
return;
|
||||
}
|
||||
if (!latest.size) deps.out("No errands run yet.");
|
||||
for (const run of latest.values()) {
|
||||
deps.out(`${run.name} ${run.kind}${run.outcome !== run.kind ? ` (${run.outcome})` : ""} ${run.at.slice(0, 16).replace("T", " ")}${run.reason ? ` ${run.reason}` : ""}`);
|
||||
if (run.card && !run.card.done) {
|
||||
deps.out(` card ${run.card.id}: ${run.card.title}${run.card.expires_on ? ` (before ${run.card.expires_on})` : ""}`);
|
||||
for (const [i, s] of run.card.steps.entries()) deps.out(` ${i + 1}. ${s}`);
|
||||
if (run.card.open) deps.out(` open ${run.card.open}`);
|
||||
if (run.card.command) deps.out(` ${run.card.command}`);
|
||||
}
|
||||
}
|
||||
for (const [key, until] of locks) deps.out(`locked: ${key.replace("|", " account ")} until ${until}`);
|
||||
process.exitCode = EXIT.OK;
|
||||
});
|
||||
}
|
||||
|
||||
export async function runCommand(file: string, flags: RunFlags, deps: Deps): Promise<number> {
|
||||
const store = Store.fromEnv(deps.env);
|
||||
try {
|
||||
const { errand, sha256 } = loadErrand(file);
|
||||
|
||||
// Rule 1: show it before running it, and show a change to a file run before.
|
||||
for (const line of summarize(errand, "unverified: a local file, not fetched from a publisher's index")) deps.say(line);
|
||||
const approved = store.approvals()[errand.name];
|
||||
if (approved && approved.sha256 !== sha256) {
|
||||
deps.say(` this file changed since its last run (${approved.sha256.slice(0, 12)} on ${approved.at.slice(0, 10)}, now ${sha256.slice(0, 12)})`);
|
||||
if (!flags.yes && !(deps.interactive && (await deps.confirm("Run the changed file?")))) {
|
||||
deps.say("Not run. Read the change, then rerun with --yes.");
|
||||
return EXIT.STOPPED;
|
||||
}
|
||||
}
|
||||
|
||||
const metaVault = (errand.metadata as { vault?: unknown } | undefined)?.vault;
|
||||
const targetText = flags.vault ?? (typeof metaVault === "string" ? metaVault : undefined);
|
||||
const vault: Vault | null = targetText ? openVault(parseTarget(targetText), deps.logicsrc) : null;
|
||||
const fallbackVault = fileVault(store.credentialsFile(errand.name));
|
||||
const reader = vault ?? fallbackVault;
|
||||
|
||||
const candidate = flags.candidate !== undefined ? Number(flags.candidate) : undefined;
|
||||
if (candidate !== undefined && (!Number.isInteger(candidate) || candidate < 1)) throw new ErrandError("--candidate is 1, 2, 3, ...");
|
||||
const inputs = await resolveInputs(errand, {
|
||||
overrides: parseOverrides(flags.input ?? []),
|
||||
vault: reader,
|
||||
extractor: flags.extractor ? commandExtractor(flags.extractor, { errand: errand.name }) : null,
|
||||
prompt: deps.interactive ? deps.prompt : async () => null,
|
||||
random: deps.random,
|
||||
now: deps.now(),
|
||||
...(candidate !== undefined ? { candidate } : {}),
|
||||
});
|
||||
|
||||
deps.say("Values for this run (secrets masked):");
|
||||
for (const name of Object.keys(errand.inputs ?? {})) {
|
||||
const resolved = inputs.values.get(name);
|
||||
const qa = inputs.qa.get(name);
|
||||
const from = qa ? (Object.keys(qa.answers).length ? "vault" : "generated as questions are chosen") : resolved ? `${resolved.from}${resolved.origin ? ` ${resolved.origin}` : ""}` : "none";
|
||||
deps.say(` ${name} = ${qa ? "••••" : inputs.display(name)} (${from})`);
|
||||
}
|
||||
if (inputs.chosen && inputs.candidates.length > 1) deps.say(` shared secret: candidate ${inputs.candidates.indexOf(inputs.chosen) + 1} of ${inputs.candidates.length} (${inputs.chosen.year ?? ""} ${inputs.chosen.form} ${inputs.chosen.field}, ${inputs.chosen.source})`);
|
||||
deps.say(` credentials go to ${vault?.write ? vault.describe() : `${fallbackVault.describe()} (0600)`}`);
|
||||
|
||||
store.approve(errand.name, sha256, file, deps.now());
|
||||
const origin = errand.site.origins[0]!;
|
||||
const runId = `${deps.now().toISOString().replace(/[:.]/g, "-")}-${errand.name}`;
|
||||
const { record } = await runErrand({
|
||||
errand,
|
||||
source: file,
|
||||
sha256,
|
||||
inputs,
|
||||
store,
|
||||
declare: flags.declare === true,
|
||||
dryRun: flags.dryRun === true,
|
||||
headful: flags.headful === true,
|
||||
interactive: deps.interactive,
|
||||
prompt: deps.prompt,
|
||||
say: deps.say,
|
||||
now: deps.now,
|
||||
random: deps.random,
|
||||
vault,
|
||||
fallbackVault,
|
||||
account: flags.account ?? "default",
|
||||
force: flags.force === true,
|
||||
...(deps.solver ? { solver: deps.solver } : {}),
|
||||
...(deps.pollMs !== undefined ? { pollMs: deps.pollMs } : {}),
|
||||
...(deps.rereadMs !== undefined ? { rereadMs: deps.rereadMs } : {}),
|
||||
openDriver:
|
||||
deps.openDriver ??
|
||||
(() =>
|
||||
openCdpDriver({
|
||||
errand,
|
||||
headless: flags.headful !== true,
|
||||
profile: store.profile(origin),
|
||||
pageTimeoutMs: durationMs(errand.limits?.page_timeout, 30_000),
|
||||
downloadDir: store.downloadsDir(runId),
|
||||
...(flags.chrome ? { chrome: flags.chrome } : {}),
|
||||
...(deps.chromeArgs ? { args: deps.chromeArgs } : {}),
|
||||
...(deps.settleMs !== undefined ? { settleMs: deps.settleMs } : {}),
|
||||
})),
|
||||
});
|
||||
|
||||
if (flags.json) deps.out(JSON.stringify(record, null, 2));
|
||||
else deps.out(`${record.name}: ${record.kind}${record.outcome !== record.kind ? ` (${record.outcome})` : ""}${record.reason ? `: ${record.reason}` : ""}`);
|
||||
if (record.kind === "rejected") return EXIT.REJECTED;
|
||||
if (record.kind === "stopped") return record.reason?.startsWith("throttle") ? EXIT.THROTTLED : EXIT.STOPPED;
|
||||
return EXIT.OK;
|
||||
} catch (error) {
|
||||
deps.say((error as Error).message);
|
||||
return error instanceof ErrandError && /not a valid OpenErrand|is not JSON|cannot read|^--input|^--candidate|^vault target/.test((error as Error).message) ? EXIT.INVALID : EXIT.STOPPED;
|
||||
}
|
||||
}
|
||||
|
||||
export { describeTarget, type LogicsrcExec };
|
||||
285
packages/openerrand/src/driver.ts
Normal file
285
packages/openerrand/src/driver.ts
Normal file
|
|
@ -0,0 +1,285 @@
|
|||
/**
|
||||
* The browser as the run loop sees it. {@link Driver} is an interface so the
|
||||
* loop (gates, outcomes, loop detection) is tested against a scripted fake;
|
||||
* {@link openCdpDriver} is the real one, on Chrome over CDP.
|
||||
*/
|
||||
|
||||
import { existsSync } from "node:fs";
|
||||
import { join } from "node:path";
|
||||
import { type Browser, launchBrowser, plainUserAgent } from "./browser.js";
|
||||
import type { Errand, Field, Page } from "./types.js";
|
||||
import { ErrandError, sleep } from "./util.js";
|
||||
|
||||
export type FillAction = { kind: "text" | "select"; value: string } | { kind: "check" };
|
||||
|
||||
export interface DownloadedFile {
|
||||
url: string;
|
||||
filename: string;
|
||||
path: string;
|
||||
}
|
||||
|
||||
export interface Driver {
|
||||
goto(url: string): Promise<void>;
|
||||
/** The current URL, read from the browser without running script on the page. */
|
||||
url(): Promise<string>;
|
||||
read(): Promise<Page>;
|
||||
/** Which of these CSS selectors find an element on the current page. */
|
||||
selectorHits(selectors: string[]): Promise<Record<string, boolean>>;
|
||||
fill(field: Field, action: FillAction): Promise<boolean>;
|
||||
/** Press the page's forward button. Returns its label, or `?a|b` with the buttons seen when none fits. */
|
||||
submit(): Promise<string>;
|
||||
/** Wait for the page that follows a submit to load. */
|
||||
settle(): Promise<void>;
|
||||
/** Files the site handed over during this run, completed. */
|
||||
downloads(): DownloadedFile[];
|
||||
/** Run script in the page. Used only by a captcha solver the errand allows; never on an identity provider's page. */
|
||||
evaluate(expression: string): Promise<unknown>;
|
||||
close(): Promise<void>;
|
||||
}
|
||||
|
||||
const DEFAULT_LABELS = "^(submit|continue|next|send|verify|confirm|sign ?in|log ?in)$";
|
||||
const DEFAULT_NEVER = "^(back|cancel|previous)$";
|
||||
const DEFAULT_IGNORE = "#timer, .modal";
|
||||
|
||||
/** Every visible, enabled control, with the label a person would read for it. Values are never read. */
|
||||
export function readPageScript(ignore: string): string {
|
||||
return `(() => {
|
||||
const IGNORE = ${JSON.stringify(ignore)};
|
||||
const visible = (el) => !!(el.offsetWidth || el.offsetHeight || el.getClientRects().length) && getComputedStyle(el).visibility !== 'hidden';
|
||||
const clean = (s) => (s || '').replace(/\\*\\s*Required Field/gi, '').replace(/\\s+/g, ' ').trim();
|
||||
const ignored = (el) => { try { return !!el.closest(IGNORE); } catch { return false; } };
|
||||
const forLabel = (el) => el.id ? document.querySelector('label[for="' + CSS.escape(el.id) + '"]') : null;
|
||||
const labelOf = (el) => {
|
||||
const byFor = forLabel(el);
|
||||
const wrap = el.closest('label');
|
||||
const group = el.closest('fieldset, .form-group, .row');
|
||||
const legend = group && group.querySelector('legend, .col-form-label');
|
||||
return clean((byFor && byFor.innerText) || el.getAttribute('aria-label') || (wrap && wrap.innerText) || (legend && legend.innerText) || el.placeholder || '');
|
||||
};
|
||||
const selectorOf = (el) => {
|
||||
if (el.id) return '#' + CSS.escape(el.id);
|
||||
if (el.name) return el.tagName.toLowerCase() + '[name="' + CSS.escape(el.name) + '"]' + (el.type === 'radio' || el.type === 'checkbox' ? '[value="' + CSS.escape(el.value) + '"]' : '');
|
||||
return null;
|
||||
};
|
||||
const controls = [...document.querySelectorAll('input, select, textarea')]
|
||||
.filter((el) => !['hidden', 'submit', 'button', 'image', 'reset', 'file'].includes(el.type) && !el.disabled && visible(el) && !ignored(el));
|
||||
let lastSelect = null;
|
||||
const fields = [];
|
||||
for (const el of controls) {
|
||||
if (el.tagName === 'SELECT') lastSelect = el;
|
||||
const selector = selectorOf(el);
|
||||
if (!selector) continue;
|
||||
const choice = el.type === 'radio' || el.type === 'checkbox';
|
||||
const own = clean(((el.closest('label') || forLabel(el) || {}).innerText) || '');
|
||||
const group = el.closest('fieldset, .form-group');
|
||||
const label = choice ? (own || labelOf(el)) : labelOf(el);
|
||||
const printed = group ? clean([...group.querySelectorAll('p, span, div')].map((n) => n.children.length ? '' : n.innerText).join(' ')) : '';
|
||||
fields.push({
|
||||
selector,
|
||||
id: el.id || '',
|
||||
name: el.name || '',
|
||||
type: el.type,
|
||||
label,
|
||||
required: el.required || el.getAttribute('aria-required') === 'true',
|
||||
options: el.tagName === 'SELECT' ? [...el.options].map((o) => ({ value: o.value, text: o.text.trim() })) : undefined,
|
||||
question: el.tagName !== 'SELECT' && !choice ? (lastSelect && lastSelect.selectedIndex > 0 ? lastSelect.options[lastSelect.selectedIndex].text.trim() : (printed || undefined)) : undefined,
|
||||
groupChecked: el.type === 'radio' && el.name ? !!document.querySelector('input[type=radio][name="' + CSS.escape(el.name) + '"]:checked') : undefined,
|
||||
});
|
||||
}
|
||||
const errors = [...document.querySelectorAll('.alert-danger, .validation-summary-errors li, .field-validation-error, .error-message, .invalid-feedback, [role=alert]')]
|
||||
.filter((n) => visible(n) && !ignored(n)).map((n) => clean(n.innerText)).filter(Boolean);
|
||||
return { url: location.href, title: document.title, text: clean(document.body ? document.body.innerText : '').slice(0, 6000), errors: [...new Set(errors)], fields };
|
||||
})()`;
|
||||
}
|
||||
|
||||
/** Set a value with the element's native setter, then fire input, change and blur so the page's own validation sees it. */
|
||||
export function fillScript(selector: string, action: FillAction): string {
|
||||
const target = `document.querySelector(${JSON.stringify(selector)})`;
|
||||
if (action.kind === "check") {
|
||||
return `(() => { const el = ${target}; if (el && !el.checked) el.click(); return !!el && el.checked; })()`;
|
||||
}
|
||||
return `(() => {
|
||||
const el = ${target}; if (!el) return false;
|
||||
const proto = el.tagName === 'SELECT' ? HTMLSelectElement.prototype : el.tagName === 'TEXTAREA' ? HTMLTextAreaElement.prototype : HTMLInputElement.prototype;
|
||||
Object.getOwnPropertyDescriptor(proto, 'value').set.call(el, ${JSON.stringify(action.value)});
|
||||
for (const type of ['input', 'change', 'blur']) el.dispatchEvent(new Event(type, { bubbles: true }));
|
||||
return el.value === ${JSON.stringify(action.value)};
|
||||
})()`;
|
||||
}
|
||||
|
||||
/**
|
||||
* The page's own forward button: a label from `submit.labels`, else the only
|
||||
* other button; never one matching `submit.never` or inside `submit.ignore`.
|
||||
*/
|
||||
export function submitScript(submit: Errand["submit"]): string {
|
||||
const labels = submit?.labels ?? DEFAULT_LABELS;
|
||||
const never = submit?.never ?? DEFAULT_NEVER;
|
||||
const ignore = submit?.ignore ?? DEFAULT_IGNORE;
|
||||
return `(() => {
|
||||
const LABELS = new RegExp(${JSON.stringify(labels)}, 'i');
|
||||
const NEVER = new RegExp(${JSON.stringify(never)}, 'i');
|
||||
const IGNORE = ${JSON.stringify(ignore)};
|
||||
const visible = (el) => !!(el.offsetWidth || el.offsetHeight || el.getClientRects().length);
|
||||
const ignored = (el) => { try { return !!el.closest(IGNORE); } catch { return false; } };
|
||||
const text = (b) => (b.innerText || b.value || '').replace(/\\s+/g, ' ').trim();
|
||||
const buttons = [...document.querySelectorAll('button, input[type=submit], input[type=button]')]
|
||||
.filter((b) => visible(b) && !b.disabled && !ignored(b) && text(b) && !NEVER.test(text(b)));
|
||||
const pick = buttons.find((b) => LABELS.test(text(b))) || (buttons.length === 1 ? buttons[0] : null);
|
||||
if (!pick) return '?' + buttons.map(text).join('|');
|
||||
pick.click();
|
||||
return text(pick);
|
||||
})()`;
|
||||
}
|
||||
|
||||
export interface CdpDriverOptions {
|
||||
errand: Errand;
|
||||
chrome?: string;
|
||||
headless: boolean;
|
||||
profile?: string;
|
||||
pageTimeoutMs: number;
|
||||
/** Where downloads land before they are filed. */
|
||||
downloadDir?: string;
|
||||
args?: string[];
|
||||
/** Pause after each load so a page's own scripts finish (default 1500 ms). */
|
||||
settleMs?: number;
|
||||
}
|
||||
|
||||
/** The plain user agent of each Chrome binary, learned once per process. */
|
||||
const plainAgents = new Map<string, string | null>();
|
||||
|
||||
export async function openCdpDriver(options: CdpDriverOptions): Promise<Driver> {
|
||||
const launch = (extra: string[]): Promise<Browser> =>
|
||||
launchBrowser({
|
||||
...(options.chrome ? { chrome: options.chrome } : {}),
|
||||
...(options.profile ? { profile: options.profile } : {}),
|
||||
headless: options.headless,
|
||||
timeoutMs: 30_000,
|
||||
args: [...(options.args ?? []), ...extra],
|
||||
});
|
||||
// Rule 11. A CDP user-agent override does not reach every request (a
|
||||
// navigation the page's own script starts still says HeadlessChrome), so
|
||||
// the plain string is given to Chrome at launch: the browser's own user
|
||||
// agent with "HeadlessChrome" replaced by "Chrome", and nothing else.
|
||||
const key = options.chrome ?? "default";
|
||||
let browser: Browser;
|
||||
if (options.headless && plainAgents.has(key)) {
|
||||
const plain = plainAgents.get(key);
|
||||
browser = await launch(plain ? [`--user-agent=${plain}`] : []);
|
||||
} else {
|
||||
browser = await launch([]);
|
||||
if (options.headless) {
|
||||
const { userAgent } = (await browser.cdp.send("Browser.getVersion")) as { userAgent: string };
|
||||
const plain = plainUserAgent(userAgent);
|
||||
plainAgents.set(key, plain);
|
||||
if (plain) {
|
||||
await browser.close();
|
||||
browser = await launch([`--user-agent=${plain}`]);
|
||||
}
|
||||
}
|
||||
}
|
||||
const { cdp } = browser;
|
||||
const settleMs = options.settleMs ?? 1_500;
|
||||
try {
|
||||
const { targetInfos } = (await cdp.send("Target.getTargets")) as { targetInfos: Array<{ targetId: string; type: string }> };
|
||||
let targetId = targetInfos.find((t) => t.type === "page")?.targetId;
|
||||
if (!targetId) ({ targetId } = (await cdp.send("Target.createTarget", { url: "about:blank" })) as { targetId: string });
|
||||
const { sessionId } = (await cdp.send("Target.attachToTarget", { targetId, flatten: true })) as { sessionId: string };
|
||||
await cdp.send("Page.enable", {}, sessionId);
|
||||
|
||||
const downloads: DownloadedFile[] = [];
|
||||
if (options.downloadDir) {
|
||||
await cdp.send("Browser.setDownloadBehavior", { behavior: "allowAndName", downloadPath: options.downloadDir, eventsEnabled: true });
|
||||
const begun = new Map<string, { url: string; filename: string }>();
|
||||
cdp.on((event) => {
|
||||
if (event.method === "Browser.downloadWillBegin") {
|
||||
begun.set(String(event.params.guid), { url: String(event.params.url), filename: String(event.params.suggestedFilename) });
|
||||
} else if (event.method === "Browser.downloadProgress" && event.params.state === "completed") {
|
||||
const guid = String(event.params.guid);
|
||||
const meta = begun.get(guid);
|
||||
const path = join(options.downloadDir!, guid);
|
||||
if (meta && existsSync(path)) downloads.push({ ...meta, path });
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
const evaluate = async <T>(expression: string): Promise<T | null> => {
|
||||
try {
|
||||
const { result, exceptionDetails } = (await cdp.send("Runtime.evaluate", { expression, returnByValue: true, awaitPromise: true }, sessionId)) as {
|
||||
result: { value?: unknown };
|
||||
exceptionDetails?: unknown;
|
||||
};
|
||||
return exceptionDetails ? null : ((result.value ?? null) as T | null);
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
};
|
||||
const ignore = options.errand.submit?.ignore ?? DEFAULT_IGNORE;
|
||||
let pending: Promise<void> | null = null;
|
||||
|
||||
return {
|
||||
async goto(url) {
|
||||
const loaded = cdp.waitFor("Page.loadEventFired", sessionId, options.pageTimeoutMs).catch(() => undefined);
|
||||
const result = (await cdp.send("Page.navigate", { url }, sessionId)) as { errorText?: string };
|
||||
if (result.errorText) throw new ErrandError(`could not open ${url}: ${result.errorText}`);
|
||||
await loaded;
|
||||
await sleep(settleMs);
|
||||
},
|
||||
async url() {
|
||||
const { targetInfo } = (await cdp.send("Target.getTargetInfo", { targetId })) as { targetInfo: { url: string } };
|
||||
return targetInfo.url;
|
||||
},
|
||||
async read() {
|
||||
const page = await evaluate<Page>(readPageScript(ignore));
|
||||
if (!page) throw new ErrandError("could not read the page");
|
||||
return page;
|
||||
},
|
||||
async selectorHits(selectors) {
|
||||
if (!selectors.length) return {};
|
||||
const hits = await evaluate<Record<string, boolean>>(
|
||||
`(() => { const out = {}; for (const s of ${JSON.stringify(selectors)}) { try { out[s] = !!document.querySelector(s); } catch { out[s] = false; } } return out; })()`,
|
||||
);
|
||||
return hits ?? {};
|
||||
},
|
||||
async fill(field, action) {
|
||||
return (await evaluate<boolean>(fillScript(field.selector, action))) === true;
|
||||
},
|
||||
async submit() {
|
||||
const loaded = cdp.waitFor("Page.loadEventFired", sessionId, options.pageTimeoutMs).catch(() => undefined);
|
||||
const pressed = (await evaluate<string>(submitScript(options.errand.submit))) ?? "?";
|
||||
if (!pressed.startsWith("?")) pending = loaded;
|
||||
else void loaded;
|
||||
return pressed;
|
||||
},
|
||||
async settle() {
|
||||
await Promise.race([pending ?? Promise.resolve(), sleep(options.pageTimeoutMs)]);
|
||||
pending = null;
|
||||
await sleep(settleMs);
|
||||
},
|
||||
downloads: () => [...downloads],
|
||||
evaluate: (expression) => evaluate(expression),
|
||||
close: () => browser.close(),
|
||||
};
|
||||
} catch (error) {
|
||||
await browser.close();
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
/** A download is what its media type says, checked by its first bytes. Unknown types are refused. */
|
||||
export function looksLike(type: string, head: Buffer): boolean {
|
||||
const starts = (bytes: number[]): boolean => bytes.every((b, i) => head[i] === b);
|
||||
if (type === "application/pdf") return starts([0x25, 0x50, 0x44, 0x46]);
|
||||
if (type === "image/png") return starts([0x89, 0x50, 0x4e, 0x47]);
|
||||
if (type === "image/jpeg") return starts([0xff, 0xd8, 0xff]);
|
||||
if (type === "application/zip") return starts([0x50, 0x4b, 0x03, 0x04]);
|
||||
if (type === "application/json") {
|
||||
try {
|
||||
JSON.parse(head.toString("utf8"));
|
||||
return true;
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
if (type.startsWith("text/")) return !head.includes(0);
|
||||
return false;
|
||||
}
|
||||
76
packages/openerrand/src/extract.ts
Normal file
76
packages/openerrand/src/extract.ts
Normal file
|
|
@ -0,0 +1,76 @@
|
|||
/**
|
||||
* The one document extractor the runner ships: a command the principal names
|
||||
* (`--extractor "python3 extract.py ~/taxes"`), run on this machine, that
|
||||
* reads a JSON request on stdin and prints JSON records on stdout.
|
||||
*
|
||||
* Request: `{ "errand": "<name>", "documents": [{ "input", "form", "field", "match"? }] }`
|
||||
* Reply: `[{ "form", "field", "value", "year"?, "label"?, "file"?, "page"? }]`
|
||||
* (or `{ "records": [...] }`).
|
||||
*
|
||||
* The extractor reads the documents; the runner only ever sees the values it
|
||||
* prints, keeps them in memory, and records the file and page each came from
|
||||
* for the person to see. Nothing about a document is sent anywhere else.
|
||||
*/
|
||||
|
||||
import { spawn } from "node:child_process";
|
||||
import type { DocumentExtractor, DocumentRecord, DocumentRequest } from "./inputs.js";
|
||||
import { ErrandError } from "./util.js";
|
||||
|
||||
function isRecord(row: unknown): row is DocumentRecord {
|
||||
if (!row || typeof row !== "object") return false;
|
||||
const r = row as Record<string, unknown>;
|
||||
return (
|
||||
typeof r.form === "string" &&
|
||||
typeof r.field === "string" &&
|
||||
(typeof r.value === "string" || typeof r.value === "number") &&
|
||||
(r.year === undefined || Number.isInteger(r.year)) &&
|
||||
(r.label === undefined || typeof r.label === "string") &&
|
||||
(r.file === undefined || typeof r.file === "string") &&
|
||||
(r.page === undefined || Number.isInteger(r.page))
|
||||
);
|
||||
}
|
||||
|
||||
export function parseRecords(stdout: string): DocumentRecord[] {
|
||||
let parsed: unknown;
|
||||
try {
|
||||
parsed = JSON.parse(stdout);
|
||||
} catch {
|
||||
throw new ErrandError("the extractor printed something that is not JSON");
|
||||
}
|
||||
const rows = Array.isArray(parsed) ? parsed : (parsed as { records?: unknown })?.records;
|
||||
if (!Array.isArray(rows)) throw new ErrandError("the extractor did not print a list of records");
|
||||
return rows.filter(isRecord);
|
||||
}
|
||||
|
||||
export function commandExtractor(command: string, options: { errand: string; timeoutMs?: number; cwd?: string } = { errand: "" }): DocumentExtractor {
|
||||
return {
|
||||
extract(requests: DocumentRequest[]): Promise<DocumentRecord[]> {
|
||||
return new Promise((resolve, reject) => {
|
||||
const child = spawn("/bin/sh", ["-c", command], { stdio: ["pipe", "pipe", "pipe"], ...(options.cwd ? { cwd: options.cwd } : {}) });
|
||||
let stdout = "";
|
||||
let stderr = "";
|
||||
const timer = setTimeout(() => {
|
||||
child.kill();
|
||||
reject(new ErrandError(`the extractor did not finish in ${(options.timeoutMs ?? 120_000) / 1000}s`));
|
||||
}, options.timeoutMs ?? 120_000);
|
||||
child.stdout.on("data", (chunk: Buffer) => (stdout += chunk.toString()));
|
||||
child.stderr.on("data", (chunk: Buffer) => (stderr += chunk.toString()));
|
||||
child.on("error", (error) => {
|
||||
clearTimeout(timer);
|
||||
reject(new ErrandError(`could not run the extractor: ${error.message}`));
|
||||
});
|
||||
child.on("close", (code) => {
|
||||
clearTimeout(timer);
|
||||
// stderr is the extractor's own; it is not echoed, since an extractor may print what it read.
|
||||
if (code !== 0) return reject(new ErrandError(`the extractor exited with ${code}${stderr ? ` (${stderr.trim().split("\n").length} lines on stderr, not shown)` : ""}`));
|
||||
try {
|
||||
resolve(parseRecords(stdout));
|
||||
} catch (error) {
|
||||
reject(error);
|
||||
}
|
||||
});
|
||||
child.stdin.end(JSON.stringify({ errand: options.errand, documents: requests }));
|
||||
});
|
||||
},
|
||||
};
|
||||
}
|
||||
231
packages/openerrand/src/fake-site.ts
Normal file
231
packages/openerrand/src/fake-site.ts
Normal file
|
|
@ -0,0 +1,231 @@
|
|||
/**
|
||||
* A tiny fake MyFTB for the integration test: Terms, Profile, a
|
||||
* proof-of-work interstitial that clears itself, Business, Phone, Code and
|
||||
* Confirmation, served over HTTPS with a throwaway self-signed certificate.
|
||||
* Chrome reaches it as https://webapp.ftb.ca.gov through
|
||||
* `--host-resolver-rules`, with every other hostname mapped to NOTFOUND, so
|
||||
* the test runs the published example file unchanged and cannot reach the
|
||||
* real site. Every value it checks is fictional (Jane Doe, corporation
|
||||
* 1234567).
|
||||
*
|
||||
* Not part of the build.
|
||||
*/
|
||||
|
||||
import { execFileSync } from "node:child_process";
|
||||
import { mkdtempSync, readFileSync, rmSync } from "node:fs";
|
||||
import { createServer, type Server } from "node:https";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
|
||||
export const FAKE = {
|
||||
firstName: "Jane",
|
||||
lastName: "Doe",
|
||||
zip: "95814",
|
||||
addressNumbers: "1234",
|
||||
corpId: "1234567",
|
||||
netIncome: "48210",
|
||||
year: "2025",
|
||||
phone: "5555550100",
|
||||
code: "482913",
|
||||
} as const;
|
||||
|
||||
export interface FakeSite {
|
||||
port: number;
|
||||
/** Every request: method, path and user agent. */
|
||||
requests: Array<{ method: string; path: string; userAgent: string }>;
|
||||
/** Called on GET /Code and on a wrong code, so the test can relay the next code. */
|
||||
onCode: (attempt: number) => void;
|
||||
posted: Record<string, Record<string, string>>;
|
||||
close(): Promise<void>;
|
||||
}
|
||||
|
||||
const page = (title: string, body: string, error = ""): string => `<!doctype html><html><head><meta charset="utf-8"><title>${title}</title></head><body>
|
||||
<header>Franchise Tax Board e-Services</header>
|
||||
<main><h1>${title.replace(/^.*\| /, "")}</h1>
|
||||
${error ? `<div class="alert-danger" role="alert">${error}</div>` : ""}
|
||||
${body}
|
||||
</main></body></html>`;
|
||||
|
||||
const form = (action: string, inner: string, buttons = '<button type="button">Back</button> <button type="submit">Continue</button>'): string =>
|
||||
`<form method="post" action="${action}">${inner}<div class="buttons">${buttons}</div></form>`;
|
||||
|
||||
const input = (id: string, label: string, type = "text", required = true): string =>
|
||||
`<div class="form-group"><label for="${id}">${label}</label><input id="${id}" name="${id}" type="${type}"${required ? " required" : ""}></div>`;
|
||||
|
||||
const select = (id: string, label: string, options: string[], required = true): string =>
|
||||
`<div class="form-group"><label for="${id}">${label}</label><select id="${id}" name="${id}"${required ? " required" : ""}><option value="">Select</option>${options.map((o) => `<option value="${o}">${o}</option>`).join("")}</select></div>`;
|
||||
|
||||
const QUESTIONS = ["What was the name of your first pet?", "What city were you born in?", "What was your first car?", "What was the name of your elementary school?"];
|
||||
|
||||
const PAGES = {
|
||||
terms: () =>
|
||||
page(
|
||||
"Registration | Terms",
|
||||
form(
|
||||
"/MyFTBAccess/Registration/NewAccount",
|
||||
`<p>Terms of use.</p>
|
||||
<div class="form-group"><input id="ReadTerms" name="ReadTerms" type="checkbox" value="true" required><label for="ReadTerms">I have read the terms of use</label></div>
|
||||
<div class="form-group"><input id="AcceptTerms" name="AcceptTerms" type="checkbox" value="true" required><label for="AcceptTerms">I accept the terms of use</label></div>`,
|
||||
),
|
||||
),
|
||||
profile: (error = "") =>
|
||||
page(
|
||||
"Registration | Profile",
|
||||
form(
|
||||
"/MyFTBAccess/Registration/Profile",
|
||||
[
|
||||
input("FstName", "First Name"),
|
||||
input("MInitial", "Middle Initial", "text", false),
|
||||
input("LstName", "Last Name"),
|
||||
input("UserName", "User Name"),
|
||||
input("ReUserName", "Confirm User Name"),
|
||||
input("Email", "Email Address", "email"),
|
||||
input("ReEmail", "Confirm Email Address", "email"),
|
||||
input("Password", "Password", "password"),
|
||||
input("RePassword", "Confirm Password", "password"),
|
||||
...[1, 2, 3].flatMap((n) => [select(`SecQ${n}`, `Security Question ${n}`, QUESTIONS), input(`SecA${n}`, `Answer ${n}`)]),
|
||||
].join("\n"),
|
||||
),
|
||||
error,
|
||||
),
|
||||
challenge: () => `<!doctype html><html><head><title>Challenge Validation</title></head><body>
|
||||
<div id="sec-cpt-if">Checking your browser…</div>
|
||||
<script>setTimeout(() => location.replace('/MyFTBAccess/Registration/Business'), 2500);</script>
|
||||
</body></html>`,
|
||||
business: (error = "") =>
|
||||
page(
|
||||
"Registration | Business",
|
||||
form(
|
||||
"/MyFTBAccess/Registration/Business",
|
||||
`<fieldset><legend>I am registering as</legend>
|
||||
<label><input type="radio" id="RoleInd" name="Role" value="individual"> Individual</label>
|
||||
<label><input type="radio" id="RoleBus" name="Role" value="business" required> Business Representative</label></fieldset>
|
||||
${select("CoType", "Type of company", ["Corporation", "Partnership", "Limited Liability Company"])}
|
||||
${select("FormType", "Form type", ["100", "100S", "100W", "109"])}
|
||||
${select("TaxYear", "Tax year", ["2025", "2024", "2023"])}
|
||||
${input("CorpNo", "California corporation number")}
|
||||
${input("Zip", "ZIP code", "tel")}
|
||||
${input("AddrNum", "The numbers in your mailing address", "tel")}
|
||||
${input("NetInc", "Net income for tax purposes", "tel")}
|
||||
<div class="form-group"><input id="Decl" name="Decl" type="checkbox" value="true" required><label for="Decl">I declare under penalty of perjury that the information I entered is true and correct.</label></div>`,
|
||||
),
|
||||
error,
|
||||
),
|
||||
phone: () =>
|
||||
page(
|
||||
"Registration | Phone",
|
||||
form(
|
||||
"/MyFTBAccess/Registration/Phone",
|
||||
`${input("PhoneNumber", "Phone number", "tel")}
|
||||
<label><input type="radio" id="ByText" name="How" value="text"> Send me a text message</label>
|
||||
<label><input type="radio" id="ByCall" name="How" value="call"> Call me</label>
|
||||
<input id="Phone_Foreign" name="Phone_Foreign" type="text" aria-label="Foreign number">`,
|
||||
),
|
||||
),
|
||||
code: (error = "") =>
|
||||
page("Registration | Verify", form("/MyFTBAccess/Registration/Code", input("VerificationCode", "Enter the verification code", "text"), '<button type="submit">Verify</button>'), error),
|
||||
confirmation: () =>
|
||||
page(
|
||||
"Registration Confirmation",
|
||||
"<p>Registration confirmation: your MyFTB account was successfully created.</p><p>We will mail you a PIN at the address we have on file. It expires 21 days from today.</p>",
|
||||
),
|
||||
};
|
||||
|
||||
function readBody(req: import("node:http").IncomingMessage): Promise<Record<string, string>> {
|
||||
return new Promise((resolve) => {
|
||||
let body = "";
|
||||
req.on("data", (chunk: Buffer) => (body += chunk.toString()));
|
||||
req.on("end", () => resolve(Object.fromEntries(new URLSearchParams(body))));
|
||||
});
|
||||
}
|
||||
|
||||
/** A self-signed certificate for webapp.ftb.ca.gov, made with openssl in a temp dir. */
|
||||
export function selfSigned(): { key: Buffer; cert: Buffer } {
|
||||
const dir = mkdtempSync(join(tmpdir(), "openerrand-cert-"));
|
||||
execFileSync("openssl", ["req", "-x509", "-newkey", "rsa:2048", "-nodes", "-keyout", join(dir, "key.pem"), "-out", join(dir, "cert.pem"), "-days", "1", "-subj", "/CN=webapp.ftb.ca.gov"], { stdio: "ignore" });
|
||||
try {
|
||||
return { key: readFileSync(join(dir, "key.pem")), cert: readFileSync(join(dir, "cert.pem")) };
|
||||
} finally {
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
}
|
||||
|
||||
export async function startFakeSite(): Promise<FakeSite> {
|
||||
const requests: FakeSite["requests"] = [];
|
||||
const posted: FakeSite["posted"] = {};
|
||||
let codeAttempts = 0;
|
||||
const site: Partial<FakeSite> = { requests, posted, onCode: () => undefined };
|
||||
|
||||
const send = (res: import("node:http").ServerResponse, html: string, status = 200): void => {
|
||||
res.writeHead(status, { "content-type": "text/html; charset=utf-8", "cache-control": "no-store" });
|
||||
res.end(html);
|
||||
};
|
||||
const go = (res: import("node:http").ServerResponse, to: string): void => {
|
||||
res.writeHead(303, { location: to });
|
||||
res.end();
|
||||
};
|
||||
|
||||
const server: Server = createServer(selfSigned(), async (req, res) => {
|
||||
const path = (req.url ?? "/").split("?")[0]!;
|
||||
requests.push({ method: req.method ?? "GET", path, userAgent: String(req.headers["user-agent"] ?? "") });
|
||||
const base = "/MyFTBAccess/Registration";
|
||||
if (req.method === "GET") {
|
||||
if (path === `${base}/NewAccount`) return send(res, PAGES.terms());
|
||||
if (path === `${base}/Profile`) return send(res, PAGES.profile());
|
||||
if (path === `${base}/Challenge`) return send(res, PAGES.challenge());
|
||||
if (path === `${base}/Business`) return send(res, PAGES.business());
|
||||
if (path === `${base}/Phone`) return send(res, PAGES.phone());
|
||||
if (path === `${base}/Code`) {
|
||||
codeAttempts += 1;
|
||||
site.onCode!(codeAttempts);
|
||||
return send(res, PAGES.code());
|
||||
}
|
||||
if (path === `${base}/Confirmation`) return send(res, PAGES.confirmation());
|
||||
return send(res, page("Not found", "<p>Not found</p>"), 404);
|
||||
}
|
||||
const body = await readBody(req);
|
||||
posted[path] = body;
|
||||
if (path === `${base}/NewAccount`) return body.ReadTerms && body.AcceptTerms ? go(res, `${base}/Profile`) : send(res, PAGES.terms());
|
||||
if (path === `${base}/Profile`) {
|
||||
const ok =
|
||||
body.FstName === FAKE.firstName &&
|
||||
body.LstName === FAKE.lastName &&
|
||||
body.UserName &&
|
||||
body.UserName === body.ReUserName &&
|
||||
body.Email === body.ReEmail &&
|
||||
(body.Password ?? "").length >= 15 &&
|
||||
body.Password === body.RePassword &&
|
||||
new Set([body.SecQ1, body.SecQ2, body.SecQ3]).size === 3 &&
|
||||
[body.SecA1, body.SecA2, body.SecA3].every((a) => (a ?? "").length >= 3);
|
||||
return ok ? go(res, `${base}/Challenge`) : send(res, PAGES.profile("Please correct the errors on this page."));
|
||||
}
|
||||
if (path === `${base}/Business`) {
|
||||
const ok =
|
||||
body.Role === "business" &&
|
||||
body.CoType === "Corporation" &&
|
||||
body.FormType === "100S" &&
|
||||
body.TaxYear === FAKE.year &&
|
||||
body.CorpNo === FAKE.corpId &&
|
||||
body.Zip === FAKE.zip &&
|
||||
body.AddrNum === FAKE.addressNumbers &&
|
||||
body.NetInc === FAKE.netIncome &&
|
||||
body.Decl === "true";
|
||||
return ok ? go(res, `${base}/Phone`) : send(res, PAGES.business("The information you entered does not match our records."));
|
||||
}
|
||||
if (path === `${base}/Phone`) return body.PhoneNumber === FAKE.phone && body.How === "text" ? go(res, `${base}/Code`) : send(res, PAGES.phone());
|
||||
if (path === `${base}/Code`) {
|
||||
if (body.VerificationCode === FAKE.code) return go(res, `${base}/Confirmation`);
|
||||
codeAttempts += 1;
|
||||
site.onCode!(codeAttempts);
|
||||
return send(res, PAGES.code("The code you entered is incorrect. Enter the newest code we sent."));
|
||||
}
|
||||
return send(res, page("Not found", "<p>Not found</p>"), 404);
|
||||
});
|
||||
|
||||
await new Promise<void>((resolve) => server.listen(0, "127.0.0.1", resolve));
|
||||
const address = server.address();
|
||||
if (!address || typeof address === "string") throw new Error("no port");
|
||||
site.port = address.port;
|
||||
site.close = () => new Promise<void>((resolve) => server.close(() => resolve()));
|
||||
return site as FakeSite;
|
||||
}
|
||||
38
packages/openerrand/src/index.ts
Normal file
38
packages/openerrand/src/index.ts
Normal file
|
|
@ -0,0 +1,38 @@
|
|||
/**
|
||||
* @logicsrc/openerrand: the reference runner for OpenErrand 0.1.
|
||||
*
|
||||
* `logicsrc errand run <file>` is the command; this module is the library
|
||||
* behind it, for a program that wants to embed the runner (its own driver,
|
||||
* extractor, vault or prompt).
|
||||
*/
|
||||
|
||||
export * from "./types.js";
|
||||
export { ErrandError, durationMs, render, transform, templateNames, MASK } from "./util.js";
|
||||
export {
|
||||
Inputs,
|
||||
QaSet,
|
||||
resolveInputs,
|
||||
generate,
|
||||
documentValues,
|
||||
inYears,
|
||||
type Candidate,
|
||||
type DocumentExtractor,
|
||||
type DocumentRecord,
|
||||
type DocumentRequest,
|
||||
type Prompt,
|
||||
type ResolveOptions,
|
||||
type VaultReader,
|
||||
} from "./inputs.js";
|
||||
export { commandExtractor, parseRecords } from "./extract.js";
|
||||
export { act, decide, describeField, isChoice, type Action, type Decision } from "./rules.js";
|
||||
export { DEFAULT_LOCKOUT, allowedOrigins, fits, isGate, isLockout, lockoutPattern, outcomeOf, outcomeText, selectorsOf, solverPermitted, stepFor } from "./pages.js";
|
||||
export { LIMITS, checkThrottle, emptyLedger, keyFor, lockoutMs, recordAttempt, recordLockout, type Ledger, type ThrottleCheck } from "./throttle.js";
|
||||
export { Store, errandHome, type Card, type RunRecord } from "./store.js";
|
||||
export { fileVault, opencredsVault, openVault, parseEnv, parseTarget, serializeEnv, teamsVault, type LogicsrcExec, type Vault, type VaultTarget } from "./vault.js";
|
||||
export { credentialValues, expiresOn, fileDownloads, renderCard, writeCredentials } from "./outputs.js";
|
||||
export { type CaptchaContext, type CaptchaSolver } from "./captcha.js";
|
||||
export { fillScript, looksLike, openCdpDriver, readPageScript, submitScript, type Driver, type FillAction } from "./driver.js";
|
||||
export { findChrome, launchBrowser, plainUserAgent } from "./browser.js";
|
||||
export { loadErrand, sha256, summarize, validateErrand } from "./load.js";
|
||||
export { runErrand, type RunDeps, type RunResult } from "./run.js";
|
||||
export { EXIT, registerErrandCommands, runCommand, type Deps } from "./commands.js";
|
||||
174
packages/openerrand/src/inputs.test.ts
Normal file
174
packages/openerrand/src/inputs.test.ts
Normal file
|
|
@ -0,0 +1,174 @@
|
|||
import { describe, expect, it } from "vitest";
|
||||
import { type DocumentRecord, generate, inYears, resolveInputs, type VaultReader } from "./inputs.js";
|
||||
import { ftbExample } from "./testing.js";
|
||||
import type { Errand } from "./types.js";
|
||||
import { transform } from "./util.js";
|
||||
|
||||
const NOW = new Date("2026-10-04T12:00:00Z");
|
||||
|
||||
/** Fictional returns for Jane Doe and her corporation; nothing here is anyone's real data. */
|
||||
const RECORDS: DocumentRecord[] = [
|
||||
{ form: "CA 540", field: "first name", value: "Jane", year: 2025, file: "2025/540.pdf", page: 1 },
|
||||
{ form: "CA 540", field: "last name", value: "Doe", year: 2025, file: "2025/540.pdf", page: 1 },
|
||||
{ form: "CA 540", field: "street address", value: "1234 Maple St", year: 2025, file: "2025/540.pdf", page: 1 },
|
||||
{ form: "CA 540", field: "ZIP code", value: "95814-1234", year: 2025, file: "2025/540.pdf", page: 1 },
|
||||
{ form: "CA 540", field: "street address", value: "99 Old Rd", year: 2023, file: "2023/540.pdf", page: 1 },
|
||||
{ form: "CA 100S", field: "California corporation number", value: "1234567", year: 2025, file: "2025/100S.pdf", page: 1 },
|
||||
{ form: "CA 100S", field: "line 20", label: "Net income for tax purposes", value: "48,210.40", year: 2025, file: "2025/100S.pdf", page: 3 },
|
||||
{ form: "CA 100S", field: "line 15", label: "Net income (loss) for state purposes", value: "51,377", year: 2025, file: "2025/100S.pdf", page: 3 },
|
||||
{ form: "CA 100S", field: "line 20", label: "Net income for tax purposes", value: "39,875", year: 2024, file: "2024/100S.pdf", page: 3 },
|
||||
// The year in progress and a year too old: neither counts.
|
||||
{ form: "CA 100S", field: "line 20", label: "Net income for tax purposes", value: "1", year: 2026, file: "2026/100S.pdf", page: 3 },
|
||||
{ form: "CA 100S", field: "line 20", label: "Net income for tax purposes", value: "2", year: 2019, file: "2019/100S.pdf", page: 3 },
|
||||
];
|
||||
|
||||
const extractor = { extract: async () => RECORDS };
|
||||
const noVault: VaultReader = { describe: () => "none", get: async () => undefined };
|
||||
|
||||
function tiny(inputs: Errand["inputs"]): Errand {
|
||||
return {
|
||||
type: "logicsrc.openerrand",
|
||||
version: "0.1",
|
||||
name: "t",
|
||||
title: "t",
|
||||
site: { name: "x", origins: ["https://example.com"], start: ["https://example.com/"] },
|
||||
inputs,
|
||||
steps: [{ id: "form", kind: "page" }],
|
||||
outcomes: [{ name: "ok", kind: "success", text: "ok" }],
|
||||
};
|
||||
}
|
||||
|
||||
describe("the FTB example's inputs", () => {
|
||||
it("resolves documents, derives, candidates and generated credentials", async () => {
|
||||
const inputs = await resolveInputs(ftbExample(), { extractor, vault: noVault, now: NOW, overrides: { email: "jane@example.com", phone: "5555550100" } });
|
||||
expect(inputs.get("first_name")).toBe("Jane");
|
||||
expect(inputs.get("address_numbers")).toBe("1234");
|
||||
expect(inputs.get("zip")).toBe("95814");
|
||||
expect(inputs.get("corp_id")).toBe("1234567");
|
||||
// Shared secret candidates: line 20 newest first, then line 15; the year in progress and 2019 are out.
|
||||
expect(inputs.candidates.map((c) => [c.year, c.field, c.value])).toEqual([
|
||||
[2025, "line 20", "48210"],
|
||||
[2024, "line 20", "39875"],
|
||||
[2025, "line 15", "51377"],
|
||||
]);
|
||||
expect(inputs.get("net_income")).toBe("48210");
|
||||
expect(inputs.get("tax_year")).toBe("2025");
|
||||
expect(inputs.get("username")).toMatch(/^[a-z][a-z0-9]{14}$/);
|
||||
expect(inputs.get("password")).toHaveLength(24);
|
||||
expect(inputs.values.get("username")?.from).toBe("generate");
|
||||
});
|
||||
|
||||
it("--candidate chooses another candidate, and the tax year follows it", async () => {
|
||||
const inputs = await resolveInputs(ftbExample(), { extractor, now: NOW, candidate: 2, overrides: { email: "jane@example.com", phone: "5555550100" } });
|
||||
expect(inputs.get("net_income")).toBe("39875");
|
||||
expect(inputs.get("tax_year")).toBe("2024");
|
||||
});
|
||||
|
||||
it("reuses a vault login before generating one", async () => {
|
||||
const vault: VaultReader = {
|
||||
describe: () => "test",
|
||||
get: async (key) => ({ FTB_BUSINESS_USERNAME: "jdoe1", FTB_BUSINESS_SECURITY_ANSWERS: '{"Pet?":"rex"}' })[key],
|
||||
};
|
||||
const inputs = await resolveInputs(ftbExample(), { extractor, vault, now: NOW, overrides: { email: "jane@example.com", phone: "5555550100" } });
|
||||
expect(inputs.get("username")).toBe("jdoe1");
|
||||
expect(inputs.values.get("username")).toMatchObject({ from: "vault", origin: "FTB_BUSINESS_USERNAME" });
|
||||
expect(inputs.values.get("password")?.from).toBe("generate");
|
||||
expect(inputs.qa.get("security")?.answers).toEqual({ "Pet?": "rex" });
|
||||
});
|
||||
|
||||
it("masks secrets on the terminal and shows personal values to the principal", async () => {
|
||||
const inputs = await resolveInputs(ftbExample(), { extractor, now: NOW, overrides: { email: "jane@example.com", phone: "5555550100" } });
|
||||
expect(inputs.display("password")).toBe("••••");
|
||||
expect(inputs.display("net_income")).toBe("••••");
|
||||
expect(inputs.display("first_name")).toBe("Jane");
|
||||
});
|
||||
|
||||
it("stops before the browser when a required input has no value", async () => {
|
||||
await expect(resolveInputs(ftbExample(), { extractor, now: NOW })).rejects.toThrow(/no value for Email address/);
|
||||
});
|
||||
|
||||
it("checks a value against the input's pattern", async () => {
|
||||
await expect(resolveInputs(ftbExample(), { extractor, now: NOW, overrides: { email: "jane@example.com", phone: "555" } })).rejects.toThrow(/phone does not match/);
|
||||
});
|
||||
|
||||
it("refuses an --input for a name the errand does not have", async () => {
|
||||
await expect(resolveInputs(ftbExample(), { overrides: { nope: "1" } })).rejects.toThrow(/no input named nope/);
|
||||
});
|
||||
});
|
||||
|
||||
describe("sources", () => {
|
||||
it("tries sources in order: a prompt with nobody at a terminal falls through to the next", async () => {
|
||||
const e = tiny({ a: { type: "string", sensitivity: "public", sources: [{ from: "prompt" }, { from: "literal", value: "fixed" }] } });
|
||||
expect((await resolveInputs(e, { prompt: async () => null })).get("a")).toBe("fixed");
|
||||
expect((await resolveInputs(e, { prompt: async () => "typed" })).get("a")).toBe("typed");
|
||||
});
|
||||
|
||||
it("asks for a secret without echo", async () => {
|
||||
const asked: boolean[] = [];
|
||||
const e = tiny({ pw: { type: "string", sensitivity: "secret", sources: [{ from: "prompt", ask: "Password: " }] } });
|
||||
await resolveInputs(e, { prompt: async (_q, o) => (asked.push(o.secret), "x") });
|
||||
expect(asked).toEqual([true]);
|
||||
});
|
||||
|
||||
it("cuts a value to max_length", async () => {
|
||||
const e = tiny({ n: { type: "string", sensitivity: "personal", max_length: 3, sources: [{ from: "literal", value: "Alexandra" }] } });
|
||||
expect((await resolveInputs(e)).get("n")).toBe("Ale");
|
||||
});
|
||||
|
||||
it("picks the oldest when asked", async () => {
|
||||
const e = tiny({ s: { type: "string", sensitivity: "personal", sources: [{ from: "document", form: "CA 540", field: "street address", pick: "oldest" }] } });
|
||||
expect((await resolveInputs(e, { extractor, now: NOW })).get("s")).toBe("99 Old Rd");
|
||||
});
|
||||
|
||||
it("a document match tests the printed label", async () => {
|
||||
const e = tiny({ s: { type: "integer", sensitivity: "secret", sources: [{ from: "document", form: "100S", field: "line 20", match: "state purposes", transform: "whole" }] } });
|
||||
expect((await resolveInputs(e, { extractor, now: NOW })).get("s")).toBeUndefined();
|
||||
});
|
||||
|
||||
it("detects an input that depends on itself through another", async () => {
|
||||
const e = tiny({
|
||||
a: { type: "string", sensitivity: "public", sources: [{ from: "derive", input: "b", transform: "trim" }] },
|
||||
b: { type: "string", sensitivity: "public", sources: [{ from: "derive", input: "a", transform: "trim" }] },
|
||||
});
|
||||
await expect(resolveInputs(e)).rejects.toThrow(/depends on itself/);
|
||||
});
|
||||
});
|
||||
|
||||
describe("generate", () => {
|
||||
it("has at least one of each class and only the specials given", () => {
|
||||
let n = 0;
|
||||
const random = (max: number) => (n++ * 7) % max;
|
||||
for (let i = 0; i < 20; i += 1) {
|
||||
const v = generate({ from: "generate", length: 12, classes: ["lower", "upper", "digit", "special"], special: "!#" }, random);
|
||||
expect(v).toHaveLength(12);
|
||||
expect(v).toMatch(/[a-z]/);
|
||||
expect(v).toMatch(/[A-Z]/);
|
||||
expect(v).toMatch(/[2-9]/);
|
||||
expect(v).toMatch(/[!#]/);
|
||||
expect(v).not.toMatch(/[^a-zA-Z0-9!#]/);
|
||||
}
|
||||
});
|
||||
|
||||
it("starts with a letter when letters are allowed", () => {
|
||||
for (let i = 0; i < 50; i += 1) expect(generate({ from: "generate", length: 15, classes: ["lower", "digit"] })).toMatch(/^[a-z]/);
|
||||
});
|
||||
});
|
||||
|
||||
describe("transforms and years", () => {
|
||||
it("whole keeps whole units with a minus for a loss", () => {
|
||||
expect(transform("48,210.99", "whole")).toBe("48210");
|
||||
expect(transform("(1,234)", "whole")).toBe("-1234");
|
||||
expect(transform("-1,234.5", "whole")).toBe("-1234");
|
||||
expect(transform("1234 Maple St, Apt 5", "digits")).toBe("12345");
|
||||
expect(transform("95814-1234", "first:5")).toBe("95814");
|
||||
expect(transform("123456789", "last:4")).toBe("6789");
|
||||
});
|
||||
|
||||
it("counts closed years back and leaves out the year in progress unless asked", () => {
|
||||
expect(inYears(2025, { back: 5, current: false }, NOW)).toBe(true);
|
||||
expect(inYears(2021, { back: 5, current: false }, NOW)).toBe(true);
|
||||
expect(inYears(2020, { back: 5, current: false }, NOW)).toBe(false);
|
||||
expect(inYears(2026, { back: 5, current: false }, NOW)).toBe(false);
|
||||
expect(inYears(2026, { back: 5, current: true }, NOW)).toBe(true);
|
||||
});
|
||||
});
|
||||
435
packages/openerrand/src/inputs.ts
Normal file
435
packages/openerrand/src/inputs.ts
Normal file
|
|
@ -0,0 +1,435 @@
|
|||
/**
|
||||
* Inputs: every value an errand needs, resolved from its sources in order
|
||||
* before the browser opens, so a missing value stops the run before anything
|
||||
* is sent and a dry run can show what each page would receive.
|
||||
*
|
||||
* Sources and where they are served from:
|
||||
* - `document`: a {@link DocumentExtractor} hook. The runner ships one, which
|
||||
* runs a command the principal names and reads JSON from it; the document
|
||||
* never leaves the machine because the runner never sees more than the
|
||||
* values the extractor prints.
|
||||
* - `vault`: a {@link VaultReader} (logicsrc teams, OpenCreds, or the local
|
||||
* 0600 state file). Read-only here; writing happens in outputs.
|
||||
* - `prompt`: asked at the terminal, without echo for a secret.
|
||||
* - `generate`: the runner's cryptographic generator.
|
||||
* - `derive`, `candidate`, `literal`: computed from the file and other inputs.
|
||||
*
|
||||
* `--input name=value` on the command line is the person typing the value, and
|
||||
* wins over every source.
|
||||
*/
|
||||
|
||||
import { randomInt } from "node:crypto";
|
||||
import type { Errand, Input, Source } from "./types.js";
|
||||
import { ErrandError, MASK, test, transform } from "./util.js";
|
||||
|
||||
/** One value an extractor found in a document. `file`/`page` say where, for the person; they never reach a site. */
|
||||
export interface DocumentRecord {
|
||||
form: string;
|
||||
field: string;
|
||||
value: string | number;
|
||||
year?: number;
|
||||
/** The label as printed beside the value, tested against a source's `match`. */
|
||||
label?: string;
|
||||
file?: string;
|
||||
page?: number;
|
||||
}
|
||||
|
||||
export interface DocumentRequest {
|
||||
input: string;
|
||||
form: string;
|
||||
field: string;
|
||||
match?: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Pluggable local extraction. The runner calls it once per run with every
|
||||
* document source of the errand and keeps what it returns in memory only.
|
||||
*/
|
||||
export interface DocumentExtractor {
|
||||
extract(requests: DocumentRequest[]): Promise<DocumentRecord[]>;
|
||||
}
|
||||
|
||||
/** Read-only view of the principal's vault. */
|
||||
export interface VaultReader {
|
||||
/** A human description such as `teams profullstack/ftb/prod`, shown in summaries. */
|
||||
describe(): string;
|
||||
get(key: string): Promise<string | undefined>;
|
||||
}
|
||||
|
||||
export interface PromptOptions {
|
||||
secret: boolean;
|
||||
}
|
||||
|
||||
/** null means nobody is at a terminal to answer. */
|
||||
export type Prompt = (question: string, options: PromptOptions) => Promise<string | null>;
|
||||
|
||||
/** One candidate for a shared-secret input, with where it came from. */
|
||||
export interface Candidate {
|
||||
value: string;
|
||||
year?: number;
|
||||
form: string;
|
||||
field: string;
|
||||
/** `2025/100S.pdf p3`: shown to the person, never sent anywhere. */
|
||||
source: string;
|
||||
}
|
||||
|
||||
export interface Resolved {
|
||||
value: string;
|
||||
/** Which source gave it: `flag` for --input. */
|
||||
from: Source["from"] | "flag";
|
||||
/** The vault key, or the file and page a document value came from. */
|
||||
origin?: string;
|
||||
}
|
||||
|
||||
export interface ResolveOptions {
|
||||
overrides?: Record<string, string>;
|
||||
vault?: VaultReader | null;
|
||||
extractor?: DocumentExtractor | null;
|
||||
prompt?: Prompt;
|
||||
random?: (max: number) => number;
|
||||
now?: Date;
|
||||
/** 1-based choice among a shared secret's candidates; the default is the best one. */
|
||||
candidate?: number;
|
||||
}
|
||||
|
||||
const ALPHABET = {
|
||||
lower: "abcdefghijkmnopqrstuvwxyz",
|
||||
upper: "ABCDEFGHJKLMNPQRSTUVWXYZ",
|
||||
digit: "23456789",
|
||||
} as const;
|
||||
const DEFAULT_SPECIAL = "!#$*@";
|
||||
|
||||
/**
|
||||
* A fresh random value with at least one character of each class. When
|
||||
* letters are allowed the first character is a letter, because user names on
|
||||
* most sites may not start with a digit.
|
||||
*/
|
||||
export function generate(source: Extract<Source, { from: "generate" }>, random: (max: number) => number = randomInt): string {
|
||||
const sets = source.classes.map((c) => (c === "special" ? source.special ?? DEFAULT_SPECIAL : ALPHABET[c]));
|
||||
const all = sets.join("");
|
||||
const chars = sets.map((set) => set[random(set.length)]!);
|
||||
while (chars.length < source.length) chars.push(all[random(all.length)]!);
|
||||
for (let i = chars.length - 1; i > 0; i -= 1) {
|
||||
const j = random(i + 1);
|
||||
[chars[i], chars[j]] = [chars[j]!, chars[i]!];
|
||||
}
|
||||
const out = chars.slice(0, source.length);
|
||||
const letters = (source.classes.includes("lower") ? ALPHABET.lower : "") + (source.classes.includes("upper") ? ALPHABET.upper : "");
|
||||
if (letters && !letters.includes(out[0]!)) {
|
||||
const at = out.findIndex((c) => letters.includes(c));
|
||||
if (at > 0) [out[0], out[at]] = [out[at]!, out[0]!];
|
||||
}
|
||||
return out.join("");
|
||||
}
|
||||
|
||||
const norm = (s: string): string => s.toLowerCase().replace(/^(form|ca)\s+/g, "").replace(/\s+/g, " ").trim();
|
||||
|
||||
/** Same form and field, ignoring case, spacing and a leading "Form"/"CA" ("CA 100S" is "100S"). */
|
||||
function sameField(record: DocumentRecord, source: Extract<Source, { from: "document" }>): boolean {
|
||||
return norm(record.form) === norm(source.form) && norm(record.field) === norm(source.field);
|
||||
}
|
||||
|
||||
/** A closed year within `back`, and the year in progress only when `current` says so. */
|
||||
export function inYears(year: number | undefined, years: { back?: number; current?: boolean } | undefined, now: Date): boolean {
|
||||
if (!years) return true;
|
||||
if (year === undefined) return false;
|
||||
const current = now.getFullYear();
|
||||
if (year >= current && !years.current) return false;
|
||||
if (years.back !== undefined && year < current - years.back) return false;
|
||||
return true;
|
||||
}
|
||||
|
||||
/** Records that fit one document source, newest year first, transformed. */
|
||||
export function documentValues(records: readonly DocumentRecord[], source: Extract<Source, { from: "document" }>, now: Date): Candidate[] {
|
||||
const out: Candidate[] = [];
|
||||
for (const record of records) {
|
||||
if (!sameField(record, source)) continue;
|
||||
if (source.match && record.label !== undefined && !test(source.match, record.label)) continue;
|
||||
if (!inYears(record.year, source.years, now)) continue;
|
||||
const value = transform(String(record.value), source.transform);
|
||||
if (value === "") continue;
|
||||
out.push({
|
||||
value,
|
||||
...(record.year !== undefined ? { year: record.year } : {}),
|
||||
form: source.form,
|
||||
field: source.field,
|
||||
source: [record.file, record.page !== undefined ? `p${record.page}` : undefined].filter(Boolean).join(" ") || "extractor",
|
||||
});
|
||||
}
|
||||
return out.sort((a, b) => (b.year ?? 0) - (a.year ?? 0));
|
||||
}
|
||||
|
||||
function checkValue(name: string, input: Input, value: string): string {
|
||||
const cut = input.max_length !== undefined && input.type !== "qa-set" ? value.slice(0, input.max_length) : value;
|
||||
if (input.type === "integer" && !/^-?\d+$/.test(cut)) throw new ErrandError(`${name} must be a whole number`);
|
||||
if (input.type === "number" && !Number.isFinite(Number(cut))) throw new ErrandError(`${name} must be a number`);
|
||||
if (input.type === "email" && !/^[^\s@]+@[^\s@]+\.[^\s@]+$/.test(cut)) throw new ErrandError(`${name} must be an email address`);
|
||||
if (input.type === "boolean" && !/^(true|false)$/i.test(cut)) throw new ErrandError(`${name} must be true or false`);
|
||||
if (input.pattern && !test(input.pattern, cut)) throw new ErrandError(`${name} does not match ${input.pattern}`);
|
||||
return cut;
|
||||
}
|
||||
|
||||
/**
|
||||
* Security questions and answers. A vault value is the JSON object the runner
|
||||
* wrote last time; otherwise answers are generated as questions are chosen.
|
||||
*/
|
||||
export class QaSet {
|
||||
readonly answers: Record<string, string>;
|
||||
private readonly generator: Extract<Source, { from: "generate" }> | undefined;
|
||||
private readonly random: (max: number) => number;
|
||||
|
||||
constructor(answers: Record<string, string>, generator: Extract<Source, { from: "generate" }> | undefined, random: (max: number) => number) {
|
||||
this.answers = { ...answers };
|
||||
this.generator = generator;
|
||||
this.random = random;
|
||||
}
|
||||
|
||||
/** The first offered question not used yet; records an answer for it, unless `peek` asks only which it would be. */
|
||||
choose(options: ReadonlyArray<{ value: string; text: string }>, peek = false, slot?: string): { value: string; text: string } | null {
|
||||
// The same box on a page that comes back keeps the question it chose the first time.
|
||||
const before = slot ? this.slots.get(slot) : undefined;
|
||||
if (before) {
|
||||
const again = options.find((o) => o.text.trim().toLowerCase() === before);
|
||||
if (again) return again;
|
||||
}
|
||||
const picked = this.pick(options, peek);
|
||||
if (picked && slot && !peek) this.slots.set(slot, picked.text.trim().toLowerCase());
|
||||
return picked;
|
||||
}
|
||||
|
||||
private readonly slots = new Map<string, string>();
|
||||
|
||||
private pick(options: ReadonlyArray<{ value: string; text: string }>, peek: boolean): { value: string; text: string } | null {
|
||||
const used = new Set(Object.keys(this.answers).map((q) => q.toLowerCase()));
|
||||
// A stored set must keep its questions: offer the stored ones first.
|
||||
const stored = options.find((o) => o.value !== "" && o.text.trim() && Object.keys(this.answers).some((q) => q.toLowerCase() === o.text.trim().toLowerCase()) && !this.chosen.has(o.text.trim().toLowerCase()));
|
||||
if (stored) {
|
||||
if (!peek) this.chosen.add(stored.text.trim().toLowerCase());
|
||||
return stored;
|
||||
}
|
||||
if (!this.generator) return null;
|
||||
const fresh = options.find((o) => o.value !== "" && o.text.trim() && !used.has(o.text.trim().toLowerCase()));
|
||||
if (!fresh) return null;
|
||||
if (peek) return fresh;
|
||||
this.answers[fresh.text.trim()] = generate(this.generator, this.random);
|
||||
this.chosen.add(fresh.text.trim().toLowerCase());
|
||||
return fresh;
|
||||
}
|
||||
|
||||
private readonly chosen = new Set<string>();
|
||||
|
||||
/** The answer for the question a page shows beside a box, or by its position ("Answer 2"). */
|
||||
answerFor(context: string, index: number | undefined): string | null {
|
||||
const lower = context.toLowerCase();
|
||||
const known = Object.entries(this.answers).find(([question]) => lower.includes(question.toLowerCase()));
|
||||
if (known) return known[1];
|
||||
if (index !== undefined && index > 0) return Object.values(this.answers)[index - 1] ?? null;
|
||||
return null;
|
||||
}
|
||||
|
||||
toJSON(): string {
|
||||
return JSON.stringify(this.answers);
|
||||
}
|
||||
}
|
||||
|
||||
export class Inputs {
|
||||
readonly errand: Errand;
|
||||
readonly values = new Map<string, Resolved>();
|
||||
readonly qa = new Map<string, QaSet>();
|
||||
/** Every candidate of the shared-secret input, best first, and the one this run submits. */
|
||||
candidates: Candidate[] = [];
|
||||
chosen: Candidate | null = null;
|
||||
sharedSecret: string | null = null;
|
||||
|
||||
constructor(errand: Errand) {
|
||||
this.errand = errand;
|
||||
}
|
||||
|
||||
input(name: string): Input | undefined {
|
||||
return this.errand.inputs?.[name];
|
||||
}
|
||||
|
||||
/** The raw value, for the site's own field and the vault. */
|
||||
get(name: string): string | undefined {
|
||||
const qa = this.qa.get(name);
|
||||
if (qa) return qa.toJSON();
|
||||
return this.values.get(name)?.value;
|
||||
}
|
||||
|
||||
/** What the terminal may show: secrets masked, everything else as is. */
|
||||
display(name: string): string {
|
||||
const input = this.input(name);
|
||||
const value = this.get(name);
|
||||
if (value === undefined) return "(none)";
|
||||
return input?.sensitivity === "secret" ? MASK : value;
|
||||
}
|
||||
|
||||
sensitivity(name: string): Input["sensitivity"] | undefined {
|
||||
return this.input(name)?.sensitivity;
|
||||
}
|
||||
|
||||
/** Inputs whose value came from `generate`: the credentials a success writes to the vault. */
|
||||
generated(): string[] {
|
||||
return [...this.values.entries()].filter(([, v]) => v.from === "generate").map(([k]) => k);
|
||||
}
|
||||
}
|
||||
|
||||
/** Resolve every input of an errand. Throws {@link ErrandError} for a required input nothing fills. */
|
||||
export async function resolveInputs(errand: Errand, options: ResolveOptions = {}): Promise<Inputs> {
|
||||
const inputs = new Inputs(errand);
|
||||
const defs = errand.inputs ?? {};
|
||||
const random = options.random ?? randomInt;
|
||||
const now = options.now ?? new Date();
|
||||
const overrides = options.overrides ?? {};
|
||||
|
||||
for (const name of Object.keys(overrides)) {
|
||||
if (!(name in defs)) throw new ErrandError(`--input ${name}: this errand has no input named ${name}`);
|
||||
}
|
||||
|
||||
// One extractor call for the whole errand, only when a document source is not already overridden.
|
||||
const requests: DocumentRequest[] = [];
|
||||
for (const [name, input] of Object.entries(defs)) {
|
||||
if (name in overrides) continue;
|
||||
for (const source of input.sources) {
|
||||
if (source.from === "document") requests.push({ input: name, form: source.form, field: source.field, ...(source.match ? { match: source.match } : {}) });
|
||||
}
|
||||
}
|
||||
const records = requests.length && options.extractor ? await options.extractor.extract(requests) : [];
|
||||
|
||||
const resolving = new Set<string>();
|
||||
|
||||
async function resolve(name: string): Promise<void> {
|
||||
if (inputs.values.has(name) || inputs.qa.has(name)) return;
|
||||
const input = defs[name];
|
||||
if (!input) throw new ErrandError(`no input named ${name}`);
|
||||
if (resolving.has(name)) throw new ErrandError(`input ${name} depends on itself`);
|
||||
resolving.add(name);
|
||||
try {
|
||||
if (input.type === "qa-set") return await resolveQa(name, input);
|
||||
if (name in overrides) {
|
||||
inputs.values.set(name, { value: checkValue(name, input, overrides[name]!), from: "flag" });
|
||||
if (input.role === "shared-secret") {
|
||||
inputs.sharedSecret = name;
|
||||
inputs.chosen = { value: overrides[name]!, form: "", field: "", source: "--input" };
|
||||
}
|
||||
return;
|
||||
}
|
||||
if (input.role === "shared-secret") return await resolveSharedSecret(name, input);
|
||||
for (const source of input.sources) {
|
||||
const got = await fromSource(name, input, source);
|
||||
if (got) {
|
||||
inputs.values.set(name, { ...got, value: checkValue(name, input, got.value) });
|
||||
return;
|
||||
}
|
||||
}
|
||||
if (input.required) throw new ErrandError(`no value for ${input.label ?? name} (${name}); give it with --input ${name}=...`);
|
||||
} finally {
|
||||
resolving.delete(name);
|
||||
}
|
||||
}
|
||||
|
||||
async function fromSource(name: string, input: Input, source: Source): Promise<Omit<Resolved, "value"> & { value: string } | null> {
|
||||
switch (source.from) {
|
||||
case "literal":
|
||||
return { value: String(source.value), from: "literal" };
|
||||
case "vault": {
|
||||
const value = await options.vault?.get(source.key);
|
||||
return value === undefined || value === "" ? null : { value, from: "vault", origin: source.key };
|
||||
}
|
||||
case "prompt": {
|
||||
if (!options.prompt) return null;
|
||||
const answer = await options.prompt(source.ask ?? `${input.label ?? name}: `, { secret: input.sensitivity === "secret" });
|
||||
return answer === null || answer === "" ? null : { value: answer, from: "prompt" };
|
||||
}
|
||||
case "generate":
|
||||
return { value: generate(source, random), from: "generate" };
|
||||
case "derive": {
|
||||
await resolve(source.input);
|
||||
const base = inputs.get(source.input);
|
||||
if (base === undefined) return null;
|
||||
const value = transform(base, source.transform);
|
||||
return value === "" ? null : { value, from: "derive" };
|
||||
}
|
||||
case "candidate": {
|
||||
await resolve(source.input);
|
||||
const c = inputs.chosen;
|
||||
if (!c) return null;
|
||||
const part = source.part === "year" ? c.year : c[source.part];
|
||||
return part === undefined || part === "" ? null : { value: String(part), from: "candidate" };
|
||||
}
|
||||
case "document": {
|
||||
const found = documentValues(records, source, now);
|
||||
const pickOldest = source.pick === "oldest";
|
||||
const hit = pickOldest ? found[found.length - 1] : found[0];
|
||||
return hit ? { value: hit.value, from: "document", origin: hit.source } : null;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/** Candidates from document sources in order (newest first within each); other sources give one candidate. */
|
||||
async function resolveSharedSecret(name: string, input: Input): Promise<void> {
|
||||
inputs.sharedSecret = name;
|
||||
const seen = new Set<string>();
|
||||
const candidates: Candidate[] = [];
|
||||
for (const source of input.sources) {
|
||||
if (source.from === "document") {
|
||||
for (const c of documentValues(records, source, now)) {
|
||||
const key = `${c.year}:${c.form}:${c.field}:${c.value}`;
|
||||
if (seen.has(key)) continue;
|
||||
seen.add(key);
|
||||
candidates.push(c);
|
||||
}
|
||||
} else {
|
||||
const got = await fromSource(name, input, source);
|
||||
if (got) candidates.push({ value: got.value, form: "", field: "", source: got.origin ?? got.from });
|
||||
}
|
||||
}
|
||||
inputs.candidates = candidates;
|
||||
const index = (options.candidate ?? 1) - 1;
|
||||
if (index < 0 || index >= Math.max(candidates.length, 1)) throw new ErrandError(`--candidate ${options.candidate}: there are ${candidates.length} candidates`);
|
||||
const chosen = candidates[index];
|
||||
if (!chosen) {
|
||||
if (input.required !== false) throw new ErrandError(`no candidate for ${input.label ?? name} (${name}): the extractor found none, or give one with --input ${name}=...`);
|
||||
return;
|
||||
}
|
||||
inputs.chosen = chosen;
|
||||
inputs.values.set(name, { value: checkValue(name, input, chosen.value), from: "document", origin: chosen.source });
|
||||
}
|
||||
|
||||
async function resolveQa(name: string, input: Input): Promise<void> {
|
||||
let answers: Record<string, string> = {};
|
||||
let generator: Extract<Source, { from: "generate" }> | undefined;
|
||||
const raw = overrides[name];
|
||||
if (raw !== undefined) answers = parseQa(name, raw);
|
||||
else {
|
||||
for (const source of input.sources) {
|
||||
if (source.from === "vault") {
|
||||
const value = await options.vault?.get(source.key);
|
||||
if (value) {
|
||||
answers = parseQa(name, value);
|
||||
break;
|
||||
}
|
||||
} else if (source.from === "generate") {
|
||||
generator = source;
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
inputs.qa.set(name, new QaSet(answers, generator, random));
|
||||
}
|
||||
|
||||
for (const name of Object.keys(defs)) await resolve(name);
|
||||
return inputs;
|
||||
}
|
||||
|
||||
function parseQa(name: string, raw: string): Record<string, string> {
|
||||
try {
|
||||
const parsed = JSON.parse(raw) as unknown;
|
||||
if (parsed && typeof parsed === "object" && !Array.isArray(parsed) && Object.values(parsed).every((v) => typeof v === "string")) {
|
||||
return parsed as Record<string, string>;
|
||||
}
|
||||
} catch {
|
||||
// fall through
|
||||
}
|
||||
throw new ErrandError(`${name} is a qa-set: its value is a JSON object of question to answer`);
|
||||
}
|
||||
160
packages/openerrand/src/integration.test.ts
Normal file
160
packages/openerrand/src/integration.test.ts
Normal file
|
|
@ -0,0 +1,160 @@
|
|||
/**
|
||||
* The FTB worked example, run unchanged by `logicsrc errand run` in real
|
||||
* headless Chrome against a local fake site. Chrome resolves
|
||||
* webapp.ftb.ca.gov to the fake server and every other hostname to nothing,
|
||||
* so the real site is unreachable from this test. All data is fictional.
|
||||
*/
|
||||
|
||||
import { execFileSync } from "node:child_process";
|
||||
import { readFileSync, writeFileSync } from "node:fs";
|
||||
import { join } from "node:path";
|
||||
import { Command } from "commander";
|
||||
import { afterAll, beforeAll, describe, expect, it } from "vitest";
|
||||
import { findChrome } from "./browser.js";
|
||||
import { type Deps, registerErrandCommands } from "./commands.js";
|
||||
import { FAKE, type FakeSite, startFakeSite } from "./fake-site.js";
|
||||
import type { RunRecord } from "./store.js";
|
||||
import { ftbExamplePath, tempDir } from "./testing.js";
|
||||
import { type LogicsrcExec, parseEnv, serializeEnv } from "./vault.js";
|
||||
|
||||
const hasOpenssl = (() => {
|
||||
try {
|
||||
execFileSync("openssl", ["version"], { stdio: "ignore" });
|
||||
return true;
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
})();
|
||||
const chrome = findChrome();
|
||||
const enabled = Boolean(chrome && hasOpenssl && !process.env.OPENERRAND_SKIP_BROWSER);
|
||||
|
||||
describe.skipIf(!enabled)("logicsrc errand run, the FTB example, real Chrome, fake site", () => {
|
||||
let site: FakeSite;
|
||||
let home: string;
|
||||
let extractor: string;
|
||||
let remote: Record<string, string>;
|
||||
const out: string[] = [];
|
||||
const say: string[] = [];
|
||||
|
||||
beforeAll(async () => {
|
||||
site = await startFakeSite();
|
||||
if (!Number.isInteger(site.port)) throw new Error("fake site has no port");
|
||||
home = tempDir("openerrand-it-");
|
||||
// The extractor the person would point at their returns: here it prints fictional records.
|
||||
extractor = join(home, "extract.mjs");
|
||||
const records = [
|
||||
{ form: "CA 540", field: "first name", value: FAKE.firstName, year: 2025, file: "2025/540.pdf", page: 1 },
|
||||
{ form: "CA 540", field: "last name", value: FAKE.lastName, year: 2025, file: "2025/540.pdf", page: 1 },
|
||||
{ form: "CA 540", field: "street address", value: "1234 Maple St", year: 2025, file: "2025/540.pdf", page: 1 },
|
||||
{ form: "CA 540", field: "ZIP code", value: "95814-0001", year: 2025, file: "2025/540.pdf", page: 1 },
|
||||
{ form: "CA 100S", field: "California corporation number", value: FAKE.corpId, year: 2025, file: "2025/100S.pdf", page: 1 },
|
||||
{ form: "CA 100S", field: "line 20", label: "Net income for tax purposes", value: "48,210", year: 2025, file: "2025/100S.pdf", page: 3 },
|
||||
{ form: "CA 100S", field: "line 20", label: "Net income for tax purposes", value: "39,875", year: 2024, file: "2024/100S.pdf", page: 3 },
|
||||
];
|
||||
writeFileSync(extractor, `let input = ""; process.stdin.on("data", (c) => (input += c)); process.stdin.on("end", () => { JSON.parse(input); process.stdout.write(${JSON.stringify(JSON.stringify(records))}); });`);
|
||||
remote = { OTHER_TEAM_KEY: "keep me" };
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await site?.close();
|
||||
});
|
||||
|
||||
function program(): Command {
|
||||
const fakeTeams: LogicsrcExec = (args) => {
|
||||
// logicsrc teams pull/push against an in-memory vault: the pull-merge-push is what is under test.
|
||||
const file = args[args.indexOf("--env") + 1]!;
|
||||
if (args[0] === "teams" && args[1] === "pull") writeFileSync(file, serializeEnv(remote));
|
||||
else if (args[0] === "teams" && args[1] === "push") remote = parseEnv(readFileSync(file, "utf8"));
|
||||
else return { status: 1, stdout: "", stderr: "unexpected" };
|
||||
return { status: 0, stdout: "", stderr: "" };
|
||||
};
|
||||
const deps: Partial<Deps> = {
|
||||
env: { ...process.env, LOGICSRC_ERRAND_HOME: home },
|
||||
out: (line) => out.push(line),
|
||||
say: (line) => {
|
||||
say.push(line);
|
||||
// Whoever holds the phone writes the code to the file the runner names.
|
||||
if (/WAITING for the sms code/.test(line)) relayNext();
|
||||
},
|
||||
interactive: false,
|
||||
logicsrc: fakeTeams,
|
||||
chromeArgs: [
|
||||
`--host-resolver-rules=MAP webapp.ftb.ca.gov:443 127.0.0.1:${site.port}, MAP * ~NOTFOUND`,
|
||||
"--ignore-certificate-errors",
|
||||
// CI runners often lack the user namespaces Chrome's sandbox needs.
|
||||
...(process.env.CI ? ["--no-sandbox"] : []),
|
||||
],
|
||||
pollMs: 250,
|
||||
rereadMs: 300,
|
||||
settleMs: 300,
|
||||
};
|
||||
const p = new Command();
|
||||
p.name("logicsrc").enablePositionalOptions().exitOverride();
|
||||
registerErrandCommands(p.command("errand"), deps);
|
||||
return p;
|
||||
}
|
||||
|
||||
const codes = ["000000", FAKE.code];
|
||||
function relayNext(): void {
|
||||
const code = codes.shift();
|
||||
if (code) setTimeout(() => writeFileSync(join(home, "codes", "ftb-register-business.code"), `${code}\n`, { mode: 0o600 }), 300);
|
||||
}
|
||||
|
||||
const lastRecord = (): RunRecord => JSON.parse(out.filter((l) => l.startsWith("{")).at(-1)!) as RunRecord;
|
||||
const args = (...extra: string[]) => ["node", "logicsrc", "errand", "run", ftbExamplePath(), "--extractor", `${process.execPath} ${extractor}`, "--input", "email=jane@example.com", "--input", `phone=${FAKE.phone}`, "--vault", "teams:test/ftb/prod", "--json", ...extra];
|
||||
|
||||
it("a dry run stops before the business page's forward button", async () => {
|
||||
await program().parseAsync(args("--dry-run"));
|
||||
expect(process.exitCode).toBe(0);
|
||||
expect(lastRecord()).toMatchObject({ kind: "dry-run", page: "https://webapp.ftb.ca.gov/MyFTBAccess/Registration/Business" });
|
||||
expect(site.posted["/MyFTBAccess/Registration/Business"]).toBeUndefined();
|
||||
// The challenge interstitial was waited out, not solved.
|
||||
expect(site.requests.some((r) => r.path.endsWith("/Challenge"))).toBe(true);
|
||||
expect(say.join("\n")).toContain("declaration (not ticked)");
|
||||
}, 120_000);
|
||||
|
||||
it("a run with --declare registers, relays the code after a wrong one, pushes the login into the team vault, and keeps the card local", async () => {
|
||||
site.requests.length = 0;
|
||||
await program().parseAsync(args("--declare"));
|
||||
const record = lastRecord();
|
||||
expect(record, say.join("\n")).toMatchObject({ outcome: "registered", kind: "success", vault: "teams test/ftb/prod" });
|
||||
expect(process.exitCode).toBe(0);
|
||||
expect(record.card?.id).toMatch(/^pin-letter\//);
|
||||
expect(record.waiting?.what).toBe("MyFTB PIN letter");
|
||||
|
||||
// What the site received: the shared secret once, the declaration ticked, the code on the second try.
|
||||
expect(site.posted["/MyFTBAccess/Registration/Business"]).toMatchObject({ NetInc: FAKE.netIncome, TaxYear: "2025", Decl: "true", Role: "business", Zip: FAKE.zip, AddrNum: "1234" });
|
||||
expect(site.requests.filter((r) => r.method === "POST" && r.path.endsWith("/Business"))).toHaveLength(1);
|
||||
expect(site.requests.filter((r) => r.method === "POST" && r.path.endsWith("/Code"))).toHaveLength(2);
|
||||
// Rule 11: the user agent says Chrome, not HeadlessChrome.
|
||||
expect(site.requests.filter((r) => r.userAgent.includes("HeadlessChrome")).map((r) => `${r.method} ${r.path}`)).toEqual([]);
|
||||
|
||||
// The team vault: merged, nothing dropped, the generated login in it.
|
||||
const profile = site.posted["/MyFTBAccess/Registration/Profile"]!;
|
||||
expect(remote).toMatchObject({ OTHER_TEAM_KEY: "keep me", FTB_BUSINESS_USERNAME: profile.UserName, FTB_BUSINESS_PASSWORD: profile.Password, FTB_BUSINESS_EMAIL: "jane@example.com" });
|
||||
expect(Object.keys(JSON.parse(remote.FTB_BUSINESS_SECURITY_ANSWERS!))).toHaveLength(3);
|
||||
|
||||
// Rule 8: values never reach the page log or the run record; secrets never reach the terminal.
|
||||
const log = readFileSync(join(home, "pages", "ftb-register-business.jsonl"), "utf8");
|
||||
for (const value of [profile.Password!, FAKE.netIncome, FAKE.code, "jane@example.com", FAKE.phone]) {
|
||||
expect(log).not.toContain(value);
|
||||
expect(JSON.stringify(record)).not.toContain(value);
|
||||
}
|
||||
expect(say.join("\n")).not.toContain(profile.Password!);
|
||||
expect(say.join("\n")).not.toContain(FAKE.netIncome);
|
||||
|
||||
// `errand status` shows the card from the local run record.
|
||||
out.length = 0;
|
||||
await program().parseAsync(["node", "logicsrc", "errand", "status"]);
|
||||
expect(out.join("\n")).toMatch(/ftb-register-business {2}success \(registered\)/);
|
||||
expect(out.join("\n")).toContain("ftb activate business --pin <PIN from the letter>");
|
||||
}, 180_000);
|
||||
|
||||
it("a third run inside the window is refused by the throttle before Chrome starts", async () => {
|
||||
const before = site.requests.length;
|
||||
await program().parseAsync(args("--declare"));
|
||||
expect(process.exitCode).toBe(4);
|
||||
expect(site.requests.length).toBe(before);
|
||||
process.exitCode = 0;
|
||||
}, 30_000);
|
||||
});
|
||||
88
packages/openerrand/src/load.ts
Normal file
88
packages/openerrand/src/load.ts
Normal file
|
|
@ -0,0 +1,88 @@
|
|||
/**
|
||||
* Loading an errand file: parse, validate with @logicsrc/validators (the same
|
||||
* schema and semantic checks `logicsrc-validate openerrand` runs), hash, and
|
||||
* describe it for the person before anything runs (rule 1).
|
||||
*/
|
||||
|
||||
import { createHash } from "node:crypto";
|
||||
import { readFileSync } from "node:fs";
|
||||
import { validate } from "@logicsrc/validators";
|
||||
import { isGate } from "./pages.js";
|
||||
import type { Errand, Source } from "./types.js";
|
||||
import { ErrandError } from "./util.js";
|
||||
|
||||
export interface Loaded {
|
||||
errand: Errand;
|
||||
sha256: string;
|
||||
file: string;
|
||||
}
|
||||
|
||||
export function sha256(text: string): string {
|
||||
return createHash("sha256").update(text).digest("hex");
|
||||
}
|
||||
|
||||
/** Validation errors as one line each: `/steps/2/solver: a captcha solver is never allowed on a tax site`. */
|
||||
export function validateErrand(data: unknown): string[] {
|
||||
const result = validate("openerrand", data);
|
||||
if (result.ok) return [];
|
||||
return result.errors.map((e) => `${e.instancePath || "/"}: ${e.message ?? e.keyword}`);
|
||||
}
|
||||
|
||||
export function loadErrand(file: string): Loaded {
|
||||
let text: string;
|
||||
try {
|
||||
text = readFileSync(file, "utf8");
|
||||
} catch (error) {
|
||||
throw new ErrandError(`cannot read ${file}: ${(error as Error).message}`);
|
||||
}
|
||||
let data: unknown;
|
||||
try {
|
||||
data = JSON.parse(text);
|
||||
} catch (error) {
|
||||
throw new ErrandError(`${file} is not JSON: ${(error as Error).message}`);
|
||||
}
|
||||
const errors = validateErrand(data);
|
||||
if (errors.length) throw new ErrandError(`${file} is not a valid OpenErrand 0.1 file:\n${errors.map((e) => ` ${e}`).join("\n")}`);
|
||||
return { errand: data as Errand, sha256: sha256(text), file };
|
||||
}
|
||||
|
||||
function describeSource(source: Source): string {
|
||||
switch (source.from) {
|
||||
case "document":
|
||||
return `document ${source.form} ${source.field}`;
|
||||
case "vault":
|
||||
return `vault ${source.key}`;
|
||||
case "prompt":
|
||||
return "asked at the terminal";
|
||||
case "generate":
|
||||
return `generated (${source.length} chars)`;
|
||||
case "derive":
|
||||
return `from ${source.input} (${source.transform})`;
|
||||
case "candidate":
|
||||
return `the ${source.part} of the ${source.input} candidate`;
|
||||
case "literal":
|
||||
return "fixed in the file";
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* What a person reads before the first run: title, publisher, verification,
|
||||
* every gate with its why, every input with its sensitivity and sources.
|
||||
*/
|
||||
export function summarize(errand: Errand, verification: string): string[] {
|
||||
const lines = [`${errand.title} (${errand.name})`, ` site: ${errand.site.name} ${errand.site.origins.join(" ")}${errand.site.sector ? ` [${errand.site.sector}]` : ""}`];
|
||||
lines.push(` publisher: ${errand.publisher ?? "not stated"} ${verification}`);
|
||||
const gates = errand.steps.filter(isGate);
|
||||
if (gates.length) {
|
||||
lines.push(" steps that are yours:");
|
||||
for (const gate of gates) lines.push(` ${gate.kind} (${gate.id}): ${"why" in gate && gate.why ? gate.why : ("what" in gate ? gate.what : "")}`);
|
||||
}
|
||||
const inputs = Object.entries(errand.inputs ?? {});
|
||||
if (inputs.length) {
|
||||
lines.push(" inputs:");
|
||||
for (const [name, input] of inputs) {
|
||||
lines.push(` ${name} [${input.sensitivity}${input.role ? `, ${input.role}` : ""}]: ${input.sources.map(describeSource).join(", then ")}`);
|
||||
}
|
||||
}
|
||||
return lines;
|
||||
}
|
||||
110
packages/openerrand/src/outputs.test.ts
Normal file
110
packages/openerrand/src/outputs.test.ts
Normal file
|
|
@ -0,0 +1,110 @@
|
|||
import { readFileSync, statSync, writeFileSync } from "node:fs";
|
||||
import { join } from "node:path";
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { resolveInputs } from "./inputs.js";
|
||||
import { credentialValues, fileDownloads, renderCard, writeCredentials } from "./outputs.js";
|
||||
import { ftbExample, tempDir } from "./testing.js";
|
||||
import type { Errand } from "./types.js";
|
||||
import { fileVault, parseEnv, parseTarget, serializeEnv, teamsVault, type LogicsrcExec, type Vault } from "./vault.js";
|
||||
|
||||
const OVERRIDES = { email: "jane@example.com", phone: "5555550100", first_name: "Jane", last_name: "Doe", street: "1234 Maple St", zip: "95814", corp_id: "1234567", net_income: "48210", tax_year: "2025" };
|
||||
|
||||
describe("hand-off cards", () => {
|
||||
it("renders built-ins and keeps the card to public values", async () => {
|
||||
const e = ftbExample();
|
||||
const inputs = await resolveInputs(e, { overrides: OVERRIDES });
|
||||
const card = renderCard(e, "pin-letter", inputs, { expires_on: "2026-10-25" }, () => 0);
|
||||
expect(card.id).toMatch(/^pin-letter\/[a-z2-9]{6}$/);
|
||||
expect(card.steps[1]).toBe("Activate before 2026-10-25: the PIN expires 21 days after registration.");
|
||||
expect(JSON.stringify(card)).not.toMatch(/jane|5555550100|48210|Maple/i);
|
||||
});
|
||||
|
||||
it("refuses a card that names a personal or secret input, whatever the validator said", async () => {
|
||||
const e: Errand = { ...ftbExample(), handoffs: { "pin-letter": { title: "PIN for {{first_name}}", steps: ["x"] } } };
|
||||
const inputs = await resolveInputs(e, { overrides: OVERRIDES });
|
||||
expect(() => renderCard(e, "pin-letter", inputs, {}, () => 0)).toThrow(/personal/);
|
||||
});
|
||||
});
|
||||
|
||||
describe("credentials", () => {
|
||||
it("renders the keys only for the outcome named in when", async () => {
|
||||
const e = ftbExample();
|
||||
const inputs = await resolveInputs(e, { overrides: OVERRIDES });
|
||||
const values = credentialValues(e, e.outcomes.find((o) => o.name === "registered")!, inputs);
|
||||
expect(Object.keys(values).sort()).toEqual(["FTB_BUSINESS_EMAIL", "FTB_BUSINESS_PASSWORD", "FTB_BUSINESS_SECURITY_ANSWERS", "FTB_BUSINESS_USERNAME"]);
|
||||
expect(values.FTB_BUSINESS_PASSWORD).toBe(inputs.get("password"));
|
||||
expect(credentialValues(e, e.outcomes.find((o) => o.name === "rejected")!, inputs)).toEqual({});
|
||||
});
|
||||
|
||||
it("falls back to the 0600 file when the vault is read-only or the write fails, and says so", async () => {
|
||||
const dir = tempDir();
|
||||
const fallback = fileVault(join(dir, "creds.env"));
|
||||
const readOnly: Vault = { describe: () => "OpenCreds", get: async () => undefined };
|
||||
const r1 = await writeCredentials({ A: "1" }, readOnly, fallback);
|
||||
expect(r1.warning).toMatch(/read-only/);
|
||||
expect(statSync(join(dir, "creds.env")).mode & 0o777).toBe(0o600);
|
||||
const broken: Vault = { ...readOnly, write: async () => Promise.reject(new Error("offline")) };
|
||||
const r2 = await writeCredentials({ B: "2" }, broken, fallback);
|
||||
expect(r2.warning).toMatch(/offline/);
|
||||
expect(parseEnv(readFileSync(join(dir, "creds.env"), "utf8"))).toEqual({ A: "1", B: "2" });
|
||||
});
|
||||
});
|
||||
|
||||
describe("vaults", () => {
|
||||
it("round-trips values with quotes, specials and JSON", () => {
|
||||
const values = { P: "a#b$c*d@e!f", Q: '{"Pet?":"rex"}', R: "plain", S: "back\\slash" };
|
||||
expect(parseEnv(serializeEnv(values))).toEqual(values);
|
||||
});
|
||||
|
||||
it("parses targets", () => {
|
||||
expect(parseTarget("teams:profullstack/ftb/prod")).toEqual({ kind: "teams", team: "profullstack", project: "ftb", env: "prod" });
|
||||
expect(parseTarget("opencreds")).toEqual({ kind: "opencreds" });
|
||||
expect(() => parseTarget("s3://x")).toThrow(/expected/);
|
||||
});
|
||||
|
||||
it("a teams write pulls, merges and pushes, never dropping a key already there", async () => {
|
||||
let remote: Record<string, string> = { OTHER_KEY: "keep me", FTB_BUSINESS_EMAIL: "old@example.com" };
|
||||
const calls: string[] = [];
|
||||
const exec: LogicsrcExec = (args) => {
|
||||
calls.push(args.slice(0, 5).join(" "));
|
||||
const file = args[args.indexOf("--env") + 1]!;
|
||||
if (args[1] === "pull") writeFileSync(file, serializeEnv(remote));
|
||||
if (args[1] === "push") remote = parseEnv(readFileSync(file, "utf8"));
|
||||
return { status: 0, stdout: "", stderr: "" };
|
||||
};
|
||||
const vault = teamsVault({ kind: "teams", team: "profullstack", project: "ftb", env: "prod" }, exec);
|
||||
expect(await vault.get("OTHER_KEY")).toBe("keep me");
|
||||
await vault.write!({ FTB_BUSINESS_EMAIL: "jane@example.com", FTB_BUSINESS_PASSWORD: "x#y" });
|
||||
expect(remote).toEqual({ OTHER_KEY: "keep me", FTB_BUSINESS_EMAIL: "jane@example.com", FTB_BUSINESS_PASSWORD: "x#y" });
|
||||
expect(calls).toEqual(["teams pull profullstack ftb prod", "teams pull profullstack ftb prod", "teams push profullstack ftb prod"]);
|
||||
});
|
||||
|
||||
it("a failed push is an error the caller turns into the file fallback", async () => {
|
||||
const exec: LogicsrcExec = (args) => (args[1] === "push" ? { status: 1, stdout: "", stderr: "not logged in" } : { status: 0, stdout: "", stderr: "" });
|
||||
const vault = teamsVault({ kind: "teams", team: "t", project: "p", env: "e" }, exec);
|
||||
await expect(vault.write!({ A: "1" })).rejects.toThrow(/not logged in/);
|
||||
});
|
||||
});
|
||||
|
||||
describe("downloads", () => {
|
||||
it("checks the type, files it, and never overwrites different bytes", async () => {
|
||||
const e: Errand = {
|
||||
...ftbExample(),
|
||||
outputs: { downloads: [{ match: { type: "application/pdf", url: "transcript" }, to: "{{dir}}/{{tax_year}}/transcript.pdf", when: "registered" }] },
|
||||
inputs: { ...ftbExample().inputs, dir: { type: "string", sensitivity: "public", sources: [{ from: "literal", value: tempDir() }] } },
|
||||
};
|
||||
const inputs = await resolveInputs(e, { overrides: OVERRIDES });
|
||||
const dl = tempDir();
|
||||
writeFileSync(join(dl, "a"), "%PDF-1.7 fake");
|
||||
const outcome = e.outcomes.find((o) => o.name === "registered")!;
|
||||
const files = [{ url: "https://webapp.ftb.ca.gov/transcript?id=1", filename: "t.pdf", path: join(dl, "a") }];
|
||||
const [to] = fileDownloads(e, outcome, inputs, files);
|
||||
expect(to).toMatch(/2025\/transcript\.pdf$/);
|
||||
// Same bytes again: fine. Different bytes: refused.
|
||||
expect(fileDownloads(e, outcome, inputs, files)).toEqual([to]);
|
||||
writeFileSync(join(dl, "b"), "%PDF-1.7 different");
|
||||
expect(() => fileDownloads(e, outcome, inputs, [{ ...files[0]!, path: join(dl, "b") }])).toThrow(/different bytes/);
|
||||
writeFileSync(join(dl, "c"), "<html>not a pdf</html>");
|
||||
expect(() => fileDownloads(e, outcome, inputs, [{ ...files[0]!, path: join(dl, "c") }])).toThrow(/is not application\/pdf/);
|
||||
});
|
||||
});
|
||||
122
packages/openerrand/src/outputs.ts
Normal file
122
packages/openerrand/src/outputs.ts
Normal file
|
|
@ -0,0 +1,122 @@
|
|||
/**
|
||||
* What a run keeps: credentials (written before success is reported), files
|
||||
* the site handed over, and hand-off cards (kept in the local run record
|
||||
* only, never posted anywhere).
|
||||
*/
|
||||
|
||||
import { copyFileSync, existsSync, readFileSync } from "node:fs";
|
||||
import { homedir } from "node:os";
|
||||
import { dirname } from "node:path";
|
||||
import { HANDOFF_BUILTINS } from "@logicsrc/validators";
|
||||
import { type DownloadedFile, looksLike } from "./driver.js";
|
||||
import type { Inputs } from "./inputs.js";
|
||||
import { type Card, ensureDir } from "./store.js";
|
||||
import type { Errand, Outcome } from "./types.js";
|
||||
import { ErrandError, re, render, shortId, templateNames } from "./util.js";
|
||||
import type { Vault } from "./vault.js";
|
||||
|
||||
/**
|
||||
* Render a hand-off card. Only `{{expires_on}}`, `{{errand.title}}`,
|
||||
* `{{site.name}}` and public inputs may appear; anything else and the card is
|
||||
* refused, whatever the validator said about the file.
|
||||
*/
|
||||
export function renderCard(errand: Errand, id: string, inputs: Inputs, extra: { expires_on?: string }, random: (max: number) => number): Card {
|
||||
const card = errand.handoffs?.[id];
|
||||
if (!card) throw new ErrandError(`no hand-off card named ${id}`);
|
||||
const lookup = (name: string): string | undefined => {
|
||||
if (name === "expires_on") return extra.expires_on ?? "(no deadline)";
|
||||
if (name === "errand.title") return errand.title;
|
||||
if (name === "site.name") return errand.site.name;
|
||||
return inputs.get(name);
|
||||
};
|
||||
const texts = [card.title, ...card.steps, ...(card.command !== undefined ? [card.command] : [])];
|
||||
for (const text of texts) {
|
||||
for (const name of templateNames(text)) {
|
||||
if ((HANDOFF_BUILTINS as readonly string[]).includes(name)) continue;
|
||||
const sensitivity = inputs.sensitivity(name);
|
||||
if (sensitivity !== "public") throw new ErrandError(`card ${id} names {{${name}}}, which is ${sensitivity ?? "not an input"}; a card carries built-ins and public inputs only`);
|
||||
}
|
||||
}
|
||||
return {
|
||||
id: `${id}/${shortId(random)}`,
|
||||
title: render(card.title, lookup),
|
||||
...(card.open ? { open: card.open } : {}),
|
||||
steps: card.steps.map((s) => render(s, lookup)),
|
||||
...(card.command !== undefined ? { command: render(card.command, lookup) } : {}),
|
||||
...(extra.expires_on ? { expires_on: extra.expires_on } : {}),
|
||||
};
|
||||
}
|
||||
|
||||
/** The credential keys for this outcome, rendered from the file's templates. Empty when the outcome is not `when`. */
|
||||
export function credentialValues(errand: Errand, outcome: Outcome, inputs: Inputs): Record<string, string> {
|
||||
const vault = errand.outputs?.vault;
|
||||
if (!vault || (vault.when !== undefined && vault.when !== outcome.name)) return {};
|
||||
const out: Record<string, string> = {};
|
||||
for (const [key, template] of Object.entries(vault.keys)) {
|
||||
out[key] = render(template, (name) => inputs.get(name));
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
/**
|
||||
* Write credentials to the vault, falling back to the 0600 state file when
|
||||
* there is no writable vault or the write fails. Returns where they went and
|
||||
* any warning, so the caller says so.
|
||||
*/
|
||||
export async function writeCredentials(values: Record<string, string>, vault: Vault | null, fallback: Vault): Promise<{ where: string; warning?: string }> {
|
||||
if (!Object.keys(values).length) return { where: "" };
|
||||
if (vault?.write) {
|
||||
try {
|
||||
await vault.write(values);
|
||||
return { where: vault.describe() };
|
||||
} catch (error) {
|
||||
await fallback.write!(values);
|
||||
return { where: fallback.describe(), warning: `vault write failed (${(error as Error).message}); the login is only in ${fallback.describe()}` };
|
||||
}
|
||||
}
|
||||
await fallback.write!(values);
|
||||
return {
|
||||
where: fallback.describe(),
|
||||
warning: vault ? `${vault.describe()} is read-only; the login is in ${fallback.describe()} (0600)` : `no vault named; the login is in ${fallback.describe()} (0600)`,
|
||||
};
|
||||
}
|
||||
|
||||
const expand = (path: string): string => path.replace(/^~(?=\/|$)/, homedir());
|
||||
|
||||
/**
|
||||
* File what the site handed over. Each entry matches by URL, file name and
|
||||
* media type (checked against the file's first bytes); the target path is
|
||||
* rendered from public and personal inputs, never secrets, and an existing
|
||||
* file with different bytes is never overwritten.
|
||||
*/
|
||||
export function fileDownloads(errand: Errand, outcome: Outcome, inputs: Inputs, downloaded: readonly DownloadedFile[]): string[] {
|
||||
const filed: string[] = [];
|
||||
for (const entry of errand.outputs?.downloads ?? []) {
|
||||
if (entry.when !== undefined && entry.when !== outcome.name) continue;
|
||||
const file = downloaded.find(
|
||||
(d) => (entry.match.url === undefined || re(entry.match.url).test(d.url)) && (entry.match.filename === undefined || re(entry.match.filename).test(d.filename)),
|
||||
);
|
||||
if (!file) continue;
|
||||
const bytes = readFileSync(file.path);
|
||||
if (entry.match.type && !looksLike(entry.match.type, bytes)) throw new ErrandError(`${file.filename} is not ${entry.match.type}; left in ${file.path}`);
|
||||
const to = expand(
|
||||
render(entry.to, (name) => {
|
||||
if (inputs.sensitivity(name) === "secret") throw new ErrandError(`download path ${entry.to} names a secret input`);
|
||||
return inputs.get(name);
|
||||
}),
|
||||
);
|
||||
if (existsSync(to)) {
|
||||
if (!readFileSync(to).equals(bytes)) throw new ErrandError(`${to} exists with different bytes; the new file is left in ${file.path}`);
|
||||
} else {
|
||||
ensureDir(dirname(to));
|
||||
copyFileSync(file.path, to);
|
||||
}
|
||||
filed.push(to);
|
||||
}
|
||||
return filed;
|
||||
}
|
||||
|
||||
/** Run date plus an ISO 8601 duration, as YYYY-MM-DD. */
|
||||
export function expiresOn(now: Date, ms: number): string {
|
||||
return new Date(now.getTime() + ms).toISOString().slice(0, 10);
|
||||
}
|
||||
133
packages/openerrand/src/pages.test.ts
Normal file
133
packages/openerrand/src/pages.test.ts
Normal file
|
|
@ -0,0 +1,133 @@
|
|||
import { describe, expect, it } from "vitest";
|
||||
import { validateErrand } from "./load.js";
|
||||
import { isLockout, outcomeOf, solverPermitted, stepFor } from "./pages.js";
|
||||
import { ftbExample } from "./testing.js";
|
||||
import { checkThrottle, emptyLedger, keyFor, LIMITS, lockoutMs, recordAttempt, recordLockout } from "./throttle.js";
|
||||
import type { CaptchaStep, Errand } from "./types.js";
|
||||
|
||||
const page = (url: string, title = "", text = "", errors: string[] = []) => ({ url, title, text, errors, fields: [] });
|
||||
|
||||
describe("which step a page is", () => {
|
||||
const e = ftbExample();
|
||||
it("a wait step whose title and selector both fit", () => {
|
||||
expect(stepFor(e, page("https://webapp.ftb.ca.gov/x", "Challenge Validation"), { "#sec-cpt-if": true })?.id).toBe("bot-check");
|
||||
// All given parts must fit: the title alone is not enough.
|
||||
expect(stepFor(e, page("https://webapp.ftb.ca.gov/x", "Challenge Validation"), { "#sec-cpt-if": false })?.id).toBe("form");
|
||||
});
|
||||
|
||||
it("falls back to the page step with no match", () => {
|
||||
expect(stepFor(e, page("https://webapp.ftb.ca.gov/MyFTBAccess/Registration/NewAccount", "Registration"), {})?.id).toBe("form");
|
||||
});
|
||||
|
||||
it("an identity provider's origin is its gate", () => {
|
||||
const withId: Errand = { ...e, steps: [...e.steps, { id: "id-me", kind: "identity-proofing", provider: "ID.me", origins: ["https://api.id.me"], why: "yours" }] };
|
||||
expect(stepFor(withId, page("https://api.id.me/en/session"), {})?.id).toBe("id-me");
|
||||
});
|
||||
});
|
||||
|
||||
describe("outcomes", () => {
|
||||
const e = ftbExample();
|
||||
it("tests rejected outcomes first, against the errors when there are any", () => {
|
||||
const p = page("https://webapp.ftb.ca.gov/c", "Confirmation", "Registration confirmation", ["The information does not match our records."]);
|
||||
expect(outcomeOf(e, p)?.name).toBe("rejected");
|
||||
});
|
||||
|
||||
it("reads the page text when there are no errors", () => {
|
||||
expect(outcomeOf(e, page("https://webapp.ftb.ca.gov/c", "x", "Registration Confirmation. We will mail you a PIN."))?.name).toBe("registered");
|
||||
expect(outcomeOf(e, page("https://webapp.ftb.ca.gov/c", "x", "Please enter your name"))).toBeNull();
|
||||
});
|
||||
|
||||
it("needs both text and url when both are given", () => {
|
||||
const both: Errand = { ...e, outcomes: [{ name: "done", kind: "success", text: "done", url: "/finished$" }] };
|
||||
expect(outcomeOf(both, page("https://webapp.ftb.ca.gov/finished", "", "done"))?.name).toBe("done");
|
||||
expect(outcomeOf(both, page("https://webapp.ftb.ca.gov/other", "", "done"))).toBeNull();
|
||||
});
|
||||
|
||||
it("knows a lockout by the file's signature or the default", () => {
|
||||
expect(isLockout(e, "Your account has been locked for 30 minutes")).toBe(true);
|
||||
expect(isLockout(e, "You have exceeded the allowed number of attempts")).toBe(true);
|
||||
expect(isLockout(e, "Welcome")).toBe(false);
|
||||
const own: Errand = { ...e, metadata: { lockout: { text: "come back tomorrow", duration: "PT24H" } } };
|
||||
expect(isLockout(own, "Please come back tomorrow")).toBe(true);
|
||||
expect(isLockout(own, "account locked")).toBe(false);
|
||||
expect(lockoutMs(own)).toBe(24 * 3_600_000);
|
||||
expect(lockoutMs(e)).toBe(LIMITS.lockoutMs);
|
||||
});
|
||||
});
|
||||
|
||||
describe("captcha solver gating", () => {
|
||||
const captcha: CaptchaStep = { id: "cap", kind: "captcha", match: { selector: ".g-recaptcha" }, solver: "allowed" };
|
||||
const commercial = (over: Partial<Errand> = {}): Errand => ({
|
||||
type: "logicsrc.openerrand",
|
||||
version: "0.1",
|
||||
name: "newsletter",
|
||||
title: "Sign up",
|
||||
site: { name: "Shop", sector: "commercial", origins: ["https://shop.example"], start: ["https://shop.example/"] },
|
||||
inputs: { email: { type: "email", sensitivity: "personal", sources: [{ from: "prompt" }] } },
|
||||
steps: [{ id: "form", kind: "page" }, captcha],
|
||||
outcomes: [{ name: "ok", kind: "success", text: "thanks" }],
|
||||
...over,
|
||||
});
|
||||
|
||||
it("allows a solver only on a commercial or other site with nothing sensitive, as the file says", () => {
|
||||
expect(validateErrand(commercial())).toEqual([]);
|
||||
expect(solverPermitted(commercial(), captcha)).toBe(true);
|
||||
expect(solverPermitted(commercial(), { ...captcha, solver: "forbidden" })).toBe(false);
|
||||
expect(solverPermitted(commercial(), { ...captcha, solver: undefined })).toBe(false);
|
||||
});
|
||||
|
||||
for (const sector of ["government", "tax", "financial", "healthcare", "identity-provider"] as const) {
|
||||
it(`never on a ${sector} site, even when the file says allowed (and the validator rejects the file)`, () => {
|
||||
const e = commercial({ site: { name: "x", sector, origins: ["https://x.example"], start: ["https://x.example/"] } });
|
||||
expect(solverPermitted(e, captcha)).toBe(false);
|
||||
expect(validateErrand(e).join(" ")).toMatch(/captcha solver is never allowed/);
|
||||
});
|
||||
}
|
||||
|
||||
it("never without a stated sector, with a secret input, or with a declaration", () => {
|
||||
expect(solverPermitted(commercial({ site: { name: "x", origins: ["https://x.example"], start: ["https://x.example/"] } }), captcha)).toBe(false);
|
||||
expect(solverPermitted(commercial({ inputs: { pw: { type: "string", sensitivity: "secret", sources: [{ from: "prompt" }] } } }), captcha)).toBe(false);
|
||||
expect(solverPermitted(commercial({ steps: [{ id: "form", kind: "page" }, captcha, { id: "d", kind: "declare", statement: "x", why: "y" }] }), captcha)).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe("throttle", () => {
|
||||
const e = ftbExample();
|
||||
const key = keyFor(e);
|
||||
const t0 = new Date("2026-10-04T10:00:00Z");
|
||||
const at = (min: number) => new Date(t0.getTime() + min * 60_000);
|
||||
|
||||
it("spaces runs on one site 2 minutes apart", () => {
|
||||
const l = recordAttempt(emptyLedger(), key, t0);
|
||||
expect(checkThrottle(l, key, at(1))).toMatchObject({ ok: false, lockout: false });
|
||||
expect(checkThrottle(l, { ...key, account: "other", errand: "ftb-activate-business" }, at(1)).ok).toBe(false);
|
||||
expect(checkThrottle(l, key, at(2)).ok).toBe(true);
|
||||
});
|
||||
|
||||
it("allows 2 runs in 30 minutes and 4 a day per errand and account", () => {
|
||||
let l = recordAttempt(emptyLedger(), key, t0);
|
||||
l = recordAttempt(l, key, at(3));
|
||||
expect(checkThrottle(l, key, at(10))).toMatchObject({ ok: false, reason: expect.stringMatching(/2 runs/) });
|
||||
expect(checkThrottle(l, { ...key, account: "personal" }, at(10)).ok).toBe(true);
|
||||
l = recordAttempt(l, key, at(40));
|
||||
l = recordAttempt(l, key, at(80));
|
||||
expect(checkThrottle(l, key, at(200))).toMatchObject({ ok: false, reason: expect.stringMatching(/4 runs/) });
|
||||
expect(checkThrottle(l, key, at(24 * 60 + 1)).ok).toBe(true);
|
||||
});
|
||||
|
||||
it("--force lifts the caps and never a lockout", () => {
|
||||
let l = recordAttempt(emptyLedger(), key, t0);
|
||||
expect(checkThrottle(l, key, at(1), true).ok).toBe(true);
|
||||
l = recordLockout(l, key, at(1));
|
||||
expect(checkThrottle(l, key, at(10), true)).toMatchObject({ ok: false, lockout: true });
|
||||
// The lock is per site and account: another errand on the same account is held too.
|
||||
expect(checkThrottle(l, { ...key, errand: "ftb-activate-business" }, at(10), true)).toMatchObject({ ok: false, lockout: true });
|
||||
expect(checkThrottle(l, key, at(1 + 35), true).ok).toBe(true);
|
||||
});
|
||||
|
||||
it("a shorter later lock never shortens a longer one", () => {
|
||||
let l = recordLockout(emptyLedger(), key, t0, 24 * 3_600_000);
|
||||
l = recordLockout(l, key, at(5));
|
||||
expect(checkThrottle(l, key, at(60), true).ok).toBe(false);
|
||||
});
|
||||
});
|
||||
103
packages/openerrand/src/pages.ts
Normal file
103
packages/openerrand/src/pages.ts
Normal file
|
|
@ -0,0 +1,103 @@
|
|||
/**
|
||||
* Reading a page against the errand: which step it is, and whether it is an
|
||||
* outcome. Pure; the driver supplies the page and which CSS selectors hit.
|
||||
*/
|
||||
|
||||
import { NO_SOLVER_SECTORS } from "@logicsrc/validators";
|
||||
import type { CaptchaStep, Errand, GateStep, IdentityStep, Match, Outcome, Page, Step } from "./types.js";
|
||||
import { originOf, re } from "./util.js";
|
||||
|
||||
/** Every CSS selector the errand's matches name, so the driver can test them in one evaluation. */
|
||||
export function selectorsOf(errand: Errand): string[] {
|
||||
const out = new Set<string>();
|
||||
for (const step of errand.steps) {
|
||||
const match = (step as { match?: Match }).match;
|
||||
if (match?.selector) out.add(match.selector);
|
||||
}
|
||||
return [...out];
|
||||
}
|
||||
|
||||
/** All the given parts of a match must fit. */
|
||||
export function fits(match: Match, page: Pick<Page, "url" | "title" | "text">, hits: Record<string, boolean>): boolean {
|
||||
if (match.url !== undefined && !re(match.url).test(page.url)) return false;
|
||||
if (match.title !== undefined && !re(match.title).test(page.title)) return false;
|
||||
if (match.text !== undefined && !re(match.text).test(page.text)) return false;
|
||||
if (match.selector !== undefined && !hits[match.selector]) return false;
|
||||
return true;
|
||||
}
|
||||
|
||||
/**
|
||||
* The step for a page: the first step whose match fits, an identity-proofing
|
||||
* gate whose provider origin the page is on, else the first page step with no
|
||||
* match (the fallback). null when nothing fits.
|
||||
*/
|
||||
export function stepFor(errand: Errand, page: Pick<Page, "url" | "title" | "text">, hits: Record<string, boolean>): Step | null {
|
||||
const origin = originOf(page.url);
|
||||
for (const step of errand.steps) {
|
||||
if (step.kind === "identity-proofing" && step.origins.includes(origin)) return step;
|
||||
const match = (step as { match?: Match }).match;
|
||||
if (match && (step.kind === "page" || step.kind === "wait" || step.kind === "captcha" || step.kind === "identity-proofing") && fits(match, page, hits)) return step;
|
||||
}
|
||||
return errand.steps.find((s) => s.kind === "page" && !s.match) ?? null;
|
||||
}
|
||||
|
||||
export function isGate(step: Step): step is GateStep {
|
||||
return step.kind === "declare" || step.kind === "identity-proofing" || step.kind === "code" || step.kind === "mail" || step.kind === "captcha";
|
||||
}
|
||||
|
||||
export function stepById(errand: Errand, id: string): Step | undefined {
|
||||
return errand.steps.find((s) => s.id === id);
|
||||
}
|
||||
|
||||
/** The text an outcome is tested against: the page's error messages, or its text when it shows none. */
|
||||
export function outcomeText(page: Pick<Page, "errors" | "text">): string {
|
||||
return page.errors.length ? page.errors.join(" ") : page.text;
|
||||
}
|
||||
|
||||
/** Rejected outcomes first, then the others in file order. Both `text` and `url` must fit when both are given. */
|
||||
export function outcomeOf(errand: Errand, page: Pick<Page, "url" | "errors" | "text">): Outcome | null {
|
||||
const text = outcomeText(page);
|
||||
const ordered = [...errand.outcomes.filter((o) => o.kind === "rejected"), ...errand.outcomes.filter((o) => o.kind !== "rejected")];
|
||||
for (const outcome of ordered) {
|
||||
if (outcome.text !== undefined && !re(outcome.text).test(text)) continue;
|
||||
if (outcome.url !== undefined && !re(outcome.url).test(page.url)) continue;
|
||||
return outcome;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
/** Origins a run may be on: the site's, plus every identity provider's (only while that gate is handed over). */
|
||||
export function allowedOrigins(errand: Errand): Set<string> {
|
||||
const out = new Set(errand.site.origins);
|
||||
for (const step of errand.steps) if (step.kind === "identity-proofing") for (const o of (step as IdentityStep).origins) out.add(o);
|
||||
return out;
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether a captcha solver may be used for this step: the step says `allowed`
|
||||
* and the errand is outside the forbidden set (a stated sector that is not
|
||||
* government, tax, financial, healthcare or identity-provider; no declare or
|
||||
* identity-proofing step; no secret input). The validator rejects a file that
|
||||
* breaks this, and the runner checks again so a hand-edited file that skipped
|
||||
* validation still cannot reach a solver.
|
||||
*/
|
||||
export function solverPermitted(errand: Errand, step: CaptchaStep): boolean {
|
||||
if (step.solver !== "allowed") return false;
|
||||
const sector = errand.site.sector;
|
||||
if (!sector || (NO_SOLVER_SECTORS as readonly string[]).includes(sector)) return false;
|
||||
if (errand.steps.some((s) => s.kind === "declare" || s.kind === "identity-proofing")) return false;
|
||||
if (Object.values(errand.inputs ?? {}).some((i) => i.sensitivity === "secret")) return false;
|
||||
return true;
|
||||
}
|
||||
|
||||
/** The text that marks a lockout: `metadata.lockout.text` when the file gives one, else a default that covers the common wordings. */
|
||||
export const DEFAULT_LOCKOUT = "account (is |has been )?(temporarily )?locked|locked out|too many (failed |unsuccessful )?attempts|exceeded the (allowed |maximum )?number of attempts";
|
||||
|
||||
export function lockoutPattern(errand: Errand): string {
|
||||
const lockout = (errand.metadata as { lockout?: { text?: unknown } } | undefined)?.lockout;
|
||||
return typeof lockout?.text === "string" && lockout.text ? lockout.text : DEFAULT_LOCKOUT;
|
||||
}
|
||||
|
||||
export function isLockout(errand: Errand, text: string): boolean {
|
||||
return re(lockoutPattern(errand)).test(text);
|
||||
}
|
||||
63
packages/openerrand/src/prompt.ts
Normal file
63
packages/openerrand/src/prompt.ts
Normal file
|
|
@ -0,0 +1,63 @@
|
|||
/**
|
||||
* Terminal prompts. Without a terminal they answer null: the runner then
|
||||
* moves to the next source, or stops and says what it needed, rather than
|
||||
* read a secret from a pipe it cannot see the other end of.
|
||||
*/
|
||||
|
||||
import type { Prompt } from "./inputs.js";
|
||||
|
||||
function readSecret(label: string): Promise<string> {
|
||||
process.stderr.write(label);
|
||||
process.stdin.setRawMode(true);
|
||||
process.stdin.resume();
|
||||
return new Promise((resolve) => {
|
||||
let value = "";
|
||||
const onData = (chunk: Buffer): void => {
|
||||
for (const byte of chunk) {
|
||||
if (byte === 0x0d || byte === 0x0a || byte === 0x04) return finish();
|
||||
if (byte === 0x03) {
|
||||
process.stdin.setRawMode(false);
|
||||
process.stderr.write("\n");
|
||||
process.exit(130);
|
||||
}
|
||||
if (byte === 0x7f || byte === 0x08) {
|
||||
value = value.slice(0, -1);
|
||||
continue;
|
||||
}
|
||||
value += String.fromCharCode(byte);
|
||||
}
|
||||
};
|
||||
const finish = (): void => {
|
||||
process.stdin.off("data", onData);
|
||||
process.stdin.setRawMode(false);
|
||||
process.stdin.pause();
|
||||
process.stderr.write("\n");
|
||||
resolve(value.trim());
|
||||
};
|
||||
process.stdin.on("data", onData);
|
||||
});
|
||||
}
|
||||
|
||||
function readLine(label: string): Promise<string> {
|
||||
process.stderr.write(label);
|
||||
return new Promise((resolve) => {
|
||||
process.stdin.setEncoding("utf8");
|
||||
process.stdin.resume();
|
||||
process.stdin.once("data", (chunk) => {
|
||||
process.stdin.pause();
|
||||
resolve(String(chunk).trim());
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
/** Secrets do not echo. */
|
||||
export const terminalPrompt: Prompt = async (question, { secret }) => {
|
||||
if (!process.stdin.isTTY) return null;
|
||||
return secret ? readSecret(question) : readLine(question);
|
||||
};
|
||||
|
||||
export async function confirm(question: string): Promise<boolean> {
|
||||
if (!process.stdin.isTTY) return false;
|
||||
const answer = await readLine(`${question} [y/N] `);
|
||||
return /^y(es)?$/i.test(answer);
|
||||
}
|
||||
156
packages/openerrand/src/rules.test.ts
Normal file
156
packages/openerrand/src/rules.test.ts
Normal file
|
|
@ -0,0 +1,156 @@
|
|||
import { describe, expect, it } from "vitest";
|
||||
import { resolveInputs } from "./inputs.js";
|
||||
import { decide } from "./rules.js";
|
||||
import { field, ftbExample } from "./testing.js";
|
||||
import type { Errand, Rule } from "./types.js";
|
||||
|
||||
function errand(inputs: Errand["inputs"], rules: Rule[] = []): Errand {
|
||||
return {
|
||||
type: "logicsrc.openerrand",
|
||||
version: "0.1",
|
||||
name: "t",
|
||||
title: "t",
|
||||
site: { name: "Example", origins: ["https://example.com"], start: ["https://example.com/"] },
|
||||
inputs,
|
||||
rules,
|
||||
steps: [{ id: "form", kind: "page" }],
|
||||
outcomes: [{ name: "ok", kind: "success", text: "done" }],
|
||||
};
|
||||
}
|
||||
|
||||
async function ftbInputs() {
|
||||
const e = ftbExample();
|
||||
return {
|
||||
errand: e,
|
||||
inputs: await resolveInputs(e, {
|
||||
overrides: {
|
||||
email: "jane@example.com",
|
||||
phone: "5555550100",
|
||||
first_name: "Jane",
|
||||
last_name: "Doe",
|
||||
street: "1234 Maple St",
|
||||
zip: "95814",
|
||||
corp_id: "1234567",
|
||||
net_income: "48210",
|
||||
tax_year: "2025",
|
||||
},
|
||||
random: () => 0,
|
||||
}),
|
||||
};
|
||||
}
|
||||
|
||||
describe("matching: id first, label second", () => {
|
||||
it("tries every id rule before any label rule, whatever the order in the file", async () => {
|
||||
const e = errand({ name: { type: "string", sensitivity: "personal", sources: [{ from: "literal", value: "Jane" }] } }, [
|
||||
{ name: "by label", label: "name", do: { text: "label-{{name}}" } },
|
||||
{ name: "by id", id: "^FstName$", do: { text: "{{name}}" } },
|
||||
]);
|
||||
const inputs = await resolveInputs(e);
|
||||
const d = decide(e.rules!, field({ id: "FstName", label: "First name" }), inputs);
|
||||
expect(d?.rule.name).toBe("by id");
|
||||
expect(d?.action).toMatchObject({ kind: "text", value: "Jane" });
|
||||
});
|
||||
|
||||
it("an id match is final even when it skips", async () => {
|
||||
const { errand: e, inputs } = await ftbInputs();
|
||||
const d = decide(e.rules!, field({ id: "MInitial", label: "Middle initial name" }), inputs);
|
||||
expect(d?.rule.name).toBe("middle initial");
|
||||
expect(d?.action).toEqual({ kind: "skip" });
|
||||
});
|
||||
|
||||
it("an id match that cannot act leaves the field unmatched rather than trying labels", async () => {
|
||||
const e = errand({ name: { type: "string", sensitivity: "personal", sources: [{ from: "prompt" }] } }, [
|
||||
{ name: "by id", id: "^FstName$", do: { text: "{{name}}" } },
|
||||
{ name: "by label", label: "first", do: { text: "x" } },
|
||||
]);
|
||||
const inputs = await resolveInputs(e);
|
||||
const d = decide(e.rules!, field({ id: "FstName", label: "First name" }), inputs);
|
||||
expect(d?.rule.name).toBe("by id");
|
||||
expect(d?.action).toBeNull();
|
||||
expect(d?.reason).toMatch(/no value/);
|
||||
});
|
||||
|
||||
it("a label match that cannot act lets the next rule try", async () => {
|
||||
const { errand: e, inputs } = await ftbInputs();
|
||||
const radio = field({ id: "RoleInd", name: "Role", type: "radio", label: "Individual" });
|
||||
// The role rule only ticks the business representative radio.
|
||||
expect(decide(e.rules!, radio, inputs)).toBeNull();
|
||||
const business = field({ id: "RoleBus", name: "Role", type: "radio", label: "Business Representative" });
|
||||
expect(decide(e.rules!, business, inputs)?.action).toEqual({ kind: "check" });
|
||||
});
|
||||
|
||||
it("types limit a rule; a rule without types fits any", async () => {
|
||||
const { errand: e, inputs } = await ftbInputs();
|
||||
const select = field({ id: "Yr", type: "select-one", label: "Tax year", options: [{ value: "", text: "Select" }, { value: "2024", text: "2024" }, { value: "2025", text: "2025" }] });
|
||||
expect(decide(e.rules!, select, inputs)?.action).toMatchObject({ kind: "select", value: "2025" });
|
||||
const box = field({ id: "Yr2", type: "tel", label: "Tax year" });
|
||||
expect(decide(e.rules!, box, inputs)?.action).toMatchObject({ kind: "text", value: "2025" });
|
||||
expect(decide(e.rules!, field({ id: "Yr3", type: "checkbox", label: "Tax year" }), inputs)).toBeNull();
|
||||
});
|
||||
|
||||
it("a page step's own rules come first", async () => {
|
||||
const e = errand({ a: { type: "string", sensitivity: "public", sources: [{ from: "literal", value: "A" }] } }, [{ name: "errand", label: "code", do: { text: "errand" } }]);
|
||||
const inputs = await resolveInputs(e);
|
||||
const rules: Rule[] = [{ name: "step", label: "code", do: { text: "{{a}}" } }, ...e.rules!];
|
||||
expect(decide(rules, field({ id: "c", label: "Code" }), inputs)?.rule.name).toBe("step");
|
||||
});
|
||||
|
||||
it("flags secrets and the shared secret so they are masked and a dry run stops there", async () => {
|
||||
const { errand: e, inputs } = await ftbInputs();
|
||||
const d = decide(e.rules!, field({ id: "NetInc", type: "text", label: "Net income for tax purposes" }), inputs);
|
||||
expect(d?.action).toMatchObject({ kind: "text", value: "48210", secret: true, sharedSecret: true });
|
||||
const zip = decide(e.rules!, field({ id: "Zip", type: "text", label: "ZIP code" }), inputs);
|
||||
expect(zip?.action).toMatchObject({ value: "95814", secret: false, sharedSecret: false });
|
||||
});
|
||||
});
|
||||
|
||||
describe("actions", () => {
|
||||
it("split spreads a value over boxes by the digit at the end of the id", async () => {
|
||||
const e = errand({ ssn: { type: "string", sensitivity: "secret", sources: [{ from: "literal", value: "123456789" }] } }, [
|
||||
{ name: "ssn", label: "social security", do: { text: "{{ssn}}", split: [3, 2, 4] } },
|
||||
]);
|
||||
const inputs = await resolveInputs(e);
|
||||
const parts = [1, 2, 3].map((n) => decide(e.rules!, field({ id: `Ssn${n}`, label: "Social security number" }), inputs)?.action);
|
||||
expect(parts.map((p) => (p as { value: string }).value)).toEqual(["123", "45", "6789"]);
|
||||
});
|
||||
|
||||
it("select escapes the input inside a pattern and tries patterns in order", async () => {
|
||||
const e = errand({ v: { type: "string", sensitivity: "public", sources: [{ from: "literal", value: "a.b" }] } }, [
|
||||
{ name: "pick", label: "pick", types: ["select-one"], do: { select: ["^{{v}}$", "fallback"] } },
|
||||
]);
|
||||
const inputs = await resolveInputs(e);
|
||||
const opts = [{ value: "1", text: "aXb" }, { value: "2", text: "a.b" }, { value: "3", text: "fallback" }];
|
||||
expect(decide(e.rules!, field({ id: "p", type: "select-one", label: "Pick", options: opts }), inputs)?.action).toMatchObject({ value: "2" });
|
||||
const noExact = opts.filter((o) => o.value !== "2");
|
||||
expect(decide(e.rules!, field({ id: "p", type: "select-one", label: "Pick", options: noExact }), inputs)?.action).toMatchObject({ value: "3" });
|
||||
});
|
||||
|
||||
it("choose picks a different unused question each time and answer finds its answer", async () => {
|
||||
const { errand: e, inputs } = await ftbInputs();
|
||||
const options = [{ value: "", text: "Select a question" }, { value: "q1", text: "Your first pet?" }, { value: "q2", text: "Your first car?" }, { value: "q3", text: "Your first school?" }];
|
||||
const picks = [1, 2, 3].map((n) => decide(e.rules!, field({ id: `SecQ${n}`, type: "select-one", label: `Security question ${n}`, options }), inputs)?.action);
|
||||
expect(picks.map((p) => (p as { value: string }).value)).toEqual(["q1", "q2", "q3"]);
|
||||
const qa = inputs.qa.get("security")!;
|
||||
expect(Object.keys(qa.answers)).toEqual(["Your first pet?", "Your first car?", "Your first school?"]);
|
||||
const answer = decide(e.rules!, field({ id: "SecA2", type: "text", label: "Answer 2", question: "Your first car?" }), inputs);
|
||||
expect(answer?.action).toMatchObject({ kind: "text", value: qa.answers["Your first car?"], secret: true });
|
||||
});
|
||||
|
||||
it("peeking uses nothing up, and a box that comes back keeps its question", async () => {
|
||||
const { errand: e, inputs } = await ftbInputs();
|
||||
const options = [{ value: "", text: "Select" }, { value: "q1", text: "Pet?" }, { value: "q2", text: "Car?" }];
|
||||
const q1 = field({ id: "SecQ1", type: "select-one", label: "Security question 1", options });
|
||||
expect(decide(e.rules!, q1, inputs, true)?.action).toMatchObject({ value: "q1" });
|
||||
expect(decide(e.rules!, q1, inputs, true)?.action).toMatchObject({ value: "q1" });
|
||||
expect(inputs.qa.get("security")!.answers).toEqual({});
|
||||
expect(decide(e.rules!, q1, inputs)?.action).toMatchObject({ value: "q1" });
|
||||
expect(decide(e.rules!, q1, inputs)?.action).toMatchObject({ value: "q1" });
|
||||
expect(decide(e.rules!, field({ id: "SecQ2", type: "select-one", label: "Security question 2", options }), inputs)?.action).toMatchObject({ value: "q2" });
|
||||
});
|
||||
|
||||
it("a gate action names its step", async () => {
|
||||
const { errand: e, inputs } = await ftbInputs();
|
||||
expect(decide(e.rules!, field({ id: "Decl", type: "checkbox", label: "I declare under penalty of perjury" }), inputs)?.action).toEqual({ kind: "gate", step: "declaration" });
|
||||
expect(decide(e.rules!, field({ id: "Code", type: "text", label: "Enter the verification code" }), inputs)?.action).toEqual({ kind: "gate", step: "text-code" });
|
||||
});
|
||||
});
|
||||
149
packages/openerrand/src/rules.ts
Normal file
149
packages/openerrand/src/rules.ts
Normal file
|
|
@ -0,0 +1,149 @@
|
|||
/**
|
||||
* The field rules: what to do with each control on a page. Pure, so every
|
||||
* matching rule of the spec is a unit test.
|
||||
*
|
||||
* 1. Id first, label second: every rule with an `id` is tried before any
|
||||
* rule's `label`.
|
||||
* 2. The first fitting rule decides. An id match is final even when it skips
|
||||
* or cannot act; a label match that cannot act lets the next rule try.
|
||||
* 3. A page step's own rules come before the errand's (the caller passes them
|
||||
* concatenated in that order).
|
||||
* 4. Choices before text is the driver's job: it calls this for selects,
|
||||
* radios and checkboxes, reads the page again, then calls it for the rest.
|
||||
* 5. An unmatched required field stops the run: the driver's job, given null.
|
||||
*/
|
||||
|
||||
import type { Inputs } from "./inputs.js";
|
||||
import type { Field, Rule } from "./types.js";
|
||||
import { escapeRegExp, re, templateNames } from "./util.js";
|
||||
|
||||
export type Action =
|
||||
| { kind: "text"; value: string; secret: boolean; sharedSecret: boolean }
|
||||
| { kind: "select"; value: string; text: string; secret: boolean; sharedSecret: boolean }
|
||||
| { kind: "check" }
|
||||
| { kind: "skip" }
|
||||
| { kind: "gate"; step: string };
|
||||
|
||||
export interface Decision {
|
||||
rule: Rule;
|
||||
/** null when an id rule matched but could not act: the field is then unmatched, and the reason says why. */
|
||||
action: Action | null;
|
||||
reason?: string;
|
||||
}
|
||||
|
||||
const CHOICE_TYPES = new Set(["select-one", "radio", "checkbox"]);
|
||||
|
||||
export function isChoice(field: Field): boolean {
|
||||
return CHOICE_TYPES.has(field.type);
|
||||
}
|
||||
|
||||
/** The digit at the end of a field's id: `Ssn2` is part 2. */
|
||||
function partIndex(field: Field): number | undefined {
|
||||
const m = /(\d)\s*$/.exec(field.id) ?? /(\d)\s*$/.exec(field.name);
|
||||
return m ? Number(m[1]) : undefined;
|
||||
}
|
||||
|
||||
function templateFlags(template: string, inputs: Inputs): { secret: boolean; sharedSecret: boolean } {
|
||||
const names = templateNames(template);
|
||||
return {
|
||||
secret: names.some((n) => inputs.sensitivity(n) === "secret"),
|
||||
sharedSecret: names.some((n) => n === inputs.sharedSecret),
|
||||
};
|
||||
}
|
||||
|
||||
function fill(template: string, inputs: Inputs): string | null {
|
||||
let missing = false;
|
||||
const value = template.replace(/\{\{\s*([a-z][a-z0-9_.]*)\s*\}\}/g, (_, name: string) => {
|
||||
const v = inputs.get(name);
|
||||
if (v === undefined) missing = true;
|
||||
return v ?? "";
|
||||
});
|
||||
return missing ? null : value;
|
||||
}
|
||||
|
||||
/** What one rule does to one field, or why it cannot. */
|
||||
export function act(rule: Rule, field: Field, inputs: Inputs, peek = false): { action: Action | null; reason?: string } {
|
||||
const how = rule.do;
|
||||
if ("skip" in how) return { action: { kind: "skip" } };
|
||||
if ("gate" in how) return { action: { kind: "gate", step: how.gate } };
|
||||
if ("check" in how) {
|
||||
if (how.check === true) return { action: { kind: "check" } };
|
||||
return re(how.check.label).test(field.label) ? { action: { kind: "check" } } : { action: null, reason: "its label is not the one to tick" };
|
||||
}
|
||||
if ("text" in how) {
|
||||
const value = fill(how.text, inputs);
|
||||
if (value === null) return { action: null, reason: `${how.text} has no value` };
|
||||
const flags = templateFlags(how.text, inputs);
|
||||
if (how.split) {
|
||||
// A value spread over boxes 3, 2 and 4 wide: the box's own digit says which part.
|
||||
const part = partIndex(field);
|
||||
if (part !== undefined && part >= 1 && part <= how.split.length) {
|
||||
const start = how.split.slice(0, part - 1).reduce((a, b) => a + b, 0);
|
||||
return { action: { kind: "text", value: value.slice(start, start + how.split[part - 1]!), ...flags } };
|
||||
}
|
||||
}
|
||||
return { action: { kind: "text", value, ...flags } };
|
||||
}
|
||||
if ("select" in how) {
|
||||
for (const pattern of how.select) {
|
||||
let missing = false;
|
||||
const source = pattern.replace(/\{\{\s*([a-z][a-z0-9_.]*)\s*\}\}/g, (_, name: string) => {
|
||||
const v = inputs.get(name);
|
||||
if (v === undefined) missing = true;
|
||||
return escapeRegExp(v ?? "");
|
||||
});
|
||||
if (missing) continue;
|
||||
const compiled = new RegExp(source, "i");
|
||||
const option = field.options?.find((o) => o.value !== "" && (compiled.test(o.text) || compiled.test(o.value)));
|
||||
if (option) {
|
||||
const flags = how.select.reduce((acc, p) => {
|
||||
const f = templateFlags(p, inputs);
|
||||
return { secret: acc.secret || f.secret, sharedSecret: acc.sharedSecret || f.sharedSecret };
|
||||
}, { secret: false, sharedSecret: false });
|
||||
return { action: { kind: "select", value: option.value, text: option.text, ...flags } };
|
||||
}
|
||||
}
|
||||
return { action: null, reason: "no option matches" };
|
||||
}
|
||||
if ("choose" in how) {
|
||||
const qa = inputs.qa.get(how.choose);
|
||||
const option = qa?.choose(field.options ?? [], peek, field.selector);
|
||||
return option
|
||||
? { action: { kind: "select", value: option.value, text: option.text, secret: false, sharedSecret: false } }
|
||||
: { action: null, reason: "no unused question to choose" };
|
||||
}
|
||||
if ("answer" in how) {
|
||||
const qa = inputs.qa.get(how.answer);
|
||||
const labelIndex = /answer\s*(\d)/i.exec(field.label)?.[1];
|
||||
const answer = qa?.answerFor(`${field.question ?? ""} ${field.label}`, partIndex(field) ?? (labelIndex ? Number(labelIndex) : undefined));
|
||||
return answer ? { action: { kind: "text", value: answer, secret: true, sharedSecret: false } } : { action: null, reason: "no answer for this question" };
|
||||
}
|
||||
return { action: null, reason: "unknown action" };
|
||||
}
|
||||
|
||||
/**
|
||||
* The first rule whose pattern and type fit, and what it does. null when no
|
||||
* rule fits at all. `peek` decides without recording anything (a security
|
||||
* question is not used up), for a caller that only wants to know.
|
||||
*/
|
||||
export function decide(rules: readonly Rule[], field: Field, inputs: Inputs, peek = false): Decision | null {
|
||||
const haystack = `${field.id} ${field.name} ${field.label}`;
|
||||
for (const rule of rules) {
|
||||
if (!rule.id || (rule.types && !rule.types.includes(field.type as never))) continue;
|
||||
if (!re(rule.id).test(field.id)) continue;
|
||||
const { action, reason } = act(rule, field, inputs, peek);
|
||||
return { rule, action, ...(reason ? { reason } : {}) };
|
||||
}
|
||||
for (const rule of rules) {
|
||||
if (!rule.label || (rule.types && !rule.types.includes(field.type as never))) continue;
|
||||
if (!re(rule.label).test(haystack)) continue;
|
||||
const { action } = act(rule, field, inputs, peek);
|
||||
if (action) return { rule, action };
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
/** How a field is named in logs and errors: its label, type and whether it is required. Never its value. */
|
||||
export function describeField(field: Field): string {
|
||||
return `${field.label || field.name || field.id || field.selector} (${field.type}${field.required ? ", required" : ""})`;
|
||||
}
|
||||
420
packages/openerrand/src/run.test.ts
Normal file
420
packages/openerrand/src/run.test.ts
Normal file
|
|
@ -0,0 +1,420 @@
|
|||
import { readFileSync, writeFileSync } from "node:fs";
|
||||
import { describe, expect, it } from "vitest";
|
||||
import type { CaptchaSolver } from "./captcha.js";
|
||||
import { type DocumentRecord, resolveInputs } from "./inputs.js";
|
||||
import { runErrand, type RunDeps } from "./run.js";
|
||||
import { type Store } from "./store.js";
|
||||
import { FakeDriver, type FakePage, field, ftbExample, tempStore } from "./testing.js";
|
||||
import { recordLockout, keyFor } from "./throttle.js";
|
||||
import type { Errand } from "./types.js";
|
||||
import { fileVault, parseEnv } from "./vault.js";
|
||||
|
||||
const O = "https://webapp.ftb.ca.gov";
|
||||
const NOW = new Date("2026-10-04T17:20:11Z");
|
||||
|
||||
/** Fictional returns: Jane Doe, 1234 Maple St, Sacramento 95814, corporation 1234567. */
|
||||
const RECORDS: DocumentRecord[] = [
|
||||
{ form: "CA 540", field: "first name", value: "Jane", year: 2025, file: "2025/540.pdf", page: 1 },
|
||||
{ form: "CA 540", field: "last name", value: "Doe", year: 2025, file: "2025/540.pdf", page: 1 },
|
||||
{ form: "CA 540", field: "street address", value: "1234 Maple St", year: 2025, file: "2025/540.pdf", page: 1 },
|
||||
{ form: "CA 540", field: "ZIP code", value: "95814", year: 2025, file: "2025/540.pdf", page: 1 },
|
||||
{ form: "CA 100S", field: "California corporation number", value: "1234567", year: 2025, file: "2025/100S.pdf", page: 1 },
|
||||
{ form: "CA 100S", field: "line 20", label: "Net income for tax purposes", value: "48210", year: 2025, file: "2025/100S.pdf", page: 3 },
|
||||
{ form: "CA 100S", field: "line 20", label: "Net income for tax purposes", value: "39875", year: 2024, file: "2024/100S.pdf", page: 3 },
|
||||
];
|
||||
|
||||
interface SiteOptions {
|
||||
expectedIncome?: string;
|
||||
code?: string;
|
||||
lockout?: boolean;
|
||||
/** Where Terms goes instead of Profile. */
|
||||
termsTo?: string;
|
||||
extraBusinessField?: boolean;
|
||||
challenge?: number;
|
||||
}
|
||||
|
||||
/** A scripted MyFTB: Terms, Profile, (challenge), Business, Phone, Code, Confirmation. */
|
||||
function site(options: SiteOptions = {}): FakeDriver {
|
||||
const questions = [{ value: "", text: "Select" }, { value: "1", text: "First pet?" }, { value: "2", text: "First car?" }, { value: "3", text: "First school?" }];
|
||||
let challengeReads = 0;
|
||||
const pages: Record<string, FakePage | ((d: FakeDriver) => FakePage)> = {
|
||||
[`${O}/MyFTBAccess/Registration/NewAccount`]: {
|
||||
title: "Registration | Terms",
|
||||
fields: [field({ id: "ReadTerms", type: "checkbox", label: "I have read the terms", required: true }), field({ id: "AcceptTerms", type: "checkbox", label: "I accept", required: true })],
|
||||
},
|
||||
[`${O}/Profile`]: {
|
||||
title: "Registration | Profile",
|
||||
fields: [
|
||||
field({ id: "FstName", label: "First Name", required: true }),
|
||||
field({ id: "MInitial", label: "Middle Initial" }),
|
||||
field({ id: "LstName", label: "Last Name", required: true }),
|
||||
field({ id: "UserName", label: "User Name", required: true }),
|
||||
field({ id: "ReUserName", label: "Confirm User Name", required: true }),
|
||||
field({ id: "Email", type: "email", label: "Email", required: true }),
|
||||
field({ id: "ReEmail", type: "email", label: "Confirm Email", required: true }),
|
||||
field({ id: "Password", type: "password", label: "Password", required: true }),
|
||||
field({ id: "RePassword", type: "password", label: "Confirm Password", required: true }),
|
||||
field({ id: "SecQ1", type: "select-one", label: "Security Question 1", options: questions, required: true }),
|
||||
field({ id: "SecA1", type: "text", label: "Answer 1", required: true, question: "First pet?" }),
|
||||
],
|
||||
},
|
||||
[`${O}/Challenge`]: (d) => {
|
||||
challengeReads += 1;
|
||||
if (challengeReads > (options.challenge ?? 0)) {
|
||||
// The page's own script finished its proof of work and moved on.
|
||||
d.current = `${O}/Business`;
|
||||
return d.page();
|
||||
}
|
||||
return { title: "Challenge Validation", hits: ["#sec-cpt-if"], fields: [] };
|
||||
},
|
||||
[`${O}/Business`]: (d) => ({
|
||||
title: "Registration | Business",
|
||||
errors: d.current.endsWith("?err") ? ["The information you entered does not match our records."] : d.current.endsWith("?locked") ? ["Your account has been locked."] : [],
|
||||
fields: [
|
||||
field({ id: "RoleInd", name: "Role", type: "radio", label: "Individual" }),
|
||||
field({ id: "RoleBus", name: "Role", type: "radio", label: "Business Representative", required: true }),
|
||||
field({ id: "CoType", type: "select-one", label: "Type of Company", options: [{ value: "", text: "Select" }, { value: "C", text: "Corporation" }, { value: "P", text: "Partnership" }] }),
|
||||
field({ id: "FormType", type: "select-one", label: "Form type", options: [{ value: "", text: "Select" }, { value: "100", text: "100" }, { value: "100S", text: "100S" }] }),
|
||||
field({ id: "Year", type: "select-one", label: "Tax year", options: [{ value: "", text: "Select" }, { value: "2025", text: "2025" }, { value: "2024", text: "2024" }] }),
|
||||
field({ id: "CorpNo", label: "California corporation number", required: true }),
|
||||
field({ id: "Zip", label: "ZIP Code", required: true }),
|
||||
field({ id: "AddrNum", label: "Numbers in your mailing address", required: true }),
|
||||
field({ id: "NetInc", label: "Net income for tax purposes", required: true }),
|
||||
field({ id: "Decl", type: "checkbox", label: "I declare under penalty of perjury that the information is true", required: true }),
|
||||
...(options.extraBusinessField ? [field({ id: "Mystery", label: "Favourite colour", required: true })] : []),
|
||||
],
|
||||
}),
|
||||
[`${O}/Phone`]: { title: "Registration | Phone", fields: [field({ id: "Ph", type: "tel", label: "Phone Number", required: true }), field({ id: "Txt", name: "How", type: "radio", label: "Send me a text message" })] },
|
||||
[`${O}/Code`]: (d) => ({
|
||||
title: "Registration | Code",
|
||||
errors: d.current.endsWith("?wrong") ? ["The code you entered is incorrect."] : [],
|
||||
fields: [field({ id: "VerificationCode", label: "Enter the verification code", required: true })],
|
||||
}),
|
||||
[`${O}/Confirmation`]: { title: "Registration Confirmation", text: "Registration Confirmation. Your account was successfully created. We will mail you a PIN.", fields: [] },
|
||||
[`${O}/Locked`]: { title: "Locked", text: "Your account has been locked. Try again in 30 minutes.", fields: [] },
|
||||
};
|
||||
for (const url of [`${O}/Business?err`, `${O}/Business?locked`]) pages[url] = pages[`${O}/Business`]!;
|
||||
pages[`${O}/Code?wrong`] = pages[`${O}/Code`]!;
|
||||
return new FakeDriver(pages, (url, values) => {
|
||||
const path = url.replace(O, "").replace(/\?.*$/, "");
|
||||
if (path === "/MyFTBAccess/Registration/NewAccount") return options.termsTo ?? `${O}/Profile`;
|
||||
if (path === "/Profile") return options.challenge !== undefined ? `${O}/Challenge` : `${O}/Business`;
|
||||
if (path === "/Business") {
|
||||
if (options.lockout) return `${O}/Locked`;
|
||||
if (values["#Decl"] !== true) return `${O}/Business?err`;
|
||||
return values["#NetInc"] === (options.expectedIncome ?? "48210") ? `${O}/Phone` : `${O}/Business?err`;
|
||||
}
|
||||
if (path === "/Phone") return `${O}/Code`;
|
||||
if (path === "/Code") return values["#VerificationCode"] === (options.code ?? "482913") ? `${O}/Confirmation` : `${O}/Code?wrong`;
|
||||
return `${O}/Nowhere`;
|
||||
});
|
||||
}
|
||||
|
||||
async function deps(store: Store, driver: FakeDriver, over: Partial<RunDeps> & { errand?: Errand; candidate?: number } = {}): Promise<RunDeps & { lines: string[] }> {
|
||||
const errand = over.errand ?? ftbExample();
|
||||
const inputs = await resolveInputs(errand, {
|
||||
extractor: { extract: async () => RECORDS },
|
||||
overrides: Object.fromEntries(Object.entries({ email: "jane@example.com", phone: "5555550100" }).filter(([k]) => k in (errand.inputs ?? {}))),
|
||||
now: NOW,
|
||||
random: () => 3,
|
||||
...(over.candidate ? { candidate: over.candidate } : {}),
|
||||
});
|
||||
const lines: string[] = [];
|
||||
return {
|
||||
errand,
|
||||
source: "ftb-register-business.json",
|
||||
sha256: "0".repeat(64),
|
||||
inputs,
|
||||
store,
|
||||
openDriver: async () => driver,
|
||||
declare: false,
|
||||
dryRun: false,
|
||||
headful: false,
|
||||
interactive: false,
|
||||
prompt: async () => null,
|
||||
say: (line) => lines.push(line),
|
||||
now: () => NOW,
|
||||
random: () => 3,
|
||||
vault: null,
|
||||
fallbackVault: fileVault(store.credentialsFile(errand.name)),
|
||||
pollMs: 5,
|
||||
rereadMs: 0,
|
||||
lines,
|
||||
...over,
|
||||
};
|
||||
}
|
||||
|
||||
/** Write the code file when the runner says it is waiting: `codes` in order, one per wait. */
|
||||
function relay(store: Store, d: { say: (line: string) => void; lines: string[] }, codes: string[]): void {
|
||||
const say = d.say;
|
||||
d.say = (line) => {
|
||||
say(line);
|
||||
if (/WAITING for the sms code/.test(line)) {
|
||||
const code = codes.shift();
|
||||
if (code) setTimeout(() => writeFileSync(store.codeFile("ftb-register-business"), `${code}\n`), 20);
|
||||
}
|
||||
};
|
||||
}
|
||||
|
||||
describe("a full run", () => {
|
||||
it("registers with --declare, takes the code from the file after a wrong one, writes the login, and ends on the PIN-letter card", async () => {
|
||||
const store = tempStore();
|
||||
const driver = site();
|
||||
const d = await deps(store, driver, { declare: true });
|
||||
relay(store, d, ["000000", "482913"]);
|
||||
const { record } = await runErrand(d);
|
||||
|
||||
expect(record).toMatchObject({ outcome: "registered", kind: "success", page: `${O}/Confirmation`, handoff: expect.stringMatching(/^pin-letter\//) });
|
||||
expect(record.waiting).toEqual({ step: "pin-letter", what: "MyFTB PIN letter", expires_on: "2026-10-25", resume: "ftb-activate-business" });
|
||||
expect(record.card?.steps[1]).toContain("2026-10-25");
|
||||
expect(record.candidate).toEqual({ year: 2025, form: "CA 100S", field: "line 20", source: "2025/100S.pdf p3" });
|
||||
// Credentials written (to the 0600 fallback here) before success.
|
||||
const saved = parseEnv(readFileSync(store.credentialsFile("ftb-register-business"), "utf8"));
|
||||
expect(saved.FTB_BUSINESS_EMAIL).toBe("jane@example.com");
|
||||
expect(saved.FTB_BUSINESS_PASSWORD).toBe(d.inputs.get("password"));
|
||||
expect(JSON.parse(saved.FTB_BUSINESS_SECURITY_ANSWERS!)).toHaveProperty(["First pet?"]);
|
||||
// The business page was submitted exactly once, with the declaration ticked after the values.
|
||||
expect(driver.submits.filter((s) => s.url.startsWith(`${O}/Business`))).toHaveLength(1);
|
||||
const lines = d.lines.join("\n");
|
||||
expect(lines.indexOf("net income = ••••")).toBeLessThan(lines.indexOf("declaration ticked on your --declare"));
|
||||
expect(lines).toContain("the site said: The code you entered is incorrect.");
|
||||
// Rule 8: no value in the page log or the run record.
|
||||
const log = readFileSync(store.pageLog("ftb-register-business"), "utf8");
|
||||
const record_ = JSON.stringify(record);
|
||||
for (const secret of [d.inputs.get("password")!, "48210", "482913", "jane@example.com", "Maple", "5555550100"]) {
|
||||
expect(log).not.toContain(secret);
|
||||
expect(record_).not.toContain(secret);
|
||||
}
|
||||
expect(lines).not.toContain(d.inputs.get("password")!);
|
||||
expect(lines).not.toContain("48210");
|
||||
expect(store.loadLedger().attempts).toHaveLength(1);
|
||||
expect(driver.closed).toBe(true);
|
||||
});
|
||||
|
||||
it("waits out a challenge the page clears by itself", async () => {
|
||||
const store = tempStore();
|
||||
const d = await deps(store, site({ challenge: 2 }), { declare: true });
|
||||
relay(store, d, ["482913"]);
|
||||
const errand = ftbExample();
|
||||
errand.steps[0] = { ...errand.steps[0]!, poll: "PT0S" } as Errand["steps"][number];
|
||||
const { record } = await runErrand({ ...d, errand });
|
||||
expect(record.reason).toBeUndefined();
|
||||
expect(record.kind).toBe("success");
|
||||
expect(d.lines.join("\n")).toContain("proof of work");
|
||||
});
|
||||
});
|
||||
|
||||
describe("the declare gate", () => {
|
||||
it("without --declare stops on the page, prints the statement word for word, and submits nothing there", async () => {
|
||||
const store = tempStore();
|
||||
const driver = site();
|
||||
const d = await deps(store, driver);
|
||||
const { record } = await runErrand(d);
|
||||
expect(record).toMatchObject({ kind: "stopped", reason: expect.stringMatching(/^gate: declaration needs --declare/) });
|
||||
expect(driver.submits.map((s) => s.url)).not.toContain(`${O}/Business`);
|
||||
expect(driver.fills.find((f) => f.selector === "#Decl")).toBeUndefined();
|
||||
expect(d.lines).toContain(' "I declare under penalty of perjury that the information is true"');
|
||||
});
|
||||
});
|
||||
|
||||
describe("the dry run", () => {
|
||||
it("fills up to the shared-secret page, shows it masked, and stops before its forward button without counting an attempt", async () => {
|
||||
const store = tempStore();
|
||||
const driver = site();
|
||||
const { record } = await runErrand(await deps(store, driver, { dryRun: true }));
|
||||
expect(record).toMatchObject({ kind: "dry-run", page: `${O}/Business` });
|
||||
expect(driver.submits.map((s) => s.url)).toEqual([`${O}/MyFTBAccess/Registration/NewAccount`, `${O}/Profile`]);
|
||||
expect(driver.fills.find((f) => f.selector === "#Decl")).toBeUndefined();
|
||||
expect(store.loadLedger().attempts).toHaveLength(0);
|
||||
});
|
||||
});
|
||||
|
||||
describe("a rejected shared secret", () => {
|
||||
it("ends the run, retries nothing, and lists the other candidates for the person to choose", async () => {
|
||||
const store = tempStore();
|
||||
const driver = site({ expectedIncome: "39875" });
|
||||
const d = await deps(store, driver, { declare: true });
|
||||
const { record } = await runErrand(d);
|
||||
expect(record).toMatchObject({ outcome: "rejected", kind: "rejected" });
|
||||
expect(driver.submits.filter((s) => s.url.startsWith(`${O}/Business`))).toHaveLength(1);
|
||||
expect(record.others).toEqual([{ index: 2, year: 2024, form: "CA 100S", field: "line 20", source: "2024/100S.pdf p3" }]);
|
||||
expect(d.lines).toContain("Nothing was retried.");
|
||||
expect(JSON.stringify(record)).not.toContain("39875");
|
||||
});
|
||||
|
||||
it("the next run, with --candidate 2, submits the person's choice", async () => {
|
||||
const store = tempStore();
|
||||
const driver = site({ expectedIncome: "39875" });
|
||||
const d = await deps(store, driver, { declare: true, candidate: 2 });
|
||||
relay(store, d, ["482913"]);
|
||||
expect((await runErrand(d)).record.kind).toBe("success");
|
||||
});
|
||||
});
|
||||
|
||||
describe("lockouts and the throttle", () => {
|
||||
it("records a lockout page and refuses the next run even with --force", async () => {
|
||||
const store = tempStore();
|
||||
const { record } = await runErrand(await deps(store, site({ lockout: true }), { declare: true }));
|
||||
expect(record).toMatchObject({ kind: "rejected" });
|
||||
expect(Object.keys(store.loadLedger().lockedUntil)).toEqual(["https://webapp.ftb.ca.gov|default"]);
|
||||
const later = new Date(NOW.getTime() + 10 * 60_000);
|
||||
const driver = site();
|
||||
const again = await runErrand(await deps(store, driver, { declare: true, force: true, now: () => later }));
|
||||
expect(again.record).toMatchObject({ kind: "stopped", reason: expect.stringMatching(/^throttle: the site locked/) });
|
||||
expect(driver.visits).toEqual([]);
|
||||
});
|
||||
|
||||
it("refuses a third run inside 30 minutes, and --force lifts that cap", async () => {
|
||||
const store = tempStore();
|
||||
for (const min of [0, 3]) await runErrand(await deps(store, site(), { now: () => new Date(NOW.getTime() + min * 60_000) }));
|
||||
const at = () => new Date(NOW.getTime() + 6 * 60_000);
|
||||
expect((await runErrand(await deps(store, site(), { now: at }))).record.reason).toMatch(/^throttle: 2 runs/);
|
||||
expect((await runErrand(await deps(store, site(), { now: at, force: true }))).record.reason).toMatch(/^gate/);
|
||||
});
|
||||
|
||||
it("a lockout recorded by another errand on the same site and account holds this one", async () => {
|
||||
const store = tempStore();
|
||||
const other = { ...ftbExample(), name: "ftb-activate-business" };
|
||||
store.saveLedger(recordLockout(store.loadLedger(), keyFor(other), NOW));
|
||||
expect((await runErrand(await deps(store, site(), { force: true }))).record.reason).toMatch(/locked/);
|
||||
});
|
||||
});
|
||||
|
||||
describe("stops", () => {
|
||||
it("an unmatched required field stops the run and names it", async () => {
|
||||
const store = tempStore();
|
||||
const { record } = await runErrand(await deps(store, site({ extraBusinessField: true }), { declare: true }));
|
||||
expect(record.reason).toMatch(/^unmatched: no rule fills Favourite colour \(text, required\) \(id "Mystery"/);
|
||||
});
|
||||
|
||||
it("a loop stops the run", async () => {
|
||||
const store = tempStore();
|
||||
const { record } = await runErrand(await deps(store, site({ termsTo: `${O}/MyFTBAccess/Registration/NewAccount` })));
|
||||
expect(record.reason).toMatch(/^loop: the same page came back 3 times/);
|
||||
});
|
||||
|
||||
it("leaving the site stops the run", async () => {
|
||||
const store = tempStore();
|
||||
const { record } = await runErrand(await deps(store, site({ termsTo: "https://evil.example/" })));
|
||||
expect(record.reason).toMatch(/^off-site/);
|
||||
});
|
||||
|
||||
it("a validation error on a later page ends the run as rejected", async () => {
|
||||
const store = tempStore();
|
||||
const driver = site();
|
||||
// Profile submits; the business page then shows an error because nothing was declared on a page that has no declare gate.
|
||||
const errand = ftbExample();
|
||||
errand.outcomes = errand.outcomes.filter((o) => o.kind !== "rejected");
|
||||
const d = await deps(store, driver, { errand, declare: true, candidate: 2 });
|
||||
const { record } = await runErrand(d);
|
||||
expect(record).toMatchObject({ kind: "rejected", outcome: "page-errors" });
|
||||
});
|
||||
});
|
||||
|
||||
describe("identity proofing", () => {
|
||||
const errand = (): Errand => ({
|
||||
...ftbExample(),
|
||||
steps: [...ftbExample().steps, { id: "id-me", kind: "identity-proofing", provider: "ID.me", origins: ["https://api.id.me"], handoff: "pin-letter", why: "ID.me may ask for a selfie. That is yours to do." }],
|
||||
});
|
||||
|
||||
it("headless, it stops and tells the person, without reading the provider's page", async () => {
|
||||
const store = tempStore();
|
||||
const driver = site({ termsTo: "https://api.id.me/session" });
|
||||
driver.pages["https://api.id.me/session"] = { title: "ID.me", fields: [field({ id: "selfie", label: "Take a selfie", required: true })] };
|
||||
const readsBefore = () => driver.reads;
|
||||
const d = await deps(store, driver, { errand: errand() });
|
||||
const { record } = await runErrand(d);
|
||||
expect(record.reason).toMatch(/^gate: identity proofing with ID.me is yours/);
|
||||
expect(d.lines.join("\n")).toContain("ID.me: ID.me may ask for a selfie");
|
||||
expect(driver.fills.filter((f) => f.url.startsWith("https://api.id.me"))).toEqual([]);
|
||||
expect(readsBefore()).toBe(2); // Terms read twice (choices, then text); never the provider's page.
|
||||
});
|
||||
|
||||
it("headful at a terminal, it waits untouched until the site is back", async () => {
|
||||
const store = tempStore();
|
||||
const driver = site({ termsTo: "https://api.id.me/session" });
|
||||
driver.pages["https://api.id.me/session"] = { title: "ID.me", fields: [] };
|
||||
setTimeout(() => (driver.current = `${O}/Profile`), 30);
|
||||
const d = await deps(store, driver, { errand: errand(), headful: true, interactive: true });
|
||||
const { record } = await runErrand(d);
|
||||
expect(record.reason).toMatch(/^gate: declaration/);
|
||||
expect(driver.fills.filter((f) => f.url.startsWith("https://api.id.me"))).toEqual([]);
|
||||
});
|
||||
});
|
||||
|
||||
describe("captcha", () => {
|
||||
const commercial = (sector: "commercial" | "tax" = "commercial"): Errand => ({
|
||||
type: "logicsrc.openerrand",
|
||||
version: "0.1",
|
||||
name: "newsletter",
|
||||
title: "Sign up",
|
||||
site: { name: "Shop", sector, origins: ["https://shop.example"], start: ["https://shop.example/"] },
|
||||
inputs: { email: { type: "email", sensitivity: "personal", sources: [{ from: "literal", value: "jane@example.com" }] } },
|
||||
rules: [{ name: "email", label: "email", do: { text: "{{email}}" } }],
|
||||
steps: [{ id: "cap", kind: "captcha", match: { selector: ".g-recaptcha" }, solver: "allowed", why: "a test for people" }, { id: "form", kind: "page" }],
|
||||
outcomes: [{ name: "ok", kind: "success", text: "thanks" }],
|
||||
});
|
||||
const shop = (): FakeDriver => {
|
||||
let solved = false;
|
||||
const driver = new FakeDriver(
|
||||
{
|
||||
"https://shop.example/": () => (solved ? { title: "Sign up", fields: [field({ id: "e", label: "Email", required: true })] } : { title: "Check", hits: [".g-recaptcha"], fields: [] }),
|
||||
"https://shop.example/thanks": { title: "Thanks", text: "thanks", fields: [] },
|
||||
},
|
||||
() => "https://shop.example/thanks",
|
||||
);
|
||||
(driver as unknown as { solve: () => void }).solve = () => (solved = true);
|
||||
return driver;
|
||||
};
|
||||
|
||||
it("headless with no solver: hands it to the person and stops", async () => {
|
||||
const store = tempStore();
|
||||
const d = await deps(store, shop(), { errand: commercial() });
|
||||
expect((await runErrand(d)).record.reason).toMatch(/^gate: a captcha is the person's/);
|
||||
});
|
||||
|
||||
it("uses a given solver only where the errand allows one, and logs the use without the answer", async () => {
|
||||
const store = tempStore();
|
||||
const driver = shop();
|
||||
const used: string[] = [];
|
||||
const solver: CaptchaSolver = { service: "test-solver", solve: async (c) => (used.push(c.url), (driver as unknown as { solve: () => void }).solve(), true) };
|
||||
const d = await deps(store, driver, { errand: commercial(), solver });
|
||||
expect((await runErrand(d)).record.kind).toBe("success");
|
||||
expect(used).toEqual(["https://shop.example/"]);
|
||||
expect(readFileSync(store.pageLog("newsletter"), "utf8")).toContain('"captcha":{"url":"https://shop.example/","service":"test-solver"}');
|
||||
});
|
||||
|
||||
it("never calls a solver on a tax site, even when one is given and the file says allowed", async () => {
|
||||
const store = tempStore();
|
||||
const used: string[] = [];
|
||||
const solver: CaptchaSolver = { service: "test-solver", solve: async (c) => (used.push(c.url), true) };
|
||||
const d = await deps(store, shop(), { errand: commercial("tax"), solver });
|
||||
expect((await runErrand(d)).record.reason).toMatch(/^gate: a captcha/);
|
||||
expect(used).toEqual([]);
|
||||
});
|
||||
});
|
||||
|
||||
describe("resuming", () => {
|
||||
it("the errand a mail gate named marks that card done when it succeeds", async () => {
|
||||
const store = tempStore();
|
||||
store.saveRun({ errand: "x", name: "ftb-register-business", title: "t", sha256: "0", outcome: "registered", kind: "success", at: "2026-10-01T00:00:00.000Z", log: "", waiting: { step: "pin-letter", what: "PIN", resume: "ftb-activate-business" }, card: { id: "pin-letter/abc234", title: "PIN", steps: ["x"] } });
|
||||
const activate: Errand = { ...ftbExample(), name: "ftb-activate-business", inputs: {}, rules: [], steps: [{ id: "form", kind: "page" }], outcomes: [{ name: "activated", kind: "success", text: "activated" }], outputs: {}, handoffs: {} };
|
||||
const driver = new FakeDriver({ [`${O}/MyFTBAccess/Registration/NewAccount`]: { title: "Done", text: "Your account is activated.", fields: [] } }, () => "");
|
||||
const { record } = await runErrand(await deps(store, driver, { errand: activate }));
|
||||
expect(record.kind).toBe("success");
|
||||
expect(store.runs().find((r) => r.name === "ftb-register-business")?.card?.done).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
describe("the mail gate", () => {
|
||||
it("a rule that hands a field to a mail gate ends the run waiting, with the card", async () => {
|
||||
const store = tempStore();
|
||||
const errand = ftbExample();
|
||||
errand.rules = [{ name: "pin", label: "pin", do: { gate: "pin-letter" } }, ...errand.rules!];
|
||||
const driver = site({ termsTo: `${O}/Pin` });
|
||||
driver.pages[`${O}/Pin`] = { title: "Activate", fields: [field({ id: "Pin", label: "PIN from the letter", required: true })] };
|
||||
const { record } = await runErrand(await deps(store, driver, { errand }));
|
||||
expect(record).toMatchObject({ kind: "waiting", outcome: "pin-letter", waiting: { expires_on: "2026-10-25" } });
|
||||
expect(record.card?.command).toBe("ftb activate business --pin <PIN from the letter>");
|
||||
});
|
||||
});
|
||||
523
packages/openerrand/src/run.ts
Normal file
523
packages/openerrand/src/run.ts
Normal file
|
|
@ -0,0 +1,523 @@
|
|||
/**
|
||||
* The run loop: open the start page, then for each page find its step, test
|
||||
* the outcomes, fill it from the rules, hand every gate to a person, and press
|
||||
* the forward button, until the site answers or the runner has to stop.
|
||||
*
|
||||
* Everything that touches the world comes in through {@link RunDeps}, so the
|
||||
* loop is tested against a scripted fake browser and only the integration
|
||||
* test drives a real Chrome.
|
||||
*/
|
||||
|
||||
import { randomInt } from "node:crypto";
|
||||
import { existsSync, readFileSync, rmSync } from "node:fs";
|
||||
import type { CaptchaSolver } from "./captcha.js";
|
||||
import type { Driver } from "./driver.js";
|
||||
import type { Inputs, Prompt } from "./inputs.js";
|
||||
import { credentialValues, expiresOn, fileDownloads, renderCard, writeCredentials } from "./outputs.js";
|
||||
import { allowedOrigins, fits, isLockout, outcomeOf, selectorsOf, solverPermitted, stepById, stepFor } from "./pages.js";
|
||||
import { type Action, decide, describeField, isChoice } from "./rules.js";
|
||||
import type { Card, RunRecord, Store } from "./store.js";
|
||||
import { checkThrottle, keyFor, lockoutMs, recordAttempt, recordLockout } from "./throttle.js";
|
||||
import type { CaptchaStep, CodeStep, DeclareStep, Errand, Field, IdentityStep, MailStep, Outcome, Page, StopReason, WaitStep } from "./types.js";
|
||||
import { durationMs, ErrandError, MASK, originOf, re, sleep } from "./util.js";
|
||||
import type { Vault } from "./vault.js";
|
||||
|
||||
export interface RunDeps {
|
||||
errand: Errand;
|
||||
/** Where the file came from, and its SHA-256: both go in the run record. */
|
||||
source: string;
|
||||
sha256: string;
|
||||
inputs: Inputs;
|
||||
store: Store;
|
||||
openDriver: () => Promise<Driver>;
|
||||
/** The principal's consent to tick declaration boxes for this run (`--declare`). Never from a file or the environment. */
|
||||
declare: boolean;
|
||||
dryRun: boolean;
|
||||
/** A visible window a person can use. */
|
||||
headful: boolean;
|
||||
/** A person is at a terminal. */
|
||||
interactive: boolean;
|
||||
prompt: Prompt;
|
||||
/** One line for the person, on the terminal (stderr). */
|
||||
say: (line: string) => void;
|
||||
now?: () => Date;
|
||||
random?: (max: number) => number;
|
||||
vault: Vault | null;
|
||||
fallbackVault: Vault;
|
||||
solver?: CaptchaSolver;
|
||||
account?: string;
|
||||
force?: boolean;
|
||||
/** Test knobs: how often files and pages are polled, and the pause between the choice and text passes. */
|
||||
pollMs?: number;
|
||||
rereadMs?: number;
|
||||
}
|
||||
|
||||
export interface RunResult {
|
||||
record: RunRecord;
|
||||
file: string;
|
||||
}
|
||||
|
||||
class Stop extends Error {
|
||||
constructor(
|
||||
readonly reason: StopReason,
|
||||
message: string,
|
||||
readonly page?: string,
|
||||
) {
|
||||
super(message);
|
||||
}
|
||||
}
|
||||
|
||||
/** Run one errand. Never throws for anything the site or the person does; the record says what happened. */
|
||||
export async function runErrand(deps: RunDeps): Promise<RunResult> {
|
||||
const { errand, inputs, store, say } = deps;
|
||||
const now = deps.now ?? (() => new Date());
|
||||
const random = deps.random ?? randomInt;
|
||||
const pollMs = deps.pollMs ?? 2_000;
|
||||
const rereadMs = deps.rereadMs ?? 1_000;
|
||||
const log = store.pageLog(errand.name);
|
||||
const startedAt = now();
|
||||
const base = {
|
||||
errand: errand.id ?? deps.source,
|
||||
name: errand.name,
|
||||
title: errand.title,
|
||||
sha256: deps.sha256,
|
||||
at: startedAt.toISOString(),
|
||||
log,
|
||||
...(inputs.chosen ? { candidate: { ...(inputs.chosen.year !== undefined ? { year: inputs.chosen.year } : {}), form: inputs.chosen.form, field: inputs.chosen.field, source: inputs.chosen.source } } : {}),
|
||||
};
|
||||
const finish = (record: Omit<RunRecord, keyof typeof base> & Partial<RunRecord>): RunResult => {
|
||||
const full = { ...base, ...record } as RunRecord;
|
||||
return { record: full, file: store.saveRun(full) };
|
||||
};
|
||||
|
||||
// The throttle: a dry run creates nothing a site keeps, so it is not an attempt.
|
||||
const key = keyFor(errand, deps.account);
|
||||
if (!deps.dryRun) {
|
||||
const ledger = store.loadLedger();
|
||||
const check = checkThrottle(ledger, key, startedAt, deps.force);
|
||||
if (!check.ok) {
|
||||
say(`Not running: ${check.reason}. Next allowed at ${check.until.toISOString()}.`);
|
||||
return finish({ outcome: "stopped", kind: "stopped", reason: `throttle: ${check.reason}; next at ${check.until.toISOString()}` });
|
||||
}
|
||||
store.saveLedger(recordAttempt(ledger, key, startedAt));
|
||||
}
|
||||
|
||||
const codeFile = store.codeFile(errand.name);
|
||||
rmSync(codeFile, { force: true });
|
||||
|
||||
const limits = { pages: errand.limits?.pages ?? 15, samePage: errand.limits?.same_page ?? 2, pageMs: durationMs(errand.limits?.page_timeout, 30_000) };
|
||||
const selectors = selectorsOf(errand);
|
||||
const allowed = allowedOrigins(errand);
|
||||
const pageRules = (stepRules: Errand["rules"]) => [...(stepRules ?? []), ...(errand.rules ?? [])];
|
||||
|
||||
const logPage = (page: Page, extra: Record<string, unknown> = {}): void => {
|
||||
// Fields without values; the text only of a page with nothing to fill, which is a result page.
|
||||
const fields = page.fields.map(({ selector, type, label, required, options }) => ({ selector, type, label, required, ...(options ? { options: options.slice(0, 15) } : {}) }));
|
||||
store.appendPageLog(errand.name, {
|
||||
at: now().toISOString(),
|
||||
url: page.url,
|
||||
title: page.title,
|
||||
errors: page.errors,
|
||||
fields,
|
||||
...(page.fields.length ? {} : { text: page.text.slice(0, 2000) }),
|
||||
...extra,
|
||||
});
|
||||
};
|
||||
|
||||
let driver: Driver | null = null;
|
||||
try {
|
||||
driver = await deps.openDriver();
|
||||
let opened = false;
|
||||
for (const url of errand.site.start) {
|
||||
try {
|
||||
await driver.goto(url);
|
||||
opened = true;
|
||||
break;
|
||||
} catch (error) {
|
||||
say(` could not open ${url}: ${(error as Error).message}`);
|
||||
}
|
||||
}
|
||||
if (!opened) throw new Stop("error", "none of the start URLs opened");
|
||||
|
||||
let submitted = 0;
|
||||
let lastUrl = "";
|
||||
let comebacks = 0;
|
||||
let codeSubmitted = false;
|
||||
let afterWait = false;
|
||||
let refilled = false;
|
||||
|
||||
for (;;) {
|
||||
// The URL first, read without running script: an identity provider's page is never touched.
|
||||
const url = await driver.url();
|
||||
const origin = originOf(url);
|
||||
if (!errand.site.origins.includes(origin)) {
|
||||
const identity = errand.steps.find((s): s is IdentityStep => s.kind === "identity-proofing" && s.origins.includes(origin));
|
||||
if (identity) {
|
||||
await identityGate(identity, url);
|
||||
continue;
|
||||
}
|
||||
throw new Stop("off-site", `the browser left the site for ${origin || url}`, url);
|
||||
}
|
||||
|
||||
let page = await readSettled(driver);
|
||||
const hits = await driver.selectorHits(selectors);
|
||||
const step = stepFor(errand, page, hits);
|
||||
|
||||
if (step?.kind === "wait") {
|
||||
await waitStep(step);
|
||||
afterWait = true;
|
||||
continue;
|
||||
}
|
||||
if (step?.kind === "captcha") {
|
||||
await captchaGate(step, page.url);
|
||||
continue;
|
||||
}
|
||||
if (step?.kind === "identity-proofing") {
|
||||
await identityGate(step, page.url);
|
||||
continue;
|
||||
}
|
||||
|
||||
logPage(page);
|
||||
const codeFields = page.fields.filter((f) => {
|
||||
const d = decide(pageRules(step?.kind === "page" ? step.rules : undefined), f, inputs, true);
|
||||
return d?.action?.kind === "gate" && stepById(errand, d.action.step)?.kind === "code";
|
||||
});
|
||||
// A wrong or late code leaves the code page up with an error: wait for the next code, do not end the run.
|
||||
const codeRetry = codeSubmitted && page.errors.length > 0 && codeFields.length > 0;
|
||||
|
||||
const outcome = outcomeOf(errand, page);
|
||||
if (outcome?.kind === "rejected") return rejected(outcome, page);
|
||||
if (isLockout(errand, lockoutText(page))) return rejected({ name: "locked", kind: "rejected" }, page);
|
||||
if (outcome) return await succeeded(outcome, page);
|
||||
|
||||
if (page.errors.length && submitted > 0 && (errand.retry?.page_errors ?? "rejected") === "rejected" && !codeRetry) {
|
||||
if (afterWait && !refilled) {
|
||||
// The interstitial replayed the form without its values; the site never saw them. Fill it once more.
|
||||
refilled = true;
|
||||
say(" (the page came back from the bot check without its values; filling it again, once)");
|
||||
} else {
|
||||
return rejected({ name: "page-errors", kind: "rejected" }, page);
|
||||
}
|
||||
}
|
||||
afterWait = false;
|
||||
if (codeRetry) say(` the site said: ${page.errors[0]}`);
|
||||
|
||||
comebacks = page.url === lastUrl ? comebacks + 1 : 0;
|
||||
if (comebacks >= limits.samePage && !codeRetry) throw new Stop("loop", `the same page came back ${comebacks + 1} times: ${page.url} (fields logged to ${log})`, page.url);
|
||||
lastUrl = page.url;
|
||||
if (submitted >= limits.pages) throw new Stop("pages", `more than ${limits.pages} pages (fields logged to ${log})`, page.url);
|
||||
if (!step || step.kind !== "page") throw new Stop("unmatched", `no page step fits ${page.url}`, page.url);
|
||||
|
||||
say(` ${page.title || "(untitled)"} ${page.url}`);
|
||||
if (step.say) say(` ${step.say}`);
|
||||
const rules = pageRules(step.rules);
|
||||
const filled: string[] = [];
|
||||
const missing: Field[] = [];
|
||||
const declares: Array<{ field: Field; gate: DeclareStep }> = [];
|
||||
const codes: Array<{ field: Field; gate: CodeStep }> = [];
|
||||
let sharedSecretHere = false;
|
||||
|
||||
for (const pass of ["choices", "text"] as const) {
|
||||
if (pass === "text") {
|
||||
// A choice can rewrite the page around it; let that land, then read again.
|
||||
await sleep(rereadMs);
|
||||
page = await readSettled(driver);
|
||||
}
|
||||
for (const field of page.fields) {
|
||||
if ((pass === "choices") !== isChoice(field)) continue;
|
||||
const decision = decide(rules, field, inputs);
|
||||
if (!decision || !decision.action) {
|
||||
// A radio group with one already chosen is answered; a required box nothing fills stops the run.
|
||||
if (field.required && !(field.type === "radio" && field.groupChecked)) missing.push(field);
|
||||
continue;
|
||||
}
|
||||
const { rule, action } = decision;
|
||||
if (action.kind === "skip") continue;
|
||||
if (action.kind === "gate") {
|
||||
const gate = stepById(errand, action.step);
|
||||
if (gate?.kind === "declare") declares.push({ field, gate });
|
||||
else if (gate?.kind === "code") codes.push({ field, gate });
|
||||
else if (gate?.kind === "mail") return waitingOnMail(gate, page);
|
||||
else if (gate?.kind === "identity-proofing") await identityGate(gate, page.url);
|
||||
else if (gate?.kind === "captcha") await captchaGate(gate, page.url);
|
||||
continue;
|
||||
}
|
||||
if ((action.kind === "text" || action.kind === "select") && action.sharedSecret) sharedSecretHere = true;
|
||||
if (!(await driver.fill(field, action.kind === "check" ? { kind: "check" } : { kind: action.kind, value: action.value }))) {
|
||||
throw new Stop("error", `could not fill ${describeField(field)} on ${page.url}`, page.url);
|
||||
}
|
||||
filled.push(describeAction(rule.name, field, action));
|
||||
}
|
||||
}
|
||||
|
||||
for (const field of missing) {
|
||||
if (step.unmatched === "ask" && deps.interactive) {
|
||||
const answer = await deps.prompt(`${field.label || field.name}${field.options ? ` [${field.options.map((o) => o.text).join(" | ")}]` : ""}: `, { secret: field.type === "password" });
|
||||
if (answer === null) throw new Stop("unmatched", unmatchedMessage(field, page), page.url);
|
||||
const option = field.options?.find((o) => o.text === answer || o.value === answer);
|
||||
await driver.fill(field, field.type === "checkbox" || field.type === "radio" ? { kind: "check" } : { kind: field.options ? "select" : "text", value: option?.value ?? answer });
|
||||
filled.push(`${field.label || field.name} = (typed)`);
|
||||
continue;
|
||||
}
|
||||
throw new Stop("unmatched", unmatchedMessage(field, page), page.url);
|
||||
}
|
||||
|
||||
for (const line of filled) say(` ${line}`);
|
||||
|
||||
// Rule 12: a dry run stops on the first page holding a declaration or a shared secret, before its forward button.
|
||||
if (deps.dryRun && (declares.length || sharedSecretHere)) {
|
||||
for (const { field, gate } of declares) {
|
||||
say(` declaration (not ticked): "${field.label}"`);
|
||||
say(` ${gate.why}`);
|
||||
}
|
||||
say("Dry run: stopped before the forward button of the first page that sends a declaration or a shared secret.");
|
||||
return finish({ outcome: "dry-run", kind: "dry-run", page: page.url });
|
||||
}
|
||||
|
||||
for (const { field, gate } of declares) {
|
||||
if (!deps.declare) {
|
||||
say("");
|
||||
say(`This page asks the principal to state:`);
|
||||
say(` "${field.label}"`);
|
||||
say(` ${gate.why}`);
|
||||
say("The values above are what would be attested. Nothing was submitted. Rerun with --declare once you have checked them.");
|
||||
throw new Stop("gate", `declaration needs --declare: "${field.label}"`, page.url);
|
||||
}
|
||||
if (!(await driver.fill(field, { kind: "check" }))) throw new Stop("error", `could not tick ${describeField(field)}`, page.url);
|
||||
say(` declaration ticked on your --declare: "${field.label}"`);
|
||||
}
|
||||
|
||||
let filledCode = false;
|
||||
for (const { field, gate } of codes) {
|
||||
const code = await waitForCode(gate);
|
||||
if (!(await driver.fill(field, { kind: "text", value: code }))) throw new Stop("error", `could not fill ${describeField(field)}`, page.url);
|
||||
say(` code = ${MASK}`);
|
||||
filledCode = true;
|
||||
}
|
||||
|
||||
const pressed = await driver.submit();
|
||||
if (pressed.startsWith("?")) throw new Stop("no-forward", `no forward button on ${page.url}; buttons seen: ${pressed.slice(1) || "none"}`, page.url);
|
||||
say(` -> ${pressed}`);
|
||||
submitted += 1;
|
||||
codeSubmitted = filledCode;
|
||||
await driver.settle();
|
||||
}
|
||||
} catch (error) {
|
||||
if (error instanceof Stop) {
|
||||
say(`Stopped (${error.reason}): ${error.message}`);
|
||||
return finish({ outcome: "stopped", kind: "stopped", reason: `${error.reason}: ${error.message}`, ...(error.page ? { page: error.page } : {}) });
|
||||
}
|
||||
const message = (error as Error).message;
|
||||
say(`Stopped (error): ${message}`);
|
||||
return finish({ outcome: "stopped", kind: "stopped", reason: `error: ${message}` });
|
||||
} finally {
|
||||
await driver?.close();
|
||||
}
|
||||
|
||||
// -------------------------------------------------------------------------
|
||||
|
||||
/** A page caught mid-navigation cannot be read; give it a moment, twice, before calling it an error. */
|
||||
async function readSettled(d: Driver): Promise<Page> {
|
||||
for (let attempt = 0; ; attempt += 1) {
|
||||
try {
|
||||
return await d.read();
|
||||
} catch (error) {
|
||||
if (attempt >= 2) throw error;
|
||||
await sleep(Math.max(pollMs, 500));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function unmatchedMessage(field: Field, page: Page): string {
|
||||
return `no rule fills ${describeField(field)} (id "${field.id}", name "${field.name}") on ${page.url}; add a rule, or set "unmatched": "ask" and run at a terminal (fields logged to ${log})`;
|
||||
}
|
||||
|
||||
function rejected(outcome: Outcome, page: Page): RunResult {
|
||||
logPage(page, { outcome: outcome.name });
|
||||
if (!deps.dryRun && isLockout(errand, `${page.title} ${lockoutText(page)}`)) {
|
||||
store.saveLedger(recordLockout(store.loadLedger(), key, now(), lockoutMs(errand)));
|
||||
say(`The site locked the account. Nothing will run against it for ${Math.round(lockoutMs(errand) / 60_000)} minutes, --force or not.`);
|
||||
}
|
||||
say(`Rejected: ${page.errors.join(" ") || page.text.slice(0, 300)}`);
|
||||
const others = inputs.candidates
|
||||
.map((c, i) => ({ index: i + 1, ...(c.year !== undefined ? { year: c.year } : {}), form: c.form, field: c.field, source: c.source }))
|
||||
.filter((c) => !(inputs.chosen && inputs.candidates[c.index - 1] === inputs.chosen));
|
||||
if (inputs.sharedSecret) {
|
||||
// Rule 5: one shared secret per run, never retried. The person chooses the next one.
|
||||
say("Nothing was retried.");
|
||||
if (others.length) {
|
||||
say("Other candidates (choose one for the next run with --candidate N):");
|
||||
for (const c of others) say(` ${c.index}. ${c.year ?? ""} ${c.form} ${c.field} from ${c.source}`.replace(/\s+/g, " "));
|
||||
}
|
||||
}
|
||||
return finish({ outcome: outcome.name, kind: "rejected", page: page.url, ...(inputs.sharedSecret && others.length ? { others } : {}) });
|
||||
}
|
||||
|
||||
async function succeeded(outcome: Outcome, page: Page): Promise<RunResult> {
|
||||
logPage(page, { outcome: outcome.name });
|
||||
// Rule 10: credentials are written before success is reported.
|
||||
const values = credentialValues(errand, outcome, inputs);
|
||||
const saved = await writeCredentials(values, deps.vault, deps.fallbackVault);
|
||||
const downloads = fileDownloads(errand, outcome, inputs, driver?.downloads() ?? []);
|
||||
const then = outcome.then ? stepById(errand, outcome.then) : undefined;
|
||||
let card: Card | undefined;
|
||||
let waiting: RunRecord["waiting"];
|
||||
if (then?.kind === "mail") {
|
||||
const expires = then.expires ? expiresOn(now(), durationMs(then.expires, 0)) : undefined;
|
||||
card = then.handoff ? renderCard(errand, then.handoff, inputs, expires ? { expires_on: expires } : {}, random) : undefined;
|
||||
waiting = { step: then.id, what: then.what, ...(expires ? { expires_on: expires } : {}), ...(then.resume ? { resume: then.resume } : {}) };
|
||||
}
|
||||
const kind = outcome.kind === "waiting" ? "waiting" : "success";
|
||||
// The errand a mail gate said to resume with has succeeded: its card is done.
|
||||
if (kind === "success") {
|
||||
for (const earlier of store.runs()) {
|
||||
if (earlier.waiting?.resume === errand.name && earlier.card && !earlier.card.done) {
|
||||
store.saveRun({ ...earlier, card: { ...earlier.card, done: true } });
|
||||
say(` card ${earlier.card.id} is done`);
|
||||
}
|
||||
}
|
||||
}
|
||||
say(`${kind === "waiting" ? "Waiting" : "Done"}: ${outcome.name} (${errand.title}).`);
|
||||
if (saved.where) say(` credentials saved to ${saved.where}`);
|
||||
if (saved.warning) say(` ${saved.warning}`);
|
||||
for (const d of downloads) say(` filed ${d}`);
|
||||
if (waiting) say(` ${waiting.what}${waiting.expires_on ? `: act before ${waiting.expires_on}` : ""}`);
|
||||
if (card) {
|
||||
say(` card ${card.id} (logicsrc errand status shows it; it is not sent anywhere)`);
|
||||
for (const [i, s] of card.steps.entries()) say(` ${i + 1}. ${s}`);
|
||||
if (card.command) say(` ${card.command}`);
|
||||
}
|
||||
return finish({
|
||||
outcome: outcome.name,
|
||||
kind,
|
||||
page: page.url,
|
||||
...(saved.where ? { vault: saved.where } : {}),
|
||||
...(downloads.length ? { downloads } : {}),
|
||||
...(card ? { handoff: card.id, card } : {}),
|
||||
...(waiting ? { waiting } : {}),
|
||||
});
|
||||
}
|
||||
|
||||
function waitingOnMail(gate: MailStep, page: Page): RunResult {
|
||||
const expires = gate.expires ? expiresOn(now(), durationMs(gate.expires, 0)) : undefined;
|
||||
const card = gate.handoff ? renderCard(errand, gate.handoff, inputs, expires ? { expires_on: expires } : {}, random) : undefined;
|
||||
say(`Waiting: ${gate.what}${expires ? `, act before ${expires}` : ""}.`);
|
||||
if (card) say(` card ${card.id} (logicsrc errand status shows it)`);
|
||||
return finish({
|
||||
outcome: gate.id,
|
||||
kind: "waiting",
|
||||
page: page.url,
|
||||
waiting: { step: gate.id, what: gate.what, ...(expires ? { expires_on: expires } : {}), ...(gate.resume ? { resume: gate.resume } : {}) },
|
||||
...(card ? { handoff: card.id, card } : {}),
|
||||
});
|
||||
}
|
||||
|
||||
/** An interstitial the page clears by itself: poll until its match no longer fits. Never solved, never bypassed. */
|
||||
async function waitStep(step: WaitStep): Promise<void> {
|
||||
say(` ${step.say ?? `waiting for ${step.id} to clear`}`);
|
||||
const until = now().getTime() + durationMs(step.timeout, 90_000);
|
||||
const poll = durationMs(step.poll, 3_000);
|
||||
while (now().getTime() < until) {
|
||||
await sleep(Math.min(poll, pollMs * 5));
|
||||
const url = await driver!.url();
|
||||
if (!errand.site.origins.includes(originOf(url))) return;
|
||||
// The page may be navigating away as it is read; that is the interstitial clearing, so poll again.
|
||||
const page = await driver!.read().catch(() => null);
|
||||
if (!page) continue;
|
||||
if (!fits(step.match, page, await driver!.selectorHits(selectors))) {
|
||||
await sleep(Math.min(1_500, pollMs));
|
||||
return;
|
||||
}
|
||||
}
|
||||
throw new Stop("timeout", `${step.id} did not clear in ${step.timeout}`);
|
||||
}
|
||||
|
||||
/** Identity proofing is the person's: hand them the window, or stop. Nothing on the provider's pages is touched. */
|
||||
async function identityGate(step: IdentityStep, url: string): Promise<void> {
|
||||
say("");
|
||||
say(`${step.provider}: ${step.why}`);
|
||||
if (!(deps.headful && deps.interactive)) {
|
||||
const card = step.handoff ? renderCard(errand, step.handoff, inputs, {}, random) : undefined;
|
||||
if (card) say(` card ${card.id} (logicsrc errand status shows it)`);
|
||||
throw new Stop("gate", `identity proofing with ${step.provider} is yours to do; rerun with --headful at a terminal to do it in the window`, url);
|
||||
}
|
||||
say(` The window is yours. The runner waits, without touching the page, until it is back on ${errand.site.origins.join(" or ")}.`);
|
||||
const until = now().getTime() + durationMs(step.timeout, 30 * 60_000);
|
||||
while (now().getTime() < until) {
|
||||
await sleep(pollMs);
|
||||
const current = await driver!.url();
|
||||
if (errand.site.origins.includes(originOf(current))) return;
|
||||
if (!allowed.has(originOf(current))) throw new Stop("off-site", `the browser left for ${originOf(current) || current}`, current);
|
||||
}
|
||||
throw new Stop("timeout", `identity proofing did not return to the site within ${step.timeout ?? "PT30M"}`, url);
|
||||
}
|
||||
|
||||
/** A captcha is the person's unless the errand allows a solver where the rules permit one, and one was given. */
|
||||
async function captchaGate(step: CaptchaStep, url: string): Promise<void> {
|
||||
if (deps.solver && solverPermitted(errand, step)) {
|
||||
store.appendPageLog(errand.name, { at: now().toISOString(), captcha: { url, service: deps.solver.service } });
|
||||
say(` captcha: using ${deps.solver.service}, as this errand allows (logged)`);
|
||||
if (!(await deps.solver.solve({ url, evaluate: (e) => driver!.evaluate(e) }))) throw new Stop("gate", `${deps.solver.service} did not clear the captcha`, url);
|
||||
await driver!.settle();
|
||||
return;
|
||||
}
|
||||
say("");
|
||||
say(`A captcha: ${step.why ?? "a test meant for a person."}`);
|
||||
if (!(deps.headful && deps.interactive)) throw new Stop("gate", "a captcha is the person's to answer; rerun with --headful at a terminal", url);
|
||||
say(" Answer it in the window; the runner waits until it is gone.");
|
||||
const until = now().getTime() + durationMs(step.timeout, 10 * 60_000);
|
||||
while (now().getTime() < until) {
|
||||
await sleep(pollMs);
|
||||
const current = await driver!.url();
|
||||
if (!errand.site.origins.includes(originOf(current))) return;
|
||||
const page = await driver!.read();
|
||||
if (!fits(step.match, page, await driver!.selectorHits(selectors))) return;
|
||||
}
|
||||
throw new Stop("timeout", "the captcha was not answered in time", url);
|
||||
}
|
||||
|
||||
/**
|
||||
* A one-time code through a declared relay: typed at the terminal, or
|
||||
* written to the code file by whoever holds the phone. Used once, never
|
||||
* logged or shown. A code that does not fit the pattern is ignored and the
|
||||
* wait goes on.
|
||||
*/
|
||||
async function waitForCode(step: CodeStep): Promise<string> {
|
||||
const pattern = step.pattern ?? "^\\w{4,10}$";
|
||||
const timeoutMs = durationMs(step.timeout, 15 * 60_000);
|
||||
if (step.why) say(` ${step.why}`);
|
||||
if (deps.interactive && step.relay.includes("terminal")) {
|
||||
for (;;) {
|
||||
const code = await deps.prompt(`Code sent by ${step.channel}: `, { secret: false });
|
||||
if (code === null) break;
|
||||
if (re(pattern).test(code.trim())) return code.trim();
|
||||
say(" that does not look like the code; try again");
|
||||
}
|
||||
}
|
||||
if (!step.relay.includes("file")) throw new Stop("gate", `the code arrives by ${step.channel} and this errand relays it only through ${step.relay.join(", ")}; nobody is at a terminal`);
|
||||
say(` WAITING for the ${step.channel} code: write it to ${codeFile} (${Math.round(timeoutMs / 60_000)} minutes)`);
|
||||
const until = now().getTime() + timeoutMs;
|
||||
while (now().getTime() < until) {
|
||||
await sleep(pollMs);
|
||||
if (!existsSync(codeFile)) continue;
|
||||
const code = readFileSync(codeFile, "utf8").trim();
|
||||
rmSync(codeFile, { force: true });
|
||||
if (re(pattern).test(code)) return code;
|
||||
}
|
||||
throw new Stop("timeout", `no code arrived in ${step.timeout}`);
|
||||
}
|
||||
}
|
||||
|
||||
/** Where a lockout is read: the errors a page shows, or the text of a result page with nothing to fill (never a form's own warnings). */
|
||||
function lockoutText(page: Page): string {
|
||||
return page.errors.length ? page.errors.join(" ") : page.fields.length ? "" : page.text;
|
||||
}
|
||||
|
||||
function describeAction(name: string, field: Field, action: Action): string {
|
||||
if (action.kind === "text") return `${name} = ${action.secret ? MASK : action.value}`;
|
||||
if (action.kind === "select") return `${name} = ${action.secret ? MASK : action.text}`;
|
||||
if (action.kind === "check") return `${name}${field.type === "radio" ? ` = ${field.label}` : ""}`;
|
||||
return name;
|
||||
}
|
||||
|
||||
export { ErrandError };
|
||||
163
packages/openerrand/src/store.ts
Normal file
163
packages/openerrand/src/store.ts
Normal file
|
|
@ -0,0 +1,163 @@
|
|||
/**
|
||||
* Everything the runner keeps on disk, under one directory only the principal
|
||||
* can read: `$LOGICSRC_ERRAND_HOME`, else `$XDG_DATA_HOME/logicsrc/errand`,
|
||||
* else `~/.local/share/logicsrc/errand`.
|
||||
*
|
||||
* | Path | What |
|
||||
* | --- | --- |
|
||||
* | `throttle.json` | The attempt ledger and lockouts. |
|
||||
* | `approvals.json` | The SHA-256 of each errand file last run, by errand name. |
|
||||
* | `runs/<at>-<name>.json` | One run record per run. No input values, ever. |
|
||||
* | `pages/<name>.jsonl` | The page log: fields (never values) and result-page text. |
|
||||
* | `profiles/<host>/` | One Chrome profile per site, kept between runs. |
|
||||
* | `codes/<name>.code` | Where a one-time code may be written by whoever holds the phone. |
|
||||
* | `credentials/<name>.env` | Credentials, when the errand names no vault (0600). |
|
||||
*
|
||||
* Files are 0600 and directories 0700.
|
||||
*/
|
||||
|
||||
import { appendFileSync, existsSync, mkdirSync, readFileSync, readdirSync, renameSync, writeFileSync } from "node:fs";
|
||||
import { homedir } from "node:os";
|
||||
import { dirname, join } from "node:path";
|
||||
import { emptyLedger, type Ledger } from "./throttle.js";
|
||||
|
||||
export type Env = Record<string, string | undefined>;
|
||||
|
||||
export function errandHome(env: Env = process.env): string {
|
||||
if (env.LOGICSRC_ERRAND_HOME) return env.LOGICSRC_ERRAND_HOME;
|
||||
const data = env.XDG_DATA_HOME || join(env.HOME || homedir(), ".local", "share");
|
||||
return join(data, "logicsrc", "errand");
|
||||
}
|
||||
|
||||
export function ensureDir(path: string): string {
|
||||
mkdirSync(path, { recursive: true, mode: 0o700 });
|
||||
return path;
|
||||
}
|
||||
|
||||
/** Write via a temp file and rename, so a crash never leaves half a ledger. */
|
||||
export function writePrivate(path: string, text: string): void {
|
||||
ensureDir(dirname(path));
|
||||
const tmp = `${path}.${process.pid}.tmp`;
|
||||
writeFileSync(tmp, text, { mode: 0o600 });
|
||||
renameSync(tmp, path);
|
||||
}
|
||||
|
||||
function readJson<T>(path: string, fallback: T): T {
|
||||
try {
|
||||
return JSON.parse(readFileSync(path, "utf8")) as T;
|
||||
} catch {
|
||||
return fallback;
|
||||
}
|
||||
}
|
||||
|
||||
export class Store {
|
||||
readonly home: string;
|
||||
|
||||
constructor(home: string) {
|
||||
this.home = home;
|
||||
}
|
||||
|
||||
static fromEnv(env: Env = process.env): Store {
|
||||
return new Store(errandHome(env));
|
||||
}
|
||||
|
||||
path(...parts: string[]): string {
|
||||
return join(this.home, ...parts);
|
||||
}
|
||||
|
||||
loadLedger(): Ledger {
|
||||
const ledger = readJson<Ledger>(this.path("throttle.json"), emptyLedger());
|
||||
return { attempts: ledger.attempts ?? [], lockedUntil: ledger.lockedUntil ?? {} };
|
||||
}
|
||||
|
||||
saveLedger(ledger: Ledger): void {
|
||||
writePrivate(this.path("throttle.json"), `${JSON.stringify(ledger, null, 2)}\n`);
|
||||
}
|
||||
|
||||
approvals(): Record<string, { sha256: string; file: string; at: string }> {
|
||||
return readJson(this.path("approvals.json"), {});
|
||||
}
|
||||
|
||||
approve(name: string, sha256: string, file: string, at: Date): void {
|
||||
const all = this.approvals();
|
||||
all[name] = { sha256, file, at: at.toISOString() };
|
||||
writePrivate(this.path("approvals.json"), `${JSON.stringify(all, null, 2)}\n`);
|
||||
}
|
||||
|
||||
pageLog(name: string): string {
|
||||
return this.path("pages", `${name}.jsonl`);
|
||||
}
|
||||
|
||||
appendPageLog(name: string, line: unknown): void {
|
||||
const file = this.pageLog(name);
|
||||
ensureDir(dirname(file));
|
||||
appendFileSync(file, `${JSON.stringify(line)}\n`, { mode: 0o600 });
|
||||
}
|
||||
|
||||
profile(origin: string): string {
|
||||
const host = origin.replace(/^https?:\/\//, "").replace(/[^a-z0-9.-]/gi, "_");
|
||||
return ensureDir(this.path("profiles", host));
|
||||
}
|
||||
|
||||
codeFile(name: string): string {
|
||||
ensureDir(this.path("codes"));
|
||||
return this.path("codes", `${name}.code`);
|
||||
}
|
||||
|
||||
credentialsFile(name: string): string {
|
||||
return this.path("credentials", `${name}.env`);
|
||||
}
|
||||
|
||||
downloadsDir(runId: string): string {
|
||||
return ensureDir(this.path("downloads", runId));
|
||||
}
|
||||
|
||||
saveRun(record: RunRecord): string {
|
||||
const file = this.path("runs", `${record.at.replace(/[:.]/g, "-")}-${record.name}.json`);
|
||||
writePrivate(file, `${JSON.stringify(record, null, 2)}\n`);
|
||||
return file;
|
||||
}
|
||||
|
||||
runs(): RunRecord[] {
|
||||
const dir = this.path("runs");
|
||||
if (!existsSync(dir)) return [];
|
||||
return readdirSync(dir)
|
||||
.filter((f) => f.endsWith(".json"))
|
||||
.sort()
|
||||
.map((f) => readJson<RunRecord | null>(join(dir, f), null))
|
||||
.filter((r): r is RunRecord => r !== null);
|
||||
}
|
||||
}
|
||||
|
||||
/** A hand-off card as kept in the run record: built-ins and public inputs only. */
|
||||
export interface Card {
|
||||
id: string;
|
||||
title: string;
|
||||
open?: string;
|
||||
steps: string[];
|
||||
command?: string;
|
||||
expires_on?: string;
|
||||
done?: boolean;
|
||||
}
|
||||
|
||||
/** The record an agent can read after a run. It never holds an input value. */
|
||||
export interface RunRecord {
|
||||
errand: string;
|
||||
name: string;
|
||||
title: string;
|
||||
sha256: string;
|
||||
outcome: string;
|
||||
kind: "success" | "rejected" | "waiting" | "stopped" | "dry-run";
|
||||
reason?: string;
|
||||
at: string;
|
||||
page?: string;
|
||||
candidate?: { year?: number; form: string; field: string; source: string };
|
||||
/** The other candidates for a person to choose from after a rejection (no values). */
|
||||
others?: Array<{ index: number; year?: number; form: string; field: string; source: string }>;
|
||||
handoff?: string;
|
||||
card?: Card;
|
||||
waiting?: { step: string; what: string; expires_on?: string; resume?: string };
|
||||
vault?: string;
|
||||
downloads?: string[];
|
||||
log: string;
|
||||
}
|
||||
135
packages/openerrand/src/testing.ts
Normal file
135
packages/openerrand/src/testing.ts
Normal file
|
|
@ -0,0 +1,135 @@
|
|||
/**
|
||||
* Test helpers: a scripted browser and fixtures. Not part of the build.
|
||||
*/
|
||||
|
||||
import { mkdtempSync, readFileSync, rmSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { dirname, join } from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { afterAll } from "vitest";
|
||||
import type { DownloadedFile, Driver, FillAction } from "./driver.js";
|
||||
import { Store } from "./store.js";
|
||||
import type { Errand, Field, Page } from "./types.js";
|
||||
|
||||
const here = dirname(fileURLToPath(import.meta.url));
|
||||
|
||||
/** The worked example from the spec, as published in @logicsrc/schemas' fixtures. */
|
||||
export function ftbExample(): Errand {
|
||||
return JSON.parse(readFileSync(join(here, "..", "..", "schemas", "fixtures", "openerrand", "ftb-register-business.json"), "utf8")) as Errand;
|
||||
}
|
||||
|
||||
export function ftbExamplePath(): string {
|
||||
return join(here, "..", "..", "schemas", "fixtures", "openerrand", "ftb-register-business.json");
|
||||
}
|
||||
|
||||
const temps: string[] = [];
|
||||
// Registered in each test file that imports this module (vitest isolates modules per file).
|
||||
afterAll(() => {
|
||||
for (const dir of temps.splice(0)) rmSync(dir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
/** A temp dir removed after the test file finishes. */
|
||||
export function tempDir(prefix = "openerrand-test-"): string {
|
||||
const dir = mkdtempSync(join(tmpdir(), prefix));
|
||||
temps.push(dir);
|
||||
return dir;
|
||||
}
|
||||
|
||||
export function tempStore(): Store {
|
||||
return new Store(tempDir());
|
||||
}
|
||||
|
||||
export function field(partial: Partial<Field> & { id: string }): Field {
|
||||
return { selector: `#${partial.id}`, name: partial.id, type: "text", label: "", required: false, ...partial };
|
||||
}
|
||||
|
||||
export interface FakePage {
|
||||
title?: string;
|
||||
text?: string;
|
||||
errors?: string[];
|
||||
fields?: Field[];
|
||||
/** CSS selectors that find something on this page. */
|
||||
hits?: string[];
|
||||
buttons?: string[];
|
||||
}
|
||||
|
||||
/**
|
||||
* A browser over a table of pages. `submit` calls `next(url, values)` to say
|
||||
* where the site goes; `values` maps selector to what was filled (true for a
|
||||
* ticked box). Every fill and submit is recorded so a test can assert that
|
||||
* nothing was sent.
|
||||
*/
|
||||
export class FakeDriver implements Driver {
|
||||
current = "";
|
||||
values: Record<string, string | boolean> = {};
|
||||
readonly fills: Array<{ url: string; selector: string; value: string | boolean }> = [];
|
||||
readonly submits: Array<{ url: string; values: Record<string, string | boolean> }> = [];
|
||||
readonly visits: string[] = [];
|
||||
reads = 0;
|
||||
closed = false;
|
||||
files: DownloadedFile[] = [];
|
||||
|
||||
constructor(
|
||||
readonly pages: Record<string, FakePage | ((driver: FakeDriver) => FakePage)>,
|
||||
readonly next: (url: string, values: Record<string, string | boolean>, driver: FakeDriver) => string,
|
||||
) {}
|
||||
|
||||
page(): FakePage {
|
||||
const p = this.pages[this.current];
|
||||
if (!p) return { title: "Not found", text: "404", fields: [] };
|
||||
return typeof p === "function" ? p(this) : p;
|
||||
}
|
||||
|
||||
async goto(url: string): Promise<void> {
|
||||
this.current = url;
|
||||
this.values = {};
|
||||
this.visits.push(url);
|
||||
}
|
||||
|
||||
async url(): Promise<string> {
|
||||
return this.current;
|
||||
}
|
||||
|
||||
async read(): Promise<Page> {
|
||||
this.reads += 1;
|
||||
const p = this.page();
|
||||
return { url: this.current, title: p.title ?? "", text: p.text ?? "", errors: p.errors ?? [], fields: p.fields ?? [] };
|
||||
}
|
||||
|
||||
async selectorHits(selectors: string[]): Promise<Record<string, boolean>> {
|
||||
const hits = this.page().hits ?? [];
|
||||
return Object.fromEntries(selectors.map((s) => [s, hits.includes(s)]));
|
||||
}
|
||||
|
||||
async fill(f: Field, action: FillAction): Promise<boolean> {
|
||||
const value = action.kind === "check" ? true : action.value;
|
||||
this.values[f.selector] = value;
|
||||
this.fills.push({ url: this.current, selector: f.selector, value });
|
||||
return true;
|
||||
}
|
||||
|
||||
async submit(): Promise<string> {
|
||||
const buttons = this.page().buttons ?? ["Continue"];
|
||||
if (!buttons.length) return "?";
|
||||
this.submits.push({ url: this.current, values: { ...this.values } });
|
||||
const to = this.next(this.current, { ...this.values }, this);
|
||||
this.current = to;
|
||||
this.values = {};
|
||||
this.visits.push(to);
|
||||
return buttons[0]!;
|
||||
}
|
||||
|
||||
async settle(): Promise<void> {}
|
||||
|
||||
downloads(): DownloadedFile[] {
|
||||
return this.files;
|
||||
}
|
||||
|
||||
async evaluate(): Promise<unknown> {
|
||||
return null;
|
||||
}
|
||||
|
||||
async close(): Promise<void> {
|
||||
this.closed = true;
|
||||
}
|
||||
}
|
||||
103
packages/openerrand/src/throttle.ts
Normal file
103
packages/openerrand/src/throttle.ts
Normal file
|
|
@ -0,0 +1,103 @@
|
|||
/**
|
||||
* The attempt ledger. Sites escalate: a tax board restarts its 30-minute lock
|
||||
* on any attempt made inside it, others ban an account or an address for
|
||||
* longer each time. A retry loop turns one lockout into an open-ended one, so
|
||||
* every run that sends something is recorded before it starts, and a run is
|
||||
* refused while a recorded lockout lasts.
|
||||
*
|
||||
* - Window and day caps are per errand and account: 2 runs in 30 minutes, 4 a
|
||||
* day.
|
||||
* - Spacing is per site, whatever the errand or account: 2 minutes between
|
||||
* runs, because the site sees one browser and one phone.
|
||||
* - A lockout is per site and account, so an activation errand respects the
|
||||
* lock a registration errand ran into on the same account. Its length is
|
||||
* `metadata.lockout.duration` when the file gives one, else 35 minutes.
|
||||
* - `--force` lifts the caps and the spacing. It never lifts a lockout.
|
||||
*/
|
||||
|
||||
import type { Errand } from "./types.js";
|
||||
import { durationMs } from "./util.js";
|
||||
|
||||
export interface Attempt {
|
||||
site: string;
|
||||
account: string;
|
||||
errand: string;
|
||||
at: string;
|
||||
}
|
||||
|
||||
export interface Ledger {
|
||||
attempts: Attempt[];
|
||||
/** `<site>|<account>` to the time before which nothing may be sent. */
|
||||
lockedUntil: Record<string, string>;
|
||||
}
|
||||
|
||||
export const LIMITS = {
|
||||
windowMs: 30 * 60_000,
|
||||
perWindow: 2,
|
||||
perDay: 4,
|
||||
spacingMs: 2 * 60_000,
|
||||
/** A 30-minute lock and 5 more, so a skewed clock cannot restart it. */
|
||||
lockoutMs: 35 * 60_000,
|
||||
} as const;
|
||||
|
||||
const DAY = 86_400_000;
|
||||
|
||||
export function emptyLedger(): Ledger {
|
||||
return { attempts: [], lockedUntil: {} };
|
||||
}
|
||||
|
||||
export interface ThrottleKey {
|
||||
site: string;
|
||||
account: string;
|
||||
errand: string;
|
||||
}
|
||||
|
||||
export function keyFor(errand: Errand, account = "default"): ThrottleKey {
|
||||
return { site: errand.site.origins[0]!, account, errand: errand.name };
|
||||
}
|
||||
|
||||
const lockKey = (k: ThrottleKey): string => `${k.site}|${k.account}`;
|
||||
|
||||
export type ThrottleCheck = { ok: true } | { ok: false; until: Date; reason: string; lockout: boolean };
|
||||
|
||||
export function checkThrottle(ledger: Ledger, key: ThrottleKey, now: Date, force = false): ThrottleCheck {
|
||||
const t = now.getTime();
|
||||
const locked = ledger.lockedUntil[lockKey(key)];
|
||||
if (locked && Date.parse(locked) > t) {
|
||||
return { ok: false, lockout: true, until: new Date(locked), reason: `the site locked the ${key.account} account; any attempt before then can restart the lock, and --force does not lift it` };
|
||||
}
|
||||
if (force) return { ok: true };
|
||||
const today = ledger.attempts.map((a) => ({ ...a, ms: Date.parse(a.at) })).filter((a) => t - a.ms < DAY);
|
||||
const onSite = today.filter((a) => a.site === key.site);
|
||||
const last = Math.max(0, ...onSite.map((a) => a.ms));
|
||||
if (last && t - last < LIMITS.spacingMs) {
|
||||
return { ok: false, lockout: false, until: new Date(last + LIMITS.spacingMs), reason: "runs on one site are spaced 2 minutes apart" };
|
||||
}
|
||||
const mine = onSite.filter((a) => a.account === key.account && a.errand === key.errand).sort((a, b) => a.ms - b.ms);
|
||||
const recent = mine.filter((a) => t - a.ms < LIMITS.windowMs);
|
||||
if (recent.length >= LIMITS.perWindow) {
|
||||
return { ok: false, lockout: false, until: new Date(recent[0]!.ms + LIMITS.windowMs), reason: `${LIMITS.perWindow} runs of ${key.errand} in 30 minutes` };
|
||||
}
|
||||
if (mine.length >= LIMITS.perDay) {
|
||||
return { ok: false, lockout: false, until: new Date(mine[0]!.ms + DAY), reason: `${LIMITS.perDay} runs of ${key.errand} today` };
|
||||
}
|
||||
return { ok: true };
|
||||
}
|
||||
|
||||
export function recordAttempt(ledger: Ledger, key: ThrottleKey, now: Date): Ledger {
|
||||
const dayAgo = now.getTime() - DAY;
|
||||
return { ...ledger, attempts: [...ledger.attempts.filter((a) => Date.parse(a.at) > dayAgo), { ...key, at: now.toISOString() }] };
|
||||
}
|
||||
|
||||
export function lockoutMs(errand: Errand): number {
|
||||
const duration = (errand.metadata as { lockout?: { duration?: unknown } } | undefined)?.lockout?.duration;
|
||||
return typeof duration === "string" ? durationMs(duration, LIMITS.lockoutMs) : LIMITS.lockoutMs;
|
||||
}
|
||||
|
||||
export function recordLockout(ledger: Ledger, key: ThrottleKey, now: Date, ms: number = LIMITS.lockoutMs): Ledger {
|
||||
const until = new Date(now.getTime() + ms).toISOString();
|
||||
const current = ledger.lockedUntil[lockKey(key)];
|
||||
// A later lock never shortens an earlier, longer one.
|
||||
const next = current && current > until ? current : until;
|
||||
return { ...ledger, lockedUntil: { ...ledger.lockedUntil, [lockKey(key)]: next } };
|
||||
}
|
||||
210
packages/openerrand/src/types.ts
Normal file
210
packages/openerrand/src/types.ts
Normal file
|
|
@ -0,0 +1,210 @@
|
|||
/**
|
||||
* The errand file as the runner reads it. These mirror
|
||||
* `logicsrc-openerrand.schema.json` in @logicsrc/schemas; the runner never
|
||||
* trusts a file it has not first passed through @logicsrc/validators, so the
|
||||
* types describe a document that is already known to be well formed.
|
||||
*/
|
||||
|
||||
export type Sensitivity = "public" | "personal" | "secret";
|
||||
export type InputType = "string" | "integer" | "number" | "email" | "date" | "boolean" | "qa-set";
|
||||
export type Role = "shared-secret" | "credential" | "identifier";
|
||||
export type Sector = "government" | "tax" | "financial" | "healthcare" | "identity-provider" | "commercial" | "other";
|
||||
export type FieldType = "text" | "password" | "email" | "tel" | "number" | "date" | "textarea" | "select-one" | "radio" | "checkbox";
|
||||
|
||||
export type Source =
|
||||
| { from: "prompt"; ask?: string }
|
||||
| { from: "vault"; key: string }
|
||||
| {
|
||||
from: "document";
|
||||
form: string;
|
||||
field: string;
|
||||
match?: string;
|
||||
pick?: "newest" | "oldest" | "each";
|
||||
years?: { back?: number; current?: boolean };
|
||||
transform?: string;
|
||||
}
|
||||
| { from: "derive"; input: string; transform: string }
|
||||
| { from: "candidate"; input: string; part: "year" | "form" | "field" | "source" }
|
||||
| { from: "generate"; length: number; classes: Array<"lower" | "upper" | "digit" | "special">; special?: string }
|
||||
| { from: "literal"; value: string | number | boolean };
|
||||
|
||||
export interface Input {
|
||||
label?: string;
|
||||
type: InputType;
|
||||
sensitivity: Sensitivity;
|
||||
role?: Role;
|
||||
required?: boolean;
|
||||
pattern?: string;
|
||||
max_length?: number;
|
||||
count?: number;
|
||||
sources: Source[];
|
||||
}
|
||||
|
||||
export type RuleAction =
|
||||
| { text: string; split?: number[] }
|
||||
| { select: string[] }
|
||||
| { check: true | { label: string } }
|
||||
| { skip: true }
|
||||
| { gate: string }
|
||||
| { choose: string }
|
||||
| { answer: string };
|
||||
|
||||
export interface Rule {
|
||||
name: string;
|
||||
id?: string;
|
||||
label?: string;
|
||||
types?: FieldType[];
|
||||
do: RuleAction;
|
||||
}
|
||||
|
||||
export interface Match {
|
||||
url?: string;
|
||||
title?: string;
|
||||
text?: string;
|
||||
selector?: string;
|
||||
}
|
||||
|
||||
export interface PageStep {
|
||||
id: string;
|
||||
kind: "page";
|
||||
match?: Match;
|
||||
rules?: Rule[];
|
||||
unmatched?: "stop" | "ask";
|
||||
say?: string;
|
||||
}
|
||||
export interface WaitStep {
|
||||
id: string;
|
||||
kind: "wait";
|
||||
match: Match;
|
||||
timeout: string;
|
||||
poll?: string;
|
||||
say?: string;
|
||||
}
|
||||
export interface DeclareStep {
|
||||
id: string;
|
||||
kind: "declare";
|
||||
statement: string;
|
||||
why: string;
|
||||
say?: string;
|
||||
}
|
||||
export interface IdentityStep {
|
||||
id: string;
|
||||
kind: "identity-proofing";
|
||||
provider: string;
|
||||
origins: string[];
|
||||
match?: Match;
|
||||
timeout?: string;
|
||||
handoff?: string;
|
||||
why: string;
|
||||
say?: string;
|
||||
}
|
||||
export interface CodeStep {
|
||||
id: string;
|
||||
kind: "code";
|
||||
channel: "sms" | "email" | "voice" | "app";
|
||||
relay: Array<"terminal" | "file" | "page">;
|
||||
pattern?: string;
|
||||
timeout: string;
|
||||
why?: string;
|
||||
say?: string;
|
||||
}
|
||||
export interface MailStep {
|
||||
id: string;
|
||||
kind: "mail";
|
||||
what: string;
|
||||
arrives?: string;
|
||||
expires?: string;
|
||||
resume?: string;
|
||||
input?: string;
|
||||
handoff?: string;
|
||||
why?: string;
|
||||
say?: string;
|
||||
}
|
||||
export interface CaptchaStep {
|
||||
id: string;
|
||||
kind: "captcha";
|
||||
match: Match;
|
||||
solver?: "forbidden" | "allowed";
|
||||
timeout?: string;
|
||||
why?: string;
|
||||
say?: string;
|
||||
}
|
||||
|
||||
export type Step = PageStep | WaitStep | DeclareStep | IdentityStep | CodeStep | MailStep | CaptchaStep;
|
||||
export type GateStep = DeclareStep | IdentityStep | CodeStep | MailStep | CaptchaStep;
|
||||
|
||||
export interface Outcome {
|
||||
name: string;
|
||||
kind: "success" | "rejected" | "waiting";
|
||||
text?: string;
|
||||
url?: string;
|
||||
then?: string;
|
||||
}
|
||||
|
||||
export interface Handoff {
|
||||
title: string;
|
||||
open?: string;
|
||||
steps: string[];
|
||||
command?: string;
|
||||
}
|
||||
|
||||
export interface Download {
|
||||
match: { url?: string; filename?: string; type?: string };
|
||||
to: string;
|
||||
when?: string;
|
||||
}
|
||||
|
||||
export interface Errand {
|
||||
type: "logicsrc.openerrand";
|
||||
version: "0.1";
|
||||
id?: string;
|
||||
name: string;
|
||||
title: string;
|
||||
description?: string;
|
||||
publisher?: string;
|
||||
updated?: string;
|
||||
reference?: string;
|
||||
principal?: "self" | "represented";
|
||||
site: { name: string; sector?: Sector; origins: string[]; start: string[]; terms?: string };
|
||||
limits?: { pages?: number; page_timeout?: string; same_page?: number };
|
||||
inputs?: Record<string, Input>;
|
||||
rules?: Rule[];
|
||||
steps: Step[];
|
||||
submit?: { labels: string; never?: string; ignore?: string };
|
||||
outcomes: Outcome[];
|
||||
retry?: { shared_secret?: "never"; page_errors?: "rejected" | "continue" };
|
||||
outputs?: {
|
||||
vault?: { when?: string; keys: Record<string, string> };
|
||||
downloads?: Download[];
|
||||
};
|
||||
handoffs?: Record<string, Handoff>;
|
||||
metadata?: Record<string, unknown>;
|
||||
}
|
||||
|
||||
/** One form control as the runner reads it off a page. Never carries a value. */
|
||||
export interface Field {
|
||||
/** A CSS selector that finds exactly this control. */
|
||||
selector: string;
|
||||
id: string;
|
||||
name: string;
|
||||
type: string;
|
||||
label: string;
|
||||
required: boolean;
|
||||
options?: Array<{ value: string; text: string }>;
|
||||
/** For a text box: the chosen option of the nearest select before it, or the question printed beside it. */
|
||||
question?: string;
|
||||
/** For a radio: whether any radio of its group is already checked. */
|
||||
groupChecked?: boolean;
|
||||
}
|
||||
|
||||
/** What the runner reads from a page. `text` is the visible body text, cut short. */
|
||||
export interface Page {
|
||||
url: string;
|
||||
title: string;
|
||||
text: string;
|
||||
errors: string[];
|
||||
fields: Field[];
|
||||
}
|
||||
|
||||
/** The outcome the runner adds of its own, with why it stopped. */
|
||||
export type StopReason = "unmatched" | "loop" | "pages" | "timeout" | "gate" | "off-site" | "no-forward" | "throttle" | "input" | "error";
|
||||
104
packages/openerrand/src/util.ts
Normal file
104
packages/openerrand/src/util.ts
Normal file
|
|
@ -0,0 +1,104 @@
|
|||
/** Small pure helpers every part of the runner shares. */
|
||||
|
||||
export class ErrandError extends Error {
|
||||
constructor(message: string) {
|
||||
super(message);
|
||||
this.name = "ErrandError";
|
||||
}
|
||||
}
|
||||
|
||||
const cache = new Map<string, RegExp>();
|
||||
|
||||
/** Patterns in an errand file are ECMAScript regular expressions, matched case-insensitively. */
|
||||
export function re(pattern: string): RegExp {
|
||||
let compiled = cache.get(pattern);
|
||||
if (!compiled) {
|
||||
compiled = new RegExp(pattern, "i");
|
||||
cache.set(pattern, compiled);
|
||||
}
|
||||
return compiled;
|
||||
}
|
||||
|
||||
export function test(pattern: string | undefined, text: string): boolean {
|
||||
return pattern === undefined ? true : re(pattern).test(text);
|
||||
}
|
||||
|
||||
export function escapeRegExp(text: string): string {
|
||||
return text.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
|
||||
}
|
||||
|
||||
/**
|
||||
* ISO 8601 durations as the file uses them (`PT15M`, `PT90S`, `P21D`, `P1W`).
|
||||
* Months and years are counted as 30 and 365 days: no errand waits on a
|
||||
* calendar month to the day.
|
||||
*/
|
||||
export function durationMs(duration: string | undefined, fallback: number): number {
|
||||
if (!duration) return fallback;
|
||||
const m = /^P(?:(\d+(?:\.\d+)?)Y)?(?:(\d+(?:\.\d+)?)M)?(?:(\d+(?:\.\d+)?)W)?(?:(\d+(?:\.\d+)?)D)?(?:T(?:(\d+(?:\.\d+)?)H)?(?:(\d+(?:\.\d+)?)M)?(?:(\d+(?:\.\d+)?)S)?)?$/.exec(duration);
|
||||
if (!m) return fallback;
|
||||
const [, y, mo, w, d, h, mi, s] = m.map((part) => (part === undefined ? 0 : Number(part)));
|
||||
const day = 86_400_000;
|
||||
return (y! * 365 + mo! * 30 + w! * 7 + d!) * day + h! * 3_600_000 + mi! * 60_000 + s! * 1000;
|
||||
}
|
||||
|
||||
const TEMPLATE = /\{\{\s*([a-z][a-z0-9_.]*)\s*\}\}/g;
|
||||
|
||||
/** Every `{{name}}` in a template. */
|
||||
export function templateNames(text: string): string[] {
|
||||
return [...text.matchAll(TEMPLATE)].map((m) => m[1]!);
|
||||
}
|
||||
|
||||
/** Replace each `{{name}}` with `lookup(name)`; a name with no value throws, so nothing is sent half-filled. */
|
||||
export function render(text: string, lookup: (name: string) => string | undefined): string {
|
||||
return text.replace(TEMPLATE, (_, name: string) => {
|
||||
const value = lookup(name);
|
||||
if (value === undefined) throw new ErrandError(`{{${name}}} has no value`);
|
||||
return value;
|
||||
});
|
||||
}
|
||||
|
||||
/** The transforms the spec names: digits, whole, upper, lower, trim, first:N, last:N. */
|
||||
export function transform(value: string, name: string | undefined): string {
|
||||
if (!name) return value;
|
||||
if (name === "digits") return (value.match(/\d+/g) ?? []).join("");
|
||||
if (name === "whole") {
|
||||
// Whole units, a leading minus for a loss, no separators: "(1,234.56)" and "-1,234.56" are both -1234.
|
||||
const negative = /^\s*\(.*\)\s*$/.test(value) || /^\s*-/.test(value);
|
||||
const digits = value.replace(/[^0-9.]/g, "");
|
||||
const whole = digits === "" ? "" : String(Math.trunc(Number(digits)));
|
||||
return whole === "" ? "" : negative && whole !== "0" ? `-${whole}` : whole;
|
||||
}
|
||||
if (name === "upper") return value.toUpperCase();
|
||||
if (name === "lower") return value.toLowerCase();
|
||||
if (name === "trim") return value.trim();
|
||||
const first = /^first:(\d+)$/.exec(name);
|
||||
if (first) return value.slice(0, Number(first[1]));
|
||||
const last = /^last:(\d+)$/.exec(name);
|
||||
if (last) return value.slice(-Number(last[1]));
|
||||
throw new ErrandError(`unknown transform ${name}`);
|
||||
}
|
||||
|
||||
/** Secrets are shown as four dots, whatever their length, so the length leaks nothing either. */
|
||||
export const MASK = "••••";
|
||||
|
||||
/** The https origin of a URL, or "" when it has none. */
|
||||
export function originOf(url: string): string {
|
||||
try {
|
||||
const u = new URL(url);
|
||||
return u.origin === "null" ? "" : u.origin;
|
||||
} catch {
|
||||
return "";
|
||||
}
|
||||
}
|
||||
|
||||
export function sleep(ms: number): Promise<void> {
|
||||
return new Promise((resolve) => setTimeout(resolve, ms));
|
||||
}
|
||||
|
||||
/** A short opaque id for a card or a run: lowercase letters and digits. */
|
||||
export function shortId(random: (max: number) => number, length = 6): string {
|
||||
const alphabet = "abcdefghijkmnpqrstuvwxyz23456789";
|
||||
let out = "";
|
||||
while (out.length < length) out += alphabet[random(alphabet.length)];
|
||||
return out;
|
||||
}
|
||||
191
packages/openerrand/src/vault.ts
Normal file
191
packages/openerrand/src/vault.ts
Normal file
|
|
@ -0,0 +1,191 @@
|
|||
/**
|
||||
* Vaults the runner reads `vault` sources from and writes `outputs.vault` to.
|
||||
*
|
||||
* A target is named by a string, from `--vault` or the file's
|
||||
* `metadata.vault` (the schema has no slot for it, and `metadata` is where a
|
||||
* publisher puts what a runner needs):
|
||||
*
|
||||
* | Target | Reads | Writes |
|
||||
* | --- | --- | --- |
|
||||
* | `teams:<team>/<project>/<env>` | `logicsrc teams pull` | pull, merge, `logicsrc teams push` |
|
||||
* | `opencreds` | `logicsrc vault get <KEY> --field <f> --reveal` | never (read-only) |
|
||||
* | `file:<path>` | a 0600 dotenv file | the same file, 0600 |
|
||||
*
|
||||
* With no target the runner uses `file:` on its own credentials file and says
|
||||
* so, as the spec asks: a login that exists only in a terminal's scrollback is
|
||||
* a login lost.
|
||||
*
|
||||
* The teams push replaces the vault's contents with the file it is given, so
|
||||
* a write always pulls first and merges, and the plaintext lives for one call
|
||||
* in a 0700 temporary directory removed in a `finally`.
|
||||
*/
|
||||
|
||||
import { spawnSync } from "node:child_process";
|
||||
import { existsSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
|
||||
import { homedir, tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import type { VaultReader } from "./inputs.js";
|
||||
import { writePrivate } from "./store.js";
|
||||
import { ErrandError } from "./util.js";
|
||||
|
||||
export interface Vault extends VaultReader {
|
||||
/** Merge these keys in. Absent for a read-only vault. */
|
||||
write?: (values: Record<string, string>) => Promise<void>;
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// dotenv, the same dialect as logicsrc's env provider, so a file this writes
|
||||
// pushes and pulls back unchanged.
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
const QUOTED = /^(['"])(.*)\1$/s;
|
||||
|
||||
export function parseEnv(body: string): Record<string, string> {
|
||||
const out: Record<string, string> = {};
|
||||
for (const raw of body.split(/\r?\n/)) {
|
||||
const line = raw.trim();
|
||||
if (!line || line.startsWith("#")) continue;
|
||||
const bare = line.startsWith("export ") ? line.slice(7) : line;
|
||||
const eq = bare.indexOf("=");
|
||||
if (eq <= 0) continue;
|
||||
const key = bare.slice(0, eq).trim();
|
||||
let value = bare.slice(eq + 1).trim();
|
||||
const quoted = QUOTED.exec(value);
|
||||
if (quoted) {
|
||||
value = quoted[2]!;
|
||||
if (quoted[1] === '"') value = value.replace(/\\n/g, "\n").replace(/\\"/g, '"').replace(/\\\\/g, "\\");
|
||||
}
|
||||
out[key] = value;
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
export function serializeEnv(values: Record<string, string>): string {
|
||||
const quote = (v: string): string => (/[\s#'"=\\]|^$/.test(v) || v.includes("\n") ? `"${v.replace(/\\/g, "\\\\").replace(/"/g, '\\"').replace(/\n/g, "\\n")}"` : v);
|
||||
return `${Object.keys(values)
|
||||
.sort()
|
||||
.map((k) => `${k}=${quote(values[k]!)}`)
|
||||
.join("\n")}\n`;
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
export type VaultTarget =
|
||||
| { kind: "teams"; team: string; project: string; env: string }
|
||||
| { kind: "opencreds"; field?: string }
|
||||
| { kind: "file"; path: string };
|
||||
|
||||
export function parseTarget(text: string): VaultTarget {
|
||||
const teams = /^teams:([^/\s]+)\/([^/\s]+)\/([^/\s]+)$/.exec(text);
|
||||
if (teams) return { kind: "teams", team: teams[1]!, project: teams[2]!, env: teams[3]! };
|
||||
const creds = /^opencreds(?::(.+))?$/.exec(text);
|
||||
if (creds) return creds[1] ? { kind: "opencreds", field: creds[1] } : { kind: "opencreds" };
|
||||
const file = /^file:(.+)$/.exec(text);
|
||||
if (file) return { kind: "file", path: file[1]!.replace(/^~(?=\/|$)/, homedir()) };
|
||||
throw new ErrandError(`vault target ${text}: expected teams:<team>/<project>/<env>, opencreds[:<field>] or file:<path>`);
|
||||
}
|
||||
|
||||
export function describeTarget(target: VaultTarget): string {
|
||||
if (target.kind === "teams") return `teams ${target.team}/${target.project}/${target.env}`;
|
||||
if (target.kind === "opencreds") return "OpenCreds vault (read-only)";
|
||||
return target.path;
|
||||
}
|
||||
|
||||
/** Runs the logicsrc CLI. The umbrella passes one that re-enters itself; standalone it is `logicsrc` on PATH. */
|
||||
export type LogicsrcExec = (args: string[], options?: { inheritStdin?: boolean }) => { status: number; stdout: string; stderr: string };
|
||||
|
||||
export const pathLogicsrc: LogicsrcExec = (args, options) => {
|
||||
const result = spawnSync("logicsrc", args, { encoding: "utf8", stdio: [options?.inheritStdin ? "inherit" : "ignore", "pipe", "pipe"] });
|
||||
if (result.error) {
|
||||
const missing = (result.error as NodeJS.ErrnoException).code === "ENOENT";
|
||||
return { status: 127, stdout: "", stderr: missing ? "logicsrc is not installed (https://logicsrc.com)" : result.error.message };
|
||||
}
|
||||
return { status: result.status ?? 1, stdout: result.stdout ?? "", stderr: result.stderr ?? "" };
|
||||
};
|
||||
|
||||
const lastLines = (text: string): string => text.trim().split("\n").slice(-2).join(" ");
|
||||
|
||||
export function fileVault(path: string): Vault {
|
||||
return {
|
||||
describe: () => path,
|
||||
async get(key) {
|
||||
return existsSync(path) ? parseEnv(readFileSync(path, "utf8"))[key] : undefined;
|
||||
},
|
||||
async write(values) {
|
||||
const current = existsSync(path) ? parseEnv(readFileSync(path, "utf8")) : {};
|
||||
writePrivate(path, serializeEnv({ ...current, ...values }));
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
export function teamsVault(target: Extract<VaultTarget, { kind: "teams" }>, exec: LogicsrcExec = pathLogicsrc): Vault {
|
||||
let cache: Record<string, string> | null = null;
|
||||
|
||||
function pull(): Record<string, string> {
|
||||
const dir = mkdtempSync(join(tmpdir(), "logicsrc-errand-"));
|
||||
try {
|
||||
const file = join(dir, "vault.env");
|
||||
const result = exec(["teams", "pull", target.team, target.project, target.env, "--env", file, "--format", "json"]);
|
||||
if (result.status !== 0) throw new ErrandError(`logicsrc teams pull failed: ${lastLines(result.stderr || result.stdout)}`);
|
||||
return existsSync(file) ? parseEnv(readFileSync(file, "utf8")) : {};
|
||||
} finally {
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
describe: () => describeTarget(target),
|
||||
async get(key) {
|
||||
if (!cache) {
|
||||
try {
|
||||
cache = pull();
|
||||
} catch {
|
||||
// A vault that does not exist yet has nothing to reuse; the next source (usually generate) runs.
|
||||
cache = {};
|
||||
}
|
||||
}
|
||||
return cache[key];
|
||||
},
|
||||
async write(values) {
|
||||
let current: Record<string, string> = {};
|
||||
try {
|
||||
current = pull();
|
||||
} catch {
|
||||
// A vault that does not exist yet is created by the push.
|
||||
}
|
||||
const dir = mkdtempSync(join(tmpdir(), "logicsrc-errand-"));
|
||||
try {
|
||||
const file = join(dir, "vault.env");
|
||||
writeFileSync(file, serializeEnv({ ...current, ...values }), { mode: 0o600 });
|
||||
const result = exec(["teams", "push", target.team, target.project, target.env, "--env", file, "--format", "json"]);
|
||||
if (result.status !== 0) throw new ErrandError(`logicsrc teams push failed: ${lastLines(result.stderr || result.stdout)}`);
|
||||
cache = { ...current, ...values };
|
||||
} finally {
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* The personal OpenCreds vault, read-only: an item named by the key, the
|
||||
* field `login.password` unless the target names another. Unlocking may ask
|
||||
* for the master password, so stdin is the terminal's.
|
||||
*/
|
||||
export function opencredsVault(target: Extract<VaultTarget, { kind: "opencreds" }>, exec: LogicsrcExec = pathLogicsrc): Vault {
|
||||
return {
|
||||
describe: () => describeTarget(target),
|
||||
async get(key) {
|
||||
const result = exec(["vault", "get", key, "--field", target.field ?? "login.password", "--reveal"], { inheritStdin: true });
|
||||
if (result.status !== 0) return undefined;
|
||||
const value = result.stdout.replace(/\n$/, "");
|
||||
return value === "" ? undefined : value;
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
export function openVault(target: VaultTarget, exec?: LogicsrcExec): Vault {
|
||||
if (target.kind === "teams") return teamsVault(target, exec);
|
||||
if (target.kind === "opencreds") return opencredsVault(target, exec);
|
||||
return fileVault(target.path);
|
||||
}
|
||||
9
packages/openerrand/tsconfig.json
Normal file
9
packages/openerrand/tsconfig.json
Normal file
|
|
@ -0,0 +1,9 @@
|
|||
{
|
||||
"extends": "../../tsconfig.base.json",
|
||||
"compilerOptions": {
|
||||
"rootDir": "src",
|
||||
"outDir": "dist"
|
||||
},
|
||||
"include": ["src/**/*.ts"],
|
||||
"exclude": ["src/**/*.test.ts", "src/testing.ts", "src/fake-site.ts"]
|
||||
}
|
||||
Loading…
Add table
Add a link
Reference in a new issue