feat(openontology): Phase 2 + Phase 3 — storage, REST/SSE, MCP, RDF/SHACL, adapters, TUI, explorer (#101)
Some checks are pending
CI / build (push) Waiting to run
test / test (push) Waiting to run

Everything the two shipped PRD phases deferred, minus what is called out below.

Storage (Phase 2)
  @logicsrc/openontology gains a SQLite/Turso adapter. It hydrates the read
  model at open, serves reads synchronously — a query evaluator that awaits per
  triple pattern is unusable — and buffers mutations as SQL that flush() writes
  in one transaction. Versioned idempotent migrations; indexes over subject,
  predicate, entity-valued object, status, both time axes, aliases, and external
  ids; FTS5 for label/alias search. The append-only status log is replayed on
  open, so retractions, supersessions, and merge redirects survive a reopen.

REST + SSE + OpenAPI (Phase 2)
  16 paths under /api/ontologies in logicsrc-web, described at
  /api/ontologies/openapi and referencing the published JSON Schemas rather
  than restating them. No token is read-only; a curator token can apply; an
  agent token can propose and cannot apply. Idempotency-Key on mutations,
  revision ETags, 409 on a stale base revision, and an SSE stream that emits
  the same event objects as the JSON endpoint.

MCP (Phase 2)
  OpenOntology and OpenPRD surfaces on the standards server: spec/manifest/
  schema/queries and PRD spec/index as resources, 11 ontology tools and 6 PRD
  tools, 7 prompts. Read-only by default; OPENONTOLOGY_MCP_WRITABLE=1 buys
  proposals, never applies — the denial is the shared policy layer, not a
  second rule that could drift.

Interoperability (Phase 3)
  RDF/Turtle export and import of the reified profile, plus the plain triple
  for asserted relationships so a consumer wanting only the accepted graph gets
  one. SHACL for 5 of 7 constraint kinds; `unique` and `query` are reported as
  unmapped in both the return value and the generated Turtle, because a shape
  that quietly means something narrower is worse than no shape.

Source adapters (Phase 3)
  CSV, JSON, YAML, NDJSON, Markdown, generic JSON HTTP, and GitHub. All produce
  PROPOSED change-set operations with source, evidence selector, run id, and
  confidence attached; fetch is injected so ingestion is offline and testable.
  Each declares its capabilities, so "nothing was deleted upstream" is never
  confused with "this adapter cannot see deletions" — none of the seven can.

TUI + explorer
  Keyboard-first panels (types, entities, claims, sources, queries, change
  sets, validation, audit) as plain strings that survive SSH and 60 columns;
  status is a glyph and a word, never colour alone; the key bar wraps rather
  than truncating. Wired as `logicsrc ontology tui`. A read-only web explorer
  at /openontology/explore with entity and claim views showing status, both
  clocks, confidence, sources, evidence, and append-only history — plus an
  /openprd page for the companion standard.

Bugs found and fixed while testing
  - the API built a new engine per request, so `explain` could never find a
    resultId from a prior request; engines are now cached per role
  - the TUI status bar called engine.validateOntologyPackage(), appending a
    package.validated event on every repaint; it now uses the pure validator

Verification: 76 new tests (527 total across the monorepo, all passing); full
build green; the libSQL adapter is exercised against real files, the API
through its route handlers, and MCP over an in-memory transport.

Not included: PWA review/approval write flows (they need an auth story this
deployment does not have), OWL/RDFS mappings, SPARQL/Cypher/Datalog query
adapters, and Phase 4 governed actions. The compatibility matrix marks those
"planned", not "supported".

Refs: prd/0001-add-logicsrc-openontology-spec.md

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Anthony Ettinger 2026-07-28 05:10:33 -07:00 committed by GitHub
parent 296775e003
commit da5f6f8381
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
53 changed files with 6939 additions and 23 deletions

View file

@ -68,15 +68,34 @@ Lookup works by exact id, canonical name, alias, or external id. `findEntities`
`sameAs` is a reviewable claim, not an implicit merge. Two records only become one through an approved `merge-entity` operation, and the losing id survives as a redirect.
## RDF, SHACL, OWL
## RDF and Turtle
Planned for a later phase, deliberately not faked in 0.1:
```bash
logicsrc ontology export --dir ./ethereum-ecosystem --format turtle --out graph.ttl
```
- **RDF/Turtle** — export and import of the losslessly mappable subset, using the same reified-claim shape as JSON-LD.
- **SHACL** — the constraint kinds with genuinely equivalent semantics (`required-predicate`, `cardinality`, `unique`, `allowed-values`, `domain-range`) map to shapes. Query-based constraints do not, and will be reported as unmapped.
- **OWL/RDFS** — an optional mapping for consumers needing formal reasoning. OpenOntology itself infers nothing: transitivity, symmetry, and inverses apply only when the schema declares them *and* a query asks.
Claims are reified, exactly as in JSON-LD, and an **asserted relationship claim additionally emits the plain triple** — so a consumer that only wants the current accepted graph gets one without unpacking provenance.
Until those ship, the compatibility matrix below says "planned", not "supported". Claiming compatibility that has not been implemented and tested is the thing this document exists to prevent.
Import parses the profile this exporter produces rather than pretending to be a general Turtle parser. Anything it cannot interpret is listed in `unsupported`, never dropped silently.
## SHACL
```bash
logicsrc ontology export --dir ./ethereum-ecosystem --format shacl --out shapes.ttl
```
Five constraint kinds map onto SHACL Core: `required-predicate`, `cardinality`, `allowed-values`, `domain-range`, and `temporal-bounds`. Severity carries across (`error``sh:Violation`, `warning``sh:Warning`).
Two do **not**, and are reported as unmapped in the returned value *and* as comments in the generated Turtle:
- `unique` — graph-wide uniqueness has no portable SHACL Core equivalent; it needs a `sh:SPARQLConstraint`.
- `query` — an OpenOntology saved query is a triple-pattern AST, not SPARQL.
A shape that silently means something narrower than the constraint it came from is worse than no shape, so those stay unmapped until the mapping is real.
## OWL/RDFS
Still planned. An optional mapping for consumers needing formal reasoning. OpenOntology itself infers nothing: transitivity, symmetry, and inverses apply only when the schema declares them *and* a query asks.
## Compatibility matrix
@ -88,14 +107,18 @@ Until those ship, the compatibility matrix below says "planned", not "supported"
| NDJSON | **supported** | Streaming entity/claim/source/evidence files |
| JSON-LD 1.1 export | **supported** | Reified claims, PROV-O aliases, lossy report |
| JSON-LD 1.1 import | **supported** | Round-trips the reference profile |
| PROV-O | **partial** | Provenance terms aliased; full mapping later |
| RDF / Turtle | planned | Phase 3 |
| SHACL | planned | Phase 3, constraint subset only |
| PROV-O | **partial** | Provenance terms aliased in JSON-LD and Turtle; full mapping later |
| RDF / Turtle export | **supported** | Reified claims + plain triples for asserted relationships |
| RDF / Turtle import | **supported** | Round-trips the reference profile; reports what it cannot read |
| SHACL | **partial** | 5 of 7 constraint kinds; `unique` and `query` reported as unmapped |
| OWL / RDFS | planned | Optional, for external reasoners |
| SPARQL | planned | Query AST → SPARQL adapter |
| Cypher | planned | Query AST → Cypher adapter |
| Datalog | planned | Query AST → Datalog adapter |
| SQLite / Turso | **supported** | Reference storage adapters |
| SQLite / Turso | **supported** | `createLibsqlStore`, versioned migrations, FTS5 entity search |
| REST + OpenAPI | **supported** | 16 paths, described at `/api/ontologies/openapi` |
| Server-Sent Events | **supported** | Same event objects as the JSON endpoint |
| MCP | **supported** | Resources, tools, and prompts; writes propose, never apply |
| Neo4j / vector DBs | not required | Optional adapters; never mandatory |
## Query portability

View file

@ -272,9 +272,11 @@ logicsrc ontology entity get|list|find|merge
logicsrc ontology claim get|list|history|propose|assert|dispute|retract
logicsrc ontology query run|explain|list
logicsrc ontology changeset list|create|diff|apply
logicsrc ontology import|export|audit
logicsrc ontology import|export|audit|tui
```
`logicsrc ontology tui` renders keyboard-first panels — types, entities, claims, sources, queries, change sets, validation, audit — as plain strings that survive SSH, tmux, and a 60-column terminal. Claim status is shown as a glyph *and* the word, never colour alone.
Read commands take `--format table|json|yaml|markdown|ndjson`. Write commands produce a **proposal** by default. Exit codes are stable for CI: `0` ok, `1` validation failed, `2` usage error, `3` not found, `4` denied or approval required.
`--as local|agent|reader` selects the actor role. It cannot grant an agent apply rights; the policy layer denies those outright.
@ -302,9 +304,49 @@ engine.applyOntologyChangeSet(changeSet.id);
Storage, source adapters, query engine, identity, policy, events, and signatures are all injectable interfaces. The clock and id factory are injectable too, so a build, an applied change set, and a test run produce byte-identical output under both Node.js and Bun.
## Storage
The store is an interface, so nothing about the model depends on where it lives.
```ts
import { createClient } from "@libsql/client";
import { createLibsqlStore, createOntologyEngine } from "@logicsrc/openontology";
const store = await createLibsqlStore({ client: createClient({ url }), seed: pkg });
const engine = createOntologyEngine({ store, actor });
// … apply a change set …
await store.flush(); // one transaction, then it is durable
```
The libSQL adapter hydrates the read model at open, serves reads synchronously, and buffers mutations as SQL that `flush()` writes in a single transaction. Callers that mutate must await it — the REST layer does after every applied change set. Everything persisted is append-only, so a crash before flush loses the last change set rather than corrupting history.
Migrations are versioned and idempotent. Indexes cover entity ids, types, aliases, external ids, subject, predicate, entity-valued object, status, valid time, recorded time, and sources; FTS5 backs label and alias search.
## REST, SSE, and MCP
The reference service is described by OpenAPI at `/api/ontologies/openapi` and shares the published JSON Schemas rather than restating them.
```txt
GET /api/ontologies
GET /api/ontologies/{id}/manifest | /schema | /entities | /entities/{entityId}
GET /api/ontologies/{id}/claims | /claims/{claimId}
POST /api/ontologies/{id}/query | /explain | /validate
GET /api/ontologies/{id}/changesets POST to propose
GET /api/ontologies/{id}/changesets/{id} POST .../review | /approve | /apply
GET /api/ontologies/{id}/events Accept: text/event-stream for SSE
```
No token is read-only. A curator token can apply; an agent token can propose and **cannot** apply. Mutating requests accept `Idempotency-Key`; a change set authored against a stale revision fails with 409 rather than overwriting.
The MCP server exposes the spec, manifest, schema, and saved queries as resources, plus tools for validate, get/find entities, query, explain, export, and propose. Write tools default to proposals, and applying is denied to agent actors by the same policy the SDK enforces — not by a separate rule that could drift.
## Ingestion
Seven source adapters — CSV, JSON, YAML, NDJSON, Markdown, a generic JSON HTTP endpoint, and GitHub — turn foreign data into **proposed** change-set operations with sources and evidence selectors attached. Each declares what it can and cannot do, so "nothing was deleted upstream" is never confused with "this adapter cannot see deletions." See [interoperability](./openontology-interoperability.md#source-adapters).
## Interoperability
JSON Schema Draft 2020-12 is the canonical contract. JSON-LD 1.1 is the interoperability profile, aliasing W3C PROV-O for provenance where the semantics genuinely match. Exports report every field the target format cannot carry rather than dropping it silently. See [OpenOntology interoperability](./openontology-interoperability.md).
JSON Schema Draft 2020-12 is the canonical contract. JSON-LD 1.1 and RDF/Turtle are interoperability profiles, aliasing W3C PROV-O for provenance where the semantics genuinely match, and SHACL covers five of the seven constraint kinds. Exports report every field the target format cannot carry rather than dropping it silently. See [OpenOntology interoperability](./openontology-interoperability.md).
## Conformance