bbs.profullstack/logicsrc
1
0
Fork 0
mirror of https://github.com/profullstack/logicsrc.git synced 2026-09-10 19:26:00 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
logicsrc/docs/openswarm/ipdb.md
Anthony Ettinger 4fed2681ec
Some checks are pending
CI / build (push) Waiting to run
Details
test / test (push) Waiting to run
Details
Add the LogicSRC OpenSwarm specification family (#142) 
BitTorrent distributes bytes better than any CDN and has never been a
product: nobody is paid to seed, and nothing in it is private. Every
Profullstack media property answers that with a central HTTP proxy and a
pass system bolted on, and pays for every byte it serves.

OpenSwarm is an add-on to BitTorrent, carried as BEP 10 extension
messages, that fixes both. The swarm carries AES-256-CTR ciphertext whose
integrity is pinned by two SHA-256 merkle roots in a signed manifest, so
a tracker or DHT node learns an infohash and a size. A leecher buys a
pass over x402 in USDC (the same exchange x402-gateway runs for crawl
passes, settled by CoinPay), a seeder serves inside a bounded credit
window, and the leecher signs a cumulative voucher for every verified
batch. Whoever seeds gets paid. Vanilla clients remain valid members and
browsers remain first-class peers.

Adding a file mints a key pair for it, derived from one publisher seed by
default so there is one thing to back up: the public half is the file's
identity and its BEP 46 key, the private half signs the manifest and
authorises grants and payout changes, and a separate content key
encrypts the bytes and is sealed to paying peers.

The family: a core (records, keys, hashing, transports, discovery,
events), ipfile, ippay, ipdb (a signed hash-chained catalogue with heads
on the DHT), ipaudio, ipvideo, iplive (paid relays with backpressure)
and ipname (Moshpit pins and DNS TXT). Plus the c0mpute.com integration
with seven workload types and thirteen use cases, a proposed ip CLI,
conformance profiles, a security model and an FAQ.

Registered on the site the way OpenCreds is: nav entry, docs registry,
sitemap, a /openswarm landing page, PRD 0005, and the MCP prd_next_id
expectation moved to 0006.

Specs only. No code, no schemas, no reference implementation.


Claude-Session: https://claude.ai/code/session_01YafYxayh7Gqe5MWNNQMev2

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-05 15:05:57 -07:00

386 lines
15 KiB
Markdown
Raw Blame History

# ipdb: The Replicated Catalogue
Version: **0.1** (draft)
Status: draft. Extends the OpenSwarm core ([spec.md](./spec.md)). Nothing
below is implemented.
## 1. Scope
`ipdb` is the catalogue: the place a client asks "what is this swarm", "what
has this publisher released", "what is in this playlist". It defines:
1. A **feed**: a signed, hash-linked, append-only log owned by one key.
2. An **entry**: one log item that puts or deletes a record under a key.
3. **Heads**: how the latest state of a feed is announced on the DHT.
4. **Replication**: how feeds move between peers, as `ipfile` swarms and as
gossip on swarm connections.
5. **Queries**: a JSON query shape a local replica and a gateway both answer.
It is a database of records, not of bytes. A manifest is in `ipdb`; the
file it describes is in an `ipfile` swarm.
### Non-goals
Multi-writer documents. A feed has one writer. Two people editing one
playlist is two feeds and a view over both, or a future CRDT record type.
Global search ranking. Consistency across feeds; each feed is consistent
with itself and that is all that is promised.
## 2. Terminology
| Term | Meaning |
| --- | --- |
| **feed** | A log identified by `(feed key, name)`. |
| **feed key** | An Ed25519 key. A publisher's default feed uses the publisher key itself. |
| **entry** | A record of type `ipdb.entry`, sequenced and chained. |
| **key** | A string unique within a feed that later entries overwrite. |
| **head** | The latest entry of a feed. |
| **segment** | A batch of entries stored as one file for replication. |
| **replica** | A peer holding some feeds, verified, queryable. |
| **follow** | A replica's decision to keep a feed current. |
## 3. Data model
### 3.1 Feed keys
The default feed is the publisher key with name `default`. Named feeds
derive from the publisher seed the same way files do:
```
feedSeed = HKDF-SHA256(ikm = publisher seed, salt = utf8(name), info = "openswarm:feed:v1")
feed key = Ed25519 keypair from feedSeed
```
A feed reference is written `ed25519:<hex>/<name>`; the default feed is
`ed25519:<hex>/default`. Names match `^[a-z0-9][a-z0-9-]{0,63}$`.
### 3.2 Entry
```json
{
"openswarm": "0.1",
"type": "ipdb.entry",
"feed": "ed25519:5d292428e8a68946e5225136c8b10e8f33ab78a45e1663d0730996dd2b63d59e",
"name": "default",
"seq": 12,
"prev": "sha256:ba8cd1a97b865a05cdc5840c63b086dd463624a831874c1ba0d7c4ce6f806f7a",
"op": "put",
"key": "file:ed25519:0d87e09c7fea3ad6ba6c2f3e027ea47f5b245452899910948470906704c5295d",
"kind": "ipfile.manifest",
"subject": "sha256:41d10f45e705e0526c9eeedf62b32dda3daaf552dd9b6a7b09e01776b05813bb",
"record": {
"openswarm": "0.1",
"type": "ipfile.manifest",
"file": "ed25519:0d87e09c7fea3ad6ba6c2f3e027ea47f5b245452899910948470906704c5295d",
"rev": 1,
"createdAt": "2026-09-05T18:00:00.000Z",
"sigs": []
},
"createdAt": "2026-09-05T18:00:05.000Z",
"sigs": [
{
"alg": "ed25519",
"key": "ed25519:5d292428e8a68946e5225136c8b10e8f33ab78a45e1663d0730996dd2b63d59e",
"sig": "RGRH7pDpXYcagULqj+8z05pLH99Lc6yLmfdbYgtGeRpjX5l0NcT3QSq9AWBtLw+5+JJiKoZ3s4ty74q/zBSE8w=="
}
]
}
```
(The embedded manifest is abbreviated here; in a real entry it is the
complete record with its signatures.)
| Field | Rule |
| --- | --- |
| `feed`, `name` | The feed. The entry MUST be signed by `feed`. |
| `seq` | 1 for the first entry, then `prev.seq + 1`. |
| `prev` | Record id of entry `seq - 1`, or `null` for `seq` 1. |
| `op` | `put` or `del`. |
| `key` | The logical key. Convention: `<noun>:<identifier>`, e.g. `file:ed25519:...`, `track:sha256:...`, `playlist:my-mix`. |
| `kind` | The embedded record's `type`. Absent for `del`. |
| `subject` | The embedded record's id. Absent for `del`. A replica MUST recompute it and refuse a mismatch. |
| `record` | The complete embedded record. Absent for `del`. Its own signatures MUST verify. |
An entry is at most 256 KiB. A record larger than that is published as an
`ipfile` and the entry embeds an `ipfile.manifest` pointing at it, with
`kind` naming the payload type in `record.meta.kind`.
### 3.3 State
The state of a feed is the map `key -> latest put record` after applying
entries in `seq` order, with `del` removing the key. A replica exposes both
the log (for audit and replication) and the state (for queries). A record
that was overwritten stays in the log; that is what makes a manifest
revision history free.
Across feeds there is no merge. A query over several feeds returns matches
from each; the `feed` of every result is returned with it.
### 3.4 Head
```json
{
"openswarm": "0.1",
"type": "ipdb.head",
"feed": "ed25519:5d292428e8a68946e5225136c8b10e8f33ab78a45e1663d0730996dd2b63d59e",
"name": "default",
"seq": 12,
"head": "sha256:41d10f45e705e0526c9eeedf62b32dda3daaf552dd9b6a7b09e01776b05813bb",
"segments": [
{ "from": 1, "to": 10, "file": "ed25519:d2d05fcad07ecbee6ff87c95159ec9969fce6c0fb9f41ff78c2c07ebe8a06c94" }
],
"createdAt": "2026-09-05T18:00:05.000Z",
"sigs": [
{
"alg": "ed25519",
"key": "ed25519:5d292428e8a68946e5225136c8b10e8f33ab78a45e1663d0730996dd2b63d59e",
"sig": "RGRH7pDpXYcagULqj+8z05pLH99Lc6yLmfdbYgtGeRpjX5l0NcT3QSq9AWBtLw+5+JJiKoZ3s4ty74q/zBSE8w=="
}
]
}
```
`segments` lists the `ipfile` swarms that hold sealed ranges of the log
(§5.1). Entries after the last segment are fetched by gossip (§5.2).
## 4. Publishing
`ip db put <key> <record.json>` appends an entry. `ip file add` appends one
automatically under `file:<file key>` in the default feed. `ip audio
publish`, `ip video publish` and `ip live create` do the same for their
records. A publisher never edits a feed by hand; every tool that creates a
record appends it.
Rules for the writer:
- `seq` MUST be strictly sequential. A gap is a fork.
- Two entries with the same `seq` from the same feed are a **fork**. A
replica that observes a fork MUST keep both, mark the feed forked from that
`seq`, and stop applying either branch to state until the feed publishes
an `ipdb.entry` with `op: "put"`, `key: "ipdb:resolve"`, and a record
naming the surviving branch head. Forks happen when a seed is restored on
two machines; they are detected, not prevented.
- A `del` of a key that was never put is valid and does nothing.
## 5. Replication
### 5.1 Segments as ipfile swarms
Every 1000 entries, or daily, the writer seals entries `from..to` as one
file: newline-delimited canonical JSON, one entry per line. It publishes the
file as an `ipfile` with `pricing.perGib` and `keyUsd` at `0.000000` and
`keys.public` set to the content key, so the swarm is readable by anyone
while staying an ordinary `ipfile` for seeders, gateways and c0mpute pin
jobs. The head lists the segment. Readers verify each line's chain as they
apply it, so a segment served by a stranger is safe.
### 5.2 Gossip on swarm connections
Peers that advertise `ipdb` in the BEP 10 handshake exchange heads on any
swarm connection:
| `t` | Payload | Meaning |
| --- | --- | --- |
| `heads` | `{ "f": [ { "k": <32 bytes>, "n": string, "s": int } ] }` | Feeds I follow and the seq I have. At most 64 per message. |
| `entries_req` | `{ "k": <32 bytes>, "n": string, "from": int, "to": int }` | Send me entries in the range. At most 100. |
| `entries` | `{ "r": [bytes] }` | Entries as JSON, in seq order. |
| `head` | `{ "r": bytes }` | An `ipdb.head` record, when the sender has a newer one. |
A peer sends `heads` once after `hello` and again whenever a followed feed
advances. A peer that receives a `heads` with a higher `s` for a feed it
follows sends `entries_req`. Gossip fills the tail after the last segment
and keeps replicas close to real time without any peer being special.
### 5.3 DHT announcement
The head is a BEP 44 mutable item (core §7.3):
```
k = feed key
salt = "ipdb:" + name
seq = head seq
v = bencoded { "h": <head record id, 32 bytes>, "s": <seq> }
```
A client with only a feed reference does `get` on the DHT, obtains `h`,
then asks any peer or gateway for the head record by id, then segments and
tail. A gateway also serves `GET /db/feeds/<key>/<name>/head`.
### 5.4 Following
`ip db follow ed25519:<hex>/default` marks a feed followed: the replica
keeps its head current by DHT polling (every 10 minutes, or on `head`
gossip), fetches segments, and applies entries. A c0mpute node with role
`index` follows every feed it is paid to follow (`c0mpute.md` §3).
## 6. Queries
### 6.1 Shape
```json
{
"type": "ipaudio.track",
"feeds": ["ed25519:5d292428e8a68946e5225136c8b10e8f33ab78a45e1663d0730996dd2b63d59e/default"],
"where": [
["record.artist", "==", "Ada"],
["record.durationMs", ">=", 120000]
],
"orderBy": [["record.createdAt", "desc"]],
"limit": 20,
"cursor": null
}
```
| Field | Rule |
| --- | --- |
| `type` | The embedded record type to match. REQUIRED. |
| `feeds` | Feed references to search. `null` means every followed feed. |
| `where` | Conjunction of `[path, op, value]`. Paths dot into the entry; `record.` reaches the embedded record. |
| `orderBy` | List of `[path, "asc"|"desc"]`. |
| `limit` | 1 to 200. |
| `cursor` | Opaque, from a previous result. |
Operators: `==`, `!=`, `<`, `<=`, `>`, `>=`, `in`, `contains` (array
membership or substring), `prefix`. No joins, no aggregation in 0.1; a
rating average is computed by the caller from the entries it gets back.
### 6.2 Result
```json
{
"results": [
{
"feed": "ed25519:5d292428e8a68946e5225136c8b10e8f33ab78a45e1663d0730996dd2b63d59e/default",
"key": "track:sha256:d6c3f8285b7871d6a400cba14408288a9acde679f12e1e7dc276f29ca7c493ff",
"seq": 14,
"subject": "sha256:d6c3f8285b7871d6a400cba14408288a9acde679f12e1e7dc276f29ca7c493ff",
"record": { "openswarm": "0.1", "type": "ipaudio.track", "sigs": [] }
}
],
"cursor": "eyJzZXEiOjE0fQ",
"asOf": { "ed25519:5d292428e8a68946e5225136c8b10e8f33ab78a45e1663d0730996dd2b63d59e/default": 27 }
}
```
`asOf` reports the head seq of each feed the answer was computed from, so a
caller can tell a stale replica from an empty one.
### 6.3 Over HTTP
`POST /db/query` on a gateway with the query as the body. A gateway MAY
require a pass (`Authorization: Bearer`) and MAY sell one over x402 for
queries the way a hub sells swarm passes; the reference price is
`0.000100` USD per query. A public gateway with no charge is the common case
and is what bittorrented.com's `/dht` browse would become for `ipfile`
swarms it can resolve.
## 7. Record kinds carried
`ipdb` carries any record type. These are the ones the family defines and
the keys they use:
| Kind | Key convention | Defined in |
| --- | --- | --- |
| `ipfile.manifest` | `file:<file key>` | ipfile |
| `ipaudio.release` | `release:<record id of first revision>` | ipaudio |
| `ipaudio.track` | `track:<record id of first revision>` | ipaudio |
| `ipvideo.title` | `title:<record id of first revision>` | ipvideo |
| `iplive.channel` | `channel:<channel key>` | iplive |
| `ipname.pin` | `name:<moshpit name>` | ipname |
| `ipdb.playlist` | `playlist:<slug>` | this document §7.1 |
| `ipdb.rating` | `rating:<subject record id>` | this document §7.2 |
### 7.1 Playlist
```json
{
"openswarm": "0.1",
"type": "ipdb.playlist",
"slug": "late-night",
"title": "Late night",
"items": [
{ "kind": "ipaudio.track", "subject": "sha256:d6c3f8285b7871d6a400cba14408288a9acde679f12e1e7dc276f29ca7c493ff", "feed": "ed25519:5d292428e8a68946e5225136c8b10e8f33ab78a45e1663d0730996dd2b63d59e/default" }
],
"createdAt": "2026-09-05T19:00:00.000Z",
"sigs": []
}
```
A playlist item references a record by id and the feed it was found in, so
a reader can fetch it even if the playlist author's replica is gone. The
media-streamer `collections` table maps onto this one to one.
### 7.2 Rating
```json
{
"openswarm": "0.1",
"type": "ipdb.rating",
"subject": "sha256:d6c3f8285b7871d6a400cba14408288a9acde679f12e1e7dc276f29ca7c493ff",
"value": 4,
"createdAt": "2026-09-05T19:05:00.000Z",
"sigs": []
}
```
`value` is 1 to 5. One rating per subject per feed, by the key convention.
An aggregate is `query type ipdb.rating where record.subject == ...` over
the feeds a caller trusts; a gateway MAY cache the count and mean.
## 8. Local storage
A replica stores entries in a local database. The reference layout is
SQLite with tables `feeds(key, name, seq, head, forked_at)`,
`entries(feed, name, seq, id, prev, op, key, kind, subject, json)`,
`state(feed, name, key, seq, kind, subject)`, and one FTS table over
`record.title`, `record.artist`, `record.name` for the `contains`
operator. This is the layout the `ip` CLI would use; a gateway on Postgres
does the same with jsonb. Nothing in the wire depends on it.
## 9. Security notes
- **Authenticity.** Every entry is signed by the feed key and chained. A
replica cannot insert, reorder or drop an entry without breaking the chain
to the announced head.
- **Freshness.** The DHT head carries `seq`; BEP 44 refuses a lower `seq`.
A replica that shows an old head is stale, not lying, and `asOf` says so.
- **Embedded records** verify on their own signatures. A feed cannot put
someone else's manifest under its own key and have it trusted as theirs;
the manifest's `publisher` signature is the authority, the feed entry is
the announcement.
- **Spam.** A gateway that follows any feed anyone asks for will be filled.
Following is a decision; `c0mpute` index jobs are paid per followed feed
for that reason.
- **Privacy.** A feed is public by construction. A private catalogue is a
segment published as a paid `ipfile` (no `keys.public`), and the entries
are then readable only by pass holders. The head still reveals the seq.
## 10. Events
| Type | Emitted by |
| --- | --- |
| `com.logicsrc.openswarm.ipdb.entry.appended.v1` | writer |
| `com.logicsrc.openswarm.ipdb.head.published.v1` | writer |
| `com.logicsrc.openswarm.ipdb.fork.detected.v1` | replica |
## 11. Implementations
| Piece | Exists | Where | What is new |
| --- | --- | --- | --- |
| Infohash catalogue with browse and search | Yes (central) | `media-streamer` `dht_torrents`, `/api/dht/browse`, `services/dht-search-api` | Learning what an infohash is from a signed feed instead of BEP 9 names |
| Collections, playlists, favourites | Yes (central) | `media-streamer` `collections`, `collection_items` | The `ipdb.playlist` record as the portable form |
| Signed append-only log | Partly | Moshpit `moshpit_tld_log` (server side, `seq` cursor) | The feed as a client-verifiable chain with DHT heads |
| Entry, head, segments, gossip, query shape | No | | All of it |
## 12. Conformance
A replica verifies chains and embedded signatures per §3, detects forks
per §4, replicates by segments and gossip per §5, and answers every
operator in §6.1. A writer never emits a gap. A gateway serves `POST
/db/query` and the head route in §5.3.
## 13. Version history
| Version | Date | Change |
| --- | --- | --- |
| 0.1 | 2026-09-05 | Initial draft. |
Reference in a new issue View git blame Copy permalink