Design documents
The engine architecture, in reading order, with the glossary.
Operon is an open-source (Apache-2.0) unified hybrid retrieval engine on object storage (D50): stateless compute over open formats in one bucket, with a RAM + NVMe hot tier, serving vector, full-text and GraphRAG retrieval as one planned query. It speaks its native REST/gRPC API, Arrow Flight SQL, the Qdrant API and a targeted Elasticsearch subset (D42); keeps analytics in Iceberg tables that any engine can read; reaches streams through a native streaming API and OTLP logs ingest (both in v1.0) and, from M5, the Kafka wire protocol; and runs durable agent workflows through the Resonate protocol. It replaces the Elasticsearch + Qdrant + Neo4j (+ Kafka + ClickHouse) stack that AI apps deploy today by role, not by wire protocol.
- Status: v0.4 — 2026-10-02 (§29–§38 and the owner rulings D400–D406 folded into the decision log and the plans; the rename to
loamsis pending, see Naming below) · v0.3 — 2026-09-26 (revised after the 2026-09-25 architecture review: decisions D42–D50; metastore backends, router, tenancy and erasure: D58–D70; FoundationDB dropped, streams and Kafka, one router, consistency tokens: D71–D76) - Approved: all documents (§00–§13), 2026-09-23; §14–§16 and the Fluss-derived stream features (
arrowencoding, changelog streams), 2026-09-24; the architecture review's revisions (narrowed protocol footprint, native graph, Iceberg-only analytics, native streams, pluggable metastore, new build order), 2026-09-25; §17 AI data ecosystem (D51–D56), 2026-09-25; §18 metastore backends, tenancy and the namespace router (D58–D70; D67, D69 and D70 are defaults awaiting owner confirmation), 2026-09-26; FoundationDB dropped, the native stream API core and OTLP logs ingest in M2, the Kafka gateway and the RisingWave companion in M5, one router for every resource kind, consistency tokens on every backend (D71–D76), 2026-09-26; §28's owner decisions (CloudNativePG for the showcase, the Neon fork with Loam's control plane, PgDog as an unmodified service, Loam's WAL on TiKV behind the safekeeper protocol and a benchmark gate: D230–D236), 2026-09-29 - Proposed 2026-10-01/02: §30–§38 (D281–D459, with gaps), folded into the decision log on 2026-10-02: the CLI (§30), the router and its verification (§31), Flow, the Event Fabric and House (§32) and connectors (§33), the standards stub (§34), Loam Git (§36), the desktop and mobile apps (§37), Knative, Authentik and GitOps (§38). §35 (the Cloudflare target) and §34's protocol gateway moved to the private
loam-platform. Owner rulings of 2026-10-01/02 are D400–D406: packages and the binary areloams(@loamson npm, Go modulesloams.dev/..., Java deferred), the domain is loams.dev, CloudEvents types areio.loams.dev.<domain>.<name>.v1, the open-core split stands (Knative open, no metering here), Authentik's open-source edition replaces Keycloak and Clerk for OSS, Iggy is the event-ingest layer beside the Loam WAL, and the repository becomesostrium-labs/loams. - Naming: §30–§38 and their plans use the
loamsnames. §00–§29 keep theoperonworking names; the rename PR (D33, D400) renames code, crates and the older docs together. - Open-core boundary: what stays open source in this repository and what belongs only to the managed Loam Cloud platform is set in open-core.md (D220, reconfirmed 2026-10-02 by D403). Commercial designs live in the private
loam-platformrepository.
Reading order
| # | Document | What it covers | Status |
|---|---|---|---|
| 00 | Pitch | Problem, what Operon is, why now, positioning (unified hybrid retrieval on object storage), competition, non-goals, governance/business model, launch demo | Approved |
| 01 | Architecture | Data model (streams, tables, collections, graphs, links), roles and protocol surfaces, consistency model, the MetaStore trait and its backends, S3 layout | Approved |
| 02 | Stream engine | The internal log and its APIs: WAL durability classes, write/read/failover protocols, the native streaming API and Flight ingest, OTLP logs ingest, the Kafka gateway, ecosystem integrations, changelog streams | Approved |
| 03 | Storage formats | Durable tier: Iceberg tables, Lance collections, Tantivy splits, adjacency sidecars, manifests, PK index | Approved |
| 04 | Hot tier & caching | Unified hot-tier model for every object type, incl. Iceberg + Lakekeeper hot tier | Approved |
| 05 | Query engine | DataFusion embedding, custom operators, hybrid retrieval, consistency tokens, distributed execution, Flight SQL | Approved |
| 06 | Search & vector | Qdrant and Elasticsearch-subset surfaces, Lance + hot HNSW tiers, API compatibility scope | Approved |
| 07 | Graph | Native graph for GraphRAG: mapped graphs over collections and tables, CSR/CSC sidecars, ExpandExec and shortest path, graph table functions, the expand search stage, LightRAG and LlamaIndex adapters | Approved |
| 08 | Analytics | Iceberg analytics: tables via Lakekeeper, keyed tables, materialized views, SQL over Flight SQL and the native API, external engine access | Approved |
| 09 | Links & workers | Declarative materialization (zero-ETL), exactly-once apply, background task scheduling | Approved |
| 10 | Operations | Deployment, metastore backends, multi-tenancy, security, observability, DR, upgrades, cost model | Approved |
| 11 | Buy vs build | Every dependency with license, version, verdict; avoid list; what we build (the moat) | Approved |
| 12 | Roadmap, testing, risks | Milestones M0–M6 and exit gates (v1.0 = M2, v1.1 = M2.x cloud and BYOC), testing strategy, risk register | Approved |
| 13 | Decision log | Decisions made so far and open questions | Living |
| 14 | Durable execution | Resonate protocol surface: durable promises, tasks and schedules on the bucket; phases, consistency, cost | Approved (direction); amended by §21 |
| 15 | Agent workspaces | Operon as the state plane for coding-agent sandboxes: Git on the bucket, copy-on-write environments, registry proxy, caches, sandbox runtimes, sessions as durable executions, MCP gateway with tool retrieval | Approved |
| 16 | Agent fleet demo | 100 Claude Code / Codex / opencode sessions on one host and one bucket: density, durability, tool-retrieval savings, analytics with tokscale parity | Approved |
| 17 | AI data ecosystem | Operon as a source and sink for AI labs' data pipelines: integration map (Ray Data, Polars, PySpark on Spark 4 and Sail, PyTorch/JAX, Spice, Iceberg engines), scan pinning, retained dataset tags, Python SDK extras, what is not built | Approved |
| 18 | Metastore backends, tenancy and the namespace router | Postgres, DynamoDB and TiKV backends (TiDB superseded by D124, D260); the relaxed MetaStore contract; conformance, fault matrices and CI targets (floci, Alternator, RustFS); consistency tokens on every backend; the router for millions of namespaces and every resource kind; orgs, API keys and quotas; the Authorizer trait and OpenFGA; BYOC; GDPR erasure; ids under sharding | Approved (three defaults pending) |
| 19 | Console, identity and agents | One console for OSS, Cloud and BYOC; org → projects → environments; agents as principals with short-lived tokens (federation, delegation, vending); OSS sign-in methods; the console API contract and its mock | Proposed |
| 20 | Loam Live: reactive database on TiKV | The AI-native cloud positioning; Loam Live, a Convex-style reactive database on TiKV (data model, transactions, commit journal, reactivity, QuickJS functions, the connect-rust sync API); keyspaces and the router; the TiKV metastore (TiKV only, no TiDB: D260; MySQL wire access open, Q260); the collections bridge; track R (D116–D131) | Proposed (direction approved by the owner) |
| 21 | Loam Durable: embedded Resonate and durable patterns | The Resonate server linked into the binary behind the opt-in durable feature (127.0.0.1:8001, D262); SQLite and native TiKV backends (TiDB/mysql:// legacy, D261); per-namespace instances; Loam's own durable operations (the operations API, bulk import, scheduled import); the seven Resonate patterns mapped to milestones; conformance; licensing and the upstream PR map; the linking spike; track D (D138–D147) | Proposed (direction approved by the owner) |
| 22 | Loam Commons: an open-source showcase suite | Plane, Forgejo, Zulip, PostHog and GlitchTip (not Sentry: FSL) deployed unmodified on Loam behind Keycloak OIDC and one cross-app OpenFGA model; the license matrix and obligations; SSO gaps; what Loam replaces and what it does not; provisioning sagas, unified search, the Live activity feed, OTLP and the MCP assistant; Compose and Helm; SC1 (D-SC-1 … D-SC-10) | Proposed |
| 23 | Neon and WeSQL beside Loam | Neon (Apache-2.0) and WeSQL (GPL-2.0-only) run unmodified as sidecars on RustFS; Loam as Neon's control plane (operon-neon), metastore mapping, routing by database name in the pg and MySQL listeners, pgoutput and binlog bridges into collections, a Neon branch per agent workspace; Neon proposed for the showcase apps' Postgres OLTP (amends D-SC-12, D-SC-16); upstream status (Neon nearly dormant since 2025-08); spike results; track N (D148–D157) | Proposed; amended by §28 2026-09-29 (D149 and D151 superseded, D150 and D153 amended) |
| 24 | Loam Functions: a CPU-time serverless runtime | Positioning (agent and backend functions next to Loam's data); tiers (workerd per tenant under gVisor, wasmtime, gVisor, deferred Firecracker); app contracts; the Rust Dapr server and tenant identity; suspend-on-await with Resonate; metering into the WAL; cost model against Cloudflare, Vercel and Lambda; track F (D170–D190; D170–D179, D189 and D190 decided by the owner in conversation, 2026-09-29) | Proposed |
| 25 | Clever Cloud's open-source stack and the GitOps deployment | Inventory and verdicts for every relevant Clever Cloud project (licenses, activity); Sōzu vs Envoy vs Pingora; Biscuit vs OpenFGA + OIDC and JWTs; the forked operator and ObjectStoreProvider; Argo CD, the umbrella chart and sync waves over RustFS, TiKV (TiDB Operator v2), Resonate, the edge, Loam and the runtime tiers | Proposed |
| 26 | Loam Jobs: Celery, BullMQ, PySpark and Flink | Existing jobs on Loam with minimal changes: operon-jobs and the loam.jobs.v1 connect-rust surface (enqueue, lease with fencing, complete, schedule, flow, engine jobs, watch); at-least-once semantics, idempotency, priorities, rate limits, DLQs; the Celery kombu transport, result backend and beat scheduler; @loam/bullmq as a BullMQ v6 IQueueBackend; Resonate for schedules, flows and opt-in durable tasks; PySpark on Sail with a Spark fallback; Flink SQL on RisingWave and DataStream under the Flink operator; track J (D204–D216; numbers left a gap for docs 23–25) | Proposed (direction approved by the owner) |
| 27 | Usage hooks | The contract behind §24 §7's open metering hooks: the Rust Dapr server does not meter; metric families and labels, the cgroup layout and sandbox labels, per-invocation host reports, Envoy access logs with a gateway-set tenant; metering and billing stay outside this repository and the dependency runs one way (D200–D202) | Proposed |
| 28 | Loam Postgres: a Neon fork with Loam's control plane and Loam's WAL | CloudNativePG (Postgres 17, Barman Cloud PITR to RustFS) for the showcase apps; dina-kar/neon with Loam as the primary control plane (compute spec, storage-controller hooks, proxy auth API); PgDog (AGPL-3.0) as an unmodified router; Loam's WAL replacing safekeepers: the safekeeper protocol in operon-safekeeper, TiKV 1PC as the quorum hot tier, the Loam log as the bucket copy; the pgbench merge gate; the fork-maintenance plan; phases P1–P5 (D230–D241) | Approved direction (owner, 2026-09-29; the WAL is gated on benchmarks; D237–D241 proposed) |
| 29 | WeSQL as Loam's MySQL-on-the-bucket OLTP engine | SmartEngine's transactions verified in the source (RC and RR only, snapshot isolation, point locks, no gap locks, no SERIALIZABLE, savepoint and XA gaps); W1 foreign keys in the SQL layer on TiDB's design; W2 the binlog made durable in a Loam WAL quorum (GPL-2.0-only client, Apache-2.0 acceptors); W3 failover by term fencing; W4 binlog to changelog stream to Iceberg with StarRocks as a sidecar (a port rejected); the record corrected on StarRocks, TiDB and WeSQL; when to choose TiDB instead (D273–D280) | Proposed |
| 30 | Loam CLI, installer and agent bootstrap | One loams binary (server, client CLI, stdio MCP server; D401); the AWS-style command tree; the --output json and exit-code contract; LOAM_HOME and profiles; local stacks over loams dev/standalone with an engine registry (pg analytics vs postgres OLTP companion); prebuilt variants (cli, standard, full); NVMe for the foyer H1 cache through sudo loams storage prepare and new --cache-* server flags; .env.loam and the no-secrets-through-MCP rule; the bootstrap MCP tools and mcp install for Claude Code, Codex, Cursor and Windsurf; embedded, tested docs and snippets; cargo-dist with a minisign-signed manifest, install.sh at loams.dev, self-update; keys and agent tokens after the auth plan; track CLI (D281–D299) | Proposed |
| 31 | Loam Router and verification | Sharded Loam SQL and Loam Postgres: WeSQL as the MySQL shard (not InnoDB-on-TiKV, D260); Loam's sharding control plane over unmodified PgDog and Vitess v24 (shard map record, rendering, multi-instance cutover with a Postgres fence, the in-doubt monitor); cross-shard promises; TLA+ specs with trace validation, Lean 4 kernels and the differential oracle, bit-exact simulation of sans-I/O machines with a fault catalog, contract and nemesis tests; the compatibility inventory method; track RT (D300–D322) | Proposed |
| 32 | Loam Flow, Event Fabric and House | Two logs by workload (owner direction, D405): the Loam WAL for databases and retrieval, the Event Fabric (Apache Iggy as ingest log and protocol edge, Apache Fluss as real-time tables tiered to Iceberg) for event ingestion; the CloudEvents envelope on Iggy and Fluss; bridges; Flow routes compiled to existing primitives; Loam House: chDB over Fluss and Iceberg behind the ClickHouse HTTP interface, Tier 1 engines, union reads with consistency tokens; the declared surface chsurface-1, the allowlist and the differential harness; Grafeo evaluated and not adopted; track FL (D330–D351) | Proposed (D347 reverses D45 for the House pending Q333) |
| 33 | Loam Flow connectors | The connector registry and capability schema; runtimes (native Rust, Iggy's connectors runtime, Camel in loam-connect, Debezium Server; Kestra as a companion); envelope and delivery; Arrow/ADBC bulk paths; CDC through Debezium; the 21 ★ connectors; the licence gate; the 200-connector matrix; track CN (D352–D359) | Proposed |
| 34 | Standards charter and the narrow waist | Stub since 2026-10-02 (D440): the protocol gateway is a commercial component in loam-platform (private). Keeps the vendor-neutral decisions: the standards charter, transports, Connect-RPC, loam.stream.v1 as the narrow waist, the Loam CloudEvents profile (io.loams.dev. types, D402), the high-rate path, the events → Arrow mapping, state rules (D360–D365, D372, D374, D378); the Runner trait lives in §24 §16 (D375) and runner usage in §27 §3.6 (D376) | Stub (decisions Proposed) |
| 35 | Cloudflare deployment target | Moved to loam-platform (private), 2026-10-02 (D403, D440). The Fs trait and object-store providers it held (D381, D382) are in §36 §17 | Moved |
| 36 | Loam Git | Extends §15 §3: a per-repository WAL of create-only segments on the bucket (no head pointer; R2's one-write-per-second-per-key limit), CloudEvents records, checkpoints, one-object packs, a group-committing sequencer per repository; WalStore, RefLog, BlobStore, Materializer; Smart HTTP (v2 upload-pack, v0/v1 receive-pack) on gitoxide primitives with stock git as oracle and repacker; git-remote-loam; loam-vfs scopes with write admission; the sccache cache (direct and gateway paths, trust classes); the crates mirror; the Fs trait (§17); track GT (D381, D382, D388–D399) | Proposed (amends §15 §3, approved) |
| 37 | Loams desktop and mobile apps | The console as a cordis v4 application (host, loams.yml catalog, plugin manifests, rpc.* Connect services gated on api_versions, typed slots, trust tiers with sandboxed third-party frames and attenuated tokens, live reload, editions as plugin sets); Loams Desktop on Tauri 2 (the loams CLI as sidecar and supervisor of record, capability lockdown, a Rust network bridge so tokens never reach JavaScript, PKCE and the keychain, signed updates, navigation-only deep links); Loams for iOS (SwiftUI, connect-swift) and Android (Compose, connect-kotlin) with no KMP; pairing as device credentials of a user with a pinned instance key; approvals decided with biometric-bound proofs; HPKE-sealed push through an open gateway; the app protos and a shared mock; track AP (D420–D439) | Proposed |
| 38 | Knative, Authentik and GitOps | The 2026-10-02 boundary reconfirmation (D403) and what moved to loam-platform; Knative Serving for the http-port contract (KnativeRunner, per-namespace tenancy, Kourier behind the gateway, no meter) and Knative Eventing as an adapter to Loam streams; Authentik's open-source edition as the IdP (D404: the checked feature list, the RFC 8693 exchange for Loam tokens, groups to OpenFGA, blueprints, the Enterprise guard), replacing Keycloak; GitOps with Clever's open-source operator and CKE tooling under Argo CD, new waves, a Flux layout and the k3s profile; track MT (D440–D459) | Proposed |
Glossary
| Term | Meaning |
|---|---|
| Org | Tenant and billing unit (D65): owns namespaces, API keys, role bindings and quotas, held in the ControlStore. |
| Namespace | Database boundary inside an org. Unit of isolation, quota, encryption key, routing and metastore sharding. |
| Stream | Partitioned, ordered, offset-addressed log. Every write in Operon lands in a stream (explicit or implicit); explicit streams are read and written through the native streaming API and Flight. |
| Partition | Ordered unit of a stream; offsets are dense per partition. |
| Table | Columnar analytical dataset, stored as an Iceberg table catalogued in Lakekeeper; queried through Flight SQL and the native API, and readable by any Iceberg engine. |
| Collection | Document set with vectors, full-text, and filters; stored as Lance dataset + Tantivy splits. Elasticsearch index / Qdrant collection equivalent. |
| Graph | Mapped graph whose vertex/edge types map onto tables or collections, plus CSR/CSC adjacency sidecars; traversed by 1–2 hop expansion and shortest path inside a query. |
| Link | Declared, continuously maintained materialization between objects (stream→table, stream→collection, table→collection, tables→graph). |
| Applied offset | Per link: the highest source offset reflected in the target's durable state. |
| Consistency token | Set of (stream, partition, offset) a reader requires to be visible. Returned by every write. It holds on every metastore backend and node (D76); clients may omit it. |
| Durable tier | Open formats on object storage. The only source of truth. |
| Hot tier | Derived, node-local, rebuildable acceleration structures (caches, HNSW, projections, in-RAM CSR). Never the source of truth. |
| Tail | Data committed to the log but not yet in the durable indexed form; merged into every read. |
| WAL class | Durability/latency class for a stream: standard, express, quorum (see §02). |
| Journal | A 3-replica Raft group (one node per AZ) that hosts the quorum WAL for a set of partitions. |
| Manifest | Immutable description of an object's durable state at a version; the pointer to the current manifest lives in the metastore. |
| MetaStore | The semantic metastore trait in operon-common (D47): domain operations (WAL commit, catalog, manifest-pointer CAS, leases and fencing), implemented by embedded openraft (default), Postgres and DynamoDB (M2) and TiKV (R1, D124; no TiDB, D260), under the relaxed contract of D59. |
| Scan plan | A collection resolved (at the current manifest, a manifest version, a consistency token or a dataset tag) into what an external reader needs: the manifest version, the Lance dataset URI and detached version id, the fragments with row counts, per-fragment Flight tickets, and the tail if one exists (D53, §17 §3). |
| Dataset tag | A named, immutable pointer to one collection manifest (from M4, one Iceberg snapshot) that GC retains until the tag is deleted; tagged reads never need the tail (D52, §17 §4). |
| Split | Immutable Tantivy index bundle with a hotcache footer (Quickwit design). |
| Segment encoding | How a stream's records are stored in WAL chunks and segments: kafka (the Kafka RecordBatch v2 layout, used internally) or arrow (columnar, for schema'd streams). |
| Changelog stream | A stream of row-level changes (+I, -U, +U, -D) of a keyed table or collection. |
| Durable promise | Resonate's unit of durable execution: a promise, keyed by a deterministic id, that survives process restarts and records a step's result. |
| Origin | The part of a Resonate id before the first :; all of one origin's promises and tasks are one document, committed atomically. |
| Event Fabric | Loam's event-ingestion layer: Apache Iggy (ingest log and protocol edge) and Apache Fluss (real-time Log and PK tables tiered to Iceberg), separate from the Loam WAL (D331–D333, D405). |
| Loam House | The query and serving layer of §32: chDB behind the ClickHouse HTTP interface over Fluss tables and Iceberg snapshots, plus Loam's vector and graph surfaces (D342–D347). |
| Declared surface | A versioned list of the ClickHouse features Loam supports (chsurface-1), each proven by tests; deviations are on the allowlist (D347, D348). |
| Connector manifest | A connector's declared capabilities, runtime, licence, auth and config schema (loam.flow.v1.ConnectorSpec, D352, D353). |
| Stack | A local Loam deployment the loams CLI creates and supervises from a stack.toml, over loams dev or standalone (D285). |
| Shard map | The per-database record of shards, key ranges and routing generation that Loam's router control plane keeps in the TiKV metastore and renders into PgDog or mirrors from Vitess (D304). |
| Ref log | Loam Git's per-repository WAL of create-only segments plus checkpoints on the bucket; the commit point is the create-only PUT of the next segment (D388, D389). |
| Runner | An execution target for functions behind the Runner trait: the node supervisor (default), process, Lambda and Knative runners, or external ones (D375, D441). |
Conventions
- (verify) marks a claim that came from a single secondary source or could not be confirmed during research (2026-09-22). Resolve before relying on it.
- Latency/cost numbers are design targets or published figures from reference systems, not measured Operon results.
- Dependency versions are as of 2026-09-22; see §11.