# Docs

- Guides: Using Loam
  - [Introduction](/docs): What Loam is, how the engine stores data, and what works today.
  - [Quickstart](/docs/quickstart): Build the operon binary, write records to a stream and watch a link materialize them.
  - [Roadmap](/docs/roadmap): The milestones and tracks, what each one delivers, and its status.

  - **Concepts**
  - [Data model](/docs/concepts/data-model): Namespaces and the five kinds of objects they contain.
  - [The log and consistency](/docs/concepts/log-and-consistency): How a write travels through Operon and why any read can see it.
  - [Storage and the hot tier](/docs/concepts/storage-and-hot-tier): What lives in the bucket, and what compute nodes keep to make it fast.
  - [Links and workers](/docs/concepts/links-and-workers): Declarative materialization with exactly-once apply, run by a stateless worker pool.

  - **Reference**
  - [API reference](/docs/reference/api): Interactive references for every Loam API, with the status of each.
  - [HTTP API](/docs/reference/http-api): The native JSON API that M0 serves for namespaces, streams and links.
  - [CLI](/docs/reference/cli): The operon binary's commands and flags.
- Design documents: The engine architecture
  - [Design documents](/docs/design): The engine architecture, in reading order, with the glossary.
  - [§00 Pitch](/docs/design/00-pitch): Problem, what Operon is, why now, positioning (unified hybrid retrieval on object storage), competition, non-goals, governance/business model, launch demo
  - [§01 Architecture: Data Model and System Shape](/docs/design/01-architecture): Data model (streams, tables, collections, graphs, links), roles and protocol surfaces, consistency model, the `MetaStore` trait and its backends, S3 layout
  - [§02 Stream Engine](/docs/design/02-stream-engine): The internal log and its APIs: WAL durability classes, write/read/failover protocols, the native streaming API and Flight ingest, OTLP logs ingest, the Kafka gateway, ecosystem integrations, changelog streams
  - [§03 Storage Formats (Durable Tier)](/docs/design/03-storage-formats): Durable tier: Iceberg tables, Lance collections, Tantivy splits, adjacency sidecars, manifests, PK index
  - [§04 Hot Tier & Caching](/docs/design/04-hot-tier): Unified hot-tier model for every object type, incl. Iceberg + Lakekeeper hot tier
  - [§05 Query Engine](/docs/design/05-query-engine): DataFusion embedding, custom operators, hybrid retrieval, consistency tokens, distributed execution, Flight SQL
  - [§06 Search & Vector (Elasticsearch + Qdrant Pillars)](/docs/design/06-search-and-vector): Qdrant and Elasticsearch-subset surfaces, Lance + hot HNSW tiers, API compatibility scope
  - [§07 Graph (native GraphRAG)](/docs/design/07-graph): Native graph for GraphRAG: mapped graphs over collections and tables, CSR/CSC sidecars, `ExpandExec` and shortest path, graph table functions, the `expand` search stage, LightRAG and LlamaIndex adapters
  - [§08 Analytics (Iceberg)](/docs/design/08-analytics): Iceberg analytics: tables via Lakekeeper, keyed tables, materialized views, SQL over Flight SQL and the native API, external engine access
  - [§09 Links & Workers](/docs/design/09-links-and-workers): Declarative materialization (zero-ETL), exactly-once apply, background task scheduling
  - [§10 Operations](/docs/design/10-operations): Deployment, metastore backends, multi-tenancy, security, observability, DR, upgrades, cost model
  - [§11 Buy vs Build](/docs/design/11-buy-vs-build): Every dependency with license, version, verdict; avoid list; what we build (the moat)
  - [§12 Roadmap, Testing & Risks](/docs/design/12-roadmap-testing-risks): Milestones M0–M6 and exit gates (v1.0 = M2, v1.1 = M2.x cloud and BYOC), testing strategy, risk register
  - [§13 Decision Log](/docs/design/13-decision-log): Decisions made so far and open questions
  - [§14 Durable Execution (Resonate Protocol)](/docs/design/14-durable-execution): Resonate protocol surface: durable promises, tasks and schedules on the bucket; phases, consistency, cost
  - [§15 Agent Workspaces (Sandboxes on Operon)](/docs/design/15-agent-workspaces): Operon as the state plane for coding-agent sandboxes: Git on the bucket, copy-on-write environments, registry proxy, caches, sandbox runtimes, sessions as durable executions, MCP gateway with tool retrieval
  - [§16 Demo: 100 Coding Agents on Operon](/docs/design/16-agent-fleet-demo): 100 Claude Code / Codex / opencode sessions on one host and one bucket: density, durability, tool-retrieval savings, analytics with tokscale parity
  - [§17 AI Data Ecosystem](/docs/design/17-ai-data-ecosystem): Operon as a source and sink for AI labs' data pipelines: integration map (Ray Data, Polars, PySpark on Spark 4 and Sail, PyTorch/JAX, Spice, Iceberg engines), scan pinning, retained dataset tags, Python SDK extras, what is not built
  - [§18 Metastore Backends, Tenancy and the Namespace Router](/docs/design/18-metastore-backends-and-router): Postgres, DynamoDB and TiKV backends (TiDB superseded by D124, D260); the relaxed `MetaStore` contract; conformance, fault matrices and CI targets (floci, Alternator, RustFS); consistency tokens on every backend; the router for millions of namespaces and every resource kind; orgs, API keys and quotas; the `Authorizer` trait and OpenFGA; BYOC; GDPR erasure; ids under sharding
  - [§19 Console, Identity and Agents](/docs/design/19-console-identity-and-agents): One console for OSS, Cloud and BYOC; org → projects → environments; agents as principals with short-lived tokens (federation, delegation, vending); OSS sign-in methods; the console API contract and its mock
  - [§20 Loam Live: a Reactive Database on TiKV and the TiKV Metastore](/docs/design/20-reactive-database-on-tikv): The AI-native cloud positioning; Loam Live, a Convex-style reactive database on TiKV (data model, transactions, commit journal, reactivity, QuickJS functions, the connect-rust sync API); keyspaces and the router; the TiKV metastore (TiKV only, no TiDB: D260; MySQL wire access open, Q260); the collections bridge; track R (D116–D131)
  - [§21 Loam Durable: Embedded Resonate and Durable Patterns](/docs/design/21-durable-execution): The Resonate server linked into the binary behind the opt-in `durable` feature (127.0.0.1:8001, D262); SQLite and native TiKV backends (TiDB/`mysql://` legacy, D261); per-namespace instances; Loam's own durable operations (the operations API, bulk import, scheduled import); the seven Resonate patterns mapped to milestones; conformance; licensing and the upstream PR map; the linking spike; track D (D138–D147)
  - [§22 Loam Commons: an Open-Source Showcase Suite on Loam](/docs/design/22-showcase-suite): Plane, Forgejo, Zulip, PostHog and GlitchTip (not Sentry: FSL) deployed unmodified on Loam behind Keycloak OIDC and one cross-app OpenFGA model; the license matrix and obligations; SSO gaps; what Loam replaces and what it does not; provisioning sagas, unified search, the Live activity feed, OTLP and the MCP assistant; Compose and Helm; SC1 (D-SC-1 … D-SC-10)
  - [§23 Neon and WeSQL: Postgres and MySQL on the Bucket, Beside Loam](/docs/design/23-neon-and-wesql): Neon (Apache-2.0) and WeSQL (GPL-2.0-only) run unmodified as sidecars on RustFS; Loam as Neon's control plane (`operon-neon`), metastore mapping, routing by database name in the pg and MySQL listeners, pgoutput and binlog bridges into collections, a Neon branch per agent workspace; Neon proposed for the showcase apps' Postgres OLTP (amends D-SC-12, D-SC-16); upstream status (Neon nearly dormant since 2025-08); spike results; track N (D148–D157)
  - [§24 Loam Functions: a CPU-Time Serverless Runtime](/docs/design/24-cpu-time-runtime): Positioning (agent and backend functions next to Loam's data); tiers (workerd per tenant under gVisor, wasmtime, gVisor, deferred Firecracker); app contracts; the Rust Dapr server and tenant identity; suspend-on-await with Resonate; metering into the WAL; cost model against Cloudflare, Vercel and Lambda; track F (D170–D190; D170–D179, D189 and D190 decided by the owner in conversation, 2026-09-29)
  - [§25 Clever Cloud's Open-Source Stack and the GitOps Deployment](/docs/design/25-clever-cloud-stack): Inventory and verdicts for every relevant Clever Cloud project (licenses, activity); Sōzu vs Envoy vs Pingora; Biscuit vs OpenFGA + OIDC and JWTs; the forked operator and `ObjectStoreProvider`; Argo CD, the umbrella chart and sync waves over RustFS, TiKV (TiDB Operator v2), Resonate, the edge, Loam and the runtime tiers
  - [§26 Loam Jobs: Celery, BullMQ, PySpark and Flink Jobs on Loam](/docs/design/26-jobs-api): Existing jobs on Loam with minimal changes: `operon-jobs` and the `loam.jobs.v1` connect-rust surface (enqueue, lease with fencing, complete, schedule, flow, engine jobs, watch); at-least-once semantics, idempotency, priorities, rate limits, DLQs; the Celery kombu transport, result backend and beat scheduler; `@loam/bullmq` as a BullMQ v6 `IQueueBackend`; Resonate for schedules, flows and opt-in durable tasks; PySpark on Sail with a Spark fallback; Flink SQL on RisingWave and DataStream under the Flink operator; track J (D204–D216; numbers left a gap for docs 23–25)
  - [§27 Usage Hooks: How the Engine Exposes Usage for Metering](/docs/design/27-usage-hooks): The contract behind §24 §7's open metering hooks: the Rust Dapr server does not meter; metric families and labels, the cgroup layout and sandbox labels, per-invocation host reports, Envoy access logs with a gateway-set tenant; metering and billing stay outside this repository and the dependency runs one way (D200–D202)
  - [§28 Loam Postgres: a Neon Fork with Loam's Control Plane and Loam's WAL](/docs/design/28-loam-postgres): CloudNativePG (Postgres 17, Barman Cloud PITR to RustFS) for the showcase apps; `dina-kar/neon` with Loam as the primary control plane (compute spec, storage-controller hooks, proxy auth API); PgDog (AGPL-3.0) as an unmodified router; Loam's WAL replacing safekeepers: the safekeeper protocol in `operon-safekeeper`, TiKV 1PC as the quorum hot tier, the Loam log as the bucket copy; the pgbench merge gate; the fork-maintenance plan; phases P1–P5 (D230–D241)
  - [§29 WeSQL as Loam's MySQL-on-the-Bucket OLTP Engine](/docs/design/29-wesql-oltp): SmartEngine's transactions verified in the source (RC and RR only, snapshot isolation, point locks, no gap locks, no SERIALIZABLE, savepoint and XA gaps); W1 foreign keys in the SQL layer on TiDB's design; W2 the binlog made durable in a Loam WAL quorum (GPL-2.0-only client, Apache-2.0 acceptors); W3 failover by term fencing; W4 binlog to changelog stream to Iceberg with StarRocks as a sidecar (a port rejected); the record corrected on StarRocks, TiDB and WeSQL; when to choose TiDB instead (D273–D280)
  - [§30 Loam CLI, Installer and Agent Bootstrap](/docs/design/30-loam-cli): One `loams` binary (server, client CLI, stdio MCP server; D401); the AWS-style command tree; the `--output json` and exit-code contract; `LOAM_HOME` and profiles; local stacks over `loams dev`/`standalone` with an engine registry (`pg` analytics vs `postgres` OLTP companion); prebuilt variants (`cli`, `standard`, `full`); NVMe for the foyer H1 cache through `sudo loams storage prepare` and new `--cache-*` server flags; `.env.loam` and the no-secrets-through-MCP rule; the bootstrap MCP tools and `mcp install` for Claude Code, Codex, Cursor and Windsurf; embedded, tested docs and snippets; cargo-dist with a minisign-signed manifest, `install.sh` at loams.dev, `self-update`; keys and agent tokens after the auth plan; track CLI (D281–D299)
  - [§31 Loam Router and Verification: Sharded Loam SQL and Loam Postgres, Specified, Simulated and Tested](/docs/design/31-loam-router-and-verification): Sharded Loam SQL and Loam Postgres: WeSQL as the MySQL shard (not InnoDB-on-TiKV, D260); Loam's sharding control plane over unmodified PgDog and Vitess v24 (shard map record, rendering, multi-instance cutover with a Postgres fence, the in-doubt monitor); cross-shard promises; TLA+ specs with trace validation, Lean 4 kernels and the differential oracle, bit-exact simulation of sans-I/O machines with a fault catalog, contract and nemesis tests; the compatibility inventory method; track RT (D300–D322)
  - [§32 Loam Flow, Event Fabric and House](/docs/design/32-loam-flow-fabric-house): Two logs by workload (owner direction, D405): the Loam WAL for databases and retrieval, the Event Fabric (Apache Iggy as ingest log and protocol edge, Apache Fluss as real-time tables tiered to Iceberg) for event ingestion; the CloudEvents envelope on Iggy and Fluss; bridges; Flow routes compiled to existing primitives; Loam House: chDB over Fluss and Iceberg behind the ClickHouse HTTP interface, Tier 1 engines, union reads with consistency tokens; the declared surface `chsurface-1`, the allowlist and the differential harness; Grafeo evaluated and not adopted; track FL (D330–D351)
  - [§33 Loam Flow Connectors: Registry, Capabilities and the Catalog](/docs/design/33-connectors): The connector registry and capability schema; runtimes (native Rust, Iggy's connectors runtime, Camel in `loam-connect`, Debezium Server; Kestra as a companion); envelope and delivery; Arrow/ADBC bulk paths; CDC through Debezium; the 21 ★ connectors; the licence gate; the 200-connector matrix; track CN (D352–D359)
  - [§34 The Standards Charter and the Narrow Waist (the protocol gateway moved)](/docs/design/34-protocol-gateway-and-standards): Stub since 2026-10-02 (D440): the protocol gateway is a commercial component in `loam-platform` (private). Keeps the vendor-neutral decisions: the standards charter, transports, Connect-RPC, `loam.stream.v1` as the narrow waist, the Loam CloudEvents profile (`io.loams.dev.` types, D402), the high-rate path, the events → Arrow mapping, state rules (D360–D365, D372, D374, D378); the `Runner` trait lives in §24 §16 (D375) and runner usage in §27 §3.6 (D376)
  - [§36 Loam Git: a WAL on the Bucket, Smart HTTP, Agent Scopes and a Build Cache](/docs/design/36-loam-git): Extends §15 §3: a per-repository WAL of create-only segments on the bucket (no head pointer; R2's one-write-per-second-per-key limit), CloudEvents records, checkpoints, one-object packs, a group-committing sequencer per repository; `WalStore`, `RefLog`, `BlobStore`, `Materializer`; Smart HTTP (v2 upload-pack, v0/v1 receive-pack) on gitoxide primitives with stock git as oracle and repacker; `git-remote-loam`; `loam-vfs` scopes with write admission; the sccache cache (direct and gateway paths, trust classes); the crates mirror; the `Fs` trait (§17); track GT (D381, D382, D388–D399)
  - [§37 Loams Desktop and Mobile Apps: a cordis Console, a Tauri Shell, Native Phones](/docs/design/37-desktop-and-mobile-apps): The console as a cordis v4 application (host, `loams.yml` catalog, plugin manifests, `rpc.*` Connect services gated on `api_versions`, typed slots, trust tiers with sandboxed third-party frames and attenuated tokens, live reload, editions as plugin sets); Loams Desktop on Tauri 2 (the `loams` CLI as sidecar and supervisor of record, capability lockdown, a Rust network bridge so tokens never reach JavaScript, PKCE and the keychain, signed updates, navigation-only deep links); Loams for iOS (SwiftUI, connect-swift) and Android (Compose, connect-kotlin) with no KMP; pairing as device credentials of a user with a pinned instance key; approvals decided with biometric-bound proofs; HPKE-sealed push through an open gateway; the app protos and a shared mock; track AP (D420–D439)
  - [§38 Knative, Authentik and GitOps for Self-Hosted Loam](/docs/design/38-knative-authentik-gitops): The 2026-10-02 boundary reconfirmation (D403) and what moved to `loam-platform`; Knative Serving for the `http-port` contract (`KnativeRunner`, per-namespace tenancy, Kourier behind the gateway, no meter) and Knative Eventing as an adapter to Loam streams; Authentik's open-source edition as the IdP (D404: the checked feature list, the RFC 8693 exchange for Loam tokens, groups to OpenFGA, blueprints, the Enterprise guard), replacing Keycloak; GitOps with Clever's open-source operator and CKE tooling under Argo CD, new waves, a Flux layout and the k3s profile; track MT (D440–D459)