The small crates that carry a lot: SlateDB, foyer, openraft, redb, qdrant-edge and connect-rust

Some dependencies define an architecture; others quietly hold it up. This post covers six in the second group. All six are linked into the Loam binary today.

SlateDB: an LSM tree whose disk is a bucketCopy Anchor Link

What it is. SlateDB is an embedded key-value store, an LSM tree, that writes directly to object storage. Writes go to a write-ahead log and an in-memory table; the WAL is flushed to the bucket in batches, and full memtables become sorted string tables (SSTs) in the bucket, compacted in the background. It supports a single writer per database, fenced so that a new writer cuts off an old one, which is how a stateless process can own a database safely. It is part of the Commonhaus Foundation.

How Loam uses it. Upserts and deletes need to know where a key's current row lives. Loam's PkIndex is one SlateDB database per keyed object, under ns/<ns>/pk/<object>/ in the bucket, mapping a primary key to its Lance stable row id. It is written only by the worker that owns the object, fenced by that worker's lease epoch, which matches SlateDB's single-writer model exactly. It is derived state: it records a watermark (the manifest and offsets it reflects), and a new handle replays per-commit key deltas newer than its watermark, or rebuilds from Lance if a delta is gone. Graph vertex-id maps will use the same abstraction.

Why. An index that lives in the bucket, like everything else in the engine, with no local state a node could lose. Limits: pre-1.0, with no API compatibility promise between versions, and one writer per database by design.

foyer: a hybrid RAM and disk cacheCopy Anchor Link

What it is. foyer is a hybrid cache library: an in-memory tier with pluggable eviction policies, backed by a disk tier on local SSD, behind one API. RisingWave uses it for its object-storage cache.

How Loam uses it. Loam's H1 cache holds byte ranges of objects from the bucket (Lance pages, Tantivy postings, split footers) in RAM and on NVMe, with checksums. Every durable object in Loam is immutable and named by a ULID or version, so the cache never needs invalidation: a key's bytes can never change. Losing a node loses cache warmth only. A small in-memory cache (moka) holds parsed metadata above it.

openraft and redb: the default metastoreCopy Anchor Link

What they are. openraft is an async Raft consensus library in Rust, maintained by the Databend team, which runs it in production for Databend's meta service. redb is an embedded, pure-Rust, ACID key-value store with copy-on-write B-trees and an fsync per commit.

How Loam uses them. Loam's default metastore is embedded Raft, in the style of Kafka's KRaft or ClickHouse Keeper: offsets, leases and manifest pointers change too often for S3 conditional writes, so they live in a replicated state machine. openraft provides consensus; redb holds the Raft log, the vote and the snapshot pointer, and passes openraft's storage test suite; state-machine snapshots go to the bucket; entries are encoded with postcard. operon dev runs one node; clusters run three or five voters, and every other node holds a non-voting learner replica so reads stay local. TiKV is the alternative backend for clusters that run it (TiKV post), and Postgres and DynamoDB backends are planned. All backends are held to one conformance suite with linearizability checks.

Limits. openraft 0.10 is still in alpha, and alphas break APIs, so Loam pins an exact version and moves it deliberately. A single metastore holds the whole catalog, so metadata sharding for millions of namespaces is designed but not built.

qdrant-edge: Qdrant's index code, as a libraryCopy Anchor Link

What it is. qdrant-edge packages Qdrant's vector index code as an embeddable Rust crate: HNSW graph construction and search, payload-aware (filterable) HNSW links, and scalar, product and binary quantization.

How Loam uses it. Cold vector search uses Lance's IVF index. Hot collections get an HNSW graph built with qdrant-edge from a collection manifest, stored back in the bucket as an artifact (a descriptor, the covered row set and the engine files in compressed chunks), so another node can load it instead of rebuilding. Point ids are Lance stable row ids, so a graph stays valid across compaction and split merges. Only one crate, operon-hnsw, names qdrant-edge; everything else uses Loam's HnswIndex traits, beside an exact flat engine used for testing. A differential test checks results are identical with the hot tier on and off.

Why not Qdrant itself. Qdrant's storage is designed for local disk. Loam needs its index code only, as a derived, rebuildable cache over object storage.

Limits. qdrant-edge enables insertion-ordered JSON maps, and Cargo unifies features, so the whole workspace uses that map type. Its sparse-vector index is private and local-directory-only, so Loam built its own exact sparse scoring instead.

connect-rust and buffa: one protobuf toolchainCopy Anchor Link

What they are. connect-rust implements the Connect protocol for Rust: one service definition served over Connect, gRPC and gRPC-Web on one handler, with generated clients. buffa generates the protobuf message types.

How Loam uses them. The Loam Live sync API (loam.live.v1) is protobuf, with Rust code generated at build time and clients for every platform generated with Buf: protobuf-es and Connect for the web, connect-go, connect-swift and connect-kotlin. The same toolchain is planned for the native stream API's gRPC surface and the jobs API, so every protobuf surface in Loam shares one generator and one set of client conventions. It is why Live's session protocol is a server stream plus unary calls: Connect's server streams work over HTTP/1.1 and in browsers, where full-duplex streams do not.

Limits. connect-rust is young (0.9), and on HTTP/1.1 it sends no response until a request body is complete, which rules out bidirectional streaming for browser clients anyway.

The rule behind all of themCopy Anchor Link

Every one of these is permissively licensed, Rust, and used for exactly one job behind a trait or a crate boundary Loam controls, so it can be replaced without touching the rest. That is how we try to buy rather than build: take the best open-source piece for each job, pin it, test it in our own conformance suites, contribute fixes upstream, and keep the parts that are genuinely new, such as the hybrid planner, the serving layer and the reactive core, as Loam's own.

That is the end of the "What Loam is built on" series. The platform series explains how these pieces come together.