§28 Loam Postgres: a Neon Fork with Loam's Control Plane and Loam's WAL

Status: Approved direction · 2026-09-29. On 2026-09-29 the owner amended §23 (PR #111) with these decisions:

• The showcase apps run on CloudNativePG.
• Neon is forked, with Loam's control plane as the primary control plane.
• PgDog does the routing, and only as an unmodified separate service.
• Loam's own WAL replaces Neon's safekeepers.

The owner then chose the WAL design:

• Option A. TiKV is the quorum hot tier, and Loam's log group-commits to the bucket behind it.
• A Loam crate speaks Neon's safekeeper protocol, so walproposer and the pageserver stay unmodified.
• A pgbench merge gate gates the switch-over.

These are decisions D230–D236, all owner-approved on 2026-09-29. The WAL replacement (D233) is approved direction: it becomes the default only if the benchmark gate passes. The concrete WAL design is D237–D241. They are Loam's proposals within the owner's decisions and are marked as such. Open questions are Q110–Q119. Arm A of the WAL (§7.2, 2026-09-30) adds D263–D269 and D271 (D270 is the CloudEvents decision; D263, compio for the data path, is the owner's) and Q261–Q264.

This document supersedes §23's D149 (Neon for the showcase apps) and D151 (fork only when needed). It amends D150 (the control plane grows from a client into the primary control plane) and D153 (PgDog, not Loam's pg listener, splices Postgres connections to computes). D148, D152, D154 and D155 stand. D156 and D157's WeSQL row are not affected.

Markers:

• (spike) means measured on 2026-09-29 on the development machine. That machine is a 14-core, 15 GB laptop with consumer NVMe and btrfs, and other sessions' builds were running, with a load average between 1.5 and 24. The notes are in .superpowers/research/loam-postgres-spike.md, local and not committed. Spike numbers compare paths on the same hardware. They are not production figures.
• (verify) means not checked against a primary source.
• (estimate) means computed, not measured.
• Paths of the form neon/… point into dina-kar/neon at fa504217c (2026-08-31, identical to upstream main).

1. SummaryCopy Anchor Link

2. Goals and non-goalsCopy Anchor Link

2.1 GoalsCopy Anchor Link

1. Real Postgres for the showcase apps now. Plain Postgres on CloudNativePG, with PITR to the same RustFS store. It takes no fork work (D230).
2. Serverless, branchable Postgres on the bucket as a Loam product. Neon's storage (pageserver, layers in the bucket, copy-on-write branches) runs under Loam's control plane, with no dependency on a vendor's closed services (D231, D232).
3. A WAL service that beats safekeepers where it counts.
   • Operations: no stateful safekeeper fleet, and no membership migrations.
   • Durability: the bucket copy is seconds old, not segments old.
   • Cost: TiKV nodes are shared with the metastore and Loam Live.
   • Latency: commit latency stays at or under the baseline, and the gate enforces it (D233).
4. License-clean. Apache-2.0 for everything Loam links (D11). PgDog, which is AGPL-3.0, runs only as an unmodified process (D236).

2.2 Non-goalsCopy Anchor Link

• Changing walproposer or the pageserver's WAL receiver. Option C (a Loam protocol inside walproposer) is rejected (§6.2).
• Merging branches. As in §23: Postgres cannot merge timelines (D155).
• Neon's hosted extras. The console, the Data API, Neon Auth and autoscaling VMs are out of scope. Scale-to-zero is Q113.
• Upstream contributions. Nothing is posted to neondatabase, pgdogdev or cloudnative-pg without the owner's approval.
• Modifying TiKV or PD. D126 holds: official releases, configured, never patched. Client-side work goes into Loam's client-rust fork.

3. ArchitectureCopy Anchor Link

  apps / agents / psql                                   showcase apps (Plane, Zulip, GlitchTip,
        │ postgres://…/<db>[__<branch>]                    Keycloak, OpenFGA)
        ▼                                                      │ postgres://
 ┌──────────────────────────────┐                     ┌────────┴──────────────────────────────┐
 │ PgDog (AGPL-3.0, unmodified, │                     │ CloudNativePG cluster (P1, D230)      │
 │ separate pod): pools, routes │                     │ Postgres 17.11, primary + replica     │
 │ by database name, balances   │                     │ Barman Cloud plugin ──► RustFS (PITR) │
 │ reads across replicas        │                     └───────────────────────────────────────┘
 └──────┬───────────────────────┘
        │ config + RELOAD from Loam's control plane
        ▼
 ┌──────────────────────────────┐   spec (GET …/computes/{id}/spec)   ┌─────────────────────────────┐
 │ Loam Postgres compute        │◄────────────────────────────────────┤ Loam control plane (D232)   │
 │ (fork image, compute_ctl,    │                                     │ in `operon`, state in TiKV  │
 │ walproposer, unmodified)     │                                     │ tenants / timelines /       │
 └──────┬──────────────┬────────┘                                     │ branches / computes;        │
        │ safekeeper   │ GetPage@LSN                                  │ /notify-attach,             │
        │ protocol v3  │                                              │ /notify-safekeepers,        │
        ▼              ▼                                              │ proxy auth API (optional)   │
 ┌──────────────────────────┐   ┌─────────────────────────────────┐   └──────┬──────────────────────┘
 │ Loam WAL service (P4)    │   │ pageserver (fork, unmodified    │          │ mgmt API
 │ `operon-safekeeper`,     ├──►│ WAL receiver)                   │◄─────────┤ storage controller
 │ role `wal`, stateless,   │   │ layers ──► bucket               │          │ (fork; its DB on CNPG,
 │ one pool per AZ          │   └─────────────────────────────────┘          │ TiKV later, P5)
 │ ├─ 1PC txn ──► TiKV keyspace `loam_pgwal` (3 replicas, leader in the compute's AZ)
 │ └─ group commit (250 ms) ──► Loam log (operon-log `standard`) ──► bucket
 │ publishes SafekeeperTimelineInfo ──► storage broker (fork, unmodified)
 └──────────────────────────┘
      stock safekeepers ×3 remain the default until the P4c gate passes (P2)

 RustFS: bucket `loam` (Loam, including the WAL log), bucket `neon` (layers), bucket `cnpg` (Barman)

3.1 ComponentsCopy Anchor Link

4. CloudNativePG for the showcase apps (D230, P1)Copy Anchor Link

Why. The showcase apps need Postgres now. Neon's fork work (§10) and the WAL service (§6) take quarters, and D-SC-12's Postgres front end over TiKV takes years. CloudNativePG is plain Postgres with a mature operator, PITR to object storage, and no fork. That is the same "plain Postgres as the exit" that D151 described.

Verified 2026-09-29:

Layout.

• One Cluster, commons-pg, for the suite. It has instances 2 (a primary and a streaming replica), image ghcr.io/cloudnative-pg/postgresql:17.11-standard-trixie pinned by digest, and one database and owner role per app. This keeps §22 §6.4's "one Postgres 17 with a database per app".
• Zulip needs PGroonga, which the standard image lacks. It gets an image built on the standard image with PGroonga, or a separate Cluster (Q111).
• Backups:
  • An ObjectStore named commons-rustfs sets destinationPath: s3://cnpg/commons-pg, endpointURL: http://rustfs:9000, credentials from a Secret, wal.compression: zstd and the two checksum variables above.
  • The Cluster declares plugins: [{name: barman-cloud.cloudnative-pg.io, isWALArchiver: true, parameters: {barmanObjectName: commons-rustfs}}].
  • A ScheduledBackup with method: plugin runs daily.
• PITR is a new Cluster with bootstrap.recovery.source: origin and recoveryTarget.targetTime, and externalClusters: [{name: origin, plugin: {name: barman-cloud.cloudnative-pg.io, parameters: {barmanObjectName: commons-rustfs, serverName: commons-pg}}}]. P1's test restores to a timestamp between two known writes.
• Dev without Kubernetes is not supported by CNPG. The suite's compose keeps a plain postgres:17.11 container. P1 adds deploy/cnpg/ with a kind or k3d manifest set for the PITR test.

Loam integration. The same as §23 for everything that addresses "a Postgres backend": the x/ record carries engine = postgres (D152), and the pgoutput bridge (D154) runs against CNPG's primary, with wal_level = logical set in the Cluster's postgresql.parameters. Branch-per-workspace (D155) needs Loam Postgres and is not offered on CNPG.

5. The Loam control plane (D232)Copy Anchor Link

§23's operon-neon was a client of Neon's APIs. Here Loam is the control plane that Neon's components call, so it serves as well as calls. The endpoints below come from the fork's code, which is the only specification.

5.1 What Loam callsCopy Anchor Link

5.2 What Loam servesCopy Anchor Link

The pageserver's upcalls (POST /upcall/v1/re-attach, POST /upcall/v1/validate) go to the storage controller, which issues the generations that fence S3 writes (Neon RFC 025). Loam does not replace them in P2. The storage controller runs from the fork, with its Postgres database on CNPG (Q48 answered: a small CNPG Cluster). A TiKV-backed replacement is P5 (§9).

5.3 StateCopy Anchor Link

Writes are compare-and-set on a version, as in §23 §6.2. The spec endpoint is idempotent: it returns the same spec for the same compute id until the branch record changes.

6. Replacing the safekeepers (D233–D235, D237–D241)Copy Anchor Link

6.1 What safekeepers do todayCopy Anchor Link

Read from the fork (neon/pgxn/neon/walproposer*.c, neon/safekeeper/src/, neon/pageserver/src/tenant/timeline/walreceiver/, neon/docs/walservice.md, RFCs 013, 035 and 041):

• Commit path.
  • walproposer is a background worker that poses as a synchronous standby (synchronous_standby_names = 'walproposer').
  • A committing backend waits in stock SyncRepWaitForLSN.
  • walproposer computes commitLsn as the quorum (n/2 + 1) of the safekeepers' AppendResponse.flush_lsn, counting only WAL of its own term, and releases the waiters through ProcessStandbyReply.
  • A commit returns when a majority has fdatasynced its WAL (walproposer.c GetAcknowledgedByQuorumWALPosition; walproposer_pg.c walprop_pg_process_safekeeper_feedback).
• Safekeeper write path.
  • Each AppendRequest (at most 128 KiB) is written without fsync.
  • One fdatasync covers everything queued when the inbound queue drains, or after at most 1 s (receive_wal.rs FLUSH_INTERVAL).
  • Segments are preallocated, so fdatasync suffices.
  • The control file is saved every 300 s or every segment, and at once on votes and term changes.
• Offload.
  • One elected offloader per timeline uploads full 16 MiB segments once commit_lsn passes them.
  • The current partial segment is uploaded within 15 minutes (wal_backup.rs, wal_backup_partial.rs).
  • Local WAL is removed below min(remote_consistent_lsn, backup_lsn, commit_lsn, flush_lsn).
• Consumers.
  • The pageserver finds safekeepers through the storage broker (SafekeeperTimelineInfo, pushed every 1 s).
  • It picks the one with the highest commit_lsn, preferring its own AZ.
  • It reads with START_REPLICATION PHYSICAL using the interpreted protocol (protobuf and zstd; records decoded and sharded on the safekeeper; libs/wal_decoder).
  • It reports remote_consistent_lsn back in a 'z' status update.
• The machinery around it:
  • term histories and truncation;
  • peer recovery;
  • timeline eviction;
  • partial-segment GC;
  • two-phase membership changes with generations and pull_timeline;
  • a storage-controller scheduler for safekeepers.

What the repository does not say. Neither the fork nor Neon's site publishes a safekeeper commit latency. Neon's architecture page says only that "commit latency is primarily quorum/network-bound" (neon.com/docs/introduction/architecture-overview). The one third-party figure found is ClickHouse's PostgresBench (2026-07-21): Neon at 16 vCPU ran 7,802 TPS with p50 32.8 ms and p99 56.3 ms. That is pgbench latency under load, not a commit round trip (clickhouse.com/blog/postgresbench-ha). The production baseline must therefore be measured (P4b). The estimate for three safekeepers across AZs is one cross-AZ round trip plus an NVMe fdatasync, about 1–3 ms at p50 (estimate, §6.6).

Measured on this machine (spike). The stack was deploy/neon/ with one safekeeper (fsync on, WAL in the container filesystem), Postgres 16.9, and pgbench run inside the compute:

So the safekeeper ack costs about 3.5 ms at p50 on this hardware, almost all of it the fdatasync of a consumer SSD under btrfs.

6.2 OptionsCopy Anchor Link

Chosen: A behind B (D234, D235, owner-approved). TiKV provides the quorum, and a Loam crate speaking the safekeeper protocol sits in front. C is rejected: it moves the risk into C code that every rebase must carry.

6.3 One logical acceptor per timeline (D239)Copy Anchor Link

Walproposer is Paxos over n independent acceptors. Three Loam acceptors writing to one TiKV store would triple every write for no gain, because TiKV already replicates across three AZs. So:

• The compute's neon.safekeepers names one endpoint. It is a per-AZ Service address of the WAL pool, so walproposer runs with n = 1 and quorum 1. Neon runs this way in development today: deploy/neon has one safekeeper, and so does the spike.

• The acceptor's state lives in TiKV, not in a process: term, term history, flush_lsn, commit_lsn, peer_horizon_lsn, backup_lsn and remote_consistent_lsn. Any instance of the pool can serve any timeline. An instance that picks up a connection behaves exactly like a safekeeper that restarted with its disk intact: it answers the greeting from TiKV's state, refuses a vote for a term it has already voted in, and accepts ProposerElected from the current proposer.

• Losing an instance is a reconnect, not a failover. walproposer reconnects after neon.safekeeper_reconnect_timeout (default 1 000 ms; a PGC_SIGHUP GUC with minimum 0, walproposer_pg.c; Loam sets 100 ms). Nothing is copied, and nothing needs electing on the Loam side.

• What disappears:

  • membership changes and generations (the generation is fixed at 1 with one member, the Service);
  • pull_timeline and peer recovery;
  • timeline eviction and partial-segment uploads;
  • the storage controller's safekeeper scheduler.

  That is thousands of lines of Neon's safekeeper and controller code, and the operational runbooks that go with them.

• Sidecar rejected. A WAL process in the compute pod saves one in-AZ hop (about 0.1–0.2 ms, estimate). But it dies with the compute, while the pageserver still has to read the tail after the compute stops. It also multiplies instances by computes. The shared pool is stateless and needs no compute integration.

6.4 The TiKV API (D237)Copy Anchor Link

The fence. A WAL write is valid only if the timeline's term is still the proposer's term when it lands. This is check_fence's rule in operon-meta-tikv (leases.rs): read the fence key for update in the same transaction as the write, compare the epoch, and abort on mismatch. The WAL service applies it to the timeline's head key:

append(tl, term, begin_lsn, bytes, commit_lsn):
  txn (optimistic, use_async_commit, try_one_pc):
    head = get(H(tl))                      # fence read
    if head.term != term  -> abort: reply AppendResponse{term: head.term}   # deposed
    if end_lsn <= head.flush_lsn -> fully covered retry: write no chunk; put the head only if
                                    commit_lsn grew; reply with the existing flush_lsn
    if head.flush_lsn > begin_lsn -> drop the overlapping prefix (same term: same bytes)
    if head.flush_lsn < begin_lsn -> error (gap), the proposer reconnects
    put(W(tl, max(begin_lsn, head.flush_lsn)), remaining bytes)   # one key per ≤128 KiB chunk
    put(H(tl), head{flush_lsn = end_lsn, commit_lsn = max(head.commit_lsn, commit_lsn)})
                                           # end_lsn > head.flush_lsn here: flush_lsn only advances
  commit (1PC when H and W are in one region, else async commit)
  -> AppendResponse{term, flush_lsn = end_lsn, commit_lsn, …}

• The head is written, not only read. A concurrent term bump (a vote handled by another instance) therefore conflicts write-write in either order:

  • If the bump committed before this transaction's start_ts, the fence read sees the new term.
  • If it committed after, one of the two prewrites fails.

  No timing assumption is involved, which a lease-and-wait scheme would need.

• A vote is the same transaction shape: read the head, grant the vote only if head.term < term, and write the head with the new term. The term is durable before VoteResponse, as in handle_vote_request.

• ProposerElected truncates at start_streaming_at, in one transaction. A chunk key covers a range of LSNs, so the chunk that contains start_streaming_at is rewritten with only its bytes below that LSN; every chunk starting at or above it is deleted; and the head gets the new term history and flush_lsn = start_streaming_at. Committed WAL is never truncated (start_streaming_at ≥ commit_lsn is asserted, as in handle_elected).

Why not RawKV. A RawKV write cannot check another key. The fenced raw scheme is "put the chunk, then CAS the head", which is two sequential Raft writes in atomic-CAS mode. A raw put without the CAS lets a deposed proposer's write be acknowledged. With one acceptor, a deposed compute would then count a commit that the new term truncates: an acknowledged transaction lost. The spike measured the options on one TiKV store (playground v8.5.8, API v2, default configuration, 1 000 sequential operations, two runs):

On this machine RawKV was not faster than a 1PC transaction. The fenced raw scheme was 2–4× slower at p50, and 2PC was about 3× slower than 1PC. The single-store playground has no network replication, so every figure here is "local Raft fsync plus apply plus client". The absolute values are inflated by the shared, loaded machine and must be re-measured on a three-node cluster (Q114). The ordering is the result: TxnKV 1PC (D237).

Published figures agree on the ordering:

• Async commit and 1PC (TiDB 5.0 release notes): Sysbench update-index average latency fell 41.7% (12.04 → 7.01 ms), and oltp-insert latency fell 37.3%. A single-region transaction skips the second phase altogether.
• Atomic-mode raw writes (raw_compare_and_swap_atomic, raw_batch_put_atomic) run as scheduler commands that take per-key latches (tikv/src/storage/txn/commands/atomic_store.rs). TiKV's docs say that "write operations like put or delete in atomic mode are more expensive" (tikv.org/docs/7.1/develop/rawkv/cas).
• RawKV on API v2 takes a causal timestamp from a TiKV-local TSO cache, refreshed every 100 ms, plus a concurrency-manager guard (tikv/src/storage/mod.rs). It avoids the PD round trip that a transaction's start_ts needs, but not the fence problem.
• No current TiKV page gives RawKV put p99 for WAL-sized values. The v6.1 performance overview gives throughput only: 43.2 K updates/s on YCSB workload A on 3 × 40 vCPU, and "at most 200,000 OPS within 10 ms latency" (tikv.org/docs/6.1/deploy/performance/overview). Q114 measures it.

6.5 Key layout (D238)Copy Anchor Link

• Keyspace loam_pgwal, pre-allocated like the others (deploy/tikv/pd.toml). It keeps the WAL out of the metastore's GC safe point and placement, and gives the WAL its own quotas.
• Keys per timeline tl = (tenant_id, timeline_id), 32 bytes:
  • H/<tl>: the head (postcard-encoded): term, term_history, flush_lsn, commit_lsn, peer_horizon_lsn, backup_lsn (bucket), remote_consistent_lsn, pg_version, system_id, wal_seg_size, timeline_start_lsn, version.
  • W/<tl>/<begin_lsn: u64 BE>: WAL bytes [begin_lsn, end_lsn) of one chunk, at most 128 KiB (one walproposer MAX_SEND_SIZE).
• One region range per timeline. Timeline creation asks PD to split at H/<tl> and at the next timeline's prefix. The head and the tail chunks then usually share a region, so the append is 1PC. When a size split separates them, the commit falls back to async commit: prewrites go to both regions in parallel, and the ack comes after the prewrites, so latency rises by little (estimate).
• Hotspots. Each timeline's tail is a sequential write hotspot, as a Raft log is. A timeline cannot be spread over regions without breaking ordering, and it does not need to be: one region leader handles tens of MB/s (verify). Many timelines spread across stores through PD's leader and region balancing.
• Splits. Load-based split does not help here. It counts read QPS, bytes and CPU only, over 10 s (split.qps-threshold 3000, byte-threshold 30 MiB/s; docs "Load Base Split"), so it never splits a write-hot WAL region. The per-timeline pre-split above is what spreads the load. Size splits happen at region-split-size (256 MiB since v8.4). PD merges small regions after split-merge-interval (1 h), and the trimmed timelines leave small regions for it to merge.
• Batching. While one append transaction is in flight for a timeline, arriving AppendRequests queue. The next transaction takes the whole queue, up to 1 MiB (8 chunks). That fits raft-max-size-per-msg (1 MiB) and is well under raft-entry-max-size (8 MiB). This is the safekeeper's own "fsync when the queue drains" group commit. With 5 ms per transaction and 1 MiB per batch, one timeline tops out at about 200 MiB/s (estimate). Pipelining more than one in-flight transaction per timeline is P4 optimisation work (Q115).
• Values. WAL compresses well, but Postgres' WAL may already hold compressed full-page images (wal_compression). Chunks are stored as they arrive, and the bucket copy is compressed (§6.7).

6.6 TiKV tuning and the latency budgetCopy Anchor Link

Configuration of the loam_pgwal cluster. The WAL runs on the same TiKV release as the rest of Loam (D126). When the WAL shares the metastore's cluster, it gets its own stores through placement rules (Q116). The defaults below are TiKV v8.5's, read from pingcap/docs release-8.5 (tikv-configuration-file.md).

Client side (Loam's client-rust fork).

• No RPC batching. client-rust sends every request as its own unary gRPC call (src/store/request.rs). It has no BatchCommands stream like client-go's (max-batch-size 128, max-batch-wait-time). One append per timeline in flight is unaffected. Many timelines per WAL instance may need it (Q118).
• No follower or stale reads, and no zone-aware routing. Every request goes to the leader (upstream PR #562). The leader placement rule is what keeps the WAL in the AZ.
• Keep a TSO prefetch. A 1PC transaction still needs a start_ts from PD. The fork's TSO oracle batches requests (R1 plan row F1). With PD's leader in another AZ, a TSO costs one cross-AZ round trip, so the WAL pool prefetches.
• The async-commit and 1PC fixes are already in the fork. max_commit_ts, and the 1PC fallback that commits instead of failing, are in docs/plans/r1-dependency-spike.md §(i). The WAL relies on both.

Latency budget (estimate). One commit record. The TiKV leader is in the compute's AZ and the followers are in two other AZs. The disks are NVMe with power-loss protection. AWS inter-AZ RTTs measured 0.39–2.42 ms across regions, "the vast majority" under 1 ms (bitsand.cloud), and AWS itself says "single-digit millisecond".

The honest reading.

• The extra steps. At equal hardware, the Loam WAL's commit path has one more read, one TSO and TiKV's apply step. The p50 is likely about 0.5–1.5 ms worse than three safekeepers.

• What the spike showed. On this laptop:

  • the single safekeeper acked in 3.57 ms at p50 and 22 ms at p99;
  • a fenced 1PC write to a single TiKV store took 6.5–7.1 ms at p50 and 25–26 ms at p99.

  That is about 2× at p50, and close at p99, both dominated by a consumer SSD's fsync.

• What the gate compares. The gate (§7) is on p99. There, async IO, Raft Engine and PLP disks can bring the Loam WAL level with a safekeeper whose fsync is the tail.

• Failover. A TiKV leader crash stalls its timelines for the election timeout, where losing one of three safekeepers stalls nothing. The P4b fault run measures commit latency through a leader kill as well as steady state (§7).

Where Loam wins without argument:

• Operations. A stateless pool on a TiKV cluster that Loam already runs, instead of a stateful three-node safekeeper fleet per region with membership changes, eviction and a scheduler.
• Durability. The bucket copy lags by the group-commit interval (250 ms). Safekeepers upload full 16 MiB segments, and partial segments only every 15 minutes. After a total loss of the hot tier, the RPO is 250 ms instead of up to 16 MiB or 15 minutes of WAL.
• Cost. Idle timelines cost nothing in the WAL tier, because their keys are trimmed. Safekeeper nodes are provisioned for peak, with three NVMe volumes each.
• Throughput. Group commit is per timeline, and TiKV scales out across stores.

If the gate fails on p99, the WAL stays behind its feature and stock safekeepers remain the default (D233). Nothing else in this document depends on the WAL.

6.7 How the pageserver and computes read it; retention; recovery (D240)Copy Anchor Link

• Replication for readers. The WAL service implements START_REPLICATION [PHYSICAL] X/Y [(term='N')] like send_wal.rs:
  • Without a term, it serves up to commit_lsn, for the pageserver and replicas.
  • With a term, it serves up to flush_lsn within that term, for the compute's neon_walreader and recovery.
  • The pageserver hard-codes the interpreted protocol (timeline.rs launch_wal_receiver: protobuf, zstd level 1, per-shard filtering). The WAL service therefore decodes WAL with Neon's wal_decoder crate (Apache-2.0).
  • Confirmed in P4a: vanilla ingestion was removed from the pageserver (neon #12126). A compute on the WAL service without an interpreted sender stalls on its first read of a page it wrote, because the pageserver never gets the WAL. Until the WAL service links wal_decoder, operon-safekeeper's feeder acts as walproposer towards one stock safekeeper and streams it the committed WAL, off the commit path. That safekeeper runs with --no-sync, publishes to the broker and serves the pageserver. This is interim (Q112).
• Sources, in order:
  1. the in-memory tail of recently appended chunks (written through at append time);
  2. TiKV (scan W/<tl>/begin..);
  3. the Loam log in the bucket for ranges already trimmed from TiKV.
• Feedback. The pageserver's 'z' updates carry remote_consistent_lsn. They are written into the head, coalesced to at most one write per second per timeline, and returned to the compute in AppendResponse.pageserver_feedback, which drives walproposer's backpressure (max_replication_*_lag). Hot-standby feedback is passed through the same way.
• Discovery. Every instance publishes SafekeeperTimelineInfo for the timelines it serves to the unmodified storage broker, every second: safekeeper_id = the pool's logical id, safekeeper_connstr = the pool's Service, and commit_lsn, flush_lsn, remote_consistent_lsn and availability_zone from the head. It also answers SafekeeperDiscoveryRequest for any timeline in loam_pgwal.
• Group commit to the bucket.
  • A per-timeline task appends committed chunks to the Loam log (operon-log standard class, D72's idempotent producer = (timeline, term), sequence = begin_lsn), with a 250 ms flush interval.
  • After each acknowledged log append, it advances head.backup_lsn.
  • The stream is pgwal/<tenant> with one partition per timeline slot (Q119), so retention, segmenting, encryption (D96) and GC come from the log unchanged.
• Trimming. Chunks below min(backup_lsn, remote_consistent_lsn, commit_lsn) are deleted from TiKV in batched transactions. MVCC GC for the keyspace (operon-tikv gc.rs) reclaims them. A timeline with no compute and no lag holds no TiKV keys except its head.
• Recovery.
  • An instance crash is a reconnect (§6.3).
  • A TiKV leader failure is a Raft election (the election timeout). Appends stall and then resume, and nothing acknowledged is lost.
  • Losing a TiKV AZ keeps quorum.
  • Losing TiKV entirely loses at most the last group-commit interval of acknowledged WAL. The timeline is recovered from the bucket's copy up to backup_lsn, and the compute must be restarted from that LSN (recover_from_bucket, an operator action). With safekeepers, the same event loses everything since the last uploaded segment.

6.8 The public Rust API (D235)Copy Anchor Link

A trait-level sketch, for P4a to refine:

/// A timeline: (tenant, timeline) in Neon's ids.
pub struct TimelineId { pub tenant: [u8; 16], pub timeline: [u8; 16] }
pub type Lsn = u64;
pub type Term = u64;

/// The acceptor's durable state: what a safekeeper keeps in its control file.
pub struct AcceptorState {
    pub term: Term,
    pub term_history: Vec<(Term, Lsn)>,
    pub flush_lsn: Lsn,
    pub commit_lsn: Lsn,
    pub peer_horizon_lsn: Lsn,
    pub backup_lsn: Lsn,
    pub remote_consistent_lsn: Lsn,
    pub timeline_start_lsn: Lsn,
    pub server: ServerInfo, // pg_version, system_id, wal_seg_size
}

pub enum Fenced { Term { current: Term } }

/// The storage behind the acceptor. TiKV implements it (P4a); an in-memory
/// implementation drives the protocol tests and the TLA+-derived traces.
#[async_trait]
pub trait WalStore: Send + Sync + 'static {
    async fn create(&self, tl: &TimelineId, server: ServerInfo, start: Lsn) -> Result<AcceptorState>;
    async fn load(&self, tl: &TimelineId) -> Result<Option<AcceptorState>>;
    /// Grant a vote if `term > state.term`; durable before returning.
    async fn vote(&self, tl: &TimelineId, term: Term) -> Result<(bool, AcceptorState)>;
    /// Adopt the elected proposer's history; truncate above `start`.
    async fn elected(&self, tl: &TimelineId, term: Term, start: Lsn,
                     history: Vec<(Term, Lsn)>) -> Result<Result<AcceptorState, Fenced>>;
    /// Fenced, contiguous append of `[begin, begin + wal.len())`; durable on Ok.
    async fn append(&self, tl: &TimelineId, term: Term, begin: Lsn, wal: &[Bytes],
                    commit_lsn: Lsn, truncate_lsn: Lsn) -> Result<Result<AcceptorState, Fenced>>;
    /// Read WAL `[from, to)` from the tail cache, TiKV or the bucket.
    fn read(&self, tl: &TimelineId, from: Lsn, to: Lsn) -> BoxStream<'static, Result<Bytes>>;
    async fn record_feedback(&self, tl: &TimelineId, fb: PageserverFeedback) -> Result<()>;
    /// Delete WAL below `lsn` once `backup_lsn` and `remote_consistent_lsn` have passed it.
    async fn trim(&self, tl: &TimelineId, lsn: Lsn) -> Result<()>;
}

/// The service: the safekeeper wire protocol over a `WalStore`.
pub struct WalService<S: WalStore> { /* … */ }
impl<S: WalStore> WalService<S> {
    pub fn new(store: S, broker: BrokerClient, config: WalServiceConfig) -> Self;
    /// `START_WAL_PUSH` (proposer) and `START_REPLICATION` (readers) on one listener,
    /// plus `IDENTIFY_SYSTEM` and `TIMELINE_STATUS`, over the Postgres protocol.
    pub async fn serve(self, listener: TcpListener, shutdown: CancellationToken) -> Result<()>;
}

• Crate layout.
  • proto: the message codecs for v2 and v3. It is pure Rust, with no Postgres headers.
  • acceptor: the state machine, ported from neon/safekeeper/src/safekeeper.rs with its tests.
  • tikv: the WalStore implementation.
  • send: readers, and the interpreted sender behind the feature interpreted.
  • broker: the storage-broker gRPC client, built from storage_broker/proto/broker.proto.
• The interpreted sender depends on the fork's wal_decoder, postgres_ffi and utils, as a git dependency pinned to a fork revision. postgres_ffi's build.rs runs bindgen against Postgres server headers (POSTGRES_INSTALL_DIR/v1x/include/postgresql/server), so that feature needs the fork's Postgres headers at build time. CI builds them once per fork pin and caches them. The default Operon build does not enable the feature (Q112).
• Attribution. Ported code keeps Neon's Apache-2.0 notice in NOTICE.

6.9 Mapping the safekeeper protocol to Loam operationsCopy Anchor Link

6.10 The upgrade pathCopy Anchor Link

1. P2: the fork runs with stock safekeepers under Loam's control plane (x/….wal = safekeepers).
2. P4a–P4b: the WAL service runs beside them on the benchmark topology only.
3. P4c: a database is switched by setting wal = loam in its x/ record. The switch runs:
   • At a quiesced point: stop the compute and wait until the safekeepers' commit_lsn = flush_lsn.
   • Loam's control plane creates the timeline in loam_pgwal at that LSN (start_lsn, with the term history copied from the safekeepers' TIMELINE_STATUS).
   • The compute spec's safekeeper_connstrings becomes the WAL pool's Service, and the compute starts: sync-safekeepers runs against one acceptor, then the basebackup.
   • The safekeeper timeline is deleted after the pageserver's remote_consistent_lsn passes the switch point.
4. Rollback is the same procedure in reverse.
5. Branches inherit the parent database's setting.

7. The benchmark gate (P4b, D233)Copy Anchor Link

The gate. On the same hardware and topology, with pgbench against stock safekeepers ×3 versus the Loam WAL:

• p99 commit latency (Loam) ≤ p99 commit latency (safekeepers), and
• throughput (Loam) ≥ throughput (safekeepers), for every workload below, within the run-to-run noise measured by repeating the baseline 3 times. P4c merges only when both hold.

Topology (deploy/loam-pg-bench/: a compose file for a single host, and a Kubernetes manifest for three nodes):

On a single host, tc netem adds a fixed delay between containers to model AZs: 0.5 ms one way for "cross-AZ" pairs. Both runs use the same delays.

Workloads (scripts/loam-pg-bench/):

The bulk gate is a sustained write (owner decision, 2026-10-01) because a 250 MB burst measures the drive cache: on the same disk the stock safekeepers reach 118 to 182 MB/s on 250 MB and 22 to 26 MB/s on 1 GB.

Each run lasts 5 minutes after a 1-minute warm-up, with pgbench -l per-transaction logs. p50, p90, p99 and p99.9 come from the logs, not from pgbench's averages. The spike's scripts, pgb.sh and pgb-async.sh, are the starting point.

Fault run (reported, not part of the pass/fail gate unless the owner makes it one): commit-16 for 5 minutes while one TiKV leader (or one safekeeper, for the baseline) is killed at minute 2. It records the longest commit stall and the p99.9.

Recording. Each run writes bench/results/<date>-<git-sha>-<variant>.json, containing:

• the topology and versions (fork sha, TiKV version, kernel, disk model);
• the settings;
• the percentiles and TPS per workload;
• the TiKV latency-breakdown metrics (propose, append, commit and apply waits) and the safekeeper flush_wal histograms.

A summary table goes into the P4c PR description. The run is a manual workflow (workflow_dispatch) on a dedicated runner, because shared CI runners are too noisy for p99s. It is also runnable locally. The spike's single-host numbers (§6.1, §6.4) are the first data point, not the gate.

7.1 First results: laptop, 2026-09-29 (P4a/P4b; not gate data)Copy Anchor Link

Implemented: crates/operon-safekeeper:

• the v3 codecs, the acceptor and WalStore;
• the TiKV store, as fenced 1PC transactions;
• the service, HTTP API, feeder and the loam-wal binary.

The harness is in deploy/loam-pg-bench and scripts/loam-pg-bench, and the raw results in bench/results/.

Verified: an unmodified Neon compute (compute-node-v16 at 77e22e4):

• starts through sync-safekeepers against loam-wal on TiKV v8.5.8;
• commits through it;
• is ingested by the pageserver through the feeder.

Setup:

• Host: one laptop, 14 cores, 15 GB of RAM, a consumer Samsung NVMe without power-loss protection, btrfs.
• Placement: every tier on the same disk, with no injected cross-AZ delay.
• Other load: other work shared the host.
• Runs: 60 s per workload after a 10 s warm-up. Baseline and candidate were interleaved three times each.
• Compute settings: shared_buffers = 2GB, compute fsync = off.

Means over the three repeats:

rf 1 compares one safekeeper with one TiKV store; rf 3 compares three safekeepers with three stores.

These are not gate results. They are laptop data points outside the §7 protocol: one host, a consumer disk, 60 s runs instead of 5 minutes, -s 10, and no tpcb-64. The gate itself needs the §7 topology on a dedicated runner (the manual loam-pg-bench workflow).

On this laptop, the comparison script's criteria fail for every workload at both replication factors. The Loam WAL's p99 is about 1.5–2.6× the safekeepers', and its throughput about 0.2–0.6× (bench/results/gate-rf*.md). This gap is far wider than the noise, so a §7 run is not worth doing until the levers below are in. Meanwhile, D233 applies: the WAL stays behind its feature, and P4c does not start.

Where the time goes. TiKV's own metrics, from a commit-1 run:

• Raft log persist dominates: p50 ≤ 2.6 ms, p99 up to 82 ms. That is the same consumer-SSD fsync the safekeeper pays, but with TiKV's scheduler, apply and 1PC steps added, plus three client RPCs per append: TSO, the fenced head read and the prewrite.
• Single-flight appends per timeline: they cap group commit. bulk shows it most, at 12–20 MB/s against 77–98 MB/s.
• The interim feeder adds a second WAL stream on the same disk.

Before the gate can pass, the levers are:

1. Pipelined appends (Q115). Several appends in flight per timeline, fenced without serialising on the head key. This is the main lever for commit-16, tpcb and bulk.
2. No PD round trip per append. Reuse the previous commit timestamp as the next start_ts. This is safe here: a stale start_ts only causes extra write conflicts. It needs a begin_at(ts) in the client-rust fork.
3. The in-process interpreted sender (Q112). It removes the feeder's second stream.
4. Re-run on the §7 topology. Server NVMe with PLP (fsync in µs, not ms), three nodes and real or netem AZ delays (Q114). The owner's < 5 ms p99 target is a server-hardware number. Both variants on this laptop are far above it.

7.2 Arm A: a native leaderless WAL on local NVMe (D263–D269 and D271–D272, 2026-09-30)Copy Anchor Link

On 2026-09-30 the owner decided that the Loam WAL targets native leaderless-WAL performance instead of tuning the TiKV hot tier. Two arms are measured against the same gate:

• Arm A, this section, takes TiKV off the commit path.
• Arm B keeps TiKV and removes its per-append overheads.

The owner also asked for Arm A's store to be a tiered, low-level I/O layer, with runtime detection and fallback (D266), and chose compio as its runtime (D263). The decisions below are Loam's proposals within that direction. They supersede D239 (one logical acceptor per timeline) for Arm A only, and leave D237–D238 in force for Arm B.

The causes in §7.1, and how Arm A removes each one:

The design (target; D272 records the compio path as built, which leaves out the custom OpCodes and registered buffers below)Copy Anchor Link

• D263. compio is the runtime of the Arm A data path (owner decision, 2026-09-30; final). The Arm A build enables compio's io-uring and polling features.

  • Shards. Each shard is one thread running a compio runtime, and owns a set of timelines (by hash of the timeline id, over a shard count fixed at first start). A shard also owns its own journal (D265), so group commit is per shard and needs no locks across cores.
  • What runs on the owning shard: the timeline's START_WAL_PUSH connection, its journal writes, and its durable writes. An accept thread reads the startup packet, which names the timeline, then hands the socket to the owning shard.
  • The driver is io_uring. (Target design. The built path in D272 uses O_DSYNC descriptors and no custom OpCodes or registered buffers.) compio's own opcodes cover sockets. Custom OpCodes cover what compio does not expose: WRITE/WRITE_FIXED with RWF_DSYNC, and registered buffers. The raw io-uring crate and io_uring_register on the ring's fd are used only where compio cannot help: buffer registration and file-table updates.
  • Linked writes. The safe compio 0.19 API used here cannot pass the IO_LINK flag on an operation. So the WRITE → FSYNC pair for FUA-less devices is two back-to-back operations on the shard, and the FSYNC is submitted only after the WRITE completes successfully. An AppendResponse never precedes durability. (The low-level Extra / Proactor::push_with_extra path could link the pair; it is a later optimisation.)
  • When the polling feature is enabled and the driver selects it, compio uses its epoll driver when io_uring is unavailable; durable writes use the pwritev2 pool.
  • tokio stays for the control plane: the TiKV metadata client, S3 offload, the admin HTTP API (axum) and the feeder. It is bridged to the shards by bounded channels, off the commit path. The existing tokio WAL service stays as the fallback front end (--runtime tokio), and is benchmarked as tier (a).

  Crate	License	Latest release (crates.io, 2026-09-30)	Model	Role
  compio	MIT	0.19.2 stable, 2026-08-18 (0.20.0-beta.1, 2026-09-29)	Thread-per-core, completion-based (io_uring, epoll fallback)	The data path (D263). Apache Iggy moved to it (verify)
  io-uring (tokio-rs)	MIT OR Apache-2.0	0.7.15, 2026-09-07; already in Cargo.lock through qdrant-edge and compio	Raw rings	Only where compio does not reach
  monoio (ByteDance)	MIT OR Apache-2.0	0.2.4, 2024-08-20	Thread-per-core	Not used. No release for over two years; dropped from the benchmark (owner)
  glommio (Datadog)	Apache-2.0 OR MIT	0.9.0, 2024-03-25	Thread-per-core	Not used. No release for over two years
  tokio-uring	MIT	0.5.0, 2024-05-27	io_uring under tokio	Not used. No release for over two years

• D264. walproposer is the only sequencer.

  • The compute's neon.safekeepers names three loam-wal acceptors, and walproposer runs its own Paxos over them, exactly as over safekeepers: terms, votes, ProposerElected, and commitLsn as the quorum of flush_lsn.
  • Commit is 1 round trip and 1 durable write: a majority of acceptors has the WAL on local disk.
  • An acceptor's term check is in memory. It is raised only after the vote is durable (D268), which is the Paxos acceptor's promise.
  • The vote handler first waits until the timeline's written WAL is durable. So VoteResponse.flush_lsn is what the disk holds, as in Neon's handle_vote_request.
  • After the vote, appends of a lower term are refused, and any later response carries the higher term, so a deposed proposer steps down.
  • Losing one of the three acceptors stalls nothing, which removes §12 row 10's TiKV leader-election stall for Arm A.

• D265. One shared journal per shard, for every timeline the shard owns.

  • Precedent: TiKV's raft-engine, which puts all regions in one log.
  • Segments. The journal is a directory of fixed-size segment files, 64 MiB by default. A background thread prepares the next two:
    • it fallocates each one, then pre-zeroes it with large direct writes and one fdatasync, so the first append never converts an unwritten extent (a metadata journal commit);
    • after that, segments are recycled instead of created.
    • On btrfs the directory gets NOCOW (FS_NOCOW_FL, as systemd-journald does), so overwrites stay in place.
  • Records are framed: kind, timeline, term, LSN, length, and a CRC32C seeded with the segment's sequence number, so a recycled segment's old records never validate. The kinds are:
    • Append: WAL bytes, plus the proposer's commit_lsn.
    • Truncate: a logical truncation of one timeline, on election.
    • Progress: commit_lsn, backup_lsn and remote_consistent_lsn learned off the append path.
  • Flush units. Records are written in flush units aligned to 4 KiB, zero-padded to the block size. This is what O_DIRECT needs. It also lets several units be in flight at once without two writes ever touching the same block. The cost is space: at most one block per unit, on WAL that is offloaded and recycled.
  • Group commit. One writer coalesces every pending record, from every timeline and every connection on the shard, into the next unit (up to 1 MiB), and keeps up to --io-depth units in flight. The shard's durable position is the end of the longest prefix of completed units. By contrast, a safekeeper fsyncs each timeline's segment file separately, and writes nothing while its fsync runs.
  • Recovery replays segments in order and stops at the first record that fails its CRC (the torn tail). Units complete out of order, but an ack only covers a prefix of completed units, so nothing acknowledged is behind a torn unit. Each timeline keeps only the WAL contiguous from its trimmed point, and its flush_lsn is where that WAL ends.
  • Reads. An in-memory index maps each timeline's LSNs to records. Reads are preads.

• D266. A tiered I/O layer, chosen at startup. loam-wal probes the journal directory and its block device, logs the tier it picked, and takes --io auto|uring|pwritev2|buffered as an override.

  Tier	Durable write	Used when
  uring (target design; D272 is the built path)	io_uring on the owning compio shard (D263), with registered files (segment slots updated on rollover) and registered, aligned buffers (WRITE_FIXED) on O_DIRECT segments. The write carries RWF_DSYNC: on pre-zeroed blocks the kernel sends one FUA write when the device has FUA, and write + flush when it does not. The alternative is a linked WRITE → FSYNC(DATASYNC) pair (--uring-sync linked, chosen automatically when queue/fua is 0). Optionally SQPOLL (--uring-sqpoll). IOPOLL is used only when the device has NVMe poll queues (queue/io_poll = 1)	io_uring is allowed (io_uring_setup succeeds and the opcodes probe) and the filesystem takes O_DIRECT
  pwritev2	O_DIRECT + pwritev2(RWF_DSYNC) from a small thread pool (one thread per in-flight unit)	io_uring is blocked, as by Docker's default seccomp profile and GKE Autopilot's containerd RuntimeDefault seccomp profile
  buffered	pwrite into the page cache; a sync thread runs fdatasync back to back while unsynced data exists	No O_DIRECT (tmpfs, some overlay filesystems), or chosen by --io buffered
  Future: NVMe passthrough	uring_cmd on /dev/ng*: NVMe write commands with FUA, no filesystem	A raw namespace per node. Documented only; no SPDK

  • This drive. The laptop's drive reports fua = 1, write_cache = write back, io_poll = 0 (no poll queues), and 512-byte logical blocks.
  • Network. Every connection sets TCP_NODELAY. SO_BUSY_POLL can be set with --busy-poll-us, and is logged and ignored when the kernel refuses it.
  • io_uring networking. The shard's sockets go through compio's io_uring driver. Multishot recv with provided buffer rings (compio's buffer pool) is the next step (Q264).
  • kTLS is the cheap route for issue #148: a rustls handshake, then TLS_TX/TLS_RX offload, so the data path keeps plain writes. That is a note only, not built.

• D267. Pipelined appends.

  • The service writes each group of queued AppendRequests as soon as it arrives, and keeps reading.
  • An AppendResponse goes out whenever the durable position passes some of the timeline's writes. It carries the highest durable flush_lsn.
  • Heartbeats are answered at once with the durable position.
  • Stores that are durable per call (memory, TiKV) keep the old behaviour through the trait's default method.

• D268. TiKV holds only metadata.

  • What: per (node, timeline), the acceptor record: term, term history, membership, server info, the start LSNs and trimmed_lsn.
  • When: it is written on timeline creation, vote, ProposerElected and trim.
  • How: these are 2PC transactions, which is fine at one or two writes per election, off the commit path.
  • Order on election: a Truncate record is made durable first, then the metadata. A crash between the two leaves a shorter WAL under the old history, which the next election repairs.
  • Development: a local control-file backend (write, fsync, rename, fsync the directory) serves single-node setups and tests.

• D269. Offload and trim.

  • Offloader: one acceptor per timeline, chosen by a lease in TiKV (B/<tl>: owner, expiry, backup_lsn). It uploads committed WAL to the bucket every 250 ms (or at 16 MiB).
  • Objects: pgwal/<tenant>/<timeline>/<begin>-<end>.lwal. The format follows operon-log's segment conventions: a magic and version header, a CRC32C trailer, and a zstd body. The Kafka RecordBatch framing is not used, because WAL is one byte stream per timeline.
  • backup_lsn is advanced in the lease record, and the other acceptors read it from there.
  • Trim. A segment is recycled once every timeline in it has passed min(backup_lsn, remote_consistent_lsn, commit_lsn).
  • Pinning. An idle timeline whose pageserver lags pins old segments. Rewriting its live records forward, as raft-engine purges, is left for later (Q262).

• D271. The pageserver feed stays the feeder for now (Q112).

  • Why not in-process decoding yet: it needs Neon's wal_decoder, postgres_ffi (bindgen against the fork's Postgres server headers), utils and pageserver_api, which bring Neon's workspace dependency pins into Operon's. That is not feasible within this arm.
  • Instead:
    • exactly one designated acceptor runs the feeder;
    • it feeds up to min(commit_lsn, its durable flush_lsn);
    • its stock --no-sync safekeeper keeps its data on a different filesystem from the acceptors' journals.
  • On the laptop that filesystem is on the same physical drive, so the feeder's writeback still competes for the device.

1PC, 2PC or raw: where each write goesCopy Anchor Link

The commit path is the only place where latency is gated. TiKV's modes are measured in §6.4 (single store, spike). The local-disk figures are from 2026-09-30 on the same laptop: 400 writes to a 64 MiB pre-zeroed file, with another session's build running.

What the table says.

• On this drive a FUA write costs about as much as a cache flush. So on equal filesystems, the direct tiers and fdatasync land close together.
• btrfs adds its log-tree commit to every durable write: 2.1–2.7 ms here against about 0.9 ms on ext4, on the same drive.
• A power-loss-protected drive matters more than any tier. Its durable write is microseconds (§6.6), and that cannot be shown on the laptop.
• What the tiers remove is everything else: allocation, fsyncs and renames on the commit path, a sync that waits for earlier writes, and one sync per timeline. The gate measures each tier separately (§7.3), and also on an ext4 partition, to separate the btrfs cost.

Open questions (Q261–Q264)Copy Anchor Link

The results of the Arm A gate runs are in §7.3.

8. PgDog routing (D236)Copy Anchor Link

Proposed 2026-10-01 (§31, D304–D307): sharded Loam Postgres databases use PgDog's own sharding, with the shard map in Loam's metastore rendered into pgdog.toml; cutover across several PgDog instances is orchestrated by Loam with a fence in Postgres (ALTER ROLE … NOLOGIN), because PgDog's open-source RESHARD cuts over one instance only; PgDog's 2PC is off by default and allowed only with a durable coordinator log (StatefulSet, NODE_ID, DEPLOYMENT_ID, the WAL directory on a volume); computes of 2PC databases get max_prepared_transactions in their spec. D236 is unchanged: PgDog stays unmodified.

Verified 2026-09-29.

• License: pgdogdev/pgdog is AGPL-3.0 (LICENSE is the GNU AGPL v3 text; gh api repos/pgdogdev/pgdog). There is no CLA. A closed Enterprise Edition exists (control plane, query monitoring, QoS), built from a private repo (docs.pgdog.dev/enterprise_edition).
• Activity: v0.1.60 (2026-09-24). Releases are weekly, and the last commit was 2026-09-28.
• Features (README; docs.pgdog.dev):
  • transaction and session pooling;
  • load balancing across replicas (round robin, random, least connections) with health checks;
  • failover detection from replication state (it is "not a replacement for Patroni");
  • sharding (hash compatible with Postgres partitioning, list, range, schema);
  • cross-shard queries with partial aggregate support, two-phase commit, and online resharding over logical replication;
  • prepared statements in transaction mode;
  • SCRAM, MD5 and TLS;
  • an admin database, OpenMetrics and a Helm chart (MIT).
• Routing by database name: each [[databases]] name is what clients connect to, with its own host, port, database_name and pool (docs.pgdog.dev/configuration/pgdog.toml/databases). That is exactly §23's D153 routing, without Loam writing a splice.

Deployment.

• Where it runs: PgDog is a Deployment in the Loam Postgres namespace, with an image from ghcr.io/pgdogdev/pgdog pinned by digest.
• Config: Loam's control plane renders pgdog.toml and users.toml from the x/ and X/ records into a ConfigMap and a Secret, and sends RELOAD to PgDog's admin database after each change.
  • Database names: <db> for main and <db>__<branch> for branches (this answers Q49's syntax; the options form stays unsupported).
  • Hosts: the branch's compute, plus read replicas (Neon replica computes) with role = replica.
• Auth:
  • PgDog terminates client auth (SCRAM) against users.toml. Loam provisions the same role and password into the compute spec, and PgDog uses them for server connections.
  • PgDog's passthrough auth forces auth_method = plain (docs.pgdog.dev/features/authentication), so it is not used.
  • Credentials come from the auth plan's credential store (Q30).
  • TLS on both hops, with peer verification. Clients reach PgDog only over TLS, with a certificate for PgDog's service name that clients verify (sslmode=verify-full); PgDog's non-TLS listener is not exposed. PgDog connects to computes over TLS with full verification of the compute's certificate against the cluster CA, and the compute spec accepts only TLS connections (hostssl). Where a service mesh provides mutual TLS between the pods, that satisfies the backend hop instead. Plaintext is refused on both hops.
• Pooling mode: transaction mode by default. Session mode is used for migrations, LISTEN/NOTIFY and advisory locks. Each database entry sets its own mode.
• What PgDog cannot do: it cannot wake a stopped compute, so there is no scale-to-zero. Scale-to-zero needs Neon's proxy and the wake_compute API (§5.2, Q113).

AGPL risk, and the rules Loam follows.

• Never linked or vendored. No PgDog crate appears in any Cargo.lock. deny.toml already rejects AGPL, and the rule is extended to the Loam Commons and loam-platform repos. No PgDog source is copied into any Loam repo.
• Never modified. Loam uses upstream images and configuration only. A needed fix is reported upstream (with the owner's approval) or worked around with configuration. If neither is possible, the fallback is used.
• Why:
  • AGPL-3.0 §13 requires anyone who modifies PgDog and lets users interact with it over a network to offer those users the modified source. Running Loam cloud's routing through PgDog is exactly "interaction over a network".
  • An unmodified PgDog carries no source obligation beyond the upstream source and license already being public. Images that Loam distributes (BYOC, the showcase's compose) must still ship the license and point to the corresponding upstream source.
  • Anyone who links or vendors PgDog into Loam's code would put that code under AGPL (D11).
• Recorded in:
  • LICENSES.md in the Loam Commons and deploy repos;
  • the license-check CI job (D-SC-6), which checks deploy images;
  • a deploy/pgdog/README.md that states these rules.

Fallbacks.

For production scale-to-zero, the expected shape is Neon's proxy in front, which wakes the compute and routes, and PgDog behind it per database for pooling. Q113 decides.

9. Where Loam can beat Neon beyond the WAL (P5)Copy Anchor Link

Each item below is proposed. P5 plans each one separately, after P4.

10. Maintaining the fork (D231, D241)Copy Anchor Link

The starting point, verified 2026-09-29.

The .gitmodules URLs are relative (../postgres.git), so dina-kar/neon's submodules resolve to dina-kar/postgres, which P2 must create as a fork of neondatabase/postgres (Q110).

Plan.

1. Catch-up (P2, estimate: 2–3 engineer-weeks).

   • Fork neondatabase/postgres.
   • Move the submodules to the REL_16/17_STABLE_neon heads. These are newer than main's pin and may need extension changes that were never published (Q110).
   • Merge upstream 16.15 and 17.11.
   • Build the compute images in the fork's CI.
   • Run Neon's pg_regress and the Python test_runner subset for compute and safekeepers, plus deploy/neon's smoke.
   • Drop v14 and v15, which are out of Loam's scope.

2. PG 18 enablement (P2 or later, estimate: 3–6 engineer-weeks). Neon main has no v18 wiring. It needs:

   • vendor/postgres-v18 from REL_18_STABLE_neon (18.2) merged to 18.6;
   • v18 in libs/postgres_ffi (bindgen, pg_constants, WAL record layouts);
   • libs/wal_decoder and the pageserver's per-version walingest paths;
   • pgxn/neon built against 18;
   • the compute Dockerfile and extensions.

   CNPG covers PG 18 for the showcase meanwhile.

3. Steady state (D241).

   • Nightly job: try merging upstream REL_17_STABLE and REL_18_STABLE into the _neon branches. Report conflicts. Build the compute image. Run pg_regress.
   • Release day (the second Thursday of Feb/May/Aug/Nov): merge the tagged minor, run the full suite, and publish ghcr.io/dina-kar/compute-node-v17:<minor> and -v18 within a week.
   • Out-of-cycle releases follow the same path.
   • Estimate: 2–4 engineer-days per quarter for two majors, because minor releases rarely touch the areas Neon patches (smgr, WAL redo, SLRU). Plus about 1 engineer-week a year for a new major's catch-up, and the Rust toolchain and dependency upkeep of the Neon workspace (cargo deny on the fork).

4. Owning the rest. Storage-side fixes in the pageserver, broker and controller are taken as needed, with a test per fix, like the client-rust fork (§20). The fork's README states that it is independent of Neon Inc.

11. Roadmap: phases (D233, D235)Copy Anchor Link

Each phase is small stacked PRs. The P4 phases are behind the feature loam-wal and change no default path.

12. RisksCopy Anchor Link

13. Open questionsCopy Anchor Link

14. Contradictions with earlier decisions, and how they are resolvedCopy Anchor Link

15. SourcesCopy Anchor Link

Read on 2026-09-29.

• Neon (fork dina-kar/neon at fa504217c):
  • pgxn/neon/walproposer.c, walproposer.h, walproposer_pg.c, neon_walreader.c;
  • safekeeper/src/{safekeeper.rs, receive_wal.rs, wal_storage.rs, control_file.rs, send_wal.rs, send_interpreted_wal.rs, wal_backup.rs, wal_backup_partial.rs, remove_wal.rs, timeline_manager.rs, broker.rs, handler.rs, http/routes.rs}, and safekeeper/spec/*.tla;
  • storage_broker/proto/broker.proto;
  • pageserver/src/tenant/timeline/walreceiver/{connection_manager.rs, walreceiver_connection.rs}, pageserver/src/walingest.rs;
  • libs/wal_decoder/, libs/postgres_ffi/build.rs, libs/compute_api/src/{spec.rs, responses.rs};
  • compute_tools/src/{spec.rs, config.rs, http/server.rs, compute.rs};
  • storage_controller/src/{compute_hook.rs, http.rs};
  • proxy/src/control_plane/client/cplane_proxy_v1.rs, messages.rs;
  • docs/{walservice.md, core_changes.md, safekeeper-protocol.md} (partly out of date), and RFCs 009, 013, 025, 035 and 041;
  • vendor/revisions.json, .gitmodules.
• Neon activity and statements:
  • the neondatabase/neon and neondatabase/postgres commit, branch and compare APIs;
  • Discussion #12835;
  • neon.com/docs/introduction/architecture-overview, neon.com/blog/paxos, neon.com/blog/wal-s3-lakebase-storage-for-the-era-of-agents;
  • jack-vanlightly.com (2023-11-15, 2025-02-19; no latency figures);
  • clickhouse.com/blog/postgresbench-ha.
• CloudNativePG:
  • cloudnative-pg/cloudnative-pg (license, releases v1.30.1 and v1.30.0);
  • cloudnative-pg.io/docs/devel/{supported_releases, backup, recovery};
  • cloudnative-pg/plugin-barman-cloud (releases, docs/usage, docs/object_stores);
  • cloudnative-pg/postgres-containers;
  • cncf.io/projects/cloudnativepg.
• PgDog: pgdogdev/pgdog (LICENSE, README.md, CONTRIBUTING.md, releases, CHANGELOG-ENTERPRISE.md); docs.pgdog.dev (databases, authentication, prepared statements, enterprise edition); pgdogdev/helm.
• PgBouncer: pgbouncer/pgbouncer (COPYRIGHT, the 1.26.0 release).
• Postgres: postgresql.org/support/versioning, /developer/roadmap, and the 2026-08-13 and 2026-05-14 release announcements.
• Object storage and log latency:
  • AWS S3 performance guidelines ("tens of milliseconds");
  • the S3 Express One Zone product page ("single-digit millisecond");
  • docs.warpstream.com low-latency clusters, and warpstream.com/blog (2026-02-04);
  • docs.automq.com WAL storage.
• TiKV:
  • pingcap/docs release-8.5: tikv-configuration-file.md, pd-configuration-file.md, configure-placement-rules.md, configure-load-base-split.md, latency-breakdown.md, tune-tikv-thread-performance.md, and the 5.0, 5.3 and 8.0 release notes;
  • tikv/tikv release-8.5 (src/storage/mod.rs, src/storage/txn/commands/atomic_store.rs, components/api_version/src/api_v2.rs), and tikv#10540;
  • tikv.org/docs/7.1/develop/rawkv/cas, tikv.org/docs/6.1/deploy/performance/overview;
  • the PingCAP Raft Engine blog;
  • tikv/client-rust (src/store/request.rs, PR #562), and tikv/client-go config/client.go;
  • bitsand.cloud/posts/cross-az-latencies; the AWS fault-isolation whitepaper (AZs); smalldatum.blogspot.com (2026-01, SSD power-loss protection and fsync); dina-kar/client-rust at 1f8962b (src/raw/client.rs, src/transaction/transaction.rs, src/config.rs); docs/plans/r1-dependency-spike.md §(i).
• Loam:
  • §02 (WAL classes), §04, §20 (D126, D130), §22 §6.4, §23;
  • crates/operon-log/src/writer.rs (the standard write path: one flush in flight, ack after the PUT and CommitWal);
  • crates/operon-meta-tikv/src/leases.rs (check_fence);
  • crates/operon-tikv;
  • deploy/neon/, deploy/tikv/;
  • deny.toml;
  • D11, D60, D61, D72, D96, D126, D130, D148–D157, D-SC-6, D-SC-12, D-SC-16; Q30, Q45–Q49.