§23 Neon and WeSQL: Postgres and MySQL on the Bucket, Beside Loam

Status: Proposed · 2026-09-29; amended the same day by §28 Loam Postgres (owner decisions D230–D236). D149 (Neon for the showcase apps) and D151 (fork only when needed) are superseded: the showcase apps run plain Postgres 17 on CloudNativePG (D230), and Neon is forked now as dina-kar/neon, with Loam's control plane as its primary control plane (D231, D232). D150 and D153 are amended: Loam serves the control-plane API, and PgDog, not Loam's pg listener, routes Postgres OLTP connections (D236). Q45–Q48 are answered and Q49's syntax is settled there. The rest of this document stands; where the two disagree, §28 wins. §29 (2026-10-01, Proposed) proposes to amend D156: WeSQL would get a four-milestone plan (foreign keys, a durable commit in the Loam WAL quorum, failover, an Iceberg bridge). Until §29's W1 and W2 pass their acceptance tests, D156 and the Q50 gate stand unchanged; W2's design is a proposed answer to Q50, not a closure of it.

This document comes from a conversation with the owner about running Neon (serverless Postgres whose storage lives on object storage) and WeSQL (MySQL whose storage lives on object storage) next to Loam. Two points were settled in that conversation: neither engine is linked into the Loam binary, and both run unmodified as sidecar services on the same RustFS store while Loam integrates with them from the outside. Everything else here is a proposal, decisions D148–D157 and open questions Q45–Q51. D149 proposed amending D-SC-12 and D-SC-16; the owner declined it on 2026-09-29, and D230 supersedes it: the showcase apps run plain Postgres on CloudNativePG, not Neon. The other decisions stay proposals, except where §28 amends them.

The spike of 2026-09-29 (§9) ran both engines on RustFS. Neon worked: psql, a branch created through the pageserver API, isolation between branches, pgoutput logical replication, OpenFGA and GlitchTip migrations, and recovery after the pageserver's disk was wiped. WeSQL ran basic SQL, but Forgejo's migrations failed on foreign keys, and commits made after the last snapshot were lost when the container was replaced without its local volume. The spike also found that Neon's public repository has been nearly dormant since August 2025 (§4.1). That fact shapes D151.

Markers: (spike) means measured in the spike (notes: .superpowers/research/neon-wesql-spike.md in the design worktree, not committed). (verify) means not checked against a primary source; the PR that depends on it checks it first. (estimate) means computed, not measured. Paths of the form neon/… point into neondatabase/neon main as of 2026-09-29 (last commit fa504217c, 2026-08-31). Paths of the form wesql/… point into wesql/wesql branch 8.0 (last commit 3e29b8850, 2026-08-29).

1. SummaryCopy Anchor Link

2. Goals and non-goalsCopy Anchor Link

2.1 GoalsCopy Anchor Link

1. Real Postgres for the showcase apps, on the bucket. Plane, Zulip, GlitchTip, Keycloak and OpenFGA need Postgres OLTP with interactive transactions, JSONB, arrays, SELECT … FOR UPDATE and, for Zulip, PGroonga (§22 §13a). Neon is Postgres (16.9 and 17.5 in its last build), so these apps run unmodified, and their data lives in the same RustFS store as Loam's.
2. Branching as a Loam feature. Every agent workspace gets a copy-on-write Postgres branch in milliseconds (36 ms in the spike, (spike)). This is the database counterpart of §15's Git branches and copy-on-write environments.
3. One data plane. Loam creates the databases, holds the mapping in its metastore, routes connections through its own listeners and copies changes into collections. Search, the activity feed and the assistant (§22 §8) then see the apps' data without asking the apps.
4. License-clean. Neon is Apache-2.0. WeSQL is GPL-2.0-only and runs only as a separate, unmodified process (D148).

2.2 Non-goalsCopy Anchor Link

• Linking Neon or WeSQL into Loam. Neither engine is a library, and WeSQL's GPL-2.0-only license rules out linking under D11.
• Building a Postgres or MySQL engine. Loam's listeners route and bridge. They do not execute OLTP SQL (D2 holds for the retrieval engine; D130).
• Loam Live on Neon. Loam Live stays on TiKV (§20). D149 only removes the need for a general Postgres over TiKV.
• Neon's hosted features. The console, the Data API, Neon Auth, autoscaling VMs and scale-to-zero through the proxy are out of scope in track N. Scale-to-zero is Q47.
• Merging branches. Neither Neon nor Postgres can merge data between timelines, and Loam does not pretend otherwise (D155).
• Upstream work without approval. No issues or PRs go to neondatabase or apecloud without the owner's explicit go-ahead.

3. ArchitectureCopy Anchor Link

   apps (Plane, Zulip, GlitchTip, Keycloak, OpenFGA)     agents / psql / BI         Forgejo? Matomo?
                 │ postgres://…/<db>                          │                        │ mysql://…/<db>
                 ▼                                            ▼                        ▼
 ┌──────────────────────────────────────────────────────────────────────────────────────────────────┐
 │ Loam                                                                                             │
 │  pg listener (PG1, datafusion-postgres)           mysql listener (opensrv-mysql)                 │
 │   StartupMessage.database ─┬─ mapped? ── splice ──► Neon compute     terminate auth ── WeSQL     │
 │                            └─ else ── Loam analytics (collections, search functions)             │
 │                                                                                                  │
 │  operon-neon (feature `neon`): pageserver / storage-controller API, compute specs, ComputeRuntime │
 │  Loam Durable (§21): branch sagas ──► create timeline → start compute → record → return DSN      │
 │  bridges: pgoutput (Neon) / binlog (WeSQL) ──► DocOps ──► collection implicit stream (D72, D129) │
 │  operon-meta-tikv: x/ databases, X/ branches, b/ bridge checkpoints                              │
 └──────┬───────────────────────────────┬───────────────────────────────────────┬───────────────────┘
        │ HTTP :9898 (mgmt)              │ Postgres wire :55433                  │ MySQL wire
 ┌──────┴──────────────────────┐  ┌──────┴────────────────────────────┐   ┌──────┴───────────────────┐
 │ Neon storage                │  │ Neon computes (stateless Postgres)│   │ WeSQL (single node,      │
 │ pageserver, safekeepers ×3, │◄─┤ one per (database, branch) in use │   │ SmartEngine on S3,       │
 │ storage broker, storage     │  │ compute_ctl + spec from Loam      │   │ local volume kept)       │
 │ controller                  │  └───────────────────────────────────┘   └──────┬───────────────────┘
 └──────┬──────────────────────┘                                                  │
        │ S3 (layers, WAL offload)                                                │ S3 (snapshots, binlog)
 ┌──────┴─────────────────────────────────────────────────────────────────────────┴───────────────────┐
 │ RustFS (Apache-2.0): bucket `loam` (Loam), bucket `neon`, bucket `wesql`                           │
 └────────────────────────────────────────────────────────────────────────────────────────────────────┘

3.1 ComponentsCopy Anchor Link

3.2 The Neon layout: tenants, timelines, computes (D150)Copy Anchor Link

• One Neon tenant per (namespace, database). Neon branches whole Postgres clusters: a timeline holds every database in the cluster. If Plane and Zulip shared a tenant, a Plane branch would copy Zulip's data. Separate tenants keep branches, quotas and erasure per app. They cost nothing when idle: a tenant with no compute is only layers in the bucket.
• The default timeline is main. A branch is a timeline whose ancestor_timeline_id is its parent, optionally at an ancestor_start_lsn for point-in-time branches.
• A compute per (database, branch) in use. Loam writes the spec (neon.tenant_id, neon.timeline_id, the safekeeper list, the pageserver connstring, roles, wal_level = logical) and starts compute_ctl through a ComputeRuntime. The dev runtime uses Compose or Podman (deploy/neon/). A Kubernetes runtime comes later. A compute is replaced, never restarted (§9.1).
• Ids. Tenant and timeline ids are 16 random bytes in Neon. Loam derives them deterministically, the first 16 bytes of SHA-256("neon/tenant/" + ns + "/" + db) (sha2 is already a workspace dependency) and the same for branches, so a retried saga step creates the same timeline rather than a second one (D155).

3.3 The WeSQL layoutCopy Anchor Link

One WeSQL instance per (namespace, database), each with its own bucket prefix (WESQL_OBJECTSTORE_ROOT_DIR) and a kept local volume (§9.2). WeSQL has an objectstore_branch_id, but branching was not tested in the spike and is not part of this design.

4. Upstream status, verified 2026-09-29Copy Anchor Link

4.1 NeonCopy Anchor Link

Reading. The public repository is effectively dormant, and development moved in-house. Apache-2.0 allows a fork, but Loam would own every fix, including Postgres minor releases rebased onto Neon's patched Postgres trees. This is the largest risk in this document (§12, row 1), and it is why D151 keeps plain Postgres as a working exit.

4.2 WeSQLCopy Anchor Link

Reading. WeSQL is a beta from one company, with a long gap and a recent burst that removed multi-replica HA. It is usable for experiments, not for data the suite cannot lose. This supports D156.

5. Why Neon for the showcase apps, and what it changes (D149)Copy Anchor Link

Superseded 2026-09-29 by D230 (§28 §4). The owner declined D149: the showcase apps run plain Postgres 17 on CloudNativePG with PITR to RustFS. This section is kept as the record of the proposal. The last paragraph ("If the owner declines D149") describes the path taken.

§22 wants the suite's apps on Loam end to end. D-SC-12 asks for Postgres write compatibility, and D-SC-16 points it at a Postgres front end over TiKV. That front end would have to be a Postgres-compatible OLTP engine: multi-statement transactions, FOR UPDATE, sequences, JSONB, arrays, pg_trgm, PGroonga and a catalog good enough for Django, Rails-like ORMs and Keycloak's Hibernate. That is years of work, and each app would still move only when its own test suite passed (§22 §13a).

Neon gives that compatibility today because it is Postgres, and it keeps D1's spirit: its durable state is in the bucket (layers and offloaded WAL), and its disks are caches, except for the safekeepers' WAL window. The spike ran OpenFGA's and GlitchTip's migrations on Neon unchanged (§9.1).

The proposal:

1. Showcase apps use Neon through Loam's routing (D153), and their changes flow into collections (D154).
2. The Postgres front end over TiKV narrows to Loam Live's reactive, Convex-style API (§20). The general Postgres front end is not built.
3. D-SC-16 gains an exception. "Transactional storage is TiKV only" becomes "TiKV for Loam Live, the metastore and durable state; Neon for Postgres-wire OLTP". There is still no TiDB.

If the owner declines D149, the suite keeps plain Postgres 17 (§22 §6.4) and the TiKV front end stays the long-term plan. N1–N5 still have value for agent workspaces (D155) and bridges (D154), and D151 makes routing and bridges on plain Postgres a configuration, not a rewrite.

6. Loam's integration pointsCopy Anchor Link

6.1 operon-neon (D150, N1)Copy Anchor Link

A crate behind the operon feature neon, off by default. It has no Neon code dependency: it speaks HTTP and JSON with reqwest (rustls) and serde.

• NeonClient calls the pageserver API: PUT /v1/tenant/{t}/location_config (attach with a generation), POST /v1/tenant/{t}/timeline/ (bootstrap or branch: new_timeline_id, ancestor_timeline_id, ancestor_start_lsn, pg_version, read_only), GET …/timeline, DELETE …/timeline/{tl} and GET /v1/status. Once the storage controller runs (Q48), the same calls go through the controller, which then owns generations. The request and response shapes are neon/libs/pageserver_api/src/models.rs (TimelineCreateRequest, TimelineCreateRequestMode::Branch); Loam keeps its own copies of the few structs it needs.
• ComputeSpec builds compute_ctl's JSON spec from Neon's docker-compose template: roles, databases, settings, wal_level = logical, and fsync left on outside dev.
• ComputeRuntime is a trait with start(spec) -> Endpoint, stop(id) and status(id). N1 ships a Podman/Compose runtime for dev and tests. Kubernetes follows in a later plan.
• Errors map onto Loam's error model. Neon returns JSON {"msg": …} with HTTP status codes.

6.2 Metastore records (D152, N2)Copy Anchor Link

In operon-meta-tikv's keyspace. The prefixes x/, X/ and b/ were checked as unused against crates/operon-meta-tikv/src/keys.rs at e69ab38:

Writes are compare-and-set on the version, like pointers (§20 §11.2). Secrets live in the credential store of the unified auth plan (Q30), not in the metastore.

6.3 Routing by database name (D153, N3 and N6)Copy Anchor Link

Amended 2026-09-29 by D236 (§28 §8): for Postgres OLTP, PgDog (unmodified, AGPL-3.0, a separate service) routes by database name (<db>, <db>__<branch>), with config rendered by Loam's control plane. The pg listener keeps analytics (PG1). The MySQL half below is unchanged.

Postgres. Postgres OLTP connections go to PgDog, not to Loam's pg listener (D236, §28 §8). Loam does not peek or splice:

• Config from records. Loam's control plane renders pgdog.toml and users.toml from the x/ and X/ records: one [[databases]] entry per mapped database, named <db> for main and <db>__<branch> for each branch (Q49's syntax; the options form is unsupported), pointing at that branch's compute. It writes them to a ConfigMap and a Secret and sends RELOAD to PgDog's admin database after every record change, so a new or deleted branch takes effect without restarting PgDog.
• Unmapped names have no PgDog entry. Loam's analytics stays on the pg listener exactly as PG1 plans (Ruling 1: the database is the namespace).
• Auth. PgDog terminates SCRAM against users.toml; Loam provisions the same role into the compute spec, and PgDog uses it for server connections.
• Authorization before routing. Loam renders a role into users.toml only for the databases its grants allow (the branch record's owner and the namespace's grants), so PgDog refuses an unauthorized database name before it opens a compute connection.
• TLS on both hops, with peer verification (§28 §8): clients connect to PgDog over TLS, and PgDog connects to computes over TLS that verifies the compute's identity (or an equivalent protected channel, such as mesh mTLS). Neither hop carries plaintext on the cluster network.
• PgDog stays reachable only inside the cluster (or on loopback in dev) until the unified auth plan (D111, Q30).

Proposed 2026-10-01 (§31, D320): Vitess's vtgate (Apache-2.0, unmodified, v24) fronts WeSQL instead of the Loam-built splice below, for unsharded and sharded keyspaces alike, with Loam rendering vtgate's static auth file and VSchema. If §31's RT3 compatibility gate fails, the splice stays the fallback for unsharded WeSQL only; sharded MySQL then waits for §31's D317 router. Vitess-fronted WeSQL runs with gtid_mode = ON (§31 §9.1).

MySQL. The server speaks first, and the database arrives in the client's HandshakeResponse, authenticated against a scramble from the server's greeting. Loam therefore cannot splice before authentication. It completes the handshake itself (caching_sha2_password or mysql_native_password, with credentials from the auth plan), opens its own connection to WeSQL with a stored credential and the requested schema, over TLS that verifies WeSQL's identity (the stored credential never crosses a plaintext connection), and then splices. COM_CHANGE_USER and COM_INIT_DB to another database are refused. This is N6. The simpler fallback is a dedicated port per WeSQL instance.

6.4 Change bridges (D154, N4 and N6)Copy Anchor Link

From Neon (pgoutput). One logical slot and one publication per (database, branch, bridge). The spike showed wal_level = logical on the compute, pgoutput messages for insert, update and delete, and a slot that survives replacing the compute, because Neon keeps slot state in the pageserver (spike).

• Reading. A replication connection (START_REPLICATION SLOT … LOGICAL … (proto_version '1', publication_names …)). Buy before building: supabase/etl (Apache-2.0, active, pushed 2026-09-28) implements the pgoutput protocol in Rust (verify its crate boundaries and dependency cost against deny.toml in N4). The fallback is a small decoder over a replication-capable Postgres client.
• Mapping. Each table maps to a collection by a declared mapping (primary key → _id, columns → fields, optional text and vector fields). Insert and update become upserts, delete becomes a delete by _id, and TRUNCATE becomes a delete by filter (D87).
• Exactly once. Operations are appended to the collection's implicit stream with producer id = (database, branch, bridge) and sequence = (commit LSN, ordinal in the transaction) through D72's idempotent producers. Loam sends confirmed_flush_lsn back only after the append is acknowledged. A crash re-sends from the confirmed LSN, and the producer sequence drops the duplicates. This is the same contract as D129.
• Operations. A bridge is a link-style task under worker leases (§09) and honours the unapplied-data budget (D86). A lagging bridge holds WAL on the safekeepers and pageserver, so Loam alarms on slot lag and can drop a slot past a limit, which then requires a resync (initial copy by COPY at a snapshot, then streaming, as in Postgres' own table sync).

From WeSQL (binlog). binlog_format = ROW and binlog_row_image = FULL by default, with gtid_mode = OFF (spike). The bridge registers as a replica (COM_BINLOG_DUMP from file:pos) and maps row events the same way. The sequence is (file index, position). mysql_async (MIT or Apache-2.0) has a binlog stream client (verify it against WeSQL's consensus-era binlog events, which WeSQL dropped on 2026-08-23).

6.5 A branch per agent workspace (D155, N5)Copy Anchor Link

§15 §9.2's session workflow gains a step ws_db = ctx.run(fork_database, task.db, id) beside fork_branch. As a saga on Loam Durable (§21 §6.3, pattern c):

At the end of a session, the branch is kept (for review, with a TTL), discarded (compensate steps 3 → 1), or promoted: the database's x/ record is repointed to the branch's timeline, but only if main has had no writes since the branch point. Otherwise promotion is refused, and the result lands as migrations applied to main. Idle branches cost only their layers. GC deletes expired timelines through a durable schedule (§21 §6.1).

7. LicensesCopy Anchor Link

Running separate services creates no obligation on Loam's code (as D60 does for Alternator and D-SC-6 for the suite). Images Loam redistributes must carry the upstream notices, and a patched WeSQL (which Loam does not plan) would need a GPL-2.0 source offer.

8. DeploymentCopy Anchor Link

• Dev: deploy/neon/ and deploy/wesql/ (added in this PR, both run from scratch in the spike): RustFS, the Neon storage services with one safekeeper, and computes started against explicit ids; WeSQL with the RustFS virtual-host settings. Loopback ports, default credentials.
• Production (later plan): three safekeepers on separate nodes with volumes, one or more pageservers, the storage controller with its own Postgres database (Q48), computes managed by Loam's Kubernetes ComputeRuntime, and Neon's proxy only if Q47 chooses it. Buckets or prefixes per engine, with lifecycle rules that never expire Neon layers (Neon's GC owns deletion).
• Resources (spike): storage services under 0.5 GB of RAM together; each idle compute tens of MB (verify under load); images of about 5 GB (neon) and 1.3 GB (compute-node-v16). WeSQL used 0.7 GB of RAM.

9. Spike results (2026-09-29)Copy Anchor Link

Rootless Podman 5 with docker-compose v5.2.0, RustFS 1.0.0, neon:latest and compute-node-v16:latest (both from 2025-08-26), and wesql-server:8.0.35-0.1.0_beta5.40. No Operon code was written and nothing was compiled. The full commands and outputs are in the spike notes.

9.1 Neon on RustFSCopy Anchor Link

9.2 WeSQL on RustFSCopy Anchor Link

9.3 What the spike changesCopy Anchor Link

• Neon is technically ready for the suite's apps and for workspace branches. The risk is upstream maintenance, not function (D151, §12).
• WeSQL cannot serve Forgejo. Its S3-only durability claim does not hold under the image's defaults, and upstream just removed HA. It stays a dev option (D156).
• Both compose files were promoted to deploy/ and re-run from scratch: Neon (main 10 rows, branch 5 rows after a divergent delete) and WeSQL (rows kept across a restart with the volume).

10. Testing (per PR)Copy Anchor Link

• N1: unit tests against recorded pageserver responses, and an ignored-by-default integration test against deploy/neon/ (tenant, timeline, branch, compute, SELECT 1, teardown).
• N2: records through the metastore conformance suite's CAS and absence cases, plus a TiKV playground test.
• N3 (§28's P3): rendering tests from records to pgdog.toml and users.toml (golden files); then tokio-postgres through PgDog: <db> and <db>__<branch> reach the right compute (SELECT current_setting('neon.timeline_id')); a branch created or deleted after RELOAD appears or disappears without dropping other databases' connections; a role not granted a database is refused; plaintext is refused on both hops; an unmapped name stays on Loam's analytics listener.
• N4: exactly-once under fault injection (kill the bridge between append and confirm; replace the compute mid-stream; drop the connection), checking that the collection equals the table.
• N5: saga crash tests at every step boundary; compensation leaves no timeline or compute behind; a promotion is refused when main has diverged.
• N6: WeSQL through the MySQL listener, and the binlog bridge under the same fault set as N4. Blocked on Q50.
• Suite gate (in the Commons project): OpenFGA's Postgres datastore tests and GlitchTip's test suite against Neon through PgDog, then Keycloak and Plane.

11. Roadmap: track N (D157)Copy Anchor Link

Amended 2026-09-29 (§28 §11): N1 and N2 become inputs to §28's P2b (the Loam control plane), and N3's Postgres routing becomes P3 (PgDog). N4 and N5 run on Loam Postgres; N4 also runs against CloudNativePG. N6 is unchanged.

Each PR is small, stacked and behind the neon feature, and changes no default code path. N3 no longer extends PG1's listener (D236), so it depends only on N1 and N2. N6's MySQL routing still extends the MySQL listener.

Production deployment (the storage controller, three safekeepers, the Kubernetes runtime, and the proxy if chosen) is a later plan, after Q47 and Q48.

12. RisksCopy Anchor Link

13. Open questionsCopy Anchor Link

14. Contradictions with earlier decisions, and how they are resolvedCopy Anchor Link

15. SourcesCopy Anchor Link

Read on 2026-09-29.

• Neon: github.com/neondatabase/neon (LICENSE, NOTICE, README.md, docker-compose/{docker-compose.yml,README.md,pageserver_config/,compute_wrapper/}, libs/pageserver_api/src/models.rs, libs/remote_storage/src/s3_bucket.rs (path-style with a custom endpoint), proxy/README.md, proxy/src/binary/proxy.rs (AuthBackendType), proxy/src/control_plane/client/cplane_proxy_v1.rs, storage_controller/src/main.rs (--database-url), compute/compute-node.Dockerfile, vendor/revisions.json); commit and release history through the GitHub API; ghcr.io/neondatabase/neon:latest and compute-node-v16:latest image metadata.
• Neon and Databricks: https://neon.com/blog/neon-and-databricks, https://neon.com/docs/introduction/neon-and-lakebase, https://www.databricks.com/blog/announcing-lakebase-public-preview.
• WeSQL: github.com/wesql/wesql branch 8.0 (README.md "Licensing", mysys/objstore/s3.cc, include/objstore.h, storage/smartengine/core/LICENSE, patches/, .github/workflows/cicd-pull-request.yml); commit history; Docker Hub apecloud/wesql-server tags; the image's /apecloud/mysql/scripts/{libacenv.sh,libutil.sh,libmysql.sh}.
• RustFS: rustfs/src/server/layer.rs (RUSTFS_SERVER_DOMAINS), docker-compose.yml.
• Candidates: github.com/supabase/etl (Apache-2.0), github.com/blackbeam/mysql_async.
• Loam: §02 (D72), §09, §14, §15 §9, §20 §11–§12 (D129), §21 §6.3, §22 §6.4, §13a, §13b; docs/plans/2026-09-28-pg1-postgres-wire.md; D1, D2, D11, D42, D60, D61, D86, D87, D111, D123, D130, D-SC-6, D-SC-12, D-SC-14, D-SC-16; Q30, Q-SC-9.