§34 The Standards Charter and the Narrow Waist (the protocol gateway moved)
Stub since 2026-10-02 (D440): the protocol gateway is a commercial component in `loam-platform` (private). Keeps the vendor-neutral decisions: the standards charter, transports, Connect-RPC, `loam.stream.v1` as the narrow waist, the Loam CloudEvents profile (`io.loams.dev.` types, D402), the high-rate path, the events → Arrow mapping, state rules (D360–D365, D372, D374, D378); the `Runner` trait lives in §24 §16 (D375) and runner usage in §27 §3.6 (D376)
Status: Stub · 2026-10-02. The protocol gateway is a commercial component, designed in the private loam-platform repository (owner ruling of 2026-10-02: "move Cloudflare, OpenRTB etc. commercial to private repos"; §38 D440). The OpenRTB and Google adapters, the canonical loam.rtb.v1 model, per-partner version negotiation, the bid hot path, match/merge for identity profiles, the ad-tech conformance suite and the plans GW1–GW4 left this repository for loam-platform (private) with that ruling; D366–D371, D373, D377 and D379 are loam-platform's now, under its own numbering. This repository does not depend on any of it. What stays is the vendor-neutral part that the open engine needs, kept below so that the decisions D360–D365, D372, D374, D375, D376 and D378 keep a home. They are still proposals until the owner rules on them (Q450 asks whether they get a document of their own).
1. What stays open, and where
| # | Decision | Where it lives now |
|---|---|---|
| D360 | The standards charter: every external standard Loam speaks is pinned to a spec version behind one crate, so replacing it touches one crate (HTTP/2 and mTLS, Connect/gRPC/gRPC-Web, CloudEvents 1.0.2, Arrow/Parquet/Iceberg, protobuf with Avro only at schema-registry sinks, OTel and W3C Trace Context, the Resonate protocol, loam.meter.v1). A decision-log row per standards choice; deprecation is announce → dual-run for at least one minor release → sunset; open formats only in stored data; crypto agility through key and algorithm ids; cargo deny, cargo vet on new data-plane crates, CycloneDX SBOMs, reproducible release builds | Here |
| D361 | Transports: HTTP/2 with mTLS (ALPN h2) between Loam components, h2c only on loopback; HTTP/3 only at the Envoy edge (D176, D184) until an internal-mesh flag | Here |
| D362 | Connect-RPC through connect-rust (connectrpc 0.9.1, Apache-2.0, the official Connect project's Rust implementation) for every new service (D128, D206) | Here |
| D363 | The narrow waist is loam.stream.v1.StreamService: Produce and ProduceCloudEvents (D270) are the one internal ingress for async events; Dapr pub/sub, HTTP push, runner hosts and runtime hooks are adapters into it. buf breaking (FILE) on proto/loam/{stream,events,meter} | Here |
| D364 | The Loam CloudEvents profile | §2 below |
| D365 | High-rate events bypass the dedup ledger | §3 below |
| D372 | Events → Arrow: one Arrow schema per event type, derived from the data message's protobuf descriptor | §4 below |
| D374 | State rules: the tenant scope on every stored key in Loam's existing forms (ns/<ns>/ in the bucket, keyspace plus prefix in TiKV); money as integer micros plus an ISO 4217 currency; int64 ns UTC in new protobuf contracts; TiKV only through transactions or atomic CAS | Here |
| D375 | One Runner trait (SupervisorRunner default; process, Lambda and Knative runners) | §24 §16; plan RN1 |
| D376 | Usage from every runner reaches §27's contract: one reporter per invocation, fields 12–16 of loam.meter.v1.Invocation. Hooks only: the ledger, rating and reconciliation are loam-platform's | §27 §3.6 |
| D378 | Languages: Rust for the data plane; other languages through buf-generated Connect clients; in-process embedding only after profiling; no core path depends on Go or Java SIMD | Here |
2. The Loam CloudEvents profile (D364)
| Attribute | Required | Rule |
|---|---|---|
specversion, id, source, type | yes | CloudEvents 1.0 (D270 validates them) |
id | yes | The idempotency key. Producers keep it stable across retries; D270 dedupes on SHA-256(source ‖ 0x00 ‖ id). There is no idempotencykey extension |
type | yes | <prefix>.<domain>.<name>.v<major>; a breaking change of data is a new major, so a new type. The owner ruled on 2026-10-01 that the prefix is io.loams.dev (Q361 answered); §02's dev.loam.stream.record and §27's dev.loam.meter.usage.v1 move with the rename PR |
dataschema | yes for Loam-defined types | urn:loam:proto:<full message name>; minor evolution is additive (§4), so there is no schemaversion extension |
time | yes | RFC 3339 with nanoseconds; the producer's clock |
tenantid | yes | <org>/<namespace>, the same value as x-loam-tenant (§27 §3.4). Set by the gateway or the runner host from the credential; a client-supplied value is replaced and counted (loam_events_tenantid_overwritten_total) |
traceparent | yes (may be generated) | W3C Trace Context; tracestate optional; generated at the first Loam hop if missing |
partitionkey | optional | D270: becomes the record key |
datacontenttype | optional | application/protobuf for Loam-defined types on the high-rate path, application/json at the edge |
3. The high-rate path (D365)
D270's ledger costs two metastore proposals per request and about 80 bytes per event for the window, so producers far above webhook rates (runtime usage events, and any adapter that emits an event per request) use plain produce instead:
- buffer events per
(namespace, stream)in a bounded queue (default 65 536 events or 64 MiB, whichever first); - flush every 50 ms or at 1 MiB as one
Produceof records in D270's Kafka binary-mode layout (ce_headers,content-type, key frompartitionkey, value the protobufdata); - drop the oldest batch when the queue is full and count it (
loam_events_dropped_total{reason="queue_full"}), never blocking the producer's response; - do not deduplicate at ingest; the event table (§4) is keyed on
(source, id)and removes duplicates.
Readers see the same CloudEvents either way: §02 §7.4's consume path rebuilds any record whose ce_ headers validate. High-rate ingestion from outside (telemetry, clickstreams, IoT) is the Event Fabric's job (§32 D331), not this path's.
4. Events → Arrow (D372)
No official Arrow mapping for CloudEvents exists (cloudevents/spec v1.0.2 defines JSON, protobuf and Avro; its working drafts add Avro compact, CBOR and XML; read 2026-10-01). Loam's:
| Column | Arrow type | From |
|---|---|---|
id, source, type, subject, dataschema, datacontenttype, tenantid, traceparent, tracestate | Utf8 (dictionary-encoded for source, type, tenantid) | the attributes |
time | Timestamp(Nanosecond, "UTC") | time |
ext | Map<Utf8, Utf8> | other extensions, string form |
ext_types | Map<Utf8, Utf8> | extension name → CloudEvents type for every non-string extension (D270's loam_ce_types) |
data | Struct derived from the data message's descriptor | data |
data_raw | Binary | the original data bytes, for lossless replay (on by default) |
The data struct is derived at build time from the protobuf descriptor (scalars to scalars, repeated to List, messages to Struct, map to Map); each Arrow field carries its proto path and tag in field metadata; Iceberg field ids are assigned by the catalog and matched by path, and only additive evolution passes CI. An event stream is linked (§09) to an Iceberg table (§08, M4) keyed on (source, id), partitioned by day(time), sorted by (type, time); until Iceberg v3 is on the pinned stack, time is timestamptz (µs) plus a time_ns long column (Q368).
5. Open questions kept here
| # | Question | Owner | Needed by |
|---|---|---|---|
| Q362 | Showback and single-organisation billing in the open repository, or loam-platform only. Answered by the owner on 2026-10-02: no metering in OSS (D440, D444); open dashboards over the hooks remain possible for anyone to build | Founder | Resolved |
| Q366 | Lambda CPU attribution: the bootstrap's getrusage delta capped by billed duration × memory_mb / 1 769, or billed duration (§27 §3.6) | Founder | RN1 Task 5 |
| Q367 | Cloud Run and Container Apps runners: build or document only | Founder | After RN1 |
| Q368 | Iceberg v3 timestamptz_ns by M4, or the time_ns column (§4) | Eng | Event-table plan |
| Q369 | Move operon-stream-grpc from tonic/prost to connect-rust/buffa (D128), and when | Eng | M2 stream API plan |
| Q372 | Internal HTTP/3: the condition that enables it | Eng | Later |
| Q450 | Give §1's decisions their own document, and split GW1's vendor-neutral tasks (buf breaking, the CloudEvents profile, the event Arrow mapping) into an open plan | Founder | Before GW1 starts in loam-platform (private) |
Q360, Q363–Q365, Q370, Q371, Q373 and Q374 moved to loam-platform with the gateway and the Cloudflare runner.
§33 Loam Flow Connectors: Registry, Capabilities and the Catalog
The connector registry and capability schema; runtimes (native Rust, Iggy's connectors runtime, Camel in `loam-connect`, Debezium Server; Kestra as a companion); envelope and delivery; Arrow/ADBC bulk paths; CDC through Debezium; the 21 ★ connectors; the licence gate; the 200-connector matrix; track CN (D352–D359)
§36 Loam Git: a WAL on the Bucket, Smart HTTP, Agent Scopes and a Build Cache
Extends §15 §3: a per-repository WAL of create-only segments on the bucket (no head pointer; R2's one-write-per-second-per-key limit), CloudEvents records, checkpoints, one-object packs, a group-committing sequencer per repository; `WalStore`, `RefLog`, `BlobStore`, `Materializer`; Smart HTTP (v2 upload-pack, v0/v1 receive-pack) on gitoxide primitives with stock git as oracle and repacker; `git-remote-loam`; `loam-vfs` scopes with write admission; the sccache cache (direct and gateway paths, trust classes); the crates mirror; the `Fs` trait (§17); track GT (D381, D382, D388–D399)