Durable execution: Resonate, embedded

An agent run is a long program that calls expensive, flaky things. A ten-step research agent that crashes at step seven either repeats six paid model calls or loses the run. The usual fix is a workflow engine beside everything else: Temporal, or a queue plus cron plus a table of states. That is one more stateful system with its own database.

Loam's answer is to embed one. The Resonate server is linked into the Loam binary, speaks the standard Resonate protocol, and stores its state next to the rest of the namespace. Agent code uses Resonate's own SDKs, unchanged. This post covers the programming model, how the embedding works, and the patterns Loam builds on it.

The model: durable functions and durable promisesCopy Anchor Link

Resonate's model is distributed async/await. A durable function is ordinary code that calls other functions through a context. Each call's result is recorded as a durable promise on the server. When the process dies and the function is invoked again, it replays from the top, and every call that already completed returns its recorded result instead of running again.

from resonate import Resonate

resonate = Resonate(url="http://127.0.0.1:8001")          # Loam's durable listener

@resonate.register(name="research", version=1)
async def research(ctx, question: str):
    sources = await ctx.run(search_the_web, question)      # recorded once
    notes = await ctx.run(summarize, sources)              # a paid model call, recorded once
    return await ctx.run(write_report, notes)

If the worker is killed after summarize returns, the retry replays research, gets sources and notes from their promises, and resumes at write_report. (The example uses the Python SDK that ships in Resonate's monorepo, 0.8.1, which matches the server version Loam pins.) The model call is never paid for twice. Because the function replays, it must be deterministic between calls; anything non-deterministic (time, randomness, I/O) goes inside a recorded call.

Promises are the whole state. A workflow is a tree of promises under one root, its origin, and the server commits each transition of an origin atomically. Tasks with fenced leases hand work to workers, and a task whose worker vanishes is redelivered after a retry timeout. Schedules create promises from a template on a cron. The protocol is formally specified, with an executable model in Lean 4, a TLA+ model and a trace checker that replays a real server's traffic against the model. The Resonate deep dive goes further into it.

Why embed itCopy Anchor Link

We looked at running Resonate (or Temporal) as a sidecar and chose to link it in:

• One binary, one deployment. operon dev gets durable execution with no extra process, container or role.
• In-process calls. Loam's own Rust code talks to the server without HTTP, which matters when Loam's own long operations become durable functions.
• The same tenancy, storage and, later, auth. Durable state is per namespace, like everything else.
• Buy, not build. Loam writes glue and workflows, not a durable-execution engine.

Resonate made that possible because its new server is Rust, built from plugins, with a public composition API. A spike linked it into an axum 0.8 application, ran Resonate's Python fan-out example against it (including its crash mode, where only the failed branch re-ran), and killed the whole binary with kill -9 while a human-in-the-loop workflow was suspended. After the restart, resolving the approval completed the workflow from its stored state. No upstream change was needed.

How the embedding worksCopy Anchor Link

• Composition, not main. A crate, operon-durable, builds a registry of only the plugins Loam carries and starts the server with Resonate's build, start and stop calls. It never calls Resonate's own run, which installs a global tracing subscriber and waits for signals itself.
• Configuration from Loam's flags. Loam builds Resonate's configuration in code. It reads no resonate.toml and no RESONATE_* variables, so a standalone Resonate setup on the same machine cannot leak in.
• No process-wide surprises. Resonate's HTTP gateway can abort the process on a handler panic. Embedded, that would take down all of Loam, so it is pinned off and a panic answers 500.
• Loopback only. The durable API has no authentication yet, and it lets a caller create schedules and, through the push transport, make the server send HTTP requests to addresses named in a promise. So the listener accepts only loopback addresses and refuses anything else at startup, and the push transport is off by default.
• Loam's own workflows run in process. The Resonate Rust SDK accepts a custom network. Loam implements it by calling the server directly and delivering tasks to an in-process worker, so Loam's operations are durable functions with the SDK's replay semantics, and with a shared store any node can resume any of them.

The server crates are not on crates.io yet, so Loam depends on a pinned revision of a fork. The fork is upstream plus a small set of commits, each also offered upstream: a dependency-hygiene change (newer sqlx and rusqlite, rustls instead of OpenSSL, which cleared seven of eight RustSec advisories against our cargo deny policy), TiDB support in the MySQL plugin, and error classification for retries.

StorageCopy Anchor Link

Why TiKV next: Resonate's blob server keeps one compare-and-swapped document per origin, a design checked in TLA+. Putting that on TiKV removes a SQL layer and a connection pool per tenant, and puts durable state on the same cluster as Live.

The patterns, and what Loam uses them forCopy Anchor Link

Submit, then poll. A bulk import is POST …/collections/{c}/import, which answers 202 Accepted with Location: /v1/operations/op-…. The operation id is the root promise id, and its state maps directly from the promise: pending, running, succeeded, failed, canceled. Progress needs no second store: branch promises are tagged with the operation id and counted. The result carries the consistency token that covers every write, so a client can read its own import.

Fan-out. The import plans once (list the source, fix the file set with ETags), then starts one branch per file and one step per Parquet row group or 64 MiB NDJSON slice. A crash re-runs the slice it was in. Upserts by primary key make that converge, and documents without a key get ids derived from the operation, file and row, so a re-run never duplicates.

Schedules. Each run of a scheduled import starts one durable call per file with an id derived from the schedule, key and ETag. Resonate deduplicates on the id, so a file imported earlier is found already resolved and is not read again, and a changed file (new ETag) is imported again. The durable store is the dedup ledger.

Sagas. Tenant provisioning is a sequence of steps, each with a compensation: a directory entry, a TiKV keyspace, a bucket prefix and envelope key, quotas, an authorization store, the tenant's durable store. On failure the workflow runs the compensations in reverse. Because the workflow is durable, a crash halfway through compensating resumes the compensation instead of leaving a half-provisioned tenant.

Human in the loop. A destructive request becomes an operation whose first step waits on an approval promise. An approver resolves or rejects it; without one, it times out and is rejected. The same gate serves agents whose policy marks an action as needing approval.

What we deliberately did not moveCopy Anchor Link

Loam's maintenance loops (split merges, Lance compaction, HNSW builds, garbage collection) stay lease-fenced worker tasks. They are short, idempotent and re-derived from state on every poll. A crash loses at most one run, and the next poll proposes it again. A cron would add latency and a dependency to the most important loops and give them nothing. Durability pays where one logical job outlives one task run or crosses systems: imports, erasure, re-embedding.

Limits we know aboutCopy Anchor Link

• Retention. The protocol has no delete for settled promises, so durable state grows without bound. Pruning is not free either: if a late retry re-creates a pruned id, the work runs again. Loam prunes only its own finished operations after a retention period; a general setting is proposed upstream.
• Tenancy. Resonate has no tenant concept: groups, schedules and searches are global to a server. Today the embedded server is one instance with one store. The planned isolation is one server instance per namespace, over the namespace's own store, which needs a small upstream change to serve per-tenant routes behind Loam's dispatcher, and needs the auth plan to know which tenant is calling.
• No cross-system transactions. A workflow step and a Loam write are not atomic. Steps are idempotent instead.
• Protocol versions. SDKs must match the server's protocol version (2026-04-01 for the pinned server). We document which SDK versions pass our conformance run.

Where this standsCopy Anchor Link

The durable feature is opt-in in builds today because it adds several hundred crates to a build; release builds and the managed cloud turn it on. The next post uses durable promises for something else: making waiting free.