Loams Durable

Loams Durable is durable execution inside Loams. A function written with a Resonate SDK records every step it takes as a durable promise. If the process running it dies, another worker picks the work up, replays the function, gets the recorded results of the finished steps back, and carries on from the first step that did not finish. Nothing that already completed runs again.

It is not our own engine. Loams embeds the Resonate server (Apache-2.0) in the loams binary and speaks the Resonate protocol unchanged, so the official Resonate SDKs work against it with only a URL change. The protocol, the SDKs, the storage engines and the formal model behind them are Resonate's work. What Loams adds is the embedding: one binary instead of another service, configuration from Loams' own flags, an in-process path for Loams' own code, and a TiKV storage backend for clusters.

Use it when a run lasts longer than a request and must not redo paid or side-effecting work after a crash: an agent that calls models and tools for minutes, waits for a human, or fans out sub-tasks. It is a good fit when you already run Loams and do not want a sixth system just for workflows. Do not use it yet for multi-tenant or internet-facing workloads: in 0.1.0 the durable API is unauthenticated, serves one namespace and listens on loopback only. If you need a mature workflow platform with a large ecosystem today, Temporal is the safer choice (see the comparison below).

What shipsCopy Anchor Link

Loams Durable ships in Loams 0.1.0, and has been in every image since v0.1.0-preview.1: the loams image is built with the cargo feature durable (beside live,live-tikv,graph,tikv,pgwire,mysql-wire,stream-grpc). The feature status table in docs/release/preview.md lists it as Available, and "Durable store on TiKV" as In progress.

The 0.1.0 CHANGELOG has no Durable entry beyond the upgrade work: the server itself has not changed shape since v0.1.0-preview.1.

ArchitectureCopy Anchor Link

The crate is loams-durable. It builds a Resonate server from Resonate's own plugins and nothing else:

• Gateway. resonate-gateway-http serves the protocol routes and the poll transport's SSE route (/poll/:group/:id) on one listener, port 8001 by default, Resonate's own default port, so the Resonate examples work unchanged. Move it with --durable-listen, turn the server off with --no-durable.
• Server. ResonateServer owns promises, tasks and schedules, over a storage plugin: resonate-server-sqlite for SQLite, or Loams' resonate-server-tikv for TiKV.
• Workers (transports). http-poll for SDK workers that hold an SSE connection; worker_inproc, a Loams plugin for the scheme inproc://any@loams/<node>, for Loams' own functions; http-push (Resonate calls a serverless worker) is linked but off unless --durable-push is given.
• In-process runtime. DurableRuntime runs Loams' own durable functions on the Resonate Rust SDK. Its network, InProcNetwork, hands each request straight to ResonateServer::process and receives tasks from worker_inproc. No socket and no HTTP are involved (the test no_http_is_used checks it).

The embed is careful not to touch its host. Loams calls Resonate's public composition API (resonate_base::build, then Running::start and Running::stop), never resonate_base::run, which would install a global tracing subscriber and wait for signals itself. Configuration comes only from Loams' flags (--durable-set key=value reaches any plugin setting), never from resonate.toml or RESONATE_* variables. Resonate's abort_on_panic is pinned off, so a handler panic answers 500 instead of aborting the whole loams process. Each of these has a test in crates/loams-durable/tests/embed.rs.

How it worksCopy Anchor Link

Promises, tasks and originsCopy Anchor Link

The Resonate protocol has three entities (design 14 §1, from Resonate's Lean specification):

• A promise has an id, a parameter, a value, tags and a timeout. It is pending until it is settled as resolved, rejected, rejected_canceled or rejected_timedout. Promises can register callbacks (an awaiter waiting on an awaited promise) and listeners.
• A task is the unit of work a worker claims. Its id is its promise's id, and its version is a fencing token. A task moves pending → acquired → suspended, halted or fulfilled. Workers heartbeat while they hold it.
• A schedule creates promises on a cron from a template.

Everything before the first : of an id is its origin. Every operation except search is single-origin: an awaiter and the promise it awaits share an origin, so one atomic write of one origin's state commits any transition. That is why the storage engines can be simple: SQLite commits a transaction, and the blob and TiKV engines commit one document per origin with a compare-and-set.

A crash, and the replayCopy Anchor Link

A durable function is ordinary async code. Each ctx.run(step, args) creates a child promise with a deterministic id and runs the step only if that promise is not already settled. The test crash_between_steps_resumes_without_rerunning_step_1 in tests/inproc.rs shows the whole mechanism:

1. Runtime A acquires the task for root promise crash-1 and runs crash_flow(4). Step 1 returns 5; its promise is resolved with 5.
2. Step 2 is in flight when runtime A is stopped under it, as a crash would do.
3. The task is not fulfilled, so the server redelivers it after the task retry timeout (30 seconds by default, DEFAULT_RETRY_TIMEOUT). Runtime B acquires it with a higher version, so anything A might still try is fenced off.
4. B replays crash_flow from the top. Step 1's promise is already resolved, so ctx.run returns 5 without calling the step. Step 2 runs and returns 50.
5. The root resolves with 50. The test asserts that step 1 ran once and step 2 finished once.

The other tests in that file cover the rest of the everyday patterns: failed_branch_alone_retries (a fan-out of four branches where one fails once: only that branch runs twice), sleep_survives_restart (a durable sleep across a runtime restart), and two_step_function_runs.

Replay has the usual rule of every replay-based engine: the code between steps must be deterministic, and side effects belong inside steps. A step that writes somewhere else should use its own promise id as the idempotency key of that write, so a step that crashed after writing but before settling does not write twice (design 21 §6.7). A workflow step and a Loams data write are never one atomic transaction; that is a stated non-goal.

StorageCopy Anchor Link

loams cluster refuses SQLite, because it is single-node; a cluster node without --durable-store serves no durable execution. On TiKV, any node's instance can deliver a timed-out task to its own in-process worker, so any node can resume any of Loams' own operations. The CI test promise_state_survives_two_servers_and_restart runs two servers on one TiKV keyspace and a restart.

For the TiKV layer underneath, see Metadata and TiKV.

Failure handlingCopy Anchor Link

Key decisionsCopy Anchor Link

The full log is on the decisions page; the design is design 21, which amends design 14.

• D19: durable execution through the Resonate protocol. Agents need durable runs next to their memory. Resonate is Apache-2.0, formally specified, plugin-based and already able to run on object storage, "so this is buy, not build". A Loams-specific durable SDK is a non-goal: users use Resonate's.
• D138: embed the server in the binary. The owner's direction was to link Resonate's plugins into loams rather than run a sidecar or a separate role. The linking spike ran it beside Loams' axum 0.8 app with public API only, passed Resonate's Python fan-out example and survived kill -9 of the whole binary in the human-in-the-loop example. D138 also makes the listener refuse non-loopback addresses until authentication exists, the same stricter rule Loams Live uses.
• D262: the durable feature is opt-in. Toggling it rebuilds about 480 crates, so it is off in default builds and on in release images and CI's durable jobs. A build without it has no Resonate code, and a --durable-* flag only logs a warning.
• D139, amended by D261: SQLite in dev, TiKV for clusters. D139 had chosen TiDB through Resonate's MySQL plugin for the cloud. D260 removed TiDB from the engine and D261 replaced it with a native TiKV backend, because Loams runs TiKV anyway. The mysql:// backend is legacy.
• D140: the dependencies come from a pinned fork. Resonate's server crates are not on crates.io. The fork ostrium-labs/resonate is upstream plus a dependency-hygiene commit and two fixes for TiDB, each also proposed upstream (Resonate PRs #1162, #1163 and #1164).
• D141: http-poll for SDK workers, push off, the Rust SDK in process for Loams' own code. Using the SDK keeps replay semantics "bought, not rebuilt". Push stays off until the auth plan and a per-organisation outbound allowlist exist.
• D142: one Resonate instance per namespace. Resonate has no tenant concept: groups, schedules, searches and timers are global to a server, so prefixing ids would share them. Isolation needs a whole instance per namespace. 0.1.0 serves only the default namespace.
• D144: conformance by black-box linearizability. Loams does not re-run Resonate's engine differentials (the fork's CI does); it runs Resonate's own conctrace and porcupine checker against the loams binary, because embedding changes configuration, the listener and the in-process path, not the engine.
• D143 and D145: patterns become features, on their own track. Async operations, fan-out with partial retry, schedules and idempotency keys come first, with bulk import from object storage as the first real operation. Loams' own short maintenance loops (compaction, merges, GC) deliberately stay lease-fenced worker tasks: a cron would add latency and a dependency and give them nothing.

What is compatibleCopy Anchor Link

• Protocol. Resonate server 0.10.1, protocol version 2026-04-01, from the fork at a pinned revision (31ffb201… in the workspace Cargo.toml). SDKs must match the server's protocol: design 21 records that PyPI resonate-sdk 0.7.x is refused by server 0.10.1 (400 Promise ID must be prefixed by resonate:origin), while the monorepo Python SDK 0.8.1 works.
• Tested in CI. Resonate's Rust SDK in process (crates/loams-durable/tests/inproc.rs); Resonate's linearizability checker against loams dev on SQLite at 8 clients × 600 operations on every relevant pull request (scripts/durable/conformance.sh); the TiKV backend's integration test in the TiKV job; the legacy TiDB backend nightly against TiDB v8.5.8 at 8 × 600 and 16 × 400 with three seeds.
• Tested once, not in CI. The linking spike of 2026-09-27 ran Resonate's Python fan-out/fan-in example (normal and --crash) and the human-in-the-loop example with kill -9 of the binary, with Python SDK 0.8.1. The nightly SDK example suite that D144 describes (Python and TypeScript examples) is not in CI yet, so no TypeScript, Go or Java SDK version is verified against Loams today. The Jobs plan (JB1) pins Resonate Python SDK 0.8.1 and TypeScript SDK 0.11.5 for its work; those are planned, not tested.

Compared with the alternativesCopy Anchor Link

Durable execution is a crowded field. All of these record progress so that work resumes after a failure; they differ in where state lives, how code is replayed, and how much surrounds the core.

Where the others are better today: Temporal is far more mature, has years of production use, rich visibility tooling and SDKs for many languages; if you need that today, use it. Restate gives you a dedicated, single-binary Rust server that is built for exactly this job. Inngest is the smoother path if your code is serverless functions triggered by events. DBOS needs nothing beyond Postgres if Postgres is all you run.

Where Loams Durable fits: it is the Resonate protocol, so you get Resonate's SDKs and its formally specified semantics, and it runs inside the system that already holds your agents' memory, collections and logs, with no extra service to operate. It is also the engine Loams itself is moving its long operations onto. If you want Resonate on its own, run Resonate: it is the same protocol, and everything the SDKs do works the same way against either.

What is missing and what is plannedCopy Anchor Link

• Authentication and tenancy. The durable API is unauthenticated, loopback only and single-namespace in 0.1.0. Per-namespace instances and the dispatcher that maps a credential to its namespace are milestone D2, after the unified auth plan (D142). In the compose stack, port 8001 stays inside the loams container and is not forwarded over your tailnet.
• TiKV in the images. The backend is built and tested; shipping it is the "Durable store on TiKV" row, In progress.
• The operations API, bulk import and scheduled import. D1 Tasks 7 to 9: POST …/import answering 202 with Location: /v1/operations/{id}, idempotency keys, per-file fan-out, and a cron schedule that imports only new files.
• Retention. The protocol has no delete for settled promises and no backend prunes them, so durable state grows. Retention is open (Q40); pruning is not free either, because a late retry would re-create a pruned id and run the work again.
• Later milestones. Approval gates and tenant provisioning sagas (D2); durable Live actions, the agent runtime and MCP operation tools (D3); a connect-rust transport (D4). Loams Jobs plans schedules and flows on this server (D210), and the desktop agent daemon plans durable turns on loams-durable with no HTTP listener (D793).

Try itCopy Anchor Link

Durable is in the 0.1.0 images, so the compose stack already runs it inside the loams service. Because the listener is loopback only, the simplest way to point an SDK at it is a local build. From the engine quick start and the build guides:

# `durable` turns on the embedded Resonate server (port 8001, loopback only).
# It rebuilds roughly 480 crates, so expect a long first build.
cargo build --release -p loams --locked --features durable
./target/release/loams dev

loams dev serves the Resonate protocol on the loopback interface, port 8001; point a Resonate SDK at that address. Flags: --durable-listen, --durable-store, --durable-push, --durable-set key=value, --no-durable. For the legacy MySQL-protocol store, loams durable migrate runs the schema migrations as an explicit step.

To run the same checks CI runs:

cargo test -p loams-durable --locked
cargo test -p loams --features durable --test durable --bin loams
scripts/durable/conformance.sh --self-test
scripts/durable/conformance.sh --store sqlite --clients 8 --ops 600

The conformance script needs Go 1.24 and Python 3 on PATH, and fetches the pinned fork to build Resonate's conctrace.

SourceCopy Anchor Link

• Crate: crates/loams-durable; scripts: scripts/durable
• Design: 14, Durable execution (Resonate protocol), 21, Loams Durable: embedded Resonate, 26, Jobs API §8 (how Jobs uses Resonate), 15, Agent workspaces
• Plans: D1, embedded durable execution, the D1 dependency spike, JB1, Jobs
• Upstream: resonatehq/resonate and the fork ostrium-labs/resonate. Thanks to the Resonate team: the protocol, the SDKs and the verification harness Loams relies on are theirs.