Stateless compute
Engine nodes hold only state they can rebuild, so they start, scale and fail cheaply.
A Loams engine node holds nothing that is the only copy of anything acknowledged. Its caches come from the bucket, its tail comes from the log, and the work it owns is held through leases that expire. You can kill any node at any time, with kill -9, and the cluster loses latency, not data.
Why we built it this way
Stateful systems make every operational task a data task. Adding a node to an Elasticsearch or Qdrant cluster means moving shards to it. Losing one means re-replicating what it held before you are safe again. Upgrading means a rolling restart that waits for replicas to catch up. Each of those takes time proportional to the data, and each is a window in which a second failure hurts.
When the bucket holds the data, those tasks become compute tasks:
- Scale out by starting a node. It registers, takes over part of the placement, and warms from the bucket. Nothing is copied to it first.
- Recover by starting a replacement. There is nothing to restore.
- Scale in, or to zero, by stopping nodes. A namespace nobody queries costs its bucket bytes and its metadata.
- Size compute for load, not for data. A node's disk is a cache, so you choose it for the working set, not for the total.
Our opinion: if a node can be lost, it will be, so design as if it already was. The engine is tested that way: a crash gate kills a child loams process at named failpoints on every step of the write, commit, segment, link, garbage collection and snapshot paths, and at random times under load, and checks that every acknowledged write is still there.
How it works
Five roles, one binary. loams runs any combination of five roles: gateway (protocols, authentication, routing), log (accept writes, write the WAL), query (plan and execute reads, own the hot tier), worker (segmenting, link apply, merges, builds, garbage collection) and meta (the metadata state machine). All but meta are stateless, and meta is not run at all when the metastore is TiKV. A laptop runs every role in one process (loams dev); a cluster splits them into pools that scale separately.
What each piece of state recovers from.
| On the node | Rebuilt from | How |
|---|---|---|
| H0 and H1 caches | The bucket | Range GETs on demand; published artifacts for HNSW |
| H3 tail | The log | Re-read the offsets after each manifest's applied offset |
| Collection ownership | The metastore | The node's node/<id> lease lapses after 10 s; rendezvous hashing picks the next owner |
| Task leases (a link, a merge) | The metastore | The lease expires; another worker takes it and is fenced from the old one by epoch |
| Unacknowledged writes | The client | The producer retries; idempotent producer ids drop duplicates |
Fencing is what makes takeover safe. A worker that was paused, not dead, could wake up and commit after its replacement. Every commit names the lease epoch it holds, and the metastore refuses a commit from a stale epoch. A manifest swap is also refused if the commit started too long ago, so a slow writer cannot reference objects that garbage collection has already decided to delete.
Ownership is a hint. Any query node can serve any read; the owner is simply the node most likely to have it warm. A read routed to the wrong node is slower, not wrong.
What we give up
- Cold starts. A new node has empty caches. The first queries it serves are cold, unless you prewarm with
loams warmor let it download published artifacts. Peer-to-peer cache transfer between nodes is a later optimization. - The metastore is stateful. Offsets, leases and pointers have to be durable and fast, so they live in a replicated metastore: the embedded Raft group (snapshotted to the bucket) or TiKV. Losing a majority of it stops writes until it is restored. See metadata and TiKV.
- Not every Loams product is stateless underneath. Loams Live and Loams SQL keep their data in TiKV, a stateful Raft-replicated store; their own server roles are stateless in front of it. Loams Postgres keeps recent WAL on safekeepers until it reaches the bucket. Those are deliberate choices for transactional products, covered in OLTP, OLAP and HTAP.
How other systems do it
- Elasticsearch, Qdrant, Kafka (classic) are stateful: data lives on node disks and moves when the cluster changes.
- WarpStream runs stateless agents in front of object storage with a separate metadata service. Loams' log role follows it.
- ClickHouse Cloud and StarRocks shared-data separate compute from storage for analytics; replicas can scale independently.
- Neon splits Postgres into stateless computes and a storage layer; computes start in under a second because they hold no data.
- Kubernetes operators for stateful databases automate the data moves, but cannot remove them.
Status
Available The five roles, loams dev, standalone and cluster, the node registry, rendezvous ownership, leased and fenced workers, warm and the crash gate ship in Loams 0.1.0. Bounded-load placement, automatic replication of hot collections and cache transfer between nodes are planned.
Go deeper
- Cluster mode: roles, leases and rendezvous in practice.
- The bucket is the source of truth.
- Architecture overview.
- Design: 01 Architecture (§3 and §7), 12 Roadmap, testing and risks, decision log (D1, D28, D29, D108).
- Source:
crates/loams(roles and the binary),crates/loams-worker,crates/loams-sim(the seeded simulation).