Cluster mode

loams cluster runs the engine across several nodes. Each node runs a chosen subset of the five roles, registers itself with a lease, and takes ownership of part of the collections by rendezvous hashing. Nothing is copied between nodes when the cluster changes, because the data is in the bucket.

loams cluster --roles meta,log,query,worker,gateway --listen 10.0.0.11:8080 …
loams cluster --roles query --listen 10.0.0.21:8080 …

Why we built it this wayCopy Anchor Link

Clustering a stateful database is mostly about moving data safely: rebalancing shards, re-replicating after a failure, draining before a restart. Because the engine is stateless over the bucket, clustering reduces to three smaller questions:

1. Who is alive? Answered by leases in the metastore.
2. Who should serve this collection? Answered by a hash, the same on every node, with no coordinator.
3. What if two nodes disagree? Answered by making ownership a hint: any node can serve any read correctly, and every write commit is fenced.

Our opinion: placement should be computable, not negotiated. If every node can compute the owner of a key from the list of live nodes, there is no placement service to run, no rebalancing protocol to get wrong, and a node that joins or leaves moves only the keys it should.

How it worksCopy Anchor Link

Roles per node. --roles picks any subset of meta, log, query, worker and gateway (gateway implies log). A small cluster runs every role on three nodes; a large one splits gateways, query nodes and workers into separately scaled pools. With the TiKV metastore there is no meta role to run.

Metastore replicas. With the embedded openraft metastore, every node keeps a replica: voters on meta nodes, non-voting learners elsewhere. Local reads stay local; writes and linearizable reads go through the leader. See metadata and TiKV.

The node registry is a set of leases. Each node holds the lease node/<node_id> with a 10-second TTL, renewed every 3 seconds. Its owner string is the node's descriptor: an incarnation id, its advertised address, its roles and its zone. The live nodes are exactly the leases that exist. A node that stops renewing drops out of every other node's view within seconds.

Rendezvous ownership. For a collection, every live query node gets a score, xxh3_64(namespace ‖ collection ‖ node_id). The owners are the top r nodes by score, at most one per zone until every zone is used; r is --replication (default 1). When a node joins, it takes only the keys for which it now has the top score; when one leaves, only its keys move, each to its own next-highest node.

Read forwarding. A gateway forwards a collection read to the first owner over an internal route, with the request's consistency level and token. The read runs where it lands; it is forwarded at most once. If the owner answers Unavailable or Timeout, or cannot be reached, the gateway runs it locally. A node that failed a forward is treated as suspect for 5 seconds.

Writes need no owner. Any log node accepts any write; offsets come from the metastore. Write backpressure is checked from the collection's backlog, which every node can read.

Workers by lease. Background tasks (a link's apply loop, a merge, an HNSW build) each run under a task lease. If a worker dies, its leases expire and another worker takes the task; a stale worker that wakes up is refused by the lease epoch on its next commit.

The hot tier follows ownership. The owner loads the collection's hot structures. After an ownership change the new owner serves cold from the bucket while it warms, or downloads the published HNSW artifacts.

What we give upCopy Anchor Link

• One owner per collection by default. A very hot collection is served by one node unless you raise --replication. Raising it automatically for hot collections is planned.
• No bounded-load placement yet. Rendezvous does not look at load today; the design adds a rule that skips an overloaded top choice.
• Addresses are resolved once. --advertise is an ip:port, or a host name resolved at registration; stable host names that change address (as on Kubernetes) are a later step.
• Internal routes must stay on a private network. Node-to-node traffic (Raft, read forwarding) is meant for a private network, such as a tailnet, not the internet.

How other systems do itCopy Anchor Link

• Elasticsearch and Kafka assign shards and partitions through a master or controller that moves data when nodes change.
• Cassandra and DynamoDB use consistent hashing on a ring; data moves with the ring.
• Rendezvous (highest random weight) hashing is used for cache placement in systems such as CDNs and load balancers; Loams uses it the same way, because a Loams owner holds a cache, not data.
• turbopuffer and WarpStream keep compute nodes stateless over object storage, so membership changes move no data.

StatusCopy Anchor Link

Available loams cluster with roles, the node registry, rendezvous ownership with zones and --replication, read forwarding with local fallback, fenced task leases and write backpressure ship in Loams 0.1.0 (the release table's "Cluster mode" row). The self-hosted compose stack runs one engine node. Bounded-load placement, automatic replication of hot collections, distributed execution of large scans across nodes, a Helm chart and a Kubernetes operator are planned.

Go deeperCopy Anchor Link

• Stateless compute, the hot tier.
• The read path.
• Design: 01 Architecture (§3.1), 04 Hot tier (§5, routing and affinity), 18 Metastore backends and router (§5), decision log (D63, D75, D107, D108).
• Source: crates/loams (the cluster command, registry and forwarding), crates/loams-meta.