Consistency tokens
Read-your-writes without locks. What a token is, what a read does with it, and why it never waits for indexing.
Every write to the Loams engine is acknowledged with a consistency token: the set of (stream, partition, offset) positions the write was given in the log. Pass that token on a later read, on any node, and the read is guaranteed to reflect the write.
v1:s7/p3@918273 stream 7, partition 3, offset 918273The REST API returns it in the Loams-Consistency-Token response header and accepts it in the same request header; the hybrid query body takes it as "consistency": {"token": "v1:…"}.
Why we built it this way
The question an agent asks after writing a memory is "can the next step see it?". In a stack of separate search, vector and graph stores there is no answer: each store indexes on its own schedule, and a refresh in one says nothing about the others.
Loams can answer it because all of those views come from one ordered log. A token is just a position in that log, and every derived object records the position it has applied. So the guarantee is mechanical, not best-effort: a read with token t returns state that includes every record up to t.
We made two choices that shape how it feels to use:
- Tokens are a hard guarantee, not a required input (D76). Single-object reads are strong by default, so most clients never handle a token. You need one only when you read through a derived object, such as a collection fed by a link from an explicit stream, or when you want to pass "what I wrote" from one service to another.
- A token read never waits for indexing. It would be easy to implement tokens by blocking until the link catches up. That would make read latency depend on indexing lag, which is exactly what you cannot control under load. Instead the read merges the unindexed records straight from the log.
How it works
At write time. The metastore assigns dense offsets to each partition's records when the log node commits its WAL object. The acknowledgement carries those offsets as the token. Nothing else is needed: no session, no lock, no version vector in the client.
At read time. The node planning the read resolves a snapshot per object: the current manifest and the offsets it has applied. Then:
- If the token's offset is at or below the applied offset, the manifest already contains the write; the tail is not needed for it.
- Otherwise the read needs the records in
(applied, token]. The node makes sure its view of the log reaches the token (in a cluster, a metastore replica may need to catch up to the leader first, which is a metadata wait measured in milliseconds, not an indexing wait), then reads those records from WAL objects or segments. - The tail overlay rule merges them: a tail entry for key k at offset o overrides the indexed document if
ois at or past the manifest's applied offset for that partition. Deletes in the tail hide indexed documents. Scores are computed over the merged set.
Read levels.
| Level | What it reads | When to use it |
|---|---|---|
strong (default) | The manifest plus the tail up to the current high watermark | Almost always |
at_least a token | The manifest plus the tail up to the token | Read-your-writes across services or derived objects |
eventual | The manifest plus whatever live tail this node already holds; never builds a tail from the log | The lowest latency, when staleness is acceptable |
| pinned | A named manifest version plus a tail up to a token | Repeatable reads while that manifest is retained |
An eventual read reports how many records it did not see in its response's performance.stale_records.
Why it holds on every node. Offsets come from the metastore, order within a partition is total, and any node can read the log. So a token means the same on a node that owns the collection, on one that does not, during a namespace move and under stale routing, and on every metastore backend.
What we give up
- Per-object snapshots, not global transactions. A token makes a read reflect specific writes. It does not make two objects change atomically together, and the engine has no multi-object serializable transactions.
- The tail has to stay small. Merging the tail is cheap only while it fits in memory. Write backpressure bounds the unapplied backlog of each collection for this reason; a bulk load can override the budget, and then strong reads may fall back to building a tail from the log or answer
Unavailableuntil the link catches up. - External readers do not see the tail. A tool reading Lance files through a pinned scan sees the manifest, not the records after it.
How other systems do it
- Elasticsearch makes writes searchable at the next refresh (one second by default);
refresh=wait_forblocks the write until then. It is per index, and says nothing about other systems. - Qdrant acknowledges an upsert with an operation id and offers
wait=trueto block until it is applied. - Session consistency in databases such as Cosmos DB or MongoDB's causal sessions passes an opaque session token so a client reads its own writes. Loams tokens are the same idea, made portable: a plain position in a log that any service can pass to any other.
- Zookies in Google's Zanzibar carry a snapshot timestamp so an authorization check is at least as fresh as a change. Loams tokens play the same role for retrieval.
Status
Available Tokens on every write, the Loams-Consistency-Token header, the strong, at_least, eventual and pinned levels, tail merge and range tails ship in Loams 0.1.0, on a single node and in cluster mode. Tokens across sharded collections and tags taken at a token are designed for later milestones.
Go deeper
- The read path: where the tail merge happens in a query.
- The hot tier: H3, the in-memory tail.
- Loams Engine.
- Design: 01 Architecture (§5, consistency model), 05 Query engine (§5), decision log (D76, D77, D86).
- Source:
crates/loams-query(read levels and tail merge),crates/loams-collection.