The write path
From an API call to a durable log entry, an acknowledged token and, seconds later, a new collection manifest.
This page follows one document upsert into a collection, from the request to the manifest that indexes it. The same path serves a native write, a Qdrant upsert, an Elasticsearch _bulk request and a Flight DoPut: every protocol becomes an append to the collection's implicit stream.
The shape to remember: the client is acknowledged once the record is durable in the log, and indexing happens afterwards.
The request: steps 1 to 5
1. The gateway translates. Whatever the protocol, the gateway authenticates the request and turns it into logical operations on the collection's implicit stream: an upsert, a delete or a patch per document. The record key is the document's canonical primary key, and the partition is a hash of it, so every version of one document lands in one partition, in order.
2. Admission and backpressure. The write is admitted only while the collection's unapplied backlog (records past its last applied offset, and their bytes) is under budget: 1,000,000 records and 128 MiB by default. Past it, the request is refused with HTTP 429 or gRPC RESOURCE_EXHAUSTED and a Retry-After. A bulk load can send Loams-Backpressure: off to be admitted up to four times the budget, at the cost of slower strong reads while it catches up. A batch in one request (a native write batch, a _bulk request, a Qdrant upsert) is atomic within the stream.
3. The WAL object. A log node buffers batches from many partitions and namespaces until its flush interval (250 ms for the standard class) or 8 MiB, then writes one WAL object to the bucket under wal/standard/<node>/<ulid>.wal with If-None-Match: *, so it can never overwrite anything. Each chunk inside holds one partition's record batches in Kafka's RecordBatch v2 layout: compact, CRC-checked and well specified. Any log node can accept writes for any partition; there is no partition leader to find.
4. The commit. The log node sends the metastore one CommitWal for the whole object: which partitions, how many records, the producer sequence numbers. The metastore's sequencer assigns each chunk dense base offsets, checks producer idempotence (the last five batch sequences per producer, as Kafka does), and appends offset index entries. One commit per flush, not per partition, keeps metastore load proportional to nodes times flush rate.
5. The acknowledgement. The client gets the offsets back as a consistency token, in the response and in the Loams-Consistency-Token header. From this moment the write is durable and visible to strong reads, which merge it from the log.
What happens if something fails
| Crash point | Outcome |
|---|---|
| Before the PUT completes | Nothing durable; the producer times out and retries |
| After the PUT, before the commit | An orphan WAL object, collected after a grace period; the producer retries without duplicates |
| After the commit, before the acknowledgement | The data is durable; a retry is deduplicated by producer id and sequence |
A retried commit of the same object returns the first commit's offsets or is rejected; it is never committed twice. Commits older than 15 minutes are rejected as stale, so the deduplication records the metastore keeps stay bounded.
The materialization: steps 6 to 8
6. The link reads a batch. One link-apply task per collection holds a fenced lease (task/link/<id>). It reads records past the manifest's applied offsets, in batches of seconds or tens of megabytes, decodes them, and resolves every key through the collection's primary-key index. For each key, the latest operation in partition order wins. Records that cannot be decoded or break the schema become dead letters, written beside the manifest, and do not block the link.
7. New objects. The task writes new rows and deletes into Lance as a detached version, a new Tantivy split, updated delete bitmaps, a record of the keys it changed, and manifest v+1, which records the applied offset of every partition. Every object is a create-only PUT under a new name.
8. The pointer swap. The task compare-and-swaps collection/<id> from v to v+1 in the metastore. The swap fails if another writer got there first, if the task's lease epoch is stale, or if the commit started too long ago. Then the primary-key index is updated, repairable from the per-commit key records if a crash interrupts it.
Because the applied offsets are part of the manifest, the swap is the exactly-once point. A crash before it repeats the batch; a crash after it resumes from the new offsets.
Background work
- Segmenting. A worker rewrites WAL chunks into per-partition segments of 64 to 256 MiB, keeping the batch bytes verbatim, and swaps the offset index entries. WAL objects with no remaining references are retired and deleted after a grace period.
- Maintenance. Split merges re-index live documents from
_sourceand drop deleted ones; Lance compaction rewrites fragments; vector index builds and HNSW artifact builds publish new manifests that rebase onto concurrent link commits. - Garbage collection. Objects no retained manifest or index entry references are deleted after the grace period.
Why it is shaped like this
- Acknowledge on durability, not on indexing. Index builds are batch work, and batch work should not sit in a client's request. The tail merge makes the write visible immediately anyway.
- One PUT for many partitions. Object stores charge per request and have per-request latency; batching across partitions and namespaces is what makes the bucket affordable as a log (D25).
- Order comes from the metastore, not from a leader. Leaderless log nodes mean any node in the client's zone can take any write, and none has to be elected (the WarpStream model).
What it costs
- Latency. On regional S3 the
standardclass targets 400 to 600 ms at p99. Theexpress(zonal buckets, 2 of 3) andquorum(Raft journal) classes are designed for lower latency and are not built yet. - Bounded backlog. If links fall behind, writers see 429s. That is the price of strong reads that never wait.
Status
Available Everything on this page ships in Loams 0.1.0, on the standard WAL class. Filter writes (delete or patch by filter) ship too; re-checking the filter at apply, conditional writes and changelog streams are designed for later milestones.
Go deeper
- The log is the spine, collections and manifests.
- The read path.
- Design: 02 Stream engine (§3 and §5), 06 Search and vector (§2), 03 Storage formats (§3.3), decision log (D25, D26, D27, D30, D86).
- Source:
crates/loams-log(WAL format and writer),crates/loams-link,crates/loams-collection.