The log is the spine

Ask a team running Postgres, Kafka, Elasticsearch and a vector database one question: "I just wrote this record. Will the next search see it?" The honest answer is "probably, in a few seconds, unless a connector is behind". Nobody can give a better one, because nothing in that stack knows. Each system has its own idea of position (a WAL position, a slot's confirmed position, a Kafka offset, a connector's committed offset, an index refresh) and none of them maps onto another.

Loams starts from the opposite end. Decision D4, from the first day of the decision log: the log is the spine. Every write lands in a stream first. Tables, collections and graphs are materializations of streams, maintained by declared links. The write's position in the log is returned to the writer as a consistency token, and any later read can ask for "at least this". This post is about why we think that is the right shape for a data platform, and about what it costs, because it is not free.

Every write is a log entryCopy Anchor Link

In Loams a collection is not a mutable index that accepts writes. It is a stream plus a materialized view of that stream.

Each table, collection and graph has an implicit stream (design 02 §8). A native API write, a Flight DoPut, an Elasticsearch _bulk and a Qdrant upsert all take the same path (design 01 §4.1):

1. The gateway translates the request into a logical write against the stream.
2. A log node appends it to the WAL, according to the stream's class, and gets dense offsets from the metastore. Only the metastore assigns offsets.
3. The client is acknowledged with a token: {(stream, partition, offset)…}.
4. Workers apply links asynchronously: they build Lance fragments and Tantivy splits, append Iceberg data files, or update graph adjacency. Each target commit records the offset it has applied.

Step 4 is the one that makes this more than a queue in front of a database. A link task leases its source partitions with an epoch, reads from the last applied offset recorded in the target's latest commit, builds the target files, and commits them together with the new applied offset (design 09 §3). For a collection that commit is a manifest-pointer compare-and-set; for an Iceberg table it is a snapshot whose summary carries the offset. A zombie task with a stale epoch fails its commit. A crash re-runs from the committed offset. Work is sometimes done twice; it is never applied twice.

That is exactly-once materialization without a transaction coordinator, because the progress marker and the data are one atomic write.

What a token buysCopy Anchor Link

The default single-object read in Loams is strong: it sees every write acknowledged before the read began. It gets there by merging the tail, the records committed to the log but not yet applied to the indexed form:

The read is "the durable or hot state at the applied offset, plus the tail from the applied offset up to the requested offset" (design 01 §4.2). The tail is held in memory on query nodes, written through at produce time. If a node does not have it, it reads that range from the log directly.

Tokens matter once a read goes through something derived: a table fed from an event stream, a graph mapped over a collection, or one collection fed from another by a link. Those objects are maintained asynchronously, so "strong" on the derived object alone means little. A token pins it down: a read that presents a token sees every write the token covers, on any node (D76, design 18 §3.5). That holds on every metastore backend Loams supports, during a namespace move, and when a request is routed with a stale view of the cluster, because:

• offsets come only from the metastore, and a write is acknowledged with its token only after every partition it touched has committed;
• each partition's offsets are dense and ordered on every backend, so "visible up to offset o" means one thing;
• any serving node reads the object's applied offsets, then merges the tail up to the token, re-reading linearizably or waiting if what it has is behind. It never answers without the token's offsets.

We made tokens a hard guarantee but not a required input. Clients may omit them. Most reads do not need them. The ones that do are the reads that matter most for agents: "I just stored this memory; the next step's retrieval must see it, through search, vectors and graph expansion." In a five-store stack that sentence has no answer. In Loams it is one parameter.

Versus CDC bolted onCopy Anchor Link

The usual way to keep a search index or a warehouse in step with an application database is change data capture: a tool such as Debezium reads the database's replication log and writes change events into Kafka, and Kafka Connect sink connectors write them into each target.

This works, and teams run it at large scale. We have three objections to it as the default architecture for AI data.

It is a pipeline of independent positions. Each hop has its own offset, its own retry policy and its own lag. Debezium's own FAQ explains that when things fail, consumers may see change events more than once, so every sink has to be idempotent on its own. Exactly-once across the whole path is possible with care, per connector. It is not the default, and the guarantee is never end to end, because the first and last systems share no notion of position.

It is a copy, not a materialization. Each target holds its own copy of the data in its own format, on its own disks, with its own schema drift. When they disagree (and they do, after a connector restart or a schema change) the target is wrong and nothing records it as such. Loams' design calls this out as a correctness bug for agents: they "retrieve stale or contradictory context" (design 00 §1).

It cannot answer the read-your-writes question. There is no token to hand back, because the writer and the reader never share a log.

To be fair: CDC exists because the application database came first and nobody wanted to change it. That is a real constraint, and Loams honours it too. The point is where the log sits. With CDC it is a side effect of one database, scraped afterwards. In Loams it is the front door.

The same idea, everywhere we could apply itCopy Anchor Link

Once you have a log with tokens, other parts of the system become simpler if they use it too. We have done this in four places:

• Loams Live's commit journal. Live is a reactive document database on TiKV. Every mutation writes one journal entry, in its own transaction, at head + 1 of one of the app's journal shards (D119). Live's subscription tailer reads the entries between two ticks, and because a shard's sequence is dense, "the entries visible at tick T and not at t" is exact: no update is missed, and no timing assumption is needed. We chose this over TiKV's change feed, whose ordered delivery waits on resolved timestamps that advance about once a second.
• The collections bridge. A Live table declared searchable gets a Loams collection. A bridge task tails the commit journal and appends document operations to the collection's implicit stream with an idempotent producer id of (app, shard) and the journal sequence as the producer sequence (D129). A crash between append and checkpoint cannot duplicate. A search with after_ts waits for the bridge and reads with its token.
• Postgres and Loams SQL into collections. The same shape again: logical replication (pgoutput) from Loams Postgres becomes document operations on a collection's implicit stream, with the source's commit position as the sequence, and the slot is confirmed only after the append (D154). For Loams SQL, loams-sqlcdc reads TiKV's change stream for the branch's keyspace (D738). This is CDC, and we use it where the source is a database we do not control the write path of. But it ends in a log with tokens, not in five independent sinks.
• Loams Graph. Each committed graph write transaction is one engine-neutral change set in the graph's implicit stream, and it is acknowledged only after the append (D742). Sessions read at the durable epoch, so no read sees a write that is not yet durable. The embedded engine's local files are a rebuildable cache. This is also what keeps the graph engine replaceable: the log does not know which engine applied it.

Changelog streams close the loop the other way. From M5, any keyed table or collection can expose its row-level changes as a stream (D21), written by the same link-apply worker that already knows the old row, so systems outside Loams can consume Loams' changes with the same exactly-once rules. That one is planned, not built.

What it costsCopy Anchor Link

A log in front of everything has real costs. We would rather list them than have you find them.

Write latency is the WAL's latency. A write is acknowledged when it is in the log, not when it is indexed. On the standard class that is a PUT to S3 plus a metastore commit, with a p99 target of 400–600 ms; the planned express and quorum classes (M5 and M6) target 20–50 ms and 3–10 ms (design 02 §2). The previous post covers the trade.

Indexing lags, so reads do work. Because links apply asynchronously, a strong read merges a tail. Tail merges cost CPU and memory on query nodes. When a collection's tail outgrows its memory bound, strong reads fall back to reading the range from the log directly: slower, still correct, and Unavailable beyond a bound (design 09 §7). Reads that can tolerate staleness can ask for eventual, which skips the tail.

Writers must sometimes wait. A log can absorb writes faster than links can apply them, and an unbounded backlog would make every strong read slower. So each collection has a budget on its unapplied backlog, by default 1,000,000 records and 128 MiB. At the budget, writes are refused with HTTP 429 or gRPC RESOURCE_EXHAUSTED and a Retry-After estimated from the link's recent apply rate (D86). Every write response reports the backlog. A bulk load can opt out up to four times the budget. This is the right behaviour, and it still surprises people who expect a database to accept every write.

Atomicity stops at the stream. A native write batch, an Elasticsearch _bulk or a Qdrant upsert is atomic within one stream. There are no multi-object serializable transactions and no interactive transactions in the retrieval engine (design 01 §5). A token gives you read-your-writes across objects; it does not give you an atomic write to two of them. If you need that, it belongs in a transactional store, which for Loams means Live, Loams Postgres or Loams SQL (the KV post explains how those fit).

Materializations take space and work. A log plus its targets is more bytes than a single mutable index, until segmenting and retention trim the log. We think that is the right price for exactly-once rebuilds, but it is a price.

The opinionCopy Anchor Link

Every data system has a log; most hide it. Postgres has a WAL, Elasticsearch a translog, Kafka is one. The mistake, in our view, is treating the log as an internal detail of each store and then rebuilding a second, worse log out of connectors to keep the stores in step.

Put the log first, give its positions to clients, and make every other object a declared, checked materialization of it, and three things fall out: one durability mechanism, exactly-once maintenance of every index, and a read-your-writes answer that spans search, vectors, tables and graphs.

Where we areCopy Anchor Link

In 0.1.0, streams and links ship, with exactly-once application to collections, the native produce and fetch API, the streaming gRPC API, consistency tokens and write backpressure, all tested in CI. Live's commit journal and the Graph's log-first writes ship too. The collections bridge and the Postgres bridge are designed; Iceberg tables, graph expansion and changelog streams are planned. The design is in design 01, design 02 and design 09.