The log is the spine

In the Loams engine, every write lands in a stream before it lands anywhere else. A native write, a Qdrant upsert, an Elasticsearch _bulk request and a Flight DoPut all become records appended to a partitioned, ordered log. Collections (and, later, tables and graphs) are not written directly. They are materializations of the log, kept up to date by declared links.

The log is the one durability mechanism. Everything else is derived from it and can be rebuilt from it.

Why we built it this wayCopy Anchor Link

Most AI stacks have two or three copies of the same document: one in the search cluster, one in the vector database, one in the graph store. Each copy is written by a different client or connector, at a different time, with a different idea of order. When they disagree, your agent retrieves stale or contradictory context, and nothing can tell you which copy is right.

A single log fixes the order once. Every derived form reads the same records in the same order and records how far it has read. From that one rule we get three properties we would otherwise have to build three times:

• Exactly-once materialization. A link commits its output and the offset it reached in the same atomic step. After a crash it resumes from that offset, so nothing is applied twice or skipped.
• Read-your-writes everywhere. A write returns the offset it was given. Any later read can ask for "at least this offset" in any object derived from that stream. That is a consistency token.
• Freshness without fast indexing. Records past the last applied offset are the tail. Reads merge the tail, so a document is searchable as soon as it is acknowledged, even though the index that will hold it is built seconds later.

Our opinion: connectors and CDC jobs between your own stores are a design smell. If the engine owns the log, it can own the materialization too, and give you one answer to "is my write visible yet?".

How it worksCopy Anchor Link

Streams. A stream is a partitioned log of records (key, value, headers, timestamp). Each partition is totally ordered and every record gets a dense offset. Streams come in two kinds:

• Implicit streams back every collection. You never create them; a write to a collection is appended to its implicit stream first. The record's key is the document's primary key, so all versions of one document land in one partition, in order.
• Explicit streams are ones you create and use directly, through the native produce and fetch API and a streaming gRPC API with idempotent producers.

The write. A log node buffers batches from many partitions, writes them as one WAL object to the bucket with a create-only PUT, and asks the metastore to assign base offsets to each partition's chunk. Only then is the write acknowledged, with its offsets. The write path walks through it step by step.

Segments. A background segmenter rewrites WAL objects into per-partition segments of 64 to 256 MiB without re-encoding the record batches, and swaps the offset index entries in the metastore. Retention trims partitions metadata first; objects are deleted after a grace period.

Links. A link is a declared, continuously maintained materialization: stream → collection today, stream → table and graph adjacency in the design. A link-apply task, holding a fenced lease, reads a batch of records, resolves upserts and deletes through a primary-key index (latest write wins, in partition order), writes new Lance fragments and a Tantivy split, and commits a new manifest that records the offsets it applied. Records that cannot be decoded or violate the schema become dead letters, kept with the manifest, rather than blocking the link.

WAL classes. Each stream picks how its writes become durable:

The latencies are design targets taken from reference systems, not Loams measurements.

What we give upCopy Anchor Link

• A hop on every write. Writing to a log and then materializing costs more total work than writing straight into an index. We accept it because the log is what makes the derived forms consistent with each other.
• Lag between log and index. The link applies in batches (seconds, or 16 to 128 MiB). Reads hide this by merging the tail, but the tail is held in memory, so it has to be bounded. Each collection has a budget of unapplied records (1,000,000 records and 128 MiB by default); past it, writes are refused with HTTP 429 or gRPC RESOURCE_EXHAUSTED and a Retry-After, until the link catches up. This backpressure is a feature: it keeps strong reads cheap.
• Not a stream processor. Links run stateless transforms. Windowed joins and checkpointed state belong in a stream processor that reads Loams streams and writes back.
• Not Kafka, yet. Streams speak Loams' own produce and fetch APIs today. A Kafka wire-protocol gateway is designed for a later milestone, without Kafka transactions.

How other systems do itCopy Anchor Link

• Kafka made the log the integration point between systems, but leaves materialization to Kafka Connect and stream processors, each with its own idea of exactly-once.
• WarpStream writes a leaderless log straight to object storage and orders it in a metadata service at commit. Loams' standard class uses the same shape.
• Apache Fluss pairs a columnar log with primary-key tables and changelogs, tiered to lakehouse formats. Loams borrowed two of its ideas: Arrow-encoded segments for schema'd streams and changelog streams out of keyed objects (both designed, not built).
• Databases with a WAL (Postgres, for example) also have a log as the spine, but the log is private. In Loams the log is a first-class object you can read, and the derived forms are public, open-format files.

StatusCopy Anchor Link

Available Streams, implicit streams for collections, the standard WAL class, segmenting, retention, links into collections with exactly-once apply, dead letters and write backpressure ship in Loams 0.1.0, along with the native produce and fetch API and the streaming gRPC API. Links into Iceberg tables, the express and quorum classes, changelog streams and the Kafka wire protocol are planned.

Go deeperCopy Anchor Link

• Collections and manifests: what a link produces.
• Consistency tokens: how the offsets become a read guarantee.
• The write path: one write, step by step.
• Design: 02 Stream engine, 09 Links and workers, decision log (D4, D12, D86).
• Source: crates/loams-log, crates/loams-link, crates/loams-worker.