Architecture overview
The engine's five roles, its metastore and its bucket, and how the other Loams products sit beside it.
The Loams engine is one Rust binary, loams, that runs any combination of five roles over one object storage bucket and one metastore. Clients reach it through a handful of protocols; every protocol is translated into the same small set of internal operations. This page is the map; the pages after it follow a write and a read through it.
The pieces
Protocol surfaces. Native REST and Connect (gRPC-compatible) APIs, Arrow Flight SQL, the Qdrant REST and gRPC API, an Elasticsearch subset, and SQL over the Postgres and MySQL wire protocols. Each surface is enabled on its own listener. The rule is that compatibility is a gateway concern: no protocol leaks into storage or the query core, and each compatibility surface is scoped by the external test suites it passes (the official client libraries and framework integrations), not by feature parity.
The five roles.
| Role | Does | Holds | Scales with |
|---|---|---|---|
gateway | Protocols, authentication, admission and backpressure, routing | Nothing | Connections, request rate |
log | Buffers writes, writes WAL objects, gets offsets, serves recent fetches | Nothing durable (for the standard class) | Ingest bandwidth |
query | Plans and runs reads in DataFusion; owns the hot tier for its objects | Caches, derived structures | Query load, hot data size |
worker | Segmenting, link apply, merges, compaction, index and artifact builds, garbage collection | Nothing (leases in the metastore) | Ingest volume, backlog |
meta | The embedded Raft metadata state machine | Raft log and snapshots (to the bucket) | Metadata operation rate |
A laptop or a single VM runs all of them in one process (loams dev or loams standalone). A cluster runs them as separate pools, with any subset per node (loams cluster --roles …). See cluster mode.
The metastore. Offsets, leases, the catalog and manifest pointers, behind one semantic interface, MetaStore. The default is an embedded openraft group; in the self-hosted compose stack it is TiKV. See metadata and TiKV.
The bucket. Every durable byte: WAL objects, log segments, Lance datasets, Tantivy splits, delete bitmaps, manifests, primary-key indexes, published hot artifacts and metastore snapshots. See the bucket is the source of truth.
The five objects
The engine's data model has five object kinds in a namespace: streams (partitioned logs), collections (documents with text and vectors), links (declared materializations from a stream into another object), and, as designed, tables (Iceberg) and graphs (mapped over collections and tables, for GraphRAG expansion). Every write lands in a stream; everything else is derived. See the log is the spine.
Data flow in one paragraph each
Write. The gateway authenticates the request, checks the collection's backlog budget, and turns it into an append to the collection's implicit stream. A log node batches it into a WAL object, PUTs it to the bucket, and commits it to the metastore, which assigns offsets. The client gets those offsets back as a consistency token. Seconds later a worker applies the records to the collection and commits a new manifest. The write path.
Read. The gateway forwards the read to the collection's owner. The owner resolves the current manifest and its applied offsets, plans the query in DataFusion, serves each byte range from the fastest layer that has it, and merges the tail of unapplied records from the log. The read path.
Around the engine
The engine is the retrieval core. The self-hosted stack runs other Loams products beside it, each with its own storage choice:
| Product | What it is | Stores data in |
|---|---|---|
| Loams Live | Reactive document database | TiKV |
| Loams SQL | MySQL-compatible SQL on a patched TiDB | TiKV |
| Loams Postgres | A Neon fork | WAL on safekeepers, pages on the bucket |
| Loams House | ClickHouse-compatible analytics on chDB | Lake tables on the bucket |
| Loams Graph | GQL on Loams' Grafeo fork | The Loams log, snapshots on the bucket |
| Loams Durable | Durable promises (Resonate protocol) | SQLite |
Why this shape
Three principles explain almost every choice above:
- Object storage is the only durable source of truth for the engine, so nodes are replaceable.
- The log is the spine, so every derived form agrees with every other one and a token means the same everywhere.
- Every object has a durable tier and a hot tier, and correctness never depends on the hot tier.
What it costs
- Write latency is bounded below by an object PUT plus a metastore commit.
- Cold reads pay object storage round trips; the hot tier is how the common case stays fast.
- The metastore is the one stateful component of the engine, and it must be run well (or delegated to TiKV).
Status
Available The five roles, single-process and cluster modes, the openraft and TiKV metastores, streams, collections, links and every protocol surface listed above ship in Loams 0.1.0. Tables (Iceberg), mapped graphs with expansion, the Kafka wire protocol and OTLP ingest are planned.
Go deeper
- The write path and the read path.
- Loams Engine.
- Design: 00 Pitch, 01 Architecture, design index, decision log (D1, D4, D5, D42, D47).
- Release: Loams 0.1.0 feature status.
- Source:
crates/(start withcrates/README.md).