Metadata and TiKV

The bucket holds the engine's data; the metastore holds the small, fast-changing facts about it: which namespaces and collections exist, the offsets of every stream partition, which manifest is live for each collection, which node holds which lease, and which objects are retired and waiting for garbage collection.

Every crate reaches it through one interface, MetaStore. Two backends implement it today: an embedded openraft group, the default for loams dev and standalone, and TiKV, which the self-hosted compose stack uses.

Why we built it this wayCopy Anchor Link

Why not put metadata on the bucket too? Streams generate metadata at a high rate: an offset assignment for every flush, consumer offset commits, partition and task leases. A conditional PUT on object storage takes tens to hundreds of milliseconds and contends per key. It cannot sequence a log at that rate. So metadata needs a store built for small transactions (D10), while object data never flows through it: manifest bodies stay immutable objects on the bucket; only pointers live in the metastore.

Why a semantic interface rather than a key-value API? If every crate spoke raw keys and values, every backend would have to rebuild sequencing, fencing and compare-and-swap out of bytes, and every subtle bug would be repeated per backend. Instead MetaStore exposes the domain operations: commit a WAL object, swap a segment, trim a partition, compare-and-swap a manifest pointer, take and renew a lease, evolve a schema. Each backend implements them natively (D47). This is the Lakekeeper model: one catalog trait, several database backends. One conformance suite, including a linearizability checker, runs against every backend.

Why TiKV? It is a mature, open-source, Raft-replicated transactional key-value store that scales out by splitting ranges, and Loams already runs it for Loams Live and Loams SQL. Using it for engine metadata in clusters means one fewer stateful system to operate, not one more (D124). We do not run TiDB as a metastore; TiKV is used directly through tikv-client (D260).

Our opinion: keep the stateful part of the engine small, behind one interface, and pick a substrate you can operate. For a laptop that is an embedded Raft group with no dependencies. For a cluster that is TiKV.

How it worksCopy Anchor Link

The embedded openraft backendCopy Anchor Link

An openraft group with its Raft log in redb and its snapshots in the bucket, one set per node (the KRaft and ClickHouse Keeper style). In loams dev it is a single voter in-process. In a cluster, nodes with the meta role are voters (three or five), and every other node keeps a non-voting learner replica, so local reads stay local. Writes go to the leader; a linearizable read asks the leader for a read index and waits until the local replica has applied it. Raft traffic is postcard over HTTP on the node's listen address, on routes private to this backend.

The TiKV backendCopy Anchor Link

Selected with --meta tikv://<pd-hosts>/<keyspace>, in builds with the tikv feature (the release image has it).

• One TiKV transaction per call. Single-record writes are optimistic. The three log writes (commit_wal, swap_segment, trim_partition) are pessimistic and lock the partition heads they touch, in key order, so concurrent commits to one partition serialize instead of retrying.
• Partition groups. A WAL object's commit runs as one transaction per partition group, with a bounded number of chunks and bytes, so one huge flush cannot become one huge transaction.
• Unknown outcomes are resolved. Every write carries a commit token. If an attempt's outcome is unknown (a timeout after sending), the call is rerun under the trait's retry rules and either finds its token or commits once.
• The clock is TiKV's timestamp oracle. Reads use one snapshot at a fresh timestamp, so every read is linearizable. Time-based rules (stale commits, lease expiry, garbage collection grace) use the oracle's physical time with a bounded clock skew, rather than one monotonic clock.

The relaxed contractCopy Anchor Link

The openraft backend gives stronger guarantees than the trait promises, but callers rely only on a relaxed contract that every backend can meet (D59): WAL commits are atomic per partition group, not across all of an object's partitions; commands carry bounded-skew time stamps; garbage collection writes a claim on an object before deleting it, and any command that would make a claimed object reachable fails; composite reads follow a documented read order. Consistency tokens hold on every backend under this contract (D76).

TiKV's other roles in LoamsCopy Anchor Link

TiKV is the substrate for small transactions and metadata across Loams (D1057): Loams SQL's data (TiKV is TiDB's storage), Loams Live, the engine metastore, the metadata of the Postgres WAL service, and control planes. It is not used as a high-volume byte log: a TiKV append pays a timestamp round trip, a Raft write and an apply, which is right for records and wrong for Postgres WAL bytes.

What we give upCopy Anchor Link

• A stateful component. Whichever backend you choose, it must be replicated and backed up. Losing a majority of openraft voters, or TiKV's quorum, stalls writes and metadata-dependent reads until it recovers; cached reads continue.
• TiKV is more to run than an embedded group. PD and TiKV are separate services with their own memory needs. For one machine, openraft is simpler.
• Every node holds the catalog today. That is fine for thousands of namespaces, not for millions. A scoped change feed, nodes that cache only what they own, and a metastore sharded by namespace are designed and staged.

How other systems do itCopy Anchor Link

• Kafka moved its metadata from ZooKeeper into an internal Raft quorum (KRaft). The openraft backend is the same shape.
• WarpStream keeps its metadata in a separate hosted service with the data in your bucket.
• Lakekeeper (the Iceberg REST catalog) has one catalog trait with Postgres as its backend; Loams copied the trait-first pattern.
• Quickwit keeps split metadata in Postgres or in a file on object storage, at a much lower write rate than a log needs.

StatusCopy Anchor Link

Available The MetaStore trait, the embedded openraft backend (single node and networked, with learners) and the TiKV backend ship in Loams 0.1.0; the compose stack runs TiKV. The Postgres and DynamoDB backends, the remote metastore for BYOC with a hosted control plane, and the sharded metastore for millions of namespaces are planned.

Go deeperCopy Anchor Link

• Cluster mode: leases and node registration on the metastore.
• Stateless compute.
• Products: Loams Live, Loams SQL.
• Design: 18 Metastore backends, tenancy and the router (§2 to §4), 01 Architecture (§3.2), decision log (D10, D47, D59, D76, D107, D124, D260, D1057).
• Source: crates/loams-meta (openraft), crates/loams-meta-tikv, crates/loams-meta-conformance, crates/loams-tikv.