On this page
Every data system you run carries a small transactional database inside it. Kafka has KRaft. ClickHouse has Keeper. Elasticsearch has its cluster state. Qdrant runs Raft for its collection metadata. Your workflow engine keeps its state in Postgres or Cassandra, and the job queue in Redis or Postgres. Each one is a consensus protocol, a storage engine and an on-call rotation, built for one product and shared with nothing.
Our position is that this layer should exist once. Small, keyed, transactional state (metadata, leases, sequencers, documents, queue rows, workflow state, even SQL rows) belongs in one distributed, transactional key-value store with tenancy built in. In Loams that store is TiKV. This post explains why a plain KV is enough for all of that, why TiKV in particular, and the four places where a KV is the wrong tool and we use something else.
The title says "all you need". The honest version is "all you need for the small, hot, transactional part". The bytes go elsewhere, and the second half of this post says where.
What runs on the KV
Loams runs PD and TiKV unmodified, from official releases. Six components put their state in it, each in its own API v2 keyspace (decision D260):
- The metastore (
loams-meta-tikv, D124): the engine's catalog, stream offsets, leases, manifest pointers and link state, for clusters. Available in 0.1.0 behind--meta tikv://…. - Loams Live (D130): a reactive document database. Each mutation is one optimistic TiKV transaction, retried on conflict. Available in 0.1.0.
- Loams SQL: a MySQL-compatible database. Each branch is a keyspace, and its compute is a stateless TiDB server pool over that keyspace (D722, D723). Available in 0.1.0, on a TiDB image with one patch.
- Durable state (D261): Resonate's durable promises and tasks on a TiKV store, for clusters. In progress; 0.1.0 keeps durable state in SQLite.
- Jobs (D214): queue rows with optimistic enqueue and pessimistic leases. Designed.
- The control plane (D125): orgs, projects, keys and the namespace directory, as tables of a system Live app. Designed.
Two more are designed to use TiKV for metadata rather than data: loams-wal, the Postgres WAL service, keeps its votes, leases and timeline records there (D268), and Loams Git can keep the refs of hot repositories in it (D1025).
That is one cluster to run, back up, monitor and upgrade, where the usual stack has six stores for the same set of jobs.
Why a KV, and not a database
The obvious alternative is to put all of this in Postgres. Plenty of good products do. We keep Postgres and DynamoDB as metastore backends for teams that already run them (D58). But as the default spine, a SQL database gives us things we do not need and withholds the one thing we do.
What we do not need is SQL itself. Every one of the components above has a handful of access paths, known in advance: "get the partition head for update", "scan the documents in this index range", "lease the next ten jobs". A query planner adds nothing to those, and it adds a second representation (rows and plans) between our code and the bytes. With an order-preserving key codec, an index is a key prefix and an index query is a range scan. Live's indexes, the metastore's catalog and the job queue are all built this way.
What we do need is horizontal scale with transactions across keys that live on different machines. A single Postgres primary is a ceiling. Sharding it gives up cross-shard atomicity, which is exactly what a metastore commit (an offset assignment that touches several partitions) or a Live mutation (read some documents, write some documents) needs.
There is a subtlety in "a KV is all you need" that we learned the hard way, and wrote down as D47. The metastore's interface to the rest of Loams is not a KV interface. It is a semantic trait, MetaStore, with domain operations: commit WAL chunks, swap a segment, compare-and-set a manifest pointer, take a lease. A raw get/put trait would make every backend rebuild sequencers, fencing and CAS loops out of bytes, and get them subtly wrong in different ways. So the KV is the substrate, and each component owns a thin, tested layer of meaning above it. One conformance suite with a linearizability checker runs against every backend.
Why TiKV
We needed a store that is transactional across keys and ranges with a stated isolation level, multi-tenant at the storage layer, scale-out, self-hostable under a permissive licence, and proven on data people care about. TiKV meets each of those:
- Transactions. TiKV implements Google's Percolator design on top of multi-Raft replication: snapshot isolation across regions, with optimistic and pessimistic modes. Pessimistic transactions take locks as they read, so a contended key (a busy partition head, for example) queues instead of aborting in a loop.
- One clock. PD's timestamp oracle hands out globally increasing timestamps. Live uses it to version query results; the metastore uses it for its clock stamps. There is no hot "clock row".
- Tenancy. API v2 prefixes every key with a mode byte and a 3-byte keyspace id, so one cluster can hold up to 224 isolated keyspaces, each created and retired through PD's API. This is the feature that made us choose TiKV. A spike confirmed that a key written by our Rust client was invisible from a TiDB running in a different keyspace on the same cluster.
- Licence and track record. TiKV, PD and the Rust client are Apache-2.0. TiKV is a CNCF graduated project and is the storage layer under TiDB's production deployments.
Here is what one transaction does. The order matters, and it is what makes recovery cheap:
The commit of a single key, the primary, decides the whole transaction. If the client dies after step 4, any reader that meets a leftover lock on a secondary key checks the primary and finishes the job. If it dies before step 4, the locks are rolled back. No coordinator has to survive.
What we considered instead. FoundationDB was on our first list of metadata backends (D10) and was dropped (D71): it needs the native libfdb_c at the cluster's exact API version on every host, there is no managed offering, and TiKV plus Postgres and DynamoDB already covered its roles. It is still a fine design. Snowflake and Apple built on it for the same reason we built on TiKV: a transactional KV under a system's metadata. etcd is the other common answer, and it is designed for small data: its documentation sets a default storage quota of 2 GB and suggests 8 GB as a maximum. That fits cluster configuration, not millions of documents.
TiDB is just another tenant
The strongest argument for a KV spine is what falls out of it. TiDB, the MySQL-compatible SQL layer from PingCAP, is stateless: it keeps all of its data in TiKV. Run it in keyspace mode, and a MySQL database is one keyspace plus a pool of processes that can scale to zero.
That is Loams SQL. A branch is a keyspace named after the branch id, created through PD before TiDB first starts (D723). TiKV's Raft log is the write-ahead log (D725). We considered putting a second log in front of TiKV and rejected it: it adds latency and no durability. A Neon-like MySQL, with branches and scale to zero, came out of the store we already ran, not out of a new storage engine.
It was not free. Classic TiDB on one PD elects one DDL owner across every keyspace, so DDL in all keyspaces but one waited forever. The fix exists upstream (pingcap/tidb#60403) but is not in the 8.5 release line, so we ship ghcr.io/ostrium-labs/tidb:v8.5.8-loams.1: upstream v8.5.8 plus that one cherry-pick. The forks post covers how we keep patches like this small.
What a KV cannot do
Now the other half. A transactional KV is a poor fit for four kinds of state, and pretending otherwise is how teams end up with a slow, expensive "everything database".
1. A high-volume ordered byte log
Postgres writes its WAL as a stream of bytes that is read once, by the storage layer, and then trimmed. We tried TiKV as that log. Every append pays a timestamp fetch, a Raft log write, a RocksDB apply and that apply's later compaction: write amplification on bytes nobody reads twice. Our laptop runs (three runs, replication factor 3; design 46 §9.2) show it:
| Workload | WAL on TiKV | Safekeeper reference |
|---|---|---|
commit-1 | 58–87 TPS | 109–117 TPS |
commit-16 | 474–712 TPS | 814–983 TPS |
bulk (one 1 GB transaction) | 8–18 MB/s | 45–228 MB/s |
These are laptop numbers, not benchmarks, and the bake-off that picks the launch store has not run yet. They are clear enough on one point: a byte log wants a byte-log service. So Loams Postgres writes to loams-wal, which group-commits to a quorum on local disk and offloads to the bucket, while TiKV holds only that service's votes and leases (D1057). The rule we wrote down is "TiKV for small transactions and metadata, loams-wal for the high-volume ordered byte log, the bucket for history".
2. Large, immutable data
Documents, vectors, inverted indexes, Iceberg tables and Postgres page layers are written once and read many times. TiKV keeps every byte on local disks, three times. Object storage keeps it at object-storage prices with the provider's durability, and the engine reads it through a RAM and NVMe cache. Even the job queue follows this rule: a job's payload or result over 16 KiB goes to the object store, and the TiKV row holds a pointer (D214). The next post makes the case for the bucket in full.
3. Analytics scans
A KV stores rows in key order. Scanning a billion of them to sum one column reads every byte of every row. Analytics wants columnar files with statistics, and those live in the bucket as Iceberg and Parquet. TiDB has a columnar answer, TiFlash, which runs as a Raft learner; we do not depend on it (D131). The HTAP post explains our plan.
4. Low-latency change feeds and hot keys
TiKV has a change-data-capture service, and it works from Rust. But ordered delivery waits on resolved timestamps, which advance about once a second. Live needs to invalidate a subscription faster than that, so each Live mutation writes its own commit-journal entry inside its transaction, and the change feed is kept for paths where a second is fine.
Hot keys are the other limit. TiKV splits a busy region by load, but it cannot split a single key. A partition head that every writer touches is a hot key by design. Pessimistic locks make it queue politely; sharding is the only real fix.
What it costs us
Choosing TiKV was a trade, and these are the costs we pay for it:
- It breaks our first rule. Loams' first decision (D1) is that object storage is the only source of truth. TiKV keeps its data on local disks with Raft. We amended D1 for Live explicitly (D130) and made continuous log backup of Live keyspaces to the bucket mandatory (D131). Whether point-in-time restore covers a non-TiDB keyspace is still to be verified. TiDB's next-generation kernel keeps its storage on object storage, but its shared-storage TiKV engine is not public, so we cannot use it.
- It is a stateful cluster to operate. PD and TiKV need care, memory and upgrades in their own order (PD, then TiKV, then TiDB). One PD, one TiKV and one TiDB peaked at about 3.2 GB of memory in our playground, mostly TiKV sizing its block cache from host RAM.
- Every commit pays round trips. A timestamp fetch, then prewrite and commit: a few milliseconds per transaction in our spike, on a loaded machine. Async commit and one-phase commit cut commit latency by 30 to 50% in that spike, and they are off until our linearizability checks pass with them.
- The Rust client is pre-1.0. Its README says it is not suitable for production. We found a TSO stream that died after a PD stall, async-commit locks a reader could not resolve, and private protobuf modules. We carry a fork with the fixes, each drafted as an upstream pull request (details).
- Nobody runs MVCC garbage collection for us. TiDB advances the GC safe point for its own data. For plain transactional keyspaces nobody does, so
loams-tikvruns the cluster's GC loop itself. Forget it, and old versions pile up forever.
What other systems get right, and miss
Most systems treat this layer as an implementation detail, and that choice is defensible for each of them on its own. Kafka's move from ZooKeeper to KRaft removed an external dependency. ClickHouse Keeper replaced ZooKeeper with something the ClickHouse team could own. Each is the right call for one product.
The problem shows up when you run five such products side by side. You get five metadata stores, five backup stories and five sets of consensus edge cases, none of which can join a transaction with another. Loams can, at least in principle, commit a Live document and enqueue a job against one TiKV, because they share a store and a clock. Where it ships today, it is one cluster to watch instead of five.
We are not purists about it either. loams dev and loams standalone run an embedded openraft metastore (D10), because a laptop should not need a TiKV cluster to try the engine. TiKV is for clusters.
Where we are
In 0.1.0 the TiKV metastore, Live on TiKV and Loams SQL are available: built, tested in CI and usable, with APIs and formats that can still change between 0.x releases. Durable on TiKV is in progress. Jobs and the control plane on TiKV are designed. The commit path still uses classic two-phase commit everywhere.
If you run TiKV or TiDB in production and think we have the trade wrong, we would like to hear it. The design is in design 20 and design 58 §8, and the decision log has the reasoning behind every number above.