Where the WAL lives: databases on object storage in 2026

Most new databases now say "built on object storage", and the phrase covers designs with little in common. A search engine that acknowledges a write only after S3 has it, a Postgres that acknowledges on three local disks and copies to S3 later, and a warehouse whose writes are a pointer swap in a catalog all fit under it. They differ in latency by three orders of magnitude, and in what breaks when a node dies.

One question separates them better than any other: where does a write go before the client is told "ok"? Answer that and most of the rest follows: write latency, how many stateful machines you operate, what a node failure costs, and what you pay per byte.

This post takes the systems that matter in 2026 through that question, then reads, consistency, cost, tenancy and formats. Every external claim links to its source, mostly vendor docs and engineering blogs, so latency figures are their numbers, measured their way. The last part is about Loams: what we copied, what we chose differently, and where another system is the better pick today. Our earlier posts on why a retrieval engine belongs on object storage and the bucket as the database argue our position. This one surveys the field.

What the bucket gives you in 2026Copy Anchor Link

Three changes to S3 made this generation of systems possible. Each one removed a piece of infrastructure that earlier designs had to run beside the bucket.

Strong consistency, including LIST. Since December 2020, "all S3 GET, PUT, and LIST operations ... are now strongly consistent", and "the results of a LIST will be an accurate reflection of what's in the bucket" (AWS). Before that, a system that listed a prefix to find its newest files could miss them, so table formats kept their own consistent metadata elsewhere.

Conditional writes. On 2024-08-20 S3 added If-None-Match, so a PUT "can check for the existence of an object before creating it" (AWS). In November 2024 it added If-Match on an ETag, which is compare-and-swap on a single object (AWS). Google Cloud Storage and Azure Blob Storage had offered preconditions for years. S3 was the holdout, and everyone designed around it. SlateDB's manifest RFC is a good record of the cost: because "S3 does not" provide CAS, it specified a two-phase write through a transactional store such as DynamoDB, chosen because of "our belief that S3 will eventually get CAS" (SlateDB RFC 0001). Delta Lake's multi-cluster S3 mode "uses DynamoDB to provide the mutual exclusion that S3 is lacking" (Delta Lake, 2022). LanceDB's docs still tell concurrent writers on S3 to use a DynamoDB commit store (LanceDB). New designs no longer need that second store.

A fast, single-zone tier. S3 Express One Zone is "a high-performance, single-Availability Zone storage class" with "consistent single-digit millisecond request latency" and support for appending to existing objects (AWS). On 2025-04-10 AWS cut its storage price by 31%, PUTs by 55% and GETs by 85% (AWS). The catch is in the name: one zone. Anything that must survive the loss of a zone has to write to several Express buckets.

The arithmetic later depends on prices. These are AWS list prices for US East (N. Virginia), from the AWS price list published 2026-10-15 and read on 2026-10-18 (S3 pricing):

For comparison, EBS gp3 is $0.08 per GB-month (AWS), and EC2 traffic between zones is billed at $0.01 per GB in each direction (AWS), which in practice means $0.02 for every GB that crosses. Cloudflare R2 charges $0.015 per GB-month and nothing for egress (Cloudflare).

The write path: four places to put the WALCopy Anchor Link

1. The WAL is a PUT to S3 StandardCopy Anchor Link

The node accepting writes keeps nothing. It buffers writes for a fraction of a second, writes one object, and acknowledges once S3 has it. Durability across zones is S3's job.

• turbopuffer writes each namespace's WAL under its own prefix: "Every write adds a new file to the WAL directory inside the namespace's prefix", and a successful write is "guaranteed to be durably written to object storage." Concurrent writes to a namespace are batched, one WAL entry per second per namespace, and the docs give p50 = 165 ms for a 500 kB write (turbopuffer architecture). Its guarantees page states that "Object storage is the only stateful dependency" (turbopuffer guarantees).
• WarpStream agents write one file "every 250ms, or once it has accumulated 4MiB of writes", and each file holds "data from all the topic-partitions" the agent received in that window. WarpStream reports "a P99 write latency of ~400ms" at about 140 MiB/s (WarpStream).
• S2 Standard streams run "against an S3 Standard chunk storage bucket", and "Appends are only acknowledged when associated records are durable in a chunk" (S2 storage classes). S2 states under 500 ms end-to-end p99 for intra-region clients on Standard (S2 pricing).
• SlateDB is an embedded LSM whose WAL and SSTs all live in object storage. It lets the caller choose: a write returns "after updating the in-memory WAL and MemTable", and await_durable() waits until it is in the bucket (SlateDB). The flush interval is the latency and PUT-cost dial. S2's open-source server, s2-lite, is built on it (Cargo.toml).

The price is latency, hundreds of milliseconds per write, because that is what an S3 Standard PUT takes. The win is a write tier with no state to replicate, rebalance or recover: kill a node and nothing acknowledged is lost.

2. The WAL is a quorum of zonal bucketsCopy Anchor Link

Write to several S3 Express One Zone buckets in different zones and acknowledge on a quorum. Nodes stay stateless, durability still spans zones, and latency drops by about an order of magnitude.

• WarpStream replicates "across a quorum of multiple S3 Express single-zone buckets"; writes must "succeed to at least 2 buckets" of the three in its example, which brings produce p99 "to less than 150ms" (WarpStream docs). Its April 2025 benchmark measured "a P99 Produce latency of 169ms and a median Produce latency of just 105ms" (WarpStream).
• S2 Express uses "a quorum of three S3 Express zonal chunk storage buckets" (S2); its docs say Express streams "ack writes within 40 milliseconds" (S2 appends), and its pricing page promises under 50 ms p99 (S2 pricing).

This is the cleverest idea in the field: a multi-zone log built from a single-zone product, with no stateful process anywhere. The price is Express's per-GB upload charge on every copy, roughly three times the Standard design in the cost section below.

3. The WAL is a quorum of stateful log nodesCopy Anchor Link

For a transactional database, hundreds of milliseconds or even tens per commit is too slow. These systems acknowledge on a consensus quorum of machines with local disks, and write to the bucket off the commit path.

• Neon, and Databricks Lakebase, which is built on it, stream WAL to safekeepers: "The transaction is committed once a quorum of safekeepers has acknowledged the record", by "a Paxos-based protocol". The pageserver then "persists those materialized versions into object storage asynchronously" (Databricks, 2026-08-27). Neon's docs describe object storage as "long-term, immutable history" that compute never touches directly (Neon).
• TiDB X, PingCAP's object-storage TiKV, uses object storage "as the single source of truth for all data", yet "The Raft log is first persisted to local disk" and its chunks "are uploaded to object storage in the background" (PingCAP).
• AutoMQ offers several WALs: EBS and Regional EBS at "Sub-millisecond", NFS at "Millisecond-level", and S3 at "Hundred-millisecond". In AutoMQ Open Source "only S3-compatible storage services are supported as WAL storage options" (AutoMQ), so the low-latency WALs are commercial.

"Source of truth" means something narrower here: the bucket holds everything older than a few seconds, and the newest commits live only on the quorum until uploaded. That is a reasonable trade, and it brings back a stateful, replicated tier with its own failover.

4. No WAL: the commit is a pointer swapCopy Anchor Link

Lakehouse formats and warehouses skip the log entirely. Writers produce immutable data files, and the commit is one atomic metadata change.

• Apache Iceberg: "All changes to table state create a new metadata file and replace the old metadata with an atomic swap", and writers commit optimistically "by swapping the table's metadata file pointer from the base version to the new version." The spec leaves the atomic operation to the catalog, for example "a pointer in a metastore or database that is updated with a check-and-put operation" (Iceberg spec). Version 3 adds binary deletion vectors and row lineage. AWS sells the managed version as S3 Tables, "the first cloud object store with built-in Apache Iceberg support" (AWS, 2024-12-03).
• Snowflake was this design before the word lakehouse existed. Its 2016 paper: storage "is provided through Amazon S3", table files are "immutable, a direct consequence of using S3 for storage", and writes "produce a newer version of the table by adding and removing whole files", tracked in "the global key-value store" (SIGMOD 2016).
• DuckLake moves "all metadata structures into a SQL database, both for catalog and table data", to cut the small metadata files Iceberg and Delta write per commit (DuckLake).

Commits cost seconds and contend per table, which is fine for batch and micro-batch writers and useless for an application's request path. Systems that want both put one of the first three designs in front, as a log, and commit to the table format behind it at a slower cadence.

The read path: everyone builds the same cacheCopy Anchor Link

Reads converge more than writes do. Every serious system puts the same two tiers in front of the bucket, node RAM and local NVMe, and routes requests so the same node keeps seeing the same data.

• Snowflake (2016): "Each worker node maintains a cache of table data on local disk", holding "file headers and individual columns", and the optimizer "assigns input file sets to worker nodes using consistent hashing over table file names" (SIGMOD 2016). Most of what follows is this idea again.
• Databricks' disk cache keeps fetched Parquet "on its local storage in a fast intermediate format" and invalidates itself when files change (Databricks).
• turbopuffer caches a namespace on NVMe with memory in front after the first query: warm p50 is 14 ms and cold p50 is 874 ms on 1M documents (turbopuffer). Its launch post puts the cold path at "a maximum of three roundtrips" and an object-store round trip for objects under 1 MB at "around 250ms" at p90 (turbopuffer blog).
• LanceDB publishes the clearest before-and-after: object-storage latency of 500–1000 ms for the open-source library against 50–200 ms for Enterprise, which "uses NVMe SSDs as a hybrid cache" (LanceDB).
• Quickwit writes a small hotcache with each split so a searcher can "open a split in less than 60ms" on object storage, and its searchers are "stateless nodes: any node can answer any query about any splits" (Quickwit).
• WarpStream spreads the read cache over agents with "a consistent hashing ring", in 4 MiB aligned chunks, once per zone, so each chunk is downloaded three times (WarpStream).
• S3 Vectors, AWS's own vector index in the bucket, quotes "under one second" for infrequent queries and "around 100ms or less" for frequent ones (AWS, 2025-12-02).

Two lessons. The cache is never the source of truth anywhere on this list, so losing it changes latency, not results, and nodes stay disposable. And cold latency is set by sequential round trips, not bandwidth: three GETs make a cold query of about a second, ten make several. File layout, footers and hotcaches matter more than cache size.

Consistency: the bucket stores, something else ordersCopy Anchor Link

S3 now gives read-after-write on single objects and CAS on single keys. That is enough to commit a manifest. It is not enough to sequence a high-rate log, because a conditional PUT on one key takes as long as any other PUT and only one writer can win each round. So nearly every system splits the job. Data goes to the bucket, and ordering goes to something small and fast:

turbopuffer is the interesting exception. Its guarantees page says "Object storage is the only stateful dependency" and concurrency control is delegated to object storage (turbopuffer). Its one-WAL-entry-per-second-per-namespace limit is what that choice costs: with one ordering point per namespace, the namespace is also the throughput unit.

On the read side, the useful question is what "strong" costs. turbopuffer's default is strong: its architecture page labels "~10ms for consistent read", the round trip to object storage that eventually consistent queries skip to get under 10 ms (turbopuffer). Eventual reads can be stale by about an hour in the worst case, and a namespace with more than 128 MiB of unindexed writes does not show new writes until they are indexed (turbopuffer). That is a tail problem every system here has. Fresh writes sit in a WAL that is not yet indexed, and a strong read must merge them in. The design question is how large the unindexed tail may grow before you push back on writers or let reads go stale.

Cost: a worked exampleCopy Anchor Link

Vendor cost comparisons pick their own workloads, so here is one with every assumption stated. The workload is a log of 1 TB per day of writes (30,720 GB in a 30-day month), kept for 7 days (7,168 GB at rest), in US East at the list prices above. Only storage, network and request charges are counted. Compute, headroom, discounts and support are left out, and they are not small: a Kafka cluster's brokers, and a stateless design's agents, cost real money in both cases.

Three replicas on gp3, three zones. Storage: 7,168 GB × 3 × $0.08 = $1,720. Replication: each GB is copied to two other zones at $0.02 per crossing, so 30,720 × 2 × $0.02 = $1,229. If producers pick a leader at random, two-thirds of writes also cross a zone to reach it: $410. That gives $1,639 of network and $3,359 in all. WarpStream's own figure for this line is "$0.05/GB in AWS at retail prices" (WarpStream), which matches.

WAL on S3 Standard. Three agents, one per zone, each flushing every 250 ms: 3 × 4 PUTs a second × 2,592,000 seconds = 31.1 million PUTs, or $156. Re-packing WAL objects into 8 MiB segments adds about 3.9 million PUTs ($20), and a read cache that fetches each 4 MiB chunk once per zone adds $9 of GETs. Storage is 7,168 × $0.023 = $165. No zone crossings are billed. That comes to $349.

WAL on three Express buckets. Each agent flushes every 50 ms to all three buckets: 180 PUTs a second, 466.6 million a month, $527. Uploading three copies of every GB is 30,720 × 3 × $0.0032 = $295. Reading one copy back to offload it is $18. An hour of WAL kept in Express is $14. Then come the same segments, GETs and Standard storage as above. That comes to $1,049.

Take away the ratios, which survive other assumptions. Moving the log onto S3 Standard removes the two biggest lines, replicated block storage and cross-zone copies, and costs about a tenth. Buying back an order of magnitude of write latency with Express costs about three times the Standard design and still less than a third of the replicated cluster. The Express line is dominated by requests and per-GB charges, so it is sensitive to the flush interval. Requests also explain why everyone batches across partitions and tenants: a design that writes one object per partition per flush multiplies the PUT line by the partition count.

Managed prices are another matter. Neon lists storage at $0.35 per GB-month (Neon), fifteen times S3 Standard, which pays for history, branching, operations and margin rather than bytes. turbopuffer's plans start at $16 and $256 monthly minimums (turbopuffer). The bucket sets the floor; what a vendor charges above it is a business decision.

Multi-tenancy: a cold tenant should cost its bytesCopy Anchor Link

Object storage changes the economics of tenants more than of anything else. When a tenant's data is a prefix in a bucket and no node holds it until it is queried, an idle tenant costs its storage and nothing else.

• turbopuffer: "Each namespace has its own prefix on object storage" (turbopuffer), and the design "especially shines when only subsets of the data (e.g. tenants) are active at once" (turbopuffer blog).
• Neon scales a database to zero "after 5 minutes" of inactivity and "reactivates automatically within a few hundred milliseconds" (Neon).
• TiDB X separates compute so that "different applications can use dedicated compute resources" over shared data (PingCAP).
• The bring-your-own-cloud pattern sits on top: WarpStream's agents run in your VPC and "customer data is never sent outside the customer VPC" (WarpStream), and LanceDB's BYOC "installs the system inside your VPC" (LanceDB). When the data plane is stateless, splitting it from the vendor's control plane is easy, and that is a real advantage of this whole family.

The hard part is noisy neighbours on shared caches and write batches: a WAL object that mixes tenants is cheap per request but complicates per-tenant keys and deletion. Few vendors document how they handle it, so ask.

Open formats, open codeCopy Anchor Link

Two patterns stand out. The best low-latency write paths in this survey are closed or sold: WarpStream's, AutoMQ's EBS WAL, TiDB X's. Bufstream went into a buyer's internal platform. And "open format" usually describes the analytical tail of a system, not its serving path. turbopuffer, WarpStream and S2 keep their own formats, which they are free to change, and which you cannot read without them. Kafka itself is moving: KIP-1150, diskless topics, passed its vote in March 2026, with the produce and consume designs still being worked out in follow-up KIPs (Aiven).

Where Loams fitsCopy Anchor Link

Loams is an Apache-2.0 engine for hybrid retrieval (vectors, BM25 full text, filters and graph expansion in one query) over open formats in your own bucket. The pitch calls it "the open-source turbopuffer", which says what we copied. Here is Loams on the axes above. Facts come from our design documents and the 0.1.0 release notes, and the latency figures are design targets, not measurements.

Write path: row 1, with rows 2 and 3 designed. The engine's write path is leaderless and modeled on WarpStream (design 02 §3). A log node buffers writes from every partition and namespace for up to 250 ms or 8 MiB, writes one WAL object with If-None-Match: *, then asks the metastore to assign offsets. The p99 target for this standard class is 400–600 ms. Design 02 also specifies an express class (three zonal buckets, ack on two, 20–50 ms) and a quorum class (a Raft journal on local NVMe, 3–10 ms). Only standard is built. If you need tens of milliseconds today, WarpStream and S2 ship that.

Ordering: a metastore, not the bucket. We chose the same split as WarpStream. Design 01 §3.2 says why: a conditional PUT takes "tens to hundreds of ms" and is "contended per key", which cannot keep up with per-flush offset assignment and consumer commits (design 01). Offsets, leases and manifest pointers live in a metastore behind one trait, with embedded openraft by default and TiKV available in 0.1.0. Manifest bodies stay immutable objects in the bucket.

Reads: the same cache as everyone, per object type. Every object has a durable tier in the bucket and a derived hot tier: metadata in RAM, a RAM-and-NVMe range cache, hot structures (HNSW graphs for vectors, pinned Tantivy splits for text), and an in-memory tail of committed but not yet indexed writes (design 04). Objects are placed on query nodes by rendezvous hashing, which is Snowflake's consistent hashing applied to collections. The targets are 5–20 ms p50 warm and 0.5–1 s cold, the same shape as turbopuffer's published numbers.

Consistency: strong by default, and across objects. A read merges the durable state at its applied offset with the tail up to the latest acknowledged write, and every write returns a consistency token of stream offsets. A read that carries the token sees that write in every object derived from the stream: the collection, and later its Iceberg table and graph (design 01 §5). That cross-object guarantee is the thing Loams does that the single-purpose systems in this survey do not try to do, because they have only one object type.

Formats: open at rest. Collections are Lance datasets plus Tantivy splits under one manifest, so pylance, Ray, Polars or PyTorch can read a pinned version without Loams, and that ships in 0.1.0. Iceberg tables through a REST catalog are designed and marked Planned.

Cost and tenancy: namespaces in one bucket. The namespace is the unit of tenancy, quotas, keys and cache affinity, and "A cold namespace costs only its S3 bytes" (design 01 §1). WAL objects are shared across namespaces to keep the PUT line proportional to nodes rather than tenants, which is why per-namespace keys need per-chunk envelope encryption (design 02 §3). Per-namespace authorization, tenant quotas and the namespace router are Planned in 0.1.0.

Where Loams does not fitCopy Anchor Link

Transactions. The retrieval engine is not an OLTP database (pitch §7). Loams' transactional products put their commits off the bucket, like Neon and TiDB X, and our data-access design says so directly: "Serious databases on object storage keep commits off S3." We measured the alternative. A pure bucket WAL for Postgres came in at 22 ms p50 and 148 ms p99 in a spike, an order of magnitude over safekeepers (D234, decision log). Loams Postgres, a fork of Neon, acknowledges on a quorum and writes to the bucket every 250 ms. Loams SQL, MySQL-compatible SQL on TiDB over TiKV, is not bottomless: its data lives on TiKV's NVMe with three Raft replicas, and the bucket holds backups (design 47). Decision D130 draws the boundary: object storage as the only source of truth holds for the retrieval engine, not for the transactional products. If you want MySQL whose storage is S3 today, that is TiDB X's design, and TiDB X is not open source yet. Our own S3 tier for Loams SQL waits on a spike, and no storage-engine code is written before it.

Analytics at warehouse scale. Snowflake and Databricks have a decade of optimizer, caching and governance work. Loams' Iceberg tables are planned, and our analytics service, the House, is experimental. Use a warehouse for warehouse work. Our part is to make the data readable by one.

Maturity. turbopuffer runs production search for customers such as Cursor (turbopuffer blog). Quickwit, SlateDB and Iceberg have years of adopters. Loams is at 0.1.0, and 0.x formats and APIs may still change between minor versions.

What others do betterCopy Anchor Link

• turbopuffer shows how far one stateful dependency can go. Delegating concurrency to the bucket removes a whole metastore from operations. We chose a metastore for streams' commit rates, and we pay for it in moving parts.
• WarpStream and S2 found the zonal-quorum trick and ship it. Our express class is the same idea, still on paper.
• SlateDB's await_durable puts the latency and durability trade in the API, per write, where the caller can see it. That is cleaner than a per-stream class for some uses.
• Quickwit's hotcache footer is the best example in the field of designing a file layout around cold round trips. Loams' text tier uses Tantivy splits in the same way.
• DuckLake is right that many small metadata files are a tax. Keeping hot metadata in a database and only bodies in the bucket is the same conclusion we reached.

SummaryCopy Anchor Link

Read "built on object storage" as a claim about where history lives, and ask three questions: where the write goes before the ack, who assigns order, and what a strong read must merge. In 2026 the write answers fall into four groups: hundreds of milliseconds with stateless nodes, tens of milliseconds with zonal quorums, milliseconds with stateful log nodes, or seconds with a catalog swap. Reads converge on RAM, NVMe and affinity routing. Costs come down to avoiding replicated block storage, cross-zone copies and per-record requests.

Loams sits in the first group for its retrieval engine, plans the second, and uses the third for its transactional products. It is fully open source and stores open formats, and it gives a consistency guarantee across vectors, text and graphs that the single-purpose systems do not need. It is also younger than all of them. Pick the system whose write path matches your latency budget, and check whether you can read your own data without the vendor.