Teardown: nine engines built on a bucket, and the one trick each gets right

A few years ago "built on S3" meant slow and cheap. Today a vector database serves 100 billion vectors from a bucket, Kafka-compatible logs ack in under 50 ms on one, and Postgres keeps its history in one. The bucket did not get much faster. A handful of teams learned to design around one slow, expensive, strongly consistent primitive, the object PUT, and published how.

This post takes nine of them apart: turbopuffer, WarpStream, S2, SlateDB, the Neon pageserver, Lance, Quickwit, AutoMQ and Bufstream, plus the primitive they lean on, conditional writes on S3, R2, Tigris, GCS and self-hosted stores. For each: the architecture, the one trick, and the published numbers with dates. It ends with what we are considering for Loams, which is in the same family.

Method. Public sources only: docs, blogs, papers, RFCs and code read on GitHub, nothing cloned, all read on 2026-10-18. Every number links to its source and date. Vendor numbers are vendor numbers: we have not reproduced them.

Licences. We read code to understand it and take ideas, never code, from anything proprietary, AGPL or BSL.

The constraints everyone designs aroundCopy Anchor Link

Four facts about object storage shape every system below.

1. A PUT is the latency floor. AWS's own performance guide puts small-object latency at roughly 100–200 ms.
2. Requests cost money, bytes mostly don't. Requests are billed per call, so a design that writes one file per partition every 250 ms costs about $50 a month per partition in PUTs alone (WarpStream, 2023-10-09).
3. Round trips, not CPU, set read latency. Four dependent GETs cost four first-byte latencies, whatever the CPU does.
4. Since 2024, S3 can compare-and-swap. If-None-Match: * on PUT arrived on 2024-08-20, If-Match on 2024-11-25, and conditional deletes on 2025-09-16. A bucket can now hold a commit point, and where each system puts its commit point is the most interesting design decision.

turbopuffer: a read path with a round-trip budgetCopy Anchor Link

turbopuffer is a proprietary vector and full-text search service with unusually detailed architecture docs.

Architecture. Each namespace is a prefix in object storage, and the bucket is the only dependency on the critical path; there is no separate metadata store. A write group-commits into one new WAL file per namespace, at most once a second, and acks only when that file is durable. Its diagram labels a "CAS commit point"; that the WAL head moves by conditional write is our inference. Reads are strongly consistent by default: a query checks the bucket for the latest WAL head, which costs about 10 ms. Eventual reads skip the check and may be up to about an hour stale (guarantees).

Indexing runs asynchronously, and the unindexed tail is searched exhaustively. That tail is bounded: past 128 MiB of unindexed writes, new writes stop being visible until indexing catches up, and unindexed data is capped at 2 GB (limits). Data moves from S3 to NVMe to RAM as it is queried, and any node can serve any namespace.

The vector index is centroid-based: a graph walk (HNSW, DiskANN) needs too many dependent round trips. ANN v3 (page last updated 2026-05-05) uses a cluster tree with a branching factor of about 100, holds 1-bit RaBitQ codes in RAM for 16–32× compression, and reranks fewer than 1% of candidates from full-precision vectors on NVMe. Native filtering keys attribute bitmaps by (value, cluster), so the planner only visits clusters that contain at least one match. And continuous recall reruns 1% of live queries exhaustively and tracks recall@10 as a service-level objective.

The trick. About three dependent object-store reads for a cold ANN query. Clusters, cluster-keyed filters and quantized codes with an NVMe rerank all follow from that budget.

Published numbers.

WarpStream: offsets are assigned after the uploadCopy Anchor Link

WarpStream is a proprietary, Kafka-compatible log whose agents are stateless Go processes without disks (architecture).

Architecture. There are no partition leaders: any agent can produce to or serve any partition. Each agent buffers everything it receives for up to 250 ms or 8 MiB, writes one file covering many partitions, then commits that file to a per-cluster metadata store, which assigns offsets. Only then does the producer get its ack. Order is decided at commit, not at upload.

That ordering point gives two tricks for free. Idempotence becomes a metadata problem: the metastore keeps the last five sequence numbers per producer and partition and leaves duplicates out of its index, while their bytes wait for compaction. And reads can be shared: a distributed mmap hashes each 4 MiB-aligned chunk of each file to one agent per availability zone and deduplicates concurrent requests, so each chunk leaves S3 once per zone however many consumers read it.

For lower latency it writes each file to two S3 Express buckets in different zones, then compacts into S3 Standard; a "Lightning" mode commits metadata asynchronously, giving up ordering and idempotence.

The trick. Writers never coordinate per partition, the PUT count depends on agents × flush rate rather than partitions, and the metastore turns immutable files into an ordered, deduplicated view.

Published numbers.

S2: a quorum made of bucketsCopy Anchor Link

S2 is a serverless stream store. Its service is proprietary; s2-lite and its cache, cachey, are MIT.

Architecture. One actor, the channel, sequences each stream's writes. It writes records into chunks, which are WAL objects, and acks once a chunk is durable. Chunks are later grouped into per-stream segments, so metadata stays off the write path. Reads go through cachey, a zonal cache with rendezvous hashing and bounded load. The API exposes match_seq_num, a compare-and-set that returns 412 with the current tail, and a fencing token set by a fence record in the stream itself.

S2 has two storage classes. Standard writes a chunk to one S3 Standard bucket. Express writes it to a quorum of three S3 Express buckets in different zones.

S2 also wrote up its deterministic simulation testing (2025-04-02): turmoil, libc overrides of getrandom and clock_gettime that catch nondeterminism inside dependencies, in-memory emulators of the bucket and metadata, and a CI meta-test that runs one seed twice and diffs the traces. It found 17 bugs. A linearizability post (2025-08-25) shows how to model an append that failed indefinitely in Porcupine: it may or may not have happened, and it ends after every other operation.

The trick. Durability without stateful disks. The ack waits on 2 of 3 zonal PUTs, and the single-writer actor gives a total order without a metastore round trip per append.

Published numbers.

SlateDB: the bucket as the only coordinatorCopy Anchor Link

SlateDB (Apache-2.0) is an embedded LSM tree whose only disk is object storage. Loams uses it for primary-key indexes.

Architecture. Writes go into a WAL buffer and a memtable. Each flush interval the buffer is uploaded as one WAL SST, so concurrent writers share a PUT. The manifest is a chain of numbered files, and each update writes the next number with create-if-absent.

Fencing happens through the same mechanism (RFC-0001). A new writer bumps writer_epoch in the manifest and then writes an empty, epoch-stamped WAL SST at the next WAL id. A zombie writer learns it has been fenced when its next WAL PUT loses that race. Compactors are fenced the same way. The RFC chose a two-phase write through DynamoDB while "gambling" that S3 would soon support CAS. It did. Distributed compaction works the same way, with stateless workers claiming numbered job files, and another RFC tags every object-store call with its purpose so a cache can treat a bloom-filter read differently from a scan.

The trick. One primitive does everything. Create-if-absent on consecutive ids carries the manifest, the fencing, the compaction claims and the checkpoints.

Published numbers.

Neon: generations instead of atomic operationsCopy Anchor Link

Neon's pageserver (Apache-2.0) stores Postgres pages keyed by (relation, block) and LSN. Loams maintains a fork.

Architecture. Data lives in immutable layer files (storage docs). An image layer holds a key range at one LSN, and a delta layer holds a key range over an LSN range. A read finds the newest image at or below the LSN, applies the deltas above it, and replays WAL. An index_part.json per timeline is the commit point for its layers.

The part worth stealing is generation numbers. Each attach of a tenant to a pageserver gets a new generation from the control plane, and every key the pageserver writes carries the generation as a suffix, so two attachments can never write the same key. Deletes go in strict order: unlink in the index, validate the generation with the control plane, then delete. If validation fails the object leaks and a scrubber finds it later; a leak beats a lost layer. Attached pageservers also upload heatmaps of hot layers, and secondaries download in heat order before a failover.

The trick. Split-brain safety without any atomic operation on S3. Every write goes to a key namespaced by an externally issued generation, and nothing destructive happens until that generation is checked.

Published numbers. Neon publishes its defaults rather than benchmarks, and we found no public figure for GetPage or cold-start latency, so we quote none.

Lance: one format for scans and point lookupsCopy Anchor Link

Lance (Apache-2.0) is a columnar file and table format for AI data, and the durable format of Loams collections.

Architecture. The file format has no row groups (paper, 2025-04). Each page holds one column, and the encoding depends on value size: small values go into "mini-blocks" read with one I/O, mid-sized values are interleaved with a control word for one or two I/Os, and large blobs are stored out of line. Table commits are a create-if-absent write of the next manifest. On conflict, a commit retries up to 20 times, rebasing on the winner according to a conflict matrix that sorts each conflict into rebasable, retryable or incompatible (commit.rs). Manifest names are u64::MAX − version, so one LIST returns the newest first, and each index records which fragments it covers so the rest are flat-scanned.

The trick. A commit is one create-if-absent of the next manifest, with no lock service. Random access costs one or two I/Os without row groups, so the same file serves scans and point lookups.

Published numbers. From the paper: default Parquet random access managed about 5.5 k rows/s against about 350 k rows/s when tuned (about 64×), on NVMe with 1 B rows. Lance scans ran 1.8–2.0× faster than the best Parquet configuration on most datasets.

Quickwit: one GET to open an indexCopy Anchor Link

Quickwit is a log search engine on Tantivy, now owned by Datadog.

Architecture. A split is an immutable Tantivy index plus a "hotcache" footer of under 0.1% of its size (Quickwit 101). The metastore stores the footer's byte range, so opening a split on S3 takes one ranged GET. The publisher commits each split with a checkpoint delta for its source and rejects a delta whose start doesn't match the stored checkpoint, which makes ingestion exactly-once with disposable indexers.

The trick. Keep the footer's location in the metastore, and an index on object storage opens in one GET. The request count then grows with the splits a query touches, not with documents.

Published numbers.

Conditional writes: what each store actually gives youCopy Anchor Link

Most systems above use a create-only or compare-and-swap PUT somewhere, and the stores differ in ways that matter.

• AWS S3. If-None-Match: * and If-Match are supported on PUT and CompleteMultipartUpload, and If-Match on deletes since 2025-09 (docs). The first writer to finish wins; others get 412, or 409 if a concurrent request raced. In-progress multipart uploads are invisible to the check, so a conditional complete can lose after every part has been uploaded.
• S3 Express One Zone. Single-digit-millisecond access, appends at an offset, and a price cut on 2025-04-10: storage by 31% and PUTs by 55%.
• Cloudflare R2. If-Match and If-None-Match are atomic on PUT (docs), but writes to the same key are limited to one per second. That rules out using a single R2 object as a hot commit pointer.
• Tigris. Metadata lives in FoundationDB, and conditional PUT, GET and DELETE are supported (docs). Copy-on-write bucket forks are worth watching for anyone who branches data.
• GCS. x-goog-if-generation-match: 0 creates only if absent (docs). Generation numbers are monotonic, which makes them better CAS tokens than ETags.
• Self-hosted stores. These are where to be careful. MinIO supports both headers. Its issue #21603 (opened 2025-09-24) reported that conditional PUTs proceeded without checking their condition when read quorum couldn't be reached. PR #21653 fixed it on 2025-10-24, so pin a version that includes the fix. Garage's compatibility page lists no conditional headers and no versioning. Ceph RGW still has open work on conditional writes under versioning.

Three pitfalls apply to every store:

• A timed-out PUT can return 412 on retry although the first attempt landed. For a unique name, read it back and compare a checksum before calling it a failure.
• An ETag is not a version. The same bytes give the same ETag, which permits ABA; prefer generation or version ids.
• A store that ignores the header overwrites silently. Prove the primitive on the store you are given; don't assume it.

AutoMQ and Bufstream: Kafka, two more waysCopy Anchor Link

AutoMQ is a Kafka fork with S3 as its storage layer: one WAL per node mixes all partitions and is uploaded almost at once, and small streams share objects until they grow past 16 MiB. Its most transferable idea is in the metadata: every object is registered as PREPARED before it is written and becomes COMMITTED when its commit lands. A scheduled task deletes expired uncommitted objects, so orphans are found by scanning metadata rather than listing the bucket. Its benchmarks page (read 2026-10) reports produce p50 465 ms and p99 823 ms at 1 GiB/s on its S3 WAL with 8 MiB / 250 ms batching.

Bufstream is a proprietary Kafka-compatible broker, acquired by CoreWeave in May 2026. Leaderless, zone-local brokers write time-bucketed intake files covering many partitions, ack after the PUT, and later reorganise them into archive files using ordering from a metastore (etcd, Postgres or Spanner). In Iceberg mode the archive files are Parquet with Iceberg metadata, so one copy of the data serves both Kafka fetches and analytics. Its Jepsen analysis (2024-11-12), which found five bugs, all fixed by 0.1.3. They included a stale cached high watermark after restart, etcd lease expiry that never reached the agents, and failed sends that returned offset 0. None was in storage. All were in metadata caching and lease plumbing, which is exactly where systems like these keep their hard state.

What they all agree onCopy Anchor Link

Side by side, the nine agree on more than they disagree:

• One PUT is the floor for an acked write. On S3 Standard, WarpStream, S2 and AutoMQ all land at roughly 400–800 ms p99.
• A zonal object-store quorum is the stateless way under 50 ms. WarpStream writes 2 S3 Express buckets, S2 writes 3, and SlateDB and AutoMQ let the WAL live in a separate, faster store.
• Reads are budgeted in round trips: three in turbopuffer, one per split in Quickwit, one or two per value in Lance, one per chunk per zone in WarpStream.
• Immutable objects plus a pointer. Nobody rewrites an object in place; where the pointer lives is the real choice.

What we're considering for LoamsCopy Anchor Link

Loams is in the same family. The bucket is the only source of truth, manifests are immutable objects, and only pointers live in a metastore. The log write path is modelled on WarpStream: multi-partition WAL objects written create-only, offsets assigned at commit, the last five producer sequences kept for idempotence. Collections are Lance datasets plus Tantivy splits opened through Quickwit-style footers, with a bounded unindexed tail (128 MiB, almost exactly turbopuffer's) scanned by brute force. Garbage collection checks the writer's lease before deleting, which plays the role of Neon's validate-before-delete. Many of the tricks above are already in the design docs.

The ideas below are the ones this teardown added. They are proposals under consideration, not commitments: none is approved or scheduled. Each links to the design document it would amend, and they are ranked by expected value for the effort.

1. Every node proves the store's guarantees at startup. Loams already requires conditional writes from every object-store provider (decision log, D178). The proposal adds a self-test when a store opens, in a scratch prefix: create-only twice (expect 412), If-Match with a right and a wrong ETag, an eight-way create race with one winner, and read- and list-after-write. It would also add read-back resolution for ambiguous create-only outcomes (the 412-after-timeout pitfall) to the object-store wrapper, shared with the proposed loams-seglog library (58, data access and HTAP), and Garage to the CI store matrix as an expected failure. Small effort, low risk.

2. Build the express WAL class before quorum. The stream design defines standard (S3 Standard), express (2 of 3 zonal buckets) and quorum (an openraft journal on local NVMe) (02, stream engine §2). Three systems here reach or approach the express target of 20–50 ms p99 with stateless nodes: S2 Express (ack under 40 ms), WarpStream on S3 Express (p99 169 ms at 50 ms batches) and SlateDB on S3 Express (5–10 ms writes). The proposal builds express first, makes 2-of-2 vs 2-of-3 a deployment choice, and defers quorum until someone needs under 10 ms or runs on-prem without zonal storage. It would amend the class sequencing in D12 and close the design's "(verify)" note on WarpStream, which this research confirms. Medium effort and risk.

3. A per-zone shared range cache, and heatmaps for new owners. Today each node fetches its own misses, and a node taking over a namespace serves cold while warming, with peer-to-peer transfer left as a later optimization (04, hot tier). The proposal replaces that line with WarpStream's per-zone ring over 4 MiB chunks with deduplication and bounded load (the shape of MIT-licensed cachey), plus Neon's heatmaps so a successor prefetches in heat order before it reports ready. Medium effort. The cost is an extra in-zone hop on misses; correctness is unchanged because cached ranges are immutable.

4. Continuous recall sampling. Loams has an on-demand recall endpoint (decision log, D92). turbopuffer's version shadows a sample of live ANN queries (1% by default) with an exact search at the same snapshot and exports recall@k per collection and path. Approximate paths drift quietly as deletions pile up and centroids age; this catches it in production. It would amend 06, search and vector. Small effort, about 1% extra CPU.

5. A determinism meta-test and a Jepsen-derived regression list. Loams already plans commit-path determinism with injected clocks and entropy. From S2: the same-seed-twice diff, libc-level overrides for dependencies, and Porcupine-style modelling of indefinitely failed operations. From Bufstream's Jepsen report: tests for stale cached watermarks after restart, lease expiry that never reaches its holder, sentinel offsets on errors, and filters that hide records from lagging consumers. Small effort, low risk.

6. Filter bitmaps keyed by IVF partition. On a cold collection, a moderately selective filter today uses a Lance prefiltered search over partitions chosen without regard to the filter (06 §4). turbopuffer-style (field, value, partition) bitmaps would let the planner probe only partitions with matches. That should mean fewer GETs and steadier filtered recall, to be measured on filtered benchmarks first. Medium effort; the sidecar must follow Lance re-clustering.

7. RaBitQ-first hot vectors. Hold 1-bit RaBitQ-style codes in RAM in hot HNSW artifacts and rerank from NVMe, guarded by item 4, and evaluate IVF_RQ as the default durable index for large collections (06 §5). Our arithmetic, not a measurement, says 4–8× more vectors per byte of hot RAM than int8. Medium effort; the risks are recall at low dimensions, and qdrant-edge's binary quantization not being RaBitQ.

8. Intent records for worker outputs. Garbage collection finds orphaned segments, splits and artifacts by listing prefixes, a cost that grows with the bucket. Following AutoMQ and Quickwit (Staged/Published), a worker would record an intent with a deadline before uploading, the commit would clear it, and GC would delete expired intents directly. WAL objects keep their age-based handling, so the produce path gains no metastore write, and a slow scrubber stays as a backstop. It would amend GC in 02 and 03, storage formats. Medium effort.

9. Tag cached reads by class. As in SlateDB, tag each range-cache read (footer, posting list, vector partition, fragment, segment, WAL chunk) and choose part size, admission and RAM vs NVMe per class, so a cold scan can't evict what warm queries need; add Neon's disk-usage watermark under the byte budgets (04, hot tier). Small to medium effort, low risk.

10. Single-copy arrow streams. For schema'd streams the segmenter could write Parquet and commit it to the linked Iceberg table, Bufstream-style, storing data once instead of twice. It conflicts with two current rules, one segment per index entry with a verbatim-bytes footer and no re-encoding while segmenting (02, 03), so it ranks last: a design spike at most, after the arrow encoding ships.

Considered and not proposed: turbopuffer's "no metastore" (a shared log commits far more than once a second per key, and R2 allows one write per second per key); Lance's reverse-sorted manifest names (pointers already live in the metastore); SlateDB's slot-contention fencing for the log (many writers share a partition); WarpStream Lightning's asynchronous commit (it gives up ordering and idempotence); and AutoMQ-style server-side copy for segmenting (it needs 5 MiB parts and patched offsets).

If you run one of these systems and we got a detail wrong, open an issue on the Loams repository and we will correct the post.