Your bucket is the database

The first decision in the Loams decision log, D1, is one sentence: object storage is the only durable source of truth, and compute is stateless. Everything else in the engine follows from it, and a fair amount of it is there to pay for it.

We wrote an earlier post about why a retrieval engine should live on object storage. This one goes further. Our position is that for most data an AI application holds, the bucket is the database: the bytes, the durability, the history and the permissions all live there, and a database server is a cache with opinions about layout. That is a strong claim, so this post does the arithmetic, names what it costs, and says where we break the rule ourselves.

The arithmeticCopy Anchor Link

Start with storage. These are AWS list prices for US East (N. Virginia), the figures our pitch has used since 2026-09-22; check S3 pricing and EBS pricing for today's numbers, and remember that large customers pay less than list for both.

• S3 Standard is about $0.023 per GB-month for the first 50 TB.
• EBS gp3 is $0.08 per GB-month, the rate AWS's own pricing example uses.
• A stateful search or vector cluster keeps two or three replicas of each shard on block storage, so one logical GB costs $0.16 to $0.24 a month. That is 7 to 10 times the bucket.

That is before headroom. Block volumes are provisioned, not consumed: a cluster sized to survive a node loss and a reindex runs well below full. And it is before compute. A stateful cluster must stay up to serve its data, so its nodes run around the clock whether anyone queries them or not. A namespace that lives in a bucket costs its bytes and nothing else until someone reads it.

Durability tells the same story. AWS designs S3 Standard for 99.999999999% durability over a year, storing objects across at least three Availability Zones, and designs it to survive the loss of a whole zone. An EBS volume other than io2 is designed for 99.8–99.9% durability, an annual failure rate of 0.1–0.2%. That gap is why every system on block storage replicates, and replication is software: a consensus protocol, a rebalancer, a recovery path, a set of bugs. On a bucket, the provider has already done that work, at a price that includes it.

Cross-zone traffic is the third line. A replicated cluster spread over three zones copies every write to the other two. AWS charges for data transferred between zones, which our stream design estimates at about $0.02 per GB for two replica copies (design 02 §9). Writes into S3 from the same region carry no such charge. Kafka pays this on every byte, which is why WarpStream and others moved Kafka's storage to object storage.

Requests are the line people forget. S3 charges per request, so a naive design that writes one object per record is ruinous. Our engine buffers writes from every partition and namespace on a node for up to 250 ms or 8 MiB and writes them as one WAL object (D25). Design 02's sketch puts 1 GiB/s of ingest across 20 nodes at about 80 PUTs a second, roughly $1,000 a month in requests. The design's job is to keep the request count proportional to nodes, not to records or tenants.

What you give upCopy Anchor Link

Latency. Nothing else on the list comes close.

A conditional PUT to S3 takes tens to hundreds of milliseconds and is contended per key (design 01 §3.2). When we measured a pure bucket WAL for Postgres in a spike, it came in at 22 ms p50 and 148 ms p99 (D234), an order of magnitude slower than Neon's safekeepers. A first read of a cold object pays a round trip before any byte arrives. Design 04 sets our own targets plainly: a warm vector or text query at 5–20 ms p50, a cold one served from the bucket at 0.5–1 s.

There are smaller losses too:

• Request-rate limits. S3 documents 3,500 writes and 5,500 reads per second per prefix. Our workers budget object-store requests per node so they stay under those limits (design 09 §6).
• Listing is slow. We never list the bucket to find garbage. The metastore tracks which objects are referenced, and GC deletes what it marks unreferenced after a grace period (D27).
• Mutation is awkward. Objects are written whole. So every data object in Loams is immutable and named by a ULID or a version; only pointers move (design 01 §6).

The question is not whether these costs are real. It is whether you can buy the latency back for the data that needs it, while every other byte stays at bucket prices. We think you can, and that most systems are built the other way round: they pay block-storage prices for everything so that the hot fraction is fast.

Buying the latency backCopy Anchor Link

Loams buys latency back in two places: on the write path, per stream, and on the read path, per object.

Writes: a WAL class per streamCopy Anchor Link

Every write in Loams lands in a stream first (the next post is about why). Each stream picks a WAL class (D12, design 02 §2):

• standard writes multi-partition WAL objects to S3 Standard and acknowledges after the PUT and a metastore offset commit. The p99 target is 400–600 ms. It is the cheapest, and the right class for bulk ingest, logs, and most collection writes.
• express writes the same object in parallel to three zonal buckets (S3 Express One Zone or GCS Rapid) and acknowledges on two of three. The target is 20–50 ms, it survives the loss of a zone, and it runs no stateful disks. Express storage is expensive, at about $0.11 per GB-month, but a WAL object lives only seconds before it is moved to Standard.
• quorum is a Raft journal of three log nodes, one per zone, with its WAL on local NVMe. It acknowledges on a majority fsync, targets 3–10 ms, and offloads to the bucket in the background. It pays cross-zone traffic for that speed.

Only standard is built today. express is on the roadmap for M5 and quorum for M6 (design 12); the low-latency path that does ship is Loams Postgres' loams-wal, below. The point of the design is that latency becomes a per-stream setting with a known price, not an architecture you are stuck with. A bulk import does not pay for an agent's chat memory, and the agent's memory does not wait 500 ms.

Reads: layers you are allowed to loseCopy Anchor Link

On the read side, every object has a durable tier in the bucket and a hot tier on the node (design 04, D9). The hot tier has four layers:

• H0, metadata: manifests, Iceberg metadata, Parquet footers and offset indexes, in RAM.
• H1, an object cache: byte ranges of any durable object, in RAM and then on NVMe, through foyer.
• H2, hot structures: acceleration that the bucket format does not have. For vectors that is an HNSW graph built with qdrant-edge over a Lance dataset whose durable index is IVF (D8). For text, it is Tantivy splits pinned on NVMe. For Iceberg tables, sorted hot projections.
• H3, the tail: data already committed to the log but not yet in its indexed form, held in memory and merged into every strong read.

Two rules make this safe rather than merely fast.

First, coherence is trivial by construction. Durable objects never change, so a cached byte range is never stale. Only pointers move, and nodes learn about new pointers from the metastore's watch stream. There is no invalidation protocol to get wrong.

Second, a query must return identical results with or without any hot structure (design 04 §6). Tests disable hot tiers at random and compare. A hot structure carries the source version it was built from, and is used only with a correct delta on top or not at all. Losing a node loses a cache; the new owner serves cold from the bucket while it warms, or downloads hot artifacts that were built once and published back into the bucket.

This is not our invention. It is the pattern StarRocks established for Iceberg on S3 with its data cache, and the one turbopuffer runs for search. What we add is applying it uniformly: to Lance pages, Tantivy splits and graph adjacency, not only Parquet.

Where we bend the ruleCopy Anchor Link

A rule with no exceptions is usually a rule nobody has tested. These are ours.

Commits that need single-digit milliseconds do not wait for S3. The planned quorum class will acknowledge from local NVMe. Loams Postgres acknowledges a commit once loams-wal has it on a quorum of local disks, and group-commits to the bucket every 250 ms (D714). The rest of the industry agrees: Neon and Databricks Lakebase acknowledge on Paxos safekeepers, TiDB X persists its Raft log locally and uploads in the background, OceanBase acknowledges on its log service (design 58 §1.2, with sources). The bucket holds history; it does not hold the commit point. Our first principle says this out loud: the one exception is "the seconds-long WAL tail of quorum-class streams" (design 01 §1).

Loams Live is not on the bucket at all. Live is an OLTP document database, and its source of truth is TiKV, with Raft replication on local disks. We amended D1 explicitly for it (D130) rather than pretend, and made continuous log backup of Live keyspaces to the bucket mandatory. The previous post explains why small, hot, transactional state belongs in a KV.

Metadata does not live in objects. Stream offsets, leases and manifest pointers change too often for conditional PUTs, so they live in the metastore. Manifest bodies stay in the bucket; only pointers are in meta (design 01 §3.2).

What does not bend: no acknowledged write in the retrieval engine depends on a node's disk surviving, and no query result depends on a cache.

What others get rightCopy Anchor Link

Local-disk systems are not wrong; they are optimized for a different shape. Qdrant and Elasticsearch on fast disks serve a large, constantly queried index with no warm-up and no first-read penalty, and for a single big index that is always hot, they are simpler to reason about than a cache. If your workload is one 500 GB index at a steady thousand queries a second, the bucket buys you less.

The shape we built for is the one AI applications actually produce: an index per user, per workspace, per repository or per agent, most of them idle most of the time. There, paying replicated-disk and always-on prices for the whole set to keep the busy few fast is the wrong trade. turbopuffer proved the model for search in a closed product. WarpStream proved it for Kafka. Neon proved it for Postgres history and branches. None of those implementations is open source in a form you can run for vectors, text and graphs together, which is the gap Loams exists to fill.

The opinion, stated plainlyCopy Anchor Link

If the bytes live in your bucket in open formats (Lance, Tantivy, Iceberg, Parquet), then:

• Losing every Loams node loses no data. You can delete the cluster and rebuild it from the bucket.
• Other engines can read your data without us. pylance, Ray or Polars can read a pinned Lance version today, and once Iceberg tables ship, DuckDB, Spark or Trino will read them through the REST catalog. Leaving Loams does not need an export.
• Cost follows use. Cold data costs bucket prices. Hot data costs cache, and you choose which data is hot.
• Your permissions, encryption keys and region are your own. The data sits in your account, under your bucket policy.

That is what we mean by "your bucket is the database". The engine is the part you can replace.

Where we areCopy Anchor Link

In 0.1.0, collections are stored as Lance plus Tantivy under one manifest in the bucket, and the hot tier (HNSW artifacts and pinned splits in RAM and NVMe, with delta indexes for fresh writes) ships and is tested in CI. Iceberg tables are planned. The latency numbers above are design targets taken from reference systems, not benchmarks of Loams; we will publish measured numbers with the methodology when we have them. The details are in design 02 and design 04.