The hot tier
RAM and NVMe layers that keep hot data next to compute, rebuild from the bucket, and never change an answer.
Every Loams object has two tiers. The durable tier is open-format files in the bucket: the source of truth. The hot tier is derived, node-local acceleration in RAM and on NVMe, plus an in-memory tail of records not yet indexed. The hot tier can be lost at any moment. That changes latency, never results.
Why we built it this way
Keeping everything in the bucket makes storage cheap and nodes disposable, but object storage round trips are slow: tens of milliseconds each, and a cold vector search needs several. If every query paid that, the design would only work for batch jobs.
The answer analytics systems found is a cache that knows the files it caches. StarRocks' shared-data mode serves Iceberg on S3 from a RAM and NVMe data cache close to local-disk speed; ClickHouse Cloud does the same over its own parts. We apply that pattern uniformly, not just to Parquet pages but to Lance pages, Tantivy splits and vector indexes.
Our opinion has two halves:
- Cheap by default, fast where it matters. A cold namespace should cost only its bucket bytes. A hot collection should answer in milliseconds. You should not have to buy RAM for data nobody queries.
- Correctness must never depend on the cache. If a hot structure can change an answer, it is not a cache, it is a second database. So every hot structure is derived from a specific source version, and the engine is tested by turning the hot tier off and comparing.
How it works
Four layers.
| Layer | Medium | Holds | Keyed by |
|---|---|---|---|
| H0 metadata | RAM | Manifests, split footers, offset indexes, Parquet footers | Immutable object id |
| H1 range cache | RAM, then NVMe | Byte ranges of any object: Lance pages, postings, segments | (path, offset, length) |
| H2 hot structures | RAM and NVMe | HNSW vector indexes, pinned Tantivy splits | (object, source version) |
| H3 tail | RAM | Records in the log but not yet in the manifest | (object, partition, offset range) |
Coherence is free. Durable objects are immutable, so a cached range of one is valid forever; H0 and H1 never invalidate anything. Only pointers move, and nodes learn about new manifests through the metastore's watch.
HNSW, built once, loaded many times. The durable vector index is Lance IVF, which works well from object storage. For hot collections, a worker builds an HNSW graph with qdrant-edge (Qdrant's own HNSW code, with filterable links and quantization) and publishes it to the bucket as an artifact under hot/hnsw/. The collection's owner node loads it: quantized vectors in RAM, full vectors on NVMe for rescoring. A replacement node downloads the artifact instead of rebuilding it.
Fresh writes without a rebuild. An artifact is keyed by stable row ids, so it stays valid across merges and compaction. When new rows arrive, a query uses a view: the artifact minus rows deleted since, plus a small appendable delta index of rows inserted since. The artifact is rebuilt when the delta grows past 20 % of its points (at least 10,000), or after 10 minutes stale.
Pinned text. Pinning a collection's text keeps every split of its live manifest on NVMe, verified by checksums, with split footers in RAM.
Promotion, demotion and budgets. You can pin a collection:
PUT /v1/namespaces/acme/collections/docs/hot
{"vectors": true, "text": true, "fragments": true}or warm it ahead of a deploy with POST …/collections/docs/warm. Auto-promotion is available and off by default: each owner keeps a heat estimate per collection, and above promote_min_hits it takes a lease that tells workers to build. Each node has a RAM budget, an NVMe budget and a cap on open artifacts; each namespace gets a fair share. Eviction goes by class (promoted before pinned), then by heat per byte, over-share namespaces first. A pin that does not fit is reported over_budget rather than evicting someone else.
Status you can see. GET …/collections/docs reports, per structure, off, building or ready, the source version it reflects and whether it is over budget.
What we give up
- A first query that is slow. A collection no node has touched is served cold from the bucket. The design targets are 5 to 20 ms at the median warm and half a second to a second cold. These are targets taken from reference systems, not Loams benchmarks.
- Memory you have to size. Budgets are per node. Too small, and hot collections evict each other.
- Owner affinity. The hot tier lives on the collection's owner node. Reads are routed there; a read that lands elsewhere is forwarded, or served colder locally if the owner is unreachable.
- Approximate search is approximate. HNSW and IVF can return different candidates. Scores are always from the exact kernel, so the same document gets the same score either way, but the candidate set can differ; exact paths (text, filters, aggregations, counts, exact vectors) are identical with the hot tier on and off.
How other systems do it
- StarRocks (shared-data) and ClickHouse Cloud: stateless compute over object storage with a two-level cache. The direct model for H1.
- turbopuffer: caches namespaces on NVMe and in memory in front of object storage, with cold queries served from the bucket.
- Qdrant: HNSW on local disk or in RAM is the primary index, not a cache, and it is replicated between nodes. Loams uses Qdrant's index code as a derived structure instead.
- Elasticsearch searchable snapshots: a frozen tier serves indexes from object storage through a shared cache, separate from the hot tier of local shards.
Status
Available H0, H1 in RAM, HNSW artifacts with delta indexes, pinned splits, fragment prefetch, pin and warm APIs, auto-promotion behind a setting, budgets and fair share ship in Loams 0.1.0. The NVMe tier of the H1 range cache is built but not yet exposed by a flag. Hot projections for Iceberg tables and cached graph adjacency are designed with those objects. Raising the replication of very hot collections automatically is planned.
Go deeper
- Stateless compute: why losing a hot tier is fine.
- Cluster mode: how owners are chosen.
- The read path.
- Design: 04 Hot tier and caching, decision log (D8, D9, D104, D109, D110).
- Source:
crates/loams-hot,crates/loams-cache,crates/loams-hnsw.