Collections and manifests

A collection is the Loams object for documents you search: each document has a primary key, a JSON body, any number of text, keyword, numeric and date fields, and optionally dense and sparse vectors. It is what replaces an Elasticsearch index and a Qdrant collection, and it is what the Qdrant API and the Elasticsearch subset read and write.

On the bucket, a collection is two open formats, Lance and Tantivy, bound together by one small file: the collection manifest.

Why we built it this wayCopy Anchor Link

No single open format is good at everything a retrieval collection needs.

• Lance stores columns and vectors well, versions a dataset, and has IVF vector indexes that work from object storage. Its full-text search is young.
• Tantivy, the Rust search library behind Quickwit, has mature BM25, phrase and fuzzy queries, and the aggregations an Elasticsearch-compatible API needs. Quickwit already showed how to keep Tantivy indexes on object storage as immutable "splits".

So the collection uses both (D6, D7), and the hard problem becomes keeping two formats consistent. Our answer is the oldest trick in storage: write everything new, then switch one pointer. The manifest names exactly which Lance version, which splits and which delete bitmaps make up the collection at version v. A reader that holds manifest v sees a mutually consistent snapshot, however many commits happen after it.

How it worksCopy Anchor Link

What is in Lance. Each document is a row: the canonical primary key, the original JSON as _source, the partition and offset of the record that last wrote it, one column per dense vector and one per sparse vector. Typed fields are not Lance columns; they live in Tantivy and can always be re-derived from _source. That keeps schema changes cheap: adding a field never rewrites Lance.

What is in Tantivy. Each indexing batch produces one immutable split, bundled into a single object with a footer (Quickwit's format), so opening a split costs one range GET. Each Tantivy document carries the Lance stable row id, so a hit in text search maps straight to its row in Lance.

Deletes and upserts. An upsert writes a new row and marks the old one deleted. A primary-key index (a SlateDB instance on the bucket) finds the old row; a roaring bitmap per split marks the dead Tantivy document. Lance's own deletion files mark the dead row.

The commit protocol. One link-apply task per collection, holding a fenced lease, does this for each batch:

1. Load the parent manifest v and its schema.
2. Decode the records, resolve every key through the primary-key index, and keep the latest operation per key in partition order.
3. Write new rows and deletes into Lance as a detached version built from exactly v's Lance version. A crashed writer's version is never referenced and never blocks the next one.
4. Write the new split, the changed delete bitmaps, a record of which keys changed and any dead letters, then manifest v+1. Every one of these is a create-only PUT under a new name.
5. Compare-and-swap the pointer collection/<id> from v to v+1 in the metastore. The swap is refused if the lease is stale, or if the commit started too long ago (so a slow writer cannot race garbage collection).

Maintenance (merging splits, compacting Lance, building vector indexes, publishing HNSW artifacts) commits the same way, as a maintenance manifest that rebases if a link commit got there first.

Reading. A query resolves the current manifest once, at plan time, and every operator in it reads that one version. It then merges the tail of records the manifest has not applied yet. See the read path.

Time travel and pinned scans. Superseded manifests are kept for 24 hours by default, and garbage collection keeps everything a retained manifest names. A Python tool can ask the native API for a scan plan at a manifest version and read the Lance fragments directly with pylance, Polars, Ray or a PyTorch loader. The dataset's own mainline version is deliberately empty, so a reader that ignores the manifest sees no rows rather than a wrong set of them.

What we give upCopy Anchor Link

• Two formats to maintain. Merges, compaction and garbage collection have to understand both Lance and Tantivy, and keep them aligned through row ids. It is more code than one format would be.
• One writer per collection. All link commits for a collection go through one leased task, so a single collection's write throughput is bounded by one apply loop. Sharding one collection across several manifest chains is designed (D95), not built.
• Tantivy splits need Tantivy built with its quickwit feature. Lance files are readable by any Lance reader; splits use Quickwit's sstable term dictionary, so a plain Tantivy build cannot open them.
• Lance's own commit path and cleanup are bypassed. Loams commits detached versions and runs its own garbage collection. A Lance tool that writes to the dataset directly would not be seen by Loams.

How other systems do itCopy Anchor Link

• Elasticsearch and OpenSearch keep Lucene segments on local disk, refresh them every second and replicate them shard by shard. A delete stays counted in BM25 statistics until a merge. Loams computes BM25 statistics over live documents only, so a score does not depend on merge history.
• Quickwit keeps immutable Tantivy splits on object storage with a metastore of split metadata. Loams' text side is that design, vendored.
• LanceDB builds on Lance for vectors and full text in one format. Its distributed serving layer is closed.
• Apache Iceberg uses the same pattern for tables: immutable data files, an immutable metadata tree and one catalog pointer swapped atomically. The collection manifest is that idea applied to a search collection.

StatusCopy Anchor Link

Available Collections, manifests, the commit protocol, upserts and deletes by id, split merges, Lance compaction, time travel within retention and pinned scans ship in Loams 0.1.0. Dataset tags (named, long-lived pins), filter writes re-checked at apply, branches and sharded collections are designed for later milestones.

Go deeperCopy Anchor Link

• Hybrid retrieval: how a query uses both halves of a collection.
• Open formats: who else can read these files.
• Loams Engine.
• Design: 03 Storage formats (§3, collections), 06 Search and vector, decision log (D6, D7, D53, D78, D105, D106).
• Source: crates/loams-collection, and the manifest schema in collection_manifest.proto.