Open formats
Lance, Tantivy splits and, next, Iceberg. Your data stays readable by other tools, without Loams running.
The Loams engine stores data at rest in formats other software can read: Lance for documents and vectors, Tantivy splits for full text, and, as designed, Apache Iceberg (Parquet files with Iceberg metadata) for tables. The files sit in your bucket, under a plain layout, and the parts Loams defines itself are small, versioned and documented.
Why we built it this way
A database whose files only it can read holds your data hostage, however good its export tool is. Every migration becomes a project, every new tool needs a connector, and every second copy you make for analytics or training drifts from the first.
Open formats remove the second copy. A training pipeline can read the exact Lance version your search served, at the same moment. An analyst can point DuckDB at the same tables your dashboards use. And if you stop running Loams, the files are still there and still mean what they meant.
Our opinion: integrate over protocols and formats, never by embedding other engines (D51). Loams does not link Spark, Polars or DuckDB into itself. It writes files they already understand and hands them a precise list of what to read.
How it works
Lance, pinned to file format 2.1. Each collection is one Lance dataset: primary key, the document's JSON as _source, one column per dense or sparse vector, scalar and IVF vector indexes. Lance is the only Rust format we found with columns, vectors and versioning together (D6).
Scan pinning makes Lance safe to read from outside. Loams commits detached Lance versions and leaves the dataset's mainline at an empty first version, so a reader that opens the dataset root sees no rows rather than a wrong set of them. To read a collection, ask the native API for a scan plan, at the current manifest, a manifest version or a consistency token. The plan names the dataset URI, the detached version id and the fragments with their row counts, and whether there is a tail it does not cover. pylance, Polars, Ray Data and a PyTorch loader read those fragments directly from the bucket, or fall back to Arrow Flight for the tail.
Tantivy splits, in Quickwit's bundle format. Each split is one immutable object with a footer that lets a reader open it with one range GET. Loams builds Tantivy 0.26 with its quickwit feature (an sstable term dictionary), so a reader needs the same build. Delete bitmaps for splits are separate small objects, so a reader also needs the manifest to know which documents are live.
Loams' own formats are small and self-describing. The collection manifest (protobuf), delete bitmaps (roaring), primary-key deltas and dead letters each carry a four-byte magic, a format version and a CRC32C. Readers support version N and N−1; format changes are opt-in per object and rolled forward by compaction. The manifest schema is a .proto file in the repository.
Iceberg, through a REST catalog. The design keeps tables as Iceberg v2 with v3 features (deletion vectors) enabled per table, catalogued by Lakekeeper, so Spark, Trino, DuckDB, ClickHouse and Snowflake can read them through the Iceberg REST catalog. Separately, Loams House already serves lake tables stored as Iceberg.
What we give up
- Format-shaped constraints. We cannot change Lance or Tantivy's on-disk layout to suit Loams; we work within it, and we pin versions (Lance 2.1, Tantivy 0.26.2) and upgrade deliberately.
- Open is not the same as standalone. A Lance fragment is readable anywhere, but knowing which fragments, splits and bitmaps make up the collection at a point in time takes the manifest, and knowing about records not yet indexed takes the log. Reading without Loams gives you a snapshot, not the live tail.
- Writes from outside are not seen. If another tool writes into a collection's Lance dataset, Loams will not reference it. Write through Loams.
- Two formats where a closed engine would use one. See collections and manifests for why we pay that.
How other systems do it
- Elasticsearch and Qdrant keep their own on-disk formats, readable only by the engine; you export through an API.
- turbopuffer stores data on object storage in its own format.
- LanceDB stores data in Lance, readable by any Lance reader.
- Iceberg-native warehouses (Snowflake, Databricks, StarRocks, Databend) increasingly read and write Iceberg so data is not locked to one engine. Loams takes the same position for tables, and extends it to search and vectors.
Status
Available Lance datasets, Tantivy splits, the Loams manifest and sidecar formats, and pinned scans for pylance, Ray, Polars and PyTorch ship in Loams 0.1.0. Iceberg tables in the engine, readable through a REST catalog, are Planned.
Go deeper
- Collections and manifests.
- OLTP, OLAP and HTAP: where Iceberg fits.
- Loams Engine, Loams House.
- Design: 03 Storage formats, 17 AI data ecosystem, decision log (D6, D7, D51, D53).
- Source:
collection_manifest.proto,crates/loams-quickwit(vendored split code),crates/loams-collection.