The Loams engine

The engine is the loams binary's retrieval core. It stores collections: documents with text fields, typed fields and dense or sparse vectors. Each collection is a Lance dataset plus a set of Tantivy splits in your object-storage bucket, bound together by one manifest. One query plan runs vector kNN, BM25 and filters and fuses the results. Every write goes through a log first, and background workers turn the log into those files. Your existing Qdrant and Elasticsearch clients can talk to it, and so can any ADBC or Flight SQL driver. psql and MySQL clients get read-only SQL over the same collections.

Use it when you want search and vector retrieval on storage you own, with compute you can throw away. It also fits when you want one store for an application that today runs Elasticsearch next to a vector database, or when you want the same documents readable by Lance-based data tools. Do not use it yet if you need per-namespace authorization, Kafka clients, aggregations or highlighting, multivectors, or a track record in production. It is a first release (0.1.0), and the fair comparison below says where the established systems are ahead.

StatusCopy Anchor Link

Everything on this page ships in Loams 0.1.0, the first release (2026-10-18). The feature table in Running Loams 0.1.0 marks every engine row Available: built, tested in CI and in the release images. As with all of 0.x, APIs, formats and flags may change between minor versions, and loams upgrade moves a stack from one release to the next with its data.

History: the engine core (collections, retrieval, the gateways, the hot tier, cluster mode and both metastores) first shipped in v0.1.0-preview.1 on 2026-10-12, and v0.1.0-preview.2 on 2026-10-15 did not change it. Around the engine, 0.1.0 adds two operational changes. One is TLS on the main port (LOAMS_TLS_CERT_FILE, LOAMS_TLS_KEY_FILE). The other is loams upgrade, which snapshots every store before it migrates and rolls back on a failure.

ArchitectureCopy Anchor Link

The protocol gateways (loams-qdrant, loams-es, the Flight SQL service, the Postgres and MySQL wire listeners and the native routes in crates/loams) never touch storage or the metastore themselves. Each one translates its protocol into the search IR of loams-query and calls CollectionService. That is why a Qdrant filter, an Elasticsearch bool query and a SQL WHERE clause run through the same filter evaluator and mean the same thing.

How it worksCopy Anchor Link

Collections: one manifest over two formatsCopy Anchor Link

A collection's durable form is in the bucket under ns/<ns>/collections/<cid>/:

• Lance (lance/) holds the system columns _pk, _source (the document's JSON), _ingest_partition and _ingest_offset, plus one column per dense vector and one per sparse vector. Lance's stable row ids are on. The file format is pinned to Lance 2.1. Lance holds the IVF vector index (IVF_PQ by default, IVF_RQ or IVF_HNSW_SQ on request) and a BTREE index on _pk.
• Tantivy splits (text/splits/) hold the inverted index and fast fields for every typed field. A split is Quickwit's bundle format: one object with a hotcache footer, opened with one ranged GET. Quickwit's split code is vendored in loams-quickwit.
• The manifest (manifests/) is an immutable protobuf in a small checksummed envelope (OPCM). It names the Lance version, the splits and their delete bitmaps, the index segments, the hot artifacts and the applied offsets of the collection's own stream.

Typed fields live only in Tantivy, never as Lance columns. Lance and Tantivy are joined by the stable row id, which each split stores in a _rowid fast field. A reader that loads manifest v sees a Lance version, split set and delete bitmaps that agree with each other. More on the idea: collections and manifests and open formats.

Loams commits Lance as detached versions. The dataset's mainline stays the empty version 1, and every collection manifest points at its own detached version. The only point that orders commits is the compare-and-swap of the manifest pointer in the metastore. A crashed or fenced writer's Lance version is never referenced, so it never blocks the next commit. Loams' own garbage collector, not Lance's cleanup, decides when files go.

The write path: the log is the spineCopy Anchor Link

Every collection has an implicit, partitioned stream. A key's partition is xxh3_64 of its canonical bytes, modulo the partition count. A write from any gateway becomes an operation record (Upsert, Delete or Patch) on that stream:

1. The gateway checks the collection's unapplied backlog. If it is over budget, the write is refused with HTTP 429 or gRPC RESOURCE_EXHAUSTED and a Retry-After. The defaults are 1,000,000 records and 128 MiB (--max-unapplied-records, --max-unapplied-bytes).
2. loams-log's LogWriter buffers records from many partitions and namespaces. It writes one WAL object per flush (wal/<class>/<node>/<ulid>.wal, with records in Kafka's RecordBatch v2 layout), then commits it to the metastore's sequencer, which assigns offsets.
3. Only then is the client acknowledged, with a consistency token such as v1:s7/p3@918273. The write is durable in the bucket at this point.
4. A worker's link-apply task reads the new records. It resolves each key through the primary-key index (SlateDB, loams-pk), writes a Lance fragment and a Tantivy split from the same batch, updates delete bitmaps, writes manifest v+1, and CASes the pointer collection/<cid>. Records that cannot be decoded or break the schema become dead letters, recorded per commit.

A request is atomic within its stream: a native write batch, an Elasticsearch _bulk or a Qdrant upsert either lands whole or not at all. As built, loams-log implements the standard class: object-storage PUTs plus a metastore commit. The express (zonal buckets) and quorum (Raft journal) classes are designed but not built. See the log is the spine and the write path.

Links and exactly-onceCopy Anchor Link

A link applies a stream to a target, exactly once. The mechanism is in loams-link. A task leases its link in the metastore with an epoch. It reads from the applied offsets recorded in the target's latest commit, and commits new data and new offsets together under the pointer CAS. A zombie task with a stale epoch fails its commit, and a restart resumes from the committed offsets. Duplicate work after a crash is thrown away, never applied twice.

Today the registered link targets are the collection target, which applies every collection's own stream, and a counter test target. You can create links over the native API (POST /v1/namespaces/{ns}/links). Declared links that transform a stream into another collection or an Iceberg table (CREATE LINK … TRANSFORM) are designed in §09 but not built.

The read path: strong by defaultCopy Anchor Link

A read resolves a snapshot per collection: a manifest version and its applied offsets. It then merges the tail, the acknowledged records the manifest does not hold yet, from an in-memory index on the query node. Strong reads therefore see every acknowledged write without waiting for indexing. The levels are Strong (the default), AtLeast(token), Eventual (skip the tail; lowest latency, staleness equals link lag) and Pinned. A pin holds nothing in the metastore: it stays readable while its manifest is retained (24 hours after it is superseded, by default). Backpressure exists so that a strong read never has to rebuild a large tail from the log. See consistency tokens and the read path.

Hybrid retrievalCopy Anchor Link

Every read surface compiles to an Apache DataFusion plan. loams-query adds the operators: FilterBitmapExec turns filters into roaring bitmaps, AnnExec runs vector top-k, TantivySearchExec runs BM25, SparseExec scores sparse vectors, FusionExec fuses and DocFetchExec fetches documents for the final top-k only.

The choices that make results trustworthy:

• Filters pick a strategy by selectivity. A very selective filter is scored by brute force. One that allows at most 10% of rows prefilters the index search. A broad one post-filters with over-fetch and retries once.
• Scores are exact. Every returned vector score comes from Loams' own exact kernel, never from an index or a hot artifact, so results are identical with the hot tier on and off.
• BM25 statistics are global and live-only. Deleted and shadowed documents are not counted, so a score does not depend on where a document's history sits.
• Fusion is rrf (with k, default 60), dbsf (Qdrant's distribution-based normalisation) or weighted. Ties go to the lower primary key.

The native hybrid request (POST /v1/namespaces/{ns}/query) takes retrieve, filter, fuse and limit. The same retrieval is in SQL through vector_search, text_search, hybrid_search and rrf. These are positional table functions, named after Spice's so that switching is cheap. More: hybrid retrieval.

The hot tierCopy Anchor Link

The hot tier makes queries fast. It is never needed for a correct answer: everything in it is derived and can be rebuilt from the bucket.

PUT /v1/namespaces/{ns}/collections/{c}/hot with vectors, text and fragments pins a collection. With vectors, workers build an HNSW artifact with qdrant-edge (wrapped by loams-hnsw) and publish it under hot/hnsw/, and the owning query node loads it. Artifacts are keyed by stable row id, so compaction and merges do not invalidate them. A query at a newer manifest uses a view: the artifact minus rows deleted since, plus an appendable delta index of rows inserted since. A view with more than 100,000 exclusions is not served until the rebuild lands. With text, every split of the live manifest is pinned on local NVMe. Budgets per node are --hot-ram-bytes (default 8 GiB) and --hot-nvme-bytes (default 100 GiB). Auto-promotion by heat exists but is off by default. Without a pin, Lance IVF serves vector queries. See the hot tier.

Pinned scansCopy Anchor Link

Because the Lance mainline is empty, pointing lance.dataset(uri) at a collection shows no rows. POST /v1/namespaces/{ns}/collections/{c}/scan returns a scan plan instead. It contains the manifest version, the Lance URI and detached version id, the fragments with their deletion files, the column roles and an expiry. Any Lance reader can then read the fragments straight from the bucket. Writes newer than the pinned version (the plan's tail) are reported, and the plan carries a pin that reads exactly that state through the native API or Flight SQL. Any reader built on Lance can take the plan (pylance, and through it Ray, Polars or PyTorch). The Loams-specific conveniences are M2 work: credential vending, dataset tags, a Ray datasource, Polars scan_loams() and loams.torch. Today the reader uses the object-store credentials you already have.

Cluster modeCopy Anchor Link

loams cluster --node-id … --roles … --listen … --peers … --bucket … runs any subset of the roles meta, log, query, worker and gateway (gateway implies log). Every node holds a metastore replica: a voter on meta nodes, a non-voting learner elsewhere. Each node registers with a lease node/<id> (TTL 10 seconds). A collection's owners are the top r live query nodes by an xxh3_64 rendezvous score, one per zone first (--replication, default 1). When nodes come and go, only the keys whose top node changed move.

A gateway forwards a collection read to the first owner over POST /internal/v1/reads/{op}, with the request's consistency and token. It forwards at most once. If the owner answers Unavailable or Timeout, or cannot be reached, the gateway runs the read locally, and a node that failed a forward is skipped for 5 seconds. The --listen address carries unauthenticated internal routes, so it must be on a private network. See cluster mode.

The metastoreCopy Anchor Link

Everything that changes at high rate lives in a metastore behind one semantic trait, MetaStore in loams-common: offset sequencing, leases and fencing, manifest pointers, the catalog and link state. Manifest bodies never go there. Two backends ship:

• openraft (loams-meta), the default for loams dev and standalone. The Raft log is in redb and snapshots go to the bucket. In a cluster, writes go to the leader over HTTP. A linearizable read asks the leader for a read index and waits for the local replica to apply it.
• TiKV (loams-meta-tikv, cargo feature tikv, --meta tikv://<pd>/<keyspace>). Each trait call is one TiKV transaction, and the clock is PD's TSO. The self-hosted compose stack runs the engine this way (--meta=tikv://pd:2379/loams_meta).

Both pass the same conformance suite (loams-meta-conformance), which includes a linearizability checker. See metadata and TiKV.

Wire APIsCopy Anchor Link

Each listener can be moved or turned off on its own (loams dev --help).

Requests to the Qdrant and Elasticsearch gateways go to the default namespace unless they carry a Loams-Namespace header. Each gateway lists its differences from the original in its crate documentation. For example, Qdrant writes with wait are already visible when acknowledged, and Elasticsearch _version is not dense.

Key decisions and whyCopy Anchor Link

The full log is indexed at decisions. These are the engine's load-bearing ones, with the reason as the log states it.

Two open items affect what you build on. D739 (proposed) would remove the engine's read-only mysql-wire listener (Q663); MySQL-compatible OLTP is Loams SQL. D111 deferred auth and TLS to one plan. Engine auth on the main port has since shipped, but the other listeners stay inside the stack, as the next sections say. The design behind all of this is in §01, §03, §04, §05 and §06.

What is compatibleCopy Anchor Link

Only what CI actually runs is listed as tested.

Not gated yet, and worth knowing before you rely on them:

• ADBC Flight SQL drivers. The quick start uses adbc-driver-flightsql from Python, but no CI job runs ADBC's validation suites yet (M1.7 Task 4).
• psql, psycopg, node-postgres. They were tried in the Postgres-wire spike, but no client CI job exists yet (PG1 Task 5).
• MySQL client libraries. None runs against the engine's MySQL listener. Its tests speak raw protocol bytes.
• LangChain and LlamaIndex. Their vector-store suites (Qdrant and Elasticsearch) define the gateways' scope, and the tests replay their request shapes, but the suites themselves do not run unmodified in CI yet (M1.7 Task 3).

How it comparesCopy Anchor Link

These are widely known properties of the alternatives, not benchmarks. Loams has not run its planned BEIR and recall-parity gates yet (M1.7 Tasks 5 and 6), so this page makes no speed or relevance claims.

Where Loams is ahead: the bucket holds the only copy, in formats other tools can read. Strong reads see a write as soon as it is acknowledged, through the tail. Vector scores are exact and identical whether a cache is warm or cold. One store answers Qdrant, Elasticsearch, SQL and Arrow clients.

Where the others are ahead today:

• Maturity and operations. They have years of production use. Loams has just shipped its first release.
• Qdrant shards and replicates a single collection, and serves multivectors, geo filters and non-float32 datatypes. Loams answers those with 501.
• Elasticsearch and OpenSearch have aggregations, highlighting, many language analyzers, real security and a large tool ecosystem.
• turbopuffer is a managed service already serving production workloads on object storage, and Loams borrowed ideas from its design (D86 to D98 came from a gap analysis against it).

Loams also lacks the Kafka wire protocol and per-namespace authorization, which many teams expect around a search store.

What is missing today and what is plannedCopy Anchor Link

From Running Loams 0.1.0, the plans and the design roadmap:

• Authorization and tenancy. A valid token or key can do anything the engine can do. Roles, per-namespace authorization, quotas and a namespace router are planned. Keep the Qdrant, Elasticsearch, Flight SQL, Postgres-wire and MySQL-wire listeners inside the stack: the auth modes are specified for the main port only.
• Exit gates. M1.7 is planned, not done: the conformance runner with the four framework suites, the ADBC and Spice Flight SQL gate, BEIR nDCG@10 against Elasticsearch, Recall@10 against Qdrant, hot on/off identity at scale, the recall endpoint (Task 10) and the published limits page (Task 11).
• MCP and analytics helpers. The loams-mcp server and conveniences such as to_arrow() and to_polars() are planned (M1.6). The SDKs themselves are published as 0.1.0; see SDKs.
• Postgres wire writes. INSERT, UPDATE, DELETE and COPY FROM STDIN behind --pg-allow-writes are PG1 Tasks 6 to 10. The flag does not exist yet.
• Search features. Elasticsearch aggregations, highlighting, point in time and scroll are Phase B. Ranking expressions (function_score, Qdrant formula), per-field BM25 parameters, more language analyzers, and f16, i8 and u8 vectors come in M2. Sharded collections come in M2.x. A custom sparse index is Phase B, and DiskANN is a Phase C evaluation.
• Data tools. Dataset tags, credential vending, and Ray, Polars, PyTorch and PySpark adapters for scan plans come in M2. A thin Loams ADBC driver and a Power BI connector are proposed in DA1 (D1055, D1056).
• Streams and other objects. The express and quorum WAL classes, OTLP ingest (M2) and the Kafka wire (deferred to Iggy's gateway, D408) are not built. Graph expansion and rerank in the hybrid query are M3, and Iceberg tables are M4.

Try itCopy Anchor Link

The whole stack, from Running Loams 0.1.0. It needs Docker Compose and a Tailscale auth key in .env:

git clone --branch v0.1.0 https://github.com/ostrium-labs/loams.git
cd loams/deploy/release
cp .env.example .env
docker compose up -d
docker compose ps                                   # every service running
docker compose exec tailscale tailscale status      # the stack is on your tailnet

Self-hosting over Tailscale covers the tailnet, the console and the one-time first-run setup.

The engine alone, from source, following the engine quick start. On machines with less than 32 GB of RAM, limit parallel jobs with -j 4:

cargo run --release -p loams -- dev

Then, on the same machine, with ENGINE set to the dev server's loopback address on port 8080, create a collection, write and run a hybrid query:

curl -s "$ENGINE/v1/namespaces/demo/collections" -H 'content-type: application/json' -d '{
  "name": "kb",
  "schema": {
    "fields": [
      {"name": "body", "source_path": "body", "kind": {"text": {"analyzer": "standard", "positions": true}}, "indexed": true, "fast": false},
      {"name": "tenant", "source_path": "tenant", "kind": "keyword", "indexed": true, "fast": true}
    ],
    "vectors": [{"name": "embedding", "dim": 3, "distance": "cosine"}],
    "sparse_vectors": [], "dynamic": "ignore", "max_fields": 1000
  }
}'

curl -s "$ENGINE/v1/namespaces/demo/collections/kb/documents" -H 'content-type: application/json' -d '{"ops": [
  {"upsert": {"id": 1, "source": {"body": "refund policy", "tenant": "a"}, "vectors": {"embedding": [1.0, 0.0, 0.0]}}},
  {"upsert": {"id": 2, "source": {"body": "shipping times", "tenant": "a"}, "vectors": {"embedding": [0.9, 0.1, 0.0]}}},
  {"upsert": {"id": 3, "source": {"body": "refund window", "tenant": "b"}, "vectors": {"embedding": [0.0, 0.0, 1.0]}}}
]}'

curl -s "$ENGINE/v1/namespaces/demo/query" -H 'content-type: application/json' -d '{
  "from": "collections.kb",
  "retrieve": [
    {"vector": {"field": "embedding", "query": [1.0, 0.0, 0.0], "k": 10}},
    {"text": {"field": "body", "query": "refund", "k": 10}}
  ],
  "filter": {"term": {"tenant": "a"}},
  "fuse": {"method": "rrf", "k": 60},
  "limit": 3
}'

The same query in SQL:

curl -s "$ENGINE/v1/namespaces/demo/sql" -H 'content-type: application/json' -d @- <<'JSON'
{"query": "SELECT _id, body, _score FROM rrf(vector_search('kb', [1.0, 0.0, 0.0], 'embedding', 10), text_search('kb', 'refund', 'body', 10)) LIMIT 3"}
JSON

An existing Qdrant client, pointed at the gateway on port 6333:

# pip install qdrant-client
from qdrant_client import QdrantClient, models

client = QdrantClient(url=QDRANT_URL)  # the engine's Qdrant REST listener, port 6333
client.create_collection("docs", vectors_config=models.VectorParams(size=3, distance=models.Distance.COSINE))
client.upsert("docs", points=[models.PointStruct(id=1, vector=[1.0, 0.0, 0.0], payload={"title": "refunds"})], wait=True)
print(client.query_points("docs", query=[1.0, 0.0, 0.0], limit=1))

Flight SQL through ADBC, and a pinned scan read with pylance:

# pip install adbc-driver-flightsql pyarrow
import adbc_driver_flightsql.dbapi as flight_sql

with flight_sql.connect(
    FLIGHT_SQL_URI,  # grpc://<engine host>:8082
    db_kwargs={"adbc.flight.sql.rpc.call_header.loams-namespace": "demo"},
) as conn, conn.cursor() as cur:
    cur.execute("SELECT _id, body FROM kb ORDER BY _id")
    print(cur.fetch_arrow_table())

# pip install pylance==12.0.0
import json, urllib.request
import lance

request = urllib.request.Request(
    ENGINE + "/v1/namespaces/demo/collections/kb/scan",
    data=b"{}", headers={"content-type": "application/json"},
)
plan = json.load(urllib.request.urlopen(request))
if plan["lance"] is not None:  # null until the first commit
    dataset = lance.dataset(plan["lance"]["uri"], version=plan["lance"]["version"])
    print(dataset.count_rows(), "live documents; tail:", plan["tail_records"], "records")

To run against a bucket instead of a local directory, use loams standalone --bucket s3://bucket/prefix. For several nodes, use loams cluster.

SourceCopy Anchor Link

• Crates: loams (the server, native API, Flight SQL, Postgres and MySQL wire), loams-collection, loams-query, loams-hot, loams-hnsw, loams-qdrant, loams-es, loams-log, loams-link, loams-worker, loams-pk, loams-text, loams-quickwit, loams-cache, loams-meta, loams-meta-tikv, loams-stream-grpc.
• Design: §01 architecture, §02 stream engine, §03 storage formats, §04 hot tier, §05 query engine, §06 search and vector, §09 links and workers, §17 AI data ecosystem, §18 metastore backends, §58 data access, and the decision log.
• Plans: M0.2 metastore, M0.3 log engine, M0.4 workers and links, M1.1 collection storage, M1.2 query engine, M1.3 hot tier and routing, M1.4 Qdrant API, M1.5 Elasticsearch API, M1.7 exit gates, PG1 Postgres wire.