The engine: hybrid retrieval, streams, Iceberg and the wire APIs

The retrieval engine is the oldest part of Loams and the part that exists most completely. It combines vector, sparse and full-text retrieval in one Rust binary over one bucket. Beside it, Loams Graph stores property graphs queried with GQL, and in 0.1.0 a search can expand one or two hops into a graph (experimental). This post covers its data model, how a write and a read move through it, the protocols it speaks, and the pieces that are still designs: Kafka-compatible streams and Iceberg tables. The engine page in Learn is the short version.

Five kinds of objectsCopy Anchor Link

A namespace holds five kinds of objects. Four ship in Loams 0.1.0.

The idea that ties them together is that the log is the spine (the Opinions post makes the full case). Every write, whatever protocol it arrives on, lands in a stream first. A write to a collection goes to that collection's implicit stream. Collections, graphs and tables are materializations of streams, kept up to date by links that record their applied offset atomically with every commit. That gives exactly-once materialization, and it gives every write a consistency token that means something in any read on any object.

The write pathCopy Anchor Link

A collection write is admitted only while the collection's unapplied backlog is under a budget (a million records or 128 MiB by default). Past it, the write gets HTTP 429 or gRPC RESOURCE_EXHAUSTED with a Retry-After. That backpressure is what keeps strong reads cheap: the backlog a read may need to merge always fits in memory.

Each collection is one Lance dataset for documents and vectors, plus a set of Tantivy splits for full text and typed-field indexes. A chain of immutable manifests binds them, and a compare-and-swap on the manifest pointer is the only commit point. We wrote about that in One manifest, two formats. Upserts and deletes go through a primary-key index on SlateDB, which maps a key to where its current row lives, and through per-split delete bitmaps. The "What Loams is built on" series covers each of those projects.

The read pathCopy Anchor Link

Every read compiles to an Apache DataFusion plan, whichever protocol it came in on. Loams adds its own physical operators:

A hybrid query is one plan: filter bitmap → (vector search ‖ BM25) → fusion → fetch. Nothing crosses a network hop between retrieval and fusion, and the planner sees the whole query.

Consistency. By default a read is strong: it sees every write acknowledged before it began. It does that by reading the durable state at the manifest's applied offset and merging the log tail after it. A read that carries a consistency token is guaranteed to see that write, on any node. eventual skips the tail for lower latency.

Scores that do not depend on layout. BM25 statistics are computed over every live document in every split plus the tail, not per split, and deleted or shadowed documents never count. So a document's score is the same before and after a split merge, and on a hot node or a cold one.

The hot tierCopy Anchor Link

Stateless compute over object storage is only fast with caching. Each query node owns the collections that rendezvous hashing assigns to it, and keeps four layers of derived state for them:

• H0: manifests, split footers and file metadata, in RAM.
• H1: byte ranges of durable objects in a RAM and NVMe cache (foyer). Objects are immutable, so the cache never needs invalidation.
• H2: acceleration structures, today HNSW graphs built with qdrant-edge, Qdrant's index code packaged as a library. The graphs are built from a manifest and stored back in the bucket, so another node can load them instead of rebuilding.
• H3: the in-memory tail of records already in the log but not yet indexed.

A differential test runs the same queries with the hot tier on and off and requires identical results.

The wire APIsCopy Anchor Link

Loams takes over the role each system plays in an AI stack, not every feature of its protocol. The footprint is deliberately narrow.

There is no Neo4j Bolt surface. Graphs are reached through GQL over loams.graph.v1, and from retrieval through the expand stage of a search and SQL table functions such as graph_expand. When this post was written there was no ClickHouse surface either; D347 reversed that for the House only, and Loams House (experimental in 0.1.0) speaks ClickHouse's HTTP interface on chDB. Analytics on Loams tables still go through Iceberg and Flight SQL. The Elasticsearch surface is scoped by external conformance suites, not by feature parity: no Kibana, no Painless, no full Query DSL.

The same retrieval is reachable from SQL, with the function names Spice uses:

SELECT id, _score
FROM rrf(
  vector_search('memories', [0.12, 0.34, 0.56], 'embedding', 100),
  text_search('memories', 'quarterly revenue', 'body', 100)
)
LIMIT 10;

Auth. When this post was written none of these listeners had authentication. In 0.1.0 engine auth is available: Tailscale identity by default, OIDC, hashed API keys, or loopback-only (LOAMS_AUTH_MODE). Roles, per-namespace authorization and tenant quotas are still planned.

What is still designCopy Anchor Link

Graph expansion for GraphRAGCopy Anchor Link

GraphRAG, LightRAG and Cognee retrieve with a narrow shape: seed by vector or BM25, expand one or two hops, rerank. Running a separate graph database for that one step adds a silo and an ETL path. Loams’ design keeps expansion in the query engine. When this post was written, graphs were to be mapped over collections and tables with CSR and CSC sidecars (D44). That changed: D634 and D740 made Loams Graph a stored property graph, queried with GQL and executed by an embedded Grafeo engine, with its truth in its own log and the bucket. Expansion now runs against a Loams Graph: the expand stage of a search (REST and Connect) and the SQL table functions graph_expand, graph_neighbors, graph_degree and graph_shortest_path. It is experimental in 0.1.0, and the SDKs do not expose it yet. Adapters for LightRAG and the LlamaIndex property graph are still planned.

Streams as a productCopy Anchor Link

The log exists and carries every write. Native HTTP produce and fetch, CloudEvents ingest, and idempotent produce with streaming subscribe over gRPC ship in 0.1.0. OTLP ingest and Kafka wire compatibility remain planned, so RisingWave, Flink, Spark, Kafka Connect and Debezium can read and write Loams streams unchanged. One change since this post: D408 defers a Kafka gateway inside Loams in favour of contributing to Apache Iggy's Kafka gateway, rather than building a second front end (see the Kafka wire protocol). Segments already use the Kafka RecordBatch v2 layout internally, and a test checks that the kafka-protocol crate decodes Loams’ batches and that Loams decodes its batches.

Iceberg tablesCopy Anchor Link

Tables for AI analytics (LLM calls and costs, traces, evals) will be Apache Iceberg tables in your bucket, catalogued in Lakekeeper, so DuckDB, Trino, Spark, ClickHouse and Snowflake read them without Loams. Loams adds keyed tables (one live row per key, through deletion vectors), a hot tier for Iceberg (a file index and sorted projections on NVMe) and a real-time tail, so a query sees rows seconds after they are written while external engines see snapshots at commit cadence. The Iceberg deep dive covers what that needs upstream.

Where this standsCopy Anchor Link

We have not published performance numbers, and will not until the exit benchmarks (BEIR relevance against Elasticsearch BM25, recall against Qdrant, the framework suites) run against a release. The next post moves off the bucket, to the part of Loams that needs real transactions: the reactive database on TiKV.