Five clusters, one bucket

Draw the architecture of a production retrieval-augmented application and count the stateful systems. Ours, when we started, looked like this: Elasticsearch for keyword search, Qdrant for vectors, Neo4j for the knowledge graph, Kafka for events and traces, and Temporal (or a queue, cron and a Postgres table) for long agent runs. Add ClickHouse for the dashboards and it is six. Each has its own cluster, its own replicas, its own upgrade cadence, its own security model and its own pager.

Our position is that for the retrieval part of that picture, this is the wrong shape. Vectors, full text, filters and graph expansion are one query, and they should be planned and executed as one query, over one copy of the data, in your bucket. That is what Loams' engine does. This post makes the case, shows what ships today, and is specific about where Elasticsearch, Qdrant, Neo4j, Kafka and Temporal are still the better choice.

What five clusters costCopy Anchor Link

Design 00 lists five consequences of the usual stack. We think they are in the wrong order of importance, so here they are in ours.

1. Copies drift, and drift is a correctness bug. The same entity lives in four stores, glued by connectors and CDC jobs. When one connector lags or a backfill half-fails, the vector store says one thing and the search index another. A human user shrugs at a stale search result. An agent takes the stale context as fact and acts on it. We stopped calling this an operations problem.

2. Nobody can plan the retrieval. A single "hybrid GraphRAG" retrieval is BM25 in Elasticsearch, kNN in Qdrant, a 1–2 hop expansion in Neo4j and a fusion step in Python, across three network hops. Each system optimizes its own piece. None of them can push a filter into the other's index, stop the expansion early because the fused list is already good enough, or fetch documents only for the final top ten. The plan exists only in application code, where no optimizer can see it.

3. There is no cross-store consistency. "I just wrote this memory; can the next agent step see it in search, in vectors and in the graph?" has no answer in a five-store stack. Each store has its own refresh interval and its own idea of "now".

4. Cost. Each system replicates two or three times on block storage. At about $0.08 per GB-month for EBS gp3, that is $0.16 to $0.24 of effective storage per GB, against about $0.023 for S3 Standard (design 00 §1), and the clusters are sized for peak and run around the clock. The bucket post does this arithmetic properly, with sources.

5. Agent runs are not durable without a sixth system. A crash in step 7 of a 10-step run repeats paid model calls or loses the run.

The last two are the ones people usually lead with. We put them last because they are the ones money can fix. You cannot buy your way out of drift or out of an unplannable query.

One request, one planCopy Anchor Link

Loams keeps a collection as one Lance dataset (documents, columns, vectors, scalar indexes) plus a set of Tantivy splits (the inverted index), bound together by a single manifest (D6, D7; one manifest, two formats). Vectors have two tiers: Lance's IVF index on the bucket, and a Qdrant-derived HNSW index in RAM and NVMe for hot collections, built from qdrant-edge (D8). DataFusion is the only query engine (D15).

Because all of it is one object under one manifest, one request can be one plan. This is the body of the native hybrid query (design 05 §4), trimmed:

POST /v1/namespaces/acme/query
{
  "from": "collections.memories",
  "consistency": {"token": "v1:s7/p3@918273"},
  "retrieve": [
    {"vector": {"field": "embedding", "query": [0.12, …], "k": 100}},
    {"text":   {"field": "body", "query": "refund policy for enterprise", "k": 100}}
  ],
  "filter": {"and": [{"term": {"tenant": "t42"}}, {"range": {"ts": {"gte": "now-30d"}}}]},
  "fuse": {"method": "rrf", "k": 60},
  "limit": 10
}

And this is what the planner turns it into:

A few things happen here that a five-system stack cannot do:

• One filter, built once, used by both retrievers. FilterBitmapExec turns the tenant and date predicates into a roaring bitmap from the scalar indexes. The vector search decides, by the filter's estimated selectivity, whether to score the allowed rows by brute force, pre-filter the index search, or post-filter with over-fetch (design 05 §3). The BM25 search uses the same bitmap. In the usual stack the filter is sent twice, to two systems with two different filter languages.
• Late materialization. Retrievers return row ids and scores. Documents are fetched by stable row id only for the final top k, in coalesced reads. Nobody ships a hundred full documents over the network to throw ninety of them away in Python.
• One snapshot. Both retrievers read the same manifest version, plus the same in-memory tail of writes that are acknowledged but not yet indexed. A document written a second ago is either in both result lists or in neither.
• One consistency token. Every write returns a token, a set of (stream, partition, offset) positions. A read that carries it is guaranteed to reflect those writes in every object derived from those streams, on every node and metastore backend (D76). That is the answer to "can the next agent step see it". The log post explains how.
• One place to measure. Every native search response carries a performance block: planning and execution time, rows scanned per retriever, cache hit ratios and object-store requests (D92). When a query is slow, there is one trace, not three.

The same retrieval is reachable from SQL, with function names shared with Spice (D56):

SELECT _id, body, _score
FROM rrf(vector_search('memories', [0.12, 0.34, 0.56], 'embedding', 100),
         text_search('memories', 'refund policy for enterprise', 'body', 100))
LIMIT 10;

vector_search, text_search and rrf ship in 0.1.0, and so does the native hybrid query with vector kNN, BM25, filters and RRF fusion.

The graph step, and what is not built yetCopy Anchor Link

GraphRAG frameworks (GraphRAG, LightRAG, Cognee, Graphiti, the LangChain and LlamaIndex graph integrations) mostly do the same three things: seed with vector or BM25 search, expand one or two hops filtered by edge type and time, then rerank (design 07 §1). Arbitrary-depth traversal and a full graph query language are not on that path. So we made expansion an operator in the same plan, ExpandExec, over adjacency sidecars of a graph mapped onto the collections and tables you already have (D44), and an expand stage in the hybrid query.

That stage is planned, not shipped. Today the native API refuses a body that carries expand or rerank. What ships in 0.1.0 is Loams Graph, a stored property graph queried with GQL on an embedded Grafeo engine, with writes on the Loams log and crash recovery. The design keeps one vector store: Grafeo is built without its own vector features, and vertices reference documents in collections, so GraphRAG stays "seed in a collection, expand in the graph, rerank" (D749). Wiring that into one plan is the next step, and we will not claim it before it runs.

The rest of the stackCopy Anchor Link

Retrieval is the centre, but the five-cluster picture had two more boxes.

Kafka. Every write in Loams already lands in a stream first: a partitioned, offset-addressed log on the bucket. Streams and links (declared, exactly-once materializations from a stream into a collection or table) ship in 0.1.0, with a native produce and fetch API over HTTP and gRPC. What Loams does not have is the Kafka wire protocol. We decided against building a second Kafka broker inside the engine: consumer-group rebalancing and transactions are a project of their own (D43), so a Kafka gateway in the engine is deferred, not cancelled, and Kafka clients of our Event Fabric go through Apache Iggy's Kafka gateway, with our contributions (D408).

Temporal. Loams Durable embeds a Resonate server: durable promises and tasks, used through the Resonate SDKs unmodified (D19). An agent's run loop becomes a workflow whose steps are promises, so a killed run resumes at the step it was on without repeating model calls. In 0.1.0 its state is in SQLite; the TiKV store for clusters is in progress.

ClickHouse. Analytics is a separate product, the House, ClickHouse-compatible on chDB. It ships in 0.1.0 as an experimental, opt-in profile that the engine reaches over TLS on a private network; it does not share a query plan with retrieval. Iceberg tables that DuckDB, Spark, Trino and ClickHouse can read are planned.

Where the specialists still winCopy Anchor Link

We would rather lose a user to the right tool than keep them on the wrong one. Here is where we would tell you to keep what you have.

The test we suggest is simple. If your application uses two or more of these systems for the same entities, and the retrieval step crosses them, the copies and the unplannable query are costing you more than any one system's feature gap. If you use one of them deeply and alone, you probably do not need us yet.

What we gave upCopy Anchor Link

One engine instead of five is a real trade, and these are the costs we see:

• Breadth for depth. Each specialist has years of features we do not. Our compatibility surfaces are defined by test suites, not by feature parity (D13), and the next post explains why. A feature outside that scope is refused with an error, not half-supported.
• Cold reads. A query on a collection nobody has touched for a week reads from the bucket. The hot tier buys the latency back for collections that are pinned or promoted; correctness never depends on it.
• One blast radius. Five clusters fail separately. One engine can fail all at once. Compute is stateless and losing a node loses no data, but a bug in the planner is a bug in every retrieval.
• Benchmarks we have not published. Our gates for M1 are BEIR nDCG@10 within one point of Elasticsearch's BM25 and ANN recall within 1% of a reference Qdrant at equal latency budget on the hot tier (design 06 §10). Those are the targets the engine is tested against. We will publish results when they are reproducible on hardware you can rent, not before.

Where we areCopy Anchor Link

In 0.1.0, collections, hybrid retrieval with RRF, the SQL search functions, streams and links, the hot tier, the Qdrant API and the Elasticsearch subset are available: built, tested in CI and usable, at 0.x stability. Graph expansion inside the hybrid plan, the Kafka wire protocol and Iceberg analytics are planned. Loams Graph, Durable and the House are separate products that do not yet share a plan with retrieval.

If your stack is the one on the left of the first diagram, try pointing your existing Qdrant or Elasticsearch client at a local loams dev and see what breaks. Then tell us. The design is in design 05, 06 and 07, and the engine post goes deeper on how it is built.