The Elasticsearch REST API and a Query DSL subset

Elasticsearch's REST API is less a protocol than a large, organically grown surface: hundreds of endpoints, a JSON query language with dozens of query types, aggregations, scripting and cluster management. Loams does not try to be Elasticsearch. It serves the part of that API that LangChain, LlamaIndex and the BEIR relevance harness send, compiled onto the same collections the Qdrant API and the native API use. This post is about where we drew that line and why.

What the protocol isCopy Anchor Link

Transport. Plain HTTP/1.1 with JSON bodies, usually on port 9200. Resources are paths: /{index}/_doc/{id} for one document, /{index}/_search for a search, /_bulk for a batch. Reads that carry a body are allowed on GET and POST alike, and the clients use POST.

Version negotiation. Since 8.0, the official clients send a vendor media type with a version parameter: Accept: application/vnd.elasticsearch+json; compatible-with=8, and the same in Content-Type when there is a body. A server that supports REST API compatibility answers in the requested major's shape, so a 7.x client can talk to an 8.x cluster during an upgrade (REST API compatibility). The 8.x clients also do a product check: they refuse to work with a server whose responses lack X-Elastic-Product: Elasticsearch. That header is the closest thing the API has to a handshake.

Auth. HTTP Basic with a user and password, or Authorization: ApiKey <base64(id:api_key)> for API keys (API keys). Each request is authenticated on its own; there is no session.

Bulk framing. _bulk is the one place where the body is not a single JSON document. It is NDJSON: newline-delimited JSON, with Content-Type: application/x-ndjson. Each operation is an action line (index, create, update or delete, with _index and _id metadata), followed for all but delete by a source line holding the document or the partial update. The body must end with a newline. The answer lists one result per item, in request order, and a bulk request can partly succeed: errors: true means at least one item failed, and each item carries its own status (bulk API). _msearch uses the same framing, a header line and a body line per search.

The Query DSL. A search body has a query, written as nested JSON: leaf queries (match, term, range, exists, prefix, wildcard, fuzzy) combined by compound ones (bool with must, should, filter, must_not; constant_score; function_score; script_score). Around the query sit from and size, sort, search_after, _source filtering, aggregations, highlighting and more. Vector search arrived as knn, either top-level or as a query, with k, num_candidates and a query_vector.

Retrievers. Elasticsearch 8.14 added retriever, a composable tree that replaced the older rank option for hybrid search (retrievers). A standard retriever wraps a query, a knn retriever wraps a vector search, and an rrf retriever fuses its children with Reciprocal Rank Fusion, rank_constant and rank_window_size as parameters:

POST /docs/_search
{
  "retriever": {
    "rrf": {
      "retrievers": [
        { "standard": { "query": { "match": { "text": "object storage" } } } },
        { "knn": { "field": "vec", "query_vector": [0.1, …], "k": 50, "num_candidates": 100 } }
      ],
      "rank_window_size": 50
    }
  },
  "size": 10
}

Why Loams speaks itCopy Anchor Link

The reasoning is the same as for Qdrant: people who try Loams already have code, and a lot of retrieval code is written against Elasticsearch through a framework. Loams' text search is BM25 over Tantivy splits (Tantivy and Quickwit's splits), with analyzers built to match Lucene's so rankings agree; being able to run a framework's Elasticsearch test suite unmodified against Loams is the cheapest way to prove that.

But "Elasticsearch compatible" is a trap if you mean it literally. The decision log narrowed it twice. D42 keeps a targeted Elasticsearch subset among the four surfaces Loams speaks. D48 then defines that subset by its senders, not by feature lists: Phase A is exactly what the LangChain and LlamaIndex Elasticsearch suites and BEIR send. Document APIs, _bulk, _search with the core Query DSL, knn, hybrid with RRF, _delete_by_query, the fixed script_score vector scripts and minimal index admin. Aggregations, point in time and highlighting were left out because no gated test needs them; _msearch stayed because BEIR sends every query through it. An earlier plan to gate on elasticsearch-py's own client suite, which needs a pile of cluster-management endpoints for its wipe_cluster fixture, was dropped by the same decision.

We think scoping by conformance suite is the honest way to do a compatibility layer. "We support the DSL" invites every edge case of a twenty-year-old product; "these suites pass unmodified, and here is the list of what differs" is something a user can check.

How Loams implements itCopy Anchor Link

loams-es is an axum server over CollectionService. It never touches storage or the metastore itself. Its modules follow the request path: http negotiates media types and sets the product header, names resolves index expressions and aliases, mapping turns Elasticsearch mappings into collection schemas, doc and write handle documents, bulk serves _bulk, dsl compiles the Query DSL, and search serves _search, _count and _msearch.

Negotiation. The gateway reports version 8.19.0 and sends X-Elastic-Product: Elasticsearch on every response, refusals included (D465): the pinned client checks the header and needs the 8.14+ hybrid surface. It accepts plain application/json and application/x-ndjson, and the vendor types with compatible-with=8, and answers in the vendor type when asked for it. A request for compatible-with=7 or 9, or an Accept and Content-Type that ask for different versions, is refused with a message naming the headers it got. A body with any other Content-Type is 406, as Elasticsearch does.

Mappings become schemas. Creating an index with mappings creates a collection: text fields become analyzed text fields in the splits, keyword, numbers, dates and booleans become typed columns, dense_vector becomes a vector field in Lance (the Lance post), and dynamic mapping adds fields as documents arrive. The shard count maps to collection partitions. What only Elasticsearch clients care about (the declared JSON of each field, similarity names, index settings) is kept as schema annotations, so GET /{index}/_mapping returns what you declared. Vectors are moved out of _source on write and put back on read, so a GET returns the document you indexed.

Bulk. _bulk parses the NDJSON, groups the items by index in order of first appearance, makes one write call per index, and puts the per-item results back at their request positions. A source line that is not JSON fails only its item. A batch refused for backpressure answers 429 es_rejected_execution_exception with Retry-After, which elastic-transport retries on its own (D86).

The DSL compiles to the search IR (D460). We parse the Query DSL ourselves into Loams' search IR, the same IR the Qdrant gateway and SQL table functions produce, which then plans onto DataFusion (DataFusion and Arrow). Quickwit has an Elasticsearch DSL parser, and we use it as the reference for request shapes plus two small helpers with their attribution, but its parser emits a different IR and omits forms the gated suites send. Supported in Phase A:

• Queries: match, match_phrase, multi_match, bool, term, terms, range, exists, prefix, wildcard, fuzzy, ids, query_string (simple), constant_score, match_all, match_none, and knn top-level or as a query.
• Hybrid: retriever with standard, knn and rrf, and the legacy rank: {"rrf": …} with a top-level query and knn.
• Vector scripts: script_score with the fixed cosine, dot-product and L2 sources that LangChain and elasticsearch-py's helpers send. These are recognised as text, not executed: there is no Painless.
• Body keys: from, size, sort, search_after (without a point in time), _source filtering, track_total_hits, track_scores, min_score, version, seq_no_primary_term, timeout.
• Documents and admin: _doc index, create, get and delete, _create, _update (partial document and upsert), _mget, _source, _delete_by_query and _update_by_query over the native filter writes (D87), index create, get, delete and exists, _mapping, aliases with at most one write index (D57), _refresh (a no-op), and synthetic GET /, _cluster/health and _license.

Auth. The engine's single authentication layer covers this listener. In apikey mode a Loams key is accepted as Authorization: Bearer, as the password of Basic auth, or in Elasticsearch's ApiKey scheme, raw or as the base64 id:key pair the clients build. So Elasticsearch(hosts=…, api_key="lk_…") works. As with Qdrant, the release stack keeps this listener inside the compose network for now; only the engine's main port is on your tailnet.

Namespaces. A Loams-Namespace header picks the namespace of a request; without it, the gateway's default namespace.

What is not supportedCopy Anchor Link

A search body key from the later phase is refused with a 400 illegal_argument_exception (Loams does not support [aggs] (Elasticsearch API Phase A)) rather than ignored, so a client never gets silently different results. Not served today:

• Search features: aggregations, highlighting, point in time, scroll, collapse, suggesters, post_filter, rescore, runtime_mappings, script_fields, stored_fields, docvalue_fields, fields, slice, indices_boost, terminate_after, explain and profile.
• Query types: nested, function_score (planned for M2 through the ranking-expression IR, D91), more_like_this, simple_query_string, retrievers other than standard, knn and rrf.
• Scripting: Painless, and script_score with any script other than the fixed vector sources. _update_by_query understands only params assignments and remove().
• Cluster and ops: _cat, _settings, index templates, ingest pipelines, ILM, the snapshot API (Loams has collection versions instead), cross-cluster search and replication, Kibana.

Aggregations and point in time are the first candidates for Phase B, as soon as a suite we gate on needs them. Design §06 §7 holds the full Phase A, Phase B and out-of-scope table.

GotchasCopy Anchor Link

The complete list of divergences from Elasticsearch 8.19 is the loams-es crate documentation. The ones that matter in practice:

• Wildcard deletes are refused. DELETE /* and DELETE /_all fail, as Elasticsearch 8 does by default with action.destructive_requires_name (D466). Delete by concrete name.
• Versions are not dense. _seq_no is the partition offset of the write's record and _version is _seq_no + 1. Both increase per document, but skip values.
• A multi-index search fails as a whole. Elasticsearch returns the hits of the shards that succeeded and reports the rest in _shards.failed; Loams fails the request if any member index fails (for example, a sort on a field one index does not map).
• Score ties break by _id only. A sort key other than _doc after _score is refused, and a _doc sort sorts by _id.
• Fieldless queries search fewer fields. A multi_match or query_string without fields searches the text (and, for multi_match, keyword) fields, not every field. query_string's lenient is accepted and not applied.
• Accepted, not enforced: keyword ignore_above, replica count, refresh interval, routing and preference. _refresh returns immediately because an acknowledged write is already searchable.
• Dates come in fewer formats. strict_date_optional_time, date_optional_time, the nanosecond variant and epoch_millis, and their combinations. Dynamic date detection recognises a smaller set than Elasticsearch.
• Error texts differ where Elasticsearch builds them from its internals: JSON syntax errors carry serde_json's text, not Jackson's, and parse locations are [1:1] rather than the value's line and column. Code that matches on error strings will notice; code that matches on type and status will not.
• No auth on a bare loams dev. Without LOAMS_AUTH_MODE, the engine runs unauthenticated and refuses to start unless every listener is on loopback. That is a development convenience, not a deployment mode.

The next part leaves the compatibility surfaces for the protocol Loams' own API is built on: Connect RPC.