Arrow Flight SQL and ADBC: SQL that answers in Arrow

Most database protocols were designed to move rows to a terminal. Arrow Flight was designed to move columns to a program. A client asks for a dataset, gets back a description and a ticket, and then pulls Arrow record batches over gRPC with no row-by-row decoding on either side. Flight SQL puts SQL on top of that, and ADBC puts a database-agnostic driver API on top of Flight SQL. Loams speaks all three, because the engine's query layer is already Arrow end to end.

What Flight isCopy Anchor Link

Flight is a gRPC service, arrow.flight.protocol.FlightService, with a small set of methods. The ones that matter here:

• Handshake, a bidirectional stream for an optional authentication exchange.
• GetFlightInfo(FlightDescriptor) → FlightInfo: "tell me about this dataset". The descriptor is either a PATH (a list of strings, like a file path) or a CMD (opaque bytes the server interprets).
• DoGet(Ticket) → stream FlightData: fetch the data a ticket names.
• DoPut(stream FlightData) → stream PutResult: upload a stream of batches; the first message carries the descriptor.
• DoAction and ListActions for anything else, plus GetSchema, PollFlightInfo, ListFlights and DoExchange.

The reply to GetFlightInfo holds the result's Arrow schema and a list of endpoints. Each endpoint has a ticket, which is an opaque token, and optionally a list of locations to fetch it from. An empty location list means "fetch it on the connection you already have". This is the part of Flight's design that makes it parallel: a server can split a result into several endpoints on several nodes, and a client can fetch them at the same time.

On the wire, each FlightData message is a protobuf with four fields: the descriptor (used on the first message of a DoPut), data_header, app_metadata and data_body. The header is an Arrow IPC message flatbuffer (a schema, a record batch or a dictionary batch), and the body is the column buffers that header describes. Arrow implementations put the body into gRPC's buffers without copying it into an intermediate protobuf, which is where Flight's throughput comes from: the bytes on the wire are the bytes in memory.

What Flight SQL addsCopy Anchor Link

Flight SQL is not a new RPC set. It is a convention for what goes in the descriptor and the ticket. Every request is a protobuf message from the arrow.flight.protocol.sql package, packed into a google.protobuf.Any and placed in a CMD descriptor:

• CommandStatementQuery { query } runs SQL. GetFlightInfo on it returns the schema and endpoints; the ticket inside is a TicketStatementQuery { statement_handle }, whose handle is the server's to define.
• CommandGetCatalogs, CommandGetDbSchemas, CommandGetTables, CommandGetTableTypes, CommandGetSqlInfo and the key and type-info commands are the metadata calls a JDBC-style driver needs. Each returns a result with a schema the spec fixes.
• CommandStatementUpdate, prepared statements (ActionCreatePreparedStatementRequest through DoAction), transactions and savepoints, and CommandStatementIngest for bulk loads through DoPut.

GetSqlInfo is how a server says what it supports: its name and version, whether it is read-only, whether it takes bulk ingestion, whether it supports transactions. Well-behaved clients read it before they try anything else.

Authentication is not fixed by the spec. A server can run a Handshake exchange, or read credentials from gRPC metadata on every call (an authorization header is the usual form). TLS is gRPC's: grpc+tls:// URIs in the drivers.

Where ADBC sitsCopy Anchor Link

ADBC (Arrow Database Connectivity) is an API, not a protocol. It is what JDBC and ODBC are for rows, but its result type is an Arrow stream. An application codes against ADBC's C API (or its Go, Java, Python, R or C# bindings), and a driver turns those calls into a specific database's protocol. There are drivers for PostgreSQL, SQLite, Snowflake, BigQuery and DuckDB, and one for Flight SQL, written in Go and exported through the C API. A Flight SQL server therefore gets an ADBC driver in every one of those languages without writing one.

ADBC also standardizes bulk ingest. adbc_ingest (in Python's DB-API layer) with a mode of create, append, replace or create_append becomes a Flight SQL CommandStatementIngest with table-definition options, sent through DoPut.

Why Loams speaks itCopy Anchor Link

The Loams engine plans every read as a DataFusion plan over Arrow batches (DataFusion and Arrow covers that layer). Returning those batches as JSON throws away the work. Flight SQL lets any language that has an ADBC or Flight SQL driver get the batches as they are.

D42 put Flight SQL in the narrowed protocol footprint alongside the native API, the Qdrant API and an Elasticsearch subset. D49 made it a core surface: Flight SQL with DoPut bulk ingest into both collections and streams, shipped together, with the ADBC Flight SQL drivers for Python and Go as an exit gate for the first milestone. The reasoning in D49 is short: zero-copy Arrow ingest and query for every language, and the high-throughput half of a native ingest path that replaces a Kafka gateway for most uses. D56 adds Spice: its Flight SQL connector is a client we test against, and Loams' SQL search functions use Spice's names where they overlap (vector_search, text_search, rrf).

The newest reason is BI. D1055 makes ADBC the way in for BI tools. Design 58 notes that Microsoft is moving Power BI and Fabric from embedded ODBC connectors to ADBC drivers, and that Dremio's replacement there is already the Flight SQL ADBC driver. If that is where BI connectivity goes, being a good Flight SQL server is how Loams gets there.

How Loams implements itCopy Anchor Link

The service is LoamsFlightSql in crates/loams-query/src/flight.rs, an implementation of arrow-flight's FlightSqlService trait. The loams binary serves it on its own listener, only on nodes with the gateway role, wrapped in the same authentication layer as the main port.

Statements. GetFlightInfo with CommandStatementQuery plans the SQL through the engine's read-only SQL catalog, so a typo fails at planning, not at fetch. It answers the schema and one endpoint with no location, so the client reads on the connection it has. The ticket is not a server-side cursor id. It is the whole statement: OPFS magic, a format version, the namespace, the SQL text, the consistency token and an optional pinned manifest version, encoded with postcard and closed with a CRC32C. DoGet decodes it, plans again and streams. Because the ticket carries everything, any node can serve it, and there is no cursor table to expire or replicate. The price is planning twice, which we pay gladly against a stateful cursor. A statement may plan and stream for at most 10 minutes.

Namespaces and consistency. Loams namespaces become Flight SQL catalogs, and collections are tables in a schema called collections. A request picks its namespace with the loams-namespace gRPC header (default when absent). A loams-consistency-token header makes the read wait for at least that write; without it reads are strong. With loams-pin-manifest as well, the read is pinned to that manifest version (consistency tokens explains the levels). The token and pin go into the ticket, so the DoGet reads exactly what the GetFlightInfo planned.

Metadata. GetCatalogs, GetDbSchemas, GetTables (with or without schemas), GetTableTypes (one type, TABLE) and GetSqlInfo are served. GetTables lists collections, not aliases. GetSqlInfo declares the server name loams, read-only SQL, bulk ingestion supported, no ingest transactions and no transactions.

Ingest. DoPut takes two forms. ADBC's CommandStatementIngest names a collection or a stream and an ingest mode, which flight_ingest.rs maps onto create, append or refuse. A bare PATH descriptor works too: ["collections", name], ["streams", name] or ["streams", name, partition]. Columns are mapped once per stream schema (_id, _source, vector columns and the rest), and each batch is cut into chunks of 10,000 rows by default, each written whole or not at all through the same write path as the native API. A path-descriptor put streams back one PutResult per record batch, whose app_metadata is a JSON acknowledgement: the batch index, its rows, the merged consistency token so far and, for streams, the offsets appended per partition. An ADBC ingest answers the row count. Under write backpressure the server answers RESOURCE_EXHAUSTED with a retry-after-ms header; a bulk loader may send loams-backpressure: off to opt out of the plain budget. Messages are capped at 64 MiB.

SQL stays read-only. CREATE TABLE, DROP and INSERT over Flight SQL are INVALID_ARGUMENT, and the way to write is ingest. That is deliberate: a collection's write path has its own semantics (ids, upserts, vectors), and SQL DML over it would be a second, weaker way to say the same thing.

What is not served. Prepared statements (UNIMPLEMENTED, which a test asserts), CommandStatementUpdate, transactions, substrait plans, PollFlightInfo and multi-endpoint results. Handshake returns an empty response: authentication is per call, from metadata, by the same layer as the main port. In API-key mode that is Authorization: Bearer lk_….

Tests and clients. The integration suites in crates/loams/tests/it/flight_sql.rs and flight_ingest.rs drive the server with arrow-flight's own Rust FlightSqlServiceClient 58.4: statements, catalogs, namespaces from metadata, consistency tokens, ADBC-style creates and appends, the refusals. The Python SDK's optional flight extra depends on adbc-driver-flightsql, and the engine quickstart connects with it. The milestone exit-gate plan pins adbc-driver-flightsql 1.12.0 (ADBC release 24) for Python, runs ADBC's own Go validation suites through the Go Flight SQL driver at v1.12.0, and adds Spice 2.3.2 as a client. That gate is planned in the repository, not yet in CI, so the honest statement today is: tested with the Rust Flight SQL client, documented with the Python ADBC driver.

What is plannedCopy Anchor Link

Design 58 and D1056 lay out the next step:

• adbc_driver_loams, a thin driver over the Arrow project's Flight SQL driver that adds Loams credentials (API keys, OIDC tokens), namespace and branch selection and consistency tokens as connection options, for every language the ADBC driver manager supports.
• A Power Query custom connector for Power BI and Fabric over the Flight SQL ADBC driver.
• ADBC 1.1 features on the server: statistics from the manifest and from Lance and Iceberg metadata, incremental execution, partitioned results (several endpoints a client fetches in parallel, mapped onto the scan-plan fragments of D53), cancellation and progress.
• One port. D601 proposes mounting the gRPC compatibility services, Flight SQL included, on the main port by service name, keeping :8082 as an alias.
• The House over Flight SQL. The House plan adds a Flight SQL front to Loams House, through the same checks as its HTTP interface, so Arrow clients can read ClickHouse-dialect lake tables without a ClickHouse driver (the ClickHouse HTTP post has the House side).

GotchasCopy Anchor Link

• Read-only means read-only. A tool that issues CREATE TEMP TABLE or INSERT as part of its workflow will fail. Use ADBC ingest for writes.
• Prepared statements are a common default. Some drivers and BI tools prepare every statement, even without parameters. Against Loams those calls answer UNIMPLEMENTED. With the ADBC Flight SQL driver, a plain execute of a statement without parameters does not prepare.
• default is a SQL keyword. The default namespace is the catalog default, which DataFusion's parser treats as a keyword. Quote it ("default".collections.kb) when you qualify names, as the Spice gate's plan notes.
• Set the namespace as a header. It is not part of the URI. With ADBC, set adbc.flight.sql.rpc.call_header.loams-namespace (the driver's per-call header option; header names must be lower case); with a raw Flight client, add loams-namespace to the call metadata.
• Do not rely on Handshake for auth. It returns nothing useful. Send the credential on every call: the ADBC Flight SQL driver's adbc.flight.sql.authorization_header option does that: it sets the value of the authorization header sent on all requests.
• Large rows hit the message cap. A single batch whose encoded message exceeds 64 MiB is refused. Send smaller batches; Loams cuts them into 10,000-row chunks on its side anyway.
• Do not expose the listener yet. In 0.1.0, the Flight SQL listener stays inside the compose network; the release notes ask you not to publish it until every listener is reviewed against the auth modes. Reach it over the tailnet or from inside the stack.

The Postgres and MySQL wire protocols cover clients that want rows and a familiar dialect (part 1, part 2). Flight SQL is for clients that want columns, and for anything that loads data in bulk.