Connect RPC: one handler for gRPC, gRPC-Web and plain HTTP

Every application call to Loams, from the console, the desktop app, an SDK or curl, is a remote procedure call defined in protobuf and served by one HTTP handler. That handler speaks three wire protocols at once: gRPC, gRPC-Web and Connect. This post is about the third one, and about why we chose to serve all three rather than pick one.

Three protocols, one envelopeCopy Anchor Link

gRPC was designed for HTTP/2 and leans on two HTTP/2 features: long-lived bidirectional streams, and trailers, headers sent after the body. A gRPC call is a POST to /<package>.<Service>/<Method> with content-type: application/grpc. The request and response bodies are sequences of length-prefixed messages: one flags byte (bit 0 says the message is compressed), a four-byte big-endian length, then the encoded protobuf. The deadline travels in grpc-timeout (an integer of up to eight digits and a one-letter unit: H hours, M minutes, S seconds, m milliseconds, u microseconds or n nanoseconds; the letters are case-sensitive). The outcome arrives last, in trailers: grpc-status with one of the sixteen canonical codes, and optionally grpc-message. The HTTP status is 200 even when the call fails.

That design is efficient and hard to reach from a browser. fetch gives JavaScript no access to HTTP/2 trailers, and many proxies and load balancers drop or mangle them. gRPC-Web fixes this by moving the trailers into the body. It keeps the same five-byte envelope, but a frame whose flags byte has the most significant bit set (0x80, or 0x81 if compressed) is a trailer frame, whose payload is the trailers written as an HTTP/1-style header block (name: value lines, with lower-case names). Content types are application/grpc-web+proto, application/grpc-web+json, or the -text variants that base64 the whole body for environments that cannot read binary responses. gRPC-Web works over HTTP/1.1 as well as HTTP/2, but its browser clients support only unary and server-streaming calls.

Connect goes one step further. It is a protocol from the Buf team that keeps protobuf schemas and gRPC's semantics (the same codes, the same streaming shapes) but makes the common case ordinary HTTP:

• Unary calls are plain requests. A POST to /<package>.<Service>/<Method> with content-type: application/json or application/proto, and the message as the whole body, with no envelope. Clients send Connect-Protocol-Version: 1. A successful response is HTTP 200 with the response message as the body.
• Errors are HTTP errors. A failed unary call gets a real HTTP status (not_found is 404, unavailable 503, unimplemented 501, resource_exhausted 429) and a JSON body: {"code": "...", "message": "...", "details": [{"type": "...", "value": "<base64>"}]}.
• Deadlines are Connect-Timeout-Ms, a positive integer of at most ten digits.
• Side-effect-free calls can be GETs. A method marked idempotency_level = NO_SIDE_EFFECTS may be called with GET and the message in the query string (message, encoding, base64, compression, connect=v1), which makes it cacheable by ordinary HTTP caches.
• Streams use the envelope. Streaming calls use application/connect+proto or application/connect+json and the same five-byte framing. Flag 0x01 means compressed; flag 0x02 marks the end-of-stream message, a JSON object carrying the error (if any) and trailing metadata. Streaming responses are always HTTP 200; the status is in that last frame.

So the three protocols are less different than their names suggest. They share a path scheme, a schema language, a set of status codes and a framing. They disagree about where the status goes (trailers or body) and whether unary calls use the envelope at all. A server that understands all three can decide from the content-type header alone which one a request speaks.

Handshake and authCopy Anchor Link

None of the three has a handshake of its own. There is no connection setup beyond HTTP's, and no session. Authentication is whatever HTTP header the server checks; in Loams that is authorization: Bearer <token> with an API key or an OIDC access token, or the identity the tailnet sidecar adds (see auth modes in the preview notes). Metadata is HTTP headers in all three protocols; gRPC-Web and Connect streaming responses carry trailing metadata in the body.

Why Loams speaks itCopy Anchor Link

We started the way most Rust services do: an axum REST API for the engine, and tonic for the one gRPC streaming service. Then three things happened in the same week.

The reactive database, Loams Live, needed a sync API with a server stream and unary mutations that worked from browsers, phones and servers. D121 put it on connect-rust because a server stream plus unary calls works over HTTP/1.1 and HTTP/2 and over all three protocols, which bidirectional streaming does not. Then the desktop and mobile apps (D420) needed one contract, generated into Swift, Kotlin and TypeScript. Then we set one direction for the whole API: "one API": every application call is an RPC in a loams.<service>.v1 package, served by the single loams binary on one port (D600).

Connect is the protocol that makes "one API" work without a second REST layer:

• curl works. A unary Connect call is a JSON POST. There is nothing to generate before you can try the API, and nothing to keep in step with the protos.
• Browsers work without a proxy. gRPC-Web needs an Envoy or a translating middleware in front of most gRPC servers. Connect needs nothing, and connect-rust serves gRPC-Web itself anyway.
• gRPC still works. Languages whose best client library is gRPC (Java, Ruby, PHP, Objective-C in our matrix, D612) call the same port with stock gRPC stubs, and grpcurl and Postman work against reflection.
• One schema governs everything. The protos under proto/loams/ generate the server traits, every SDK, and the reference docs. buf breaking checks compatibility on every change.

We wrote down the rule as D362: Connect-RPC through connect-rust for every new service. At the time we checked, connectrpc 0.9.1 was the official Connect project's Rust implementation and passed the full Connect conformance suite (3,600 server tests). Earlier drafts kept a fallback of tonic plus a hand-written Connect codec; it was not needed. The library itself, and buffa, are covered in the small crates post; here we only care about the wire.

How Loams implements itCopy Anchor Link

The main port's Connect wiring lives in crates/loams/src/api/connect.rs. It builds one connectrpc::Router, registers each service on it (instance, collections, documents, query, auth, admin, Live, Graph, repositories), installs grpc.health.v1, and, when reflection is on, grpc.reflection.v1. It then mounts every RPC path as its own axum route, beside the native REST routes that are being retired.

Some details that matter to a caller:

• Messages and stubs come from loams-proto, generated in build.rs with connectrpc-build from the protos under proto/loams/. The descriptor set is compiled into the binary for reflection.
• Reflection is on in loams dev and off by default elsewhere, because it publishes the API's schema to anyone who can reach the port. ListServices lists only what this process actually answers.
• Errors carry a reason. Every refusal is a Connect error with a loams.errors.v1.ErrorInfo detail whose reason is a stable snake_case string, which is what SDKs branch on. A service whose engine is not compiled into this build answers unimplemented with reason feature_not_in_variant and the variant in metadata, so a caller can tell "not in this build" from "not written yet" (not_implemented).
• Feature detection is one call: loams.instance.v1.InstanceService/GetInstance needs no auth, is marked NO_SIDE_EFFECTS, and lists the services this binary serves.

curl -s https://loams.<tailnet>.ts.net/loams.instance.v1.InstanceService/GetInstance \
  -H 'content-type: application/json' -H 'connect-protocol-version: 1' -d '{}'

The same pattern runs outside the engine. loams-agentd-connect serves loams.agentd.v1, the surface Loams Studio uses to drive the agent daemon, on a loopback Connect listener beside the daemon's newline-delimited JSON wire (D907). The web console and Studio use connect-es and connect-query in the browser.

What we support, and what we do notCopy Anchor Link

No client or bidi streaming. This is a choice, not a gap. Browsers cannot send a streaming request body in practice, URLSession cannot do full-duplex gRPC, and half-duplex calls pass through every proxy. connect-rust also sends no response on HTTP/1.1 until the request body ends, so over HTTP/1.1 a bidi call degrades to "send everything, then read everything", which is not what anyone calling a bidi method expects. Bulk upload that would want a client stream goes through Arrow Flight's DoPut (see part 3) or chunked unary writes.

SDKsCopy Anchor Link

Design 44 chooses a protocol per language: Connect where an official Connect library exists, gRPC elsewhere (D612). The SDKs then taught us that "official library" is not the only question. The C# SDK was planned on Grpc.Net.Client, which speaks gRPC and gRPC-Web but not Connect; the shared conformance corpus records 13 of its 28 required fixtures in Connect encodings, including the only case that proves a refusal arriving inside a stream envelope rather than as an HTTP status. So C# speaks Connect over System.Net.Http with a hand-written transport (D650). The C++ SDK does the same over HTTP/1.1 rather than depend on grpc++ (D639). Every SDK replays the fixture corpus in sdks/fixtures/ against a real loams dev, and a language ships only at 100% of required fixtures (D617).

GotchasCopy Anchor Link

• application/json is ambiguous. It is a Connect unary content type and the native REST content type. Middleware that shapes errors by content type has to know which router it sits in. We hit this with the hot-tier header (Loams-Hot): its refusal is applied inside the Connect router, where there is no ambiguity, rather than around the whole port.
• HTTP status is not the outcome for streams. A Connect or gRPC-Web stream that fails after it started still has status 200; the error is in the end-of-stream frame. Clients that only check response.ok miss it. Our corpus has a fixture for exactly this.
• Proxies and trailers. If you put a plain HTTP/1.1 proxy in front of the main port, gRPC (which needs HTTP/2 trailers) can break while Connect and gRPC-Web keep working. That is the main reason to prefer Connect from anything that is not a native gRPC stack.
• Timeouts have two spellings. gRPC and gRPC-Web clients send grpc-timeout with a unit (5S); Connect uses Connect-Timeout-Ms (5000, up to ten digits). Generated clients handle this; hand-written ones are where it goes wrong.
• Errors in JSON use string codes, "not_found", not the integer 5 gRPC uses. Details are base64 protobuf even in a JSON error; decode value with the type named in type. The C# SDK's base64 decoder accepts input with or without = padding for this reason (D646).
• Reflection is a disclosure. Leave it off on anything reachable from outside your tailnet, which is the default.

Next in the series: the S3 API and the bucket contract. For how the CloudEvents HTTP binding sits beside the Connect stream API, see part 15.