The Kafka wire protocol: what it takes to be a broker

Planned No Kafka client can connect to Loams today. This post explains the protocol, the one piece of it Loams already uses, and the plan, which changed in October.

The Kafka protocol is the de facto API of event streaming. RisingWave, Flink, Spark Structured Streaming, Kafka Connect, Debezium, Fluent Bit and Vector all speak it, and so do WarpStream, AutoMQ, Bufstream and Redpanda, which are Kafka-compatible brokers built on other storage. If a log speaks Kafka, that whole ecosystem can read and write it with no code of ours.

What the protocol isCopy Anchor Link

FramingCopy Anchor Link

Kafka is a binary request/response protocol over TCP. Every message, in both directions, starts with a 4-byte big-endian size, followed by that many bytes. A request's bytes start with a request header:

• request_api_key (int16): which API, such as 0 for Produce, 1 for Fetch, 3 for Metadata, 18 for ApiVersions;
• request_api_version (int16): which version of that API's schema the body uses;
• correlation_id (int32): echoed in the response, so a client can pipeline requests on one connection and match the answers;
• client_id (nullable string): a free-form name, used for quotas and logs.

The response header is just the correlation_id. Newer flexible versions (KIP-482) switch to header v2 for requests and v1 for responses, which add tagged fields: optional fields keyed by a varint tag that old readers skip. Flexible versions also encode strings and arrays with varint ("compact") lengths.

Every API is versioned independently, and every field of every schema says which versions it exists in. That is how Kafka has evolved for over a decade without a flag day, and it is why the protocol is large: a broker that wants to serve the Java client, librdkafka, franz-go and Sarama has to accept a range of versions per API, not one.

The handshake: ApiVersionsCopy Anchor Link

A client does not guess what the broker speaks. Its first request on a connection is ApiVersions (key 18), and the answer lists, for each API key the broker supports, the minimum and maximum version. The client then uses, for each API, the highest version both sides know. ApiVersions has a special rule so this can work at all: the broker always answers it with response header v0, even when the request used a version the broker does not support, so an old broker can still tell a new client what it does speak.

Apache Kafka 4.0 removed the oldest versions of many client-facing APIs (KIP-896: for example Produce v0 to v2 and Fetch v0 to v3 are gone, and a broker answers UNSUPPORTED_VERSION for them), so a new broker can start its supported ranges at the versions current clients actually send. Metadata, FindCoordinator, Heartbeat, SaslHandshake and ApiVersions keep v0.

Authentication: SASLCopy Anchor Link

Kafka authenticates with SASL, usually over TLS (SASL_SSL). After ApiVersions, the client sends SaslHandshake (key 17) naming a mechanism (PLAIN, SCRAM-SHA-256, SCRAM-SHA-512, OAUTHBEARER, or GSSAPI for Kerberos), and then exchanges the mechanism's tokens inside SaslAuthenticate (key 36) requests, which carry them as ordinary framed Kafka messages (KIP-152). Until authentication finishes, a broker refuses everything else. Authorization is per resource (topic, group, cluster) by ACLs in Apache Kafka, which a compatible broker maps onto its own model.

Metadata, Produce, FetchCopy Anchor Link

• Metadata (3) tells the client which brokers exist and which broker leads each partition of the topics it asked for. Kafka clients send each partition's produce and fetch requests to that partition's leader.
• Produce (0) carries, per topic and partition, one or more record batches, and an acks setting (0, 1, or -1 for all in-sync replicas). The answer gives each partition's base offset or an error code.
• Fetch (1) asks, per partition, for records from an offset, with max_wait_ms and min_bytes so a consumer long-polls. The answer is the stored record batches, as bytes.
• ListOffsets (2) maps a timestamp, earliest or latest to an offset.

Consumer groups add a coordinator: FindCoordinator (10), then JoinGroup (11), SyncGroup (14) and Heartbeat (12) in the classic protocol, where one member computes the assignment; or ConsumerGroupHeartbeat (68) in the new protocol of KIP-848, where the broker assigns. Committed positions go through OffsetCommit (8) and OffsetFetch (9). Idempotent producers start with InitProducerId (22), which returns a producer id and epoch.

Record batch v2Copy Anchor Link

Records travel and rest in record batches. The v2 header carries baseOffset, batchLength, partitionLeaderEpoch, magic (2), a CRC-32C, attributes (compression codec, timestamp type, transactional and control bits), lastOffsetDelta, the first and max timestamps, and producerId, producerEpoch and baseSequence for idempotence. Then come the records, each a varint-framed tuple of attributes, timestamp delta, offset delta, key, value and headers.

Two details matter to anyone storing these bytes. The CRC covers the bytes from attributes to the end, so baseOffset and partitionLeaderEpoch can be rewritten without recomputing it. And the producer's (producerId, epoch, baseSequence) is what a broker deduplicates on: a retried batch with a sequence it has already seen is acknowledged with its original offsets and not appended again.

Why Loams would speak itCopy Anchor Link

Loams streams are partitioned logs on object storage (see the log is the spine). The stream engine design calls Kafka "Loams' long-term source-compatibility protocol" (design 02 §7.2), and WarpStream, AutoMQ and Bufstream show the model: a Kafka-compatible log whose brokers are stateless because the data is in a bucket.

The reason it looked cheap for us is that the hard part is already done. loams-log writes Kafka RecordBatch v2 as the default kafka encoding of every stream: magic 2, written with baseOffset = 0, and readers and the segmenter patch bytes 0 to 8 with the absolute offset, which is free because those bytes sit outside the CRC. The WAL and the segments keep the batch bytes verbatim. So a Kafka Produce could validate the client's batch, let the sequencer assign offsets and append the bytes as they are; a Fetch could return the stored bytes with the right baseOffset. Nothing would be re-encoded. And the native stream API's idempotent producers already dedupe by (producer_id, epoch, sequence), which is the same rule Kafka's InitProducerId sets up.

The plan, and how it changedCopy Anchor Link

D74 (2026-09-26) put a Kafka gateway in milestone M5, staged:

1. ApiVersions, Metadata, Produce, Fetch and ListOffsets, with SASL over TLS, and the topic admin calls that the gated tools need, mapped onto stream admin.
2. Idempotent producers: InitProducerId mapped onto the native producer ids and epochs, so the sequencer's dedupe applies unchanged.
3. Consumer groups with committed offsets: the classic group protocol first, then KIP-848. A group's coordinator would run on the rendezvous owner of (namespace, group), its offsets in the metastore, and commits conditional on the group's generation so a stale coordinator cannot commit.

No Kafka transactions (transactional ids, AddPartitionsToTxn, EndTxn, read-committed fetches) in that milestone; they come on demand (Q4). The gate was the Kafka client test suites (librdkafka, franz-go or the Java client) passing stage by stage, plus RisingWave, Flink and Kafka Connect end to end.

Two design points from that plan are worth keeping whoever builds the front end:

• Leaderless Metadata. Any Loams log node accepts produce and fetch for any partition, so Metadata can name a node in the client's own zone as the "leader" of every partition, as WarpStream does. A Metadata request carries no rack, though: client.rack reaches a broker only in a consumer's Fetch (v11 and later, KIP-392), and producers never send it. The zone has to come from the client id (WarpStream's ws_az=<zone> convention) or from a per-zone bootstrap address, or traffic may cross zones.
• Topic and credential mapping. Whether a topic is named <namespace>.<stream> or the namespace is bound to the credential, and which SASL mechanisms carry a Loams API key, is still open (Q26).

D408 (2026-10-02) deferred the Loams-side gateway, without cancelling it. Our direction is two logs chosen by workload (D331): the Loams WAL and streams stay the log for databases and retrieval (Postgres, Loams SQL, collections, graphs, OTLP logs, trigger-rate CloudEvents), and high-rate event ingestion goes to the Event Fabric on Apache Iggy and Apache Fluss (D332, D405). Kafka clients of the Fabric use Iggy's Kafka gateway, and Loams contributes to it upstream (offset commit and fetch, produce-path hardening, SASL mapped to Iggy users), along with a Confluent Schema Registry REST subset (Q344). The gateway in the loams binary comes back only if Iggy's cannot serve consumer groups with committed offsets.

We think that is the right call. A Kafka front end is a long tail of versions, error codes and client quirks, and two of them in one product would split the testing. When D332 was written, Iggy's Kafka gateway was unreleased and had no offset commit, which is exactly the work we would have done anyway.

What exists today on the Kafka side is the other direction: loams-connector-kafka is a Kafka client, a first-party connector on librdkafka that reads topics into the Fabric (consumer-group offsets committed after the Fabric acknowledges) and writes to them with the idempotent producer or Kafka transactions, against Apache Kafka, Redpanda and Azure Event Hubs' Kafka endpoint. It is in progress and not in the release images (preview status).

Gotchas for any Kafka-compatible brokerCopy Anchor Link

• Versions are per API, not per protocol. "Supports Kafka 3.x" means nothing on the wire; the ApiVersions answer is the contract, and every client picks its own maximum. Test with several clients.
• Error codes drive client behaviour. NOT_LEADER_OR_FOLLOWER makes a client refresh metadata, OUT_OF_ORDER_SEQUENCE_NUMBER and INVALID_PRODUCER_EPOCH make an idempotent producer stop or reset. A broker that returns the wrong code gets retry storms or silent duplicates.
• Offsets are dense and never reused. Kafka clients assume offset n + 1 follows n within a partition (control batches excepted). Loams offsets are dense, which is why the mapping is direct.
• acks has to mean something. Clients set acks=1 for speed and acks=-1 for safety. Loams streams choose durability per stream through WAL classes, and the plan does not yet say how acks maps onto them; a front end has to decide, and document it, before anyone trusts it.
• Compression is inside the batch. The attributes name gzip, snappy, lz4 or zstd for the whole record set, so a broker that validates batches has to decompress them, or trust them.
• No rack on Produce. Zone affinity for producers needs a convention outside the protocol.

Read nextCopy Anchor Link

• CloudEvents, which Loams streams store in the Kafka binding's binary layout, so an event stream is also a valid Kafka topic of CloudEvents.
• OTLP, the other planned ingest protocol for streams.
• TiKV and PD and RustFS for the storage under the log.