OTLP: OpenTelemetry logs into Loams streams

Planned Loams does not accept OTLP yet. There is no OTLP receiver in the engine or the Fabric on dev. This post explains the protocol and the two designs that use it.

Every log shipper worth using speaks OTLP now: the OpenTelemetry Collector, Fluent Bit's opentelemetry output, Vector's opentelemetry sink, and the OpenTelemetry SDKs in every language. A system that accepts OTLP gets logs from all of them without writing a plugin for any of them. That is the whole argument for it.

What the protocol isCopy Anchor Link

OTLP is protobuf messages over two transports, with one request/response pair per signal.

The messagesCopy Anchor Link

Each signal has a collector service with one RPC, Export:

• opentelemetry.proto.collector.logs.v1.LogsService/Export(ExportLogsServiceRequest)
• …trace.v1.TraceService/Export(ExportTraceServiceRequest)
• …metrics.v1.MetricsService/Export(ExportMetricsServiceRequest)

The request nests: a list of ResourceLogs, each with a Resource (attributes that describe the emitter, such as service.name and host.name) and a list of ScopeLogs, each with an instrumentation scope and a list of LogRecords. A LogRecord carries time_unix_nano, observed_time_unix_nano, a severity number and text, a body (an AnyValue: string, number, bool, bytes, array or map), its own attributes, and optionally the 16-byte trace id and 8-byte span id of the span it was logged in. Grouping by resource and scope is a compression trick: the attributes that every record from one process shares are sent once.

OTLP/gRPCCopy Anchor Link

A unary gRPC call to Export on port 4317 by default, optionally gzip-compressed. Authentication is whatever the receiver requires in metadata, typically authorization: Bearer …, over TLS.

OTLP/HTTPCopy Anchor Link

A POST to /v1/logs, /v1/traces or /v1/metrics on port 4318 by default, with either:

• Content-Type: application/x-protobuf, the binary protobuf of the same request message; or
• Content-Type: application/json, the protobuf JSON mapping, with OTLP's own exceptions: trace and span ids are hex strings, not base64, and enum values are integers.

Content-Encoding: gzip is allowed. The answer is the same Export…ServiceResponse, in the request's encoding.

Success, partial success, and retriesCopy Anchor Link

This is where receivers go wrong, so it is worth being exact.

• Full success. HTTP 200 or gRPC OK, with an empty response.
• Partial success. Still a success status, with partial_success set: rejected_log_records (or spans, or data points) and an error_message. The spec says the client must not retry the request; the rejected items were rejected for a reason that a retry will not fix, such as being malformed. A receiver may also set partial_success with zero rejections and a message, as a warning.
• Retryable failure. The spec's retry tables say, for gRPC, UNAVAILABLE, CANCELLED, DEADLINE_EXCEEDED, ABORTED, OUT_OF_RANGE and DATA_LOSS are retryable, and RESOURCE_EXHAUSTED is retryable only when the server includes RetryInfo saying recovery is possible. For HTTP, 429, 502, 503 and 504 are retryable. The server signals how long to wait with RetryInfo (gRPC) or Retry-After (HTTP), and the client should honour it, with exponential backoff otherwise.
• Anything else is not retryable, and the client drops the data.

OTLP has no request ids and no producer sequence. A client that retries after a response was lost sends the same data again, and the receiver has no way to know. OTLP delivery is at-least-once by design.

Why Loams would speak itCopy Anchor Link

D73 put it simply: every log shipper speaks OTLP, and mapping it onto streams reuses the stream API core with no plugin of ours. Once logs are in a stream, a stream-to-collection link makes them searchable through the native API and the Qdrant and Elasticsearch surfaces within seconds. Logs are the most common thing teams want to put next to their AI application's data, and the cheapest one to get there.

Our decision on two logs (D331) keeps OTLP logs on the Loams WAL and streams, not on the Event Fabric's Iggy log: they are retrieval data, searched next to everything else, rather than high-rate event ingestion.

The planCopy Anchor Link

There are two designs, and they are complementary rather than competing.

The engine endpoint (design 02 §7.1, D73)Copy Anchor Link

• Transports. POST /v1/logs with protobuf or JSON, gzip allowed; OTLP/gRPC LogsService/Export. The OTLP default ports, 4318 and 4317.
• Target. The namespace and stream come from the loams-namespace and loams-stream headers (gRPC metadata), which every OTLP exporter can set; they default to the API key's namespace and a stream named otel_logs. The API key goes in Authorization: Bearer.
• Mapping. Each LogRecord becomes one stream record. The value is one self-contained JSON object: the record in OTLP's JSON encoding with its resource and scope attributes merged in, so a reader never needs the request it came from. The timestamp is time_unix_nano, else observed_time_unix_nano, else arrival time. The key is empty (round-robin) unless the stream names a key attribute such as service.name, which keeps one service's logs in order. Trace id, span id and severity number are copied into record headers too, so a link can filter without parsing the value.
• Acknowledgement. A request succeeds once all its records are committed to the log. Records that fail validation are reported in partial_success. Over quota, the endpoint answers HTTP 429 or gRPC RESOURCE_EXHAUSTED with a retry delay, which exporters retry.
• Not in scope. Traces and metrics (Q43 asks about traces for agent and durable-step spans). The first version stores JSON values in kafka-encoded streams; Arrow-encoded OTLP streams wait for the arrow stream encoding.

The no-code integrations this enables are listed in design 02 §7.3: Fluent Bit's opentelemetry output, the Collector's otlp and otlphttp exporters, and Vector's opentelemetry sink.

The Fabric receiver (design 33, connector 21)Copy Anchor Link

Design 33's first-party connector set includes an OpenTelemetry source: an OTLP/HTTP and OTLP/gRPC receiver in loams-fabric ingest that turns each log record, span or metric point into one CloudEvent (type io.loams.dev.flow.opentelemetry.<signal>.v1), at least once. Its manifest is already in the repository at connectors/registry/opentelemetry.yaml, declaring key and mTLS auth and OTLP's protobuf and JSON encodings; the receiver code it points to is not on dev yet. Until the unified auth plan exposes it through the edge, it listens on the ingest service's loopback listener only (design 33 §2.2).

The engine endpoint is the path for logs you want to search; the Fabric receiver is the path for telemetry that flows on into other sinks.

GotchasCopy Anchor Link

• Duplicates are normal. Because OTLP has no idempotency key, a receiver that answers slowly gets the same batch twice. Anything downstream that counts logs must tolerate it, or deduplicate on content.
• Never answer partial success for a transient problem. The client will not retry it, so the data is gone. Over quota is a 429 or RESOURCE_EXHAUSTED with retry information, not a partial rejection.
• JSON is not plain protobuf JSON. Hex ids and integer enums are OTLP's rules. A receiver built on a generic protobuf JSON parser misreads trace ids.
• Resource attributes are the useful ones. service.name lives on the resource, not on the record. A receiver that keeps only the LogRecord loses the field everyone filters by, which is why the plan merges resource and scope attributes into every record.
• Timestamps can be zero. time_unix_nano is optional; observed_time_unix_nano is the collector's clock. Pick a rule and document it, as the plan does.
• Bodies are not always strings. AnyValue can be a map. Log search should index structured bodies, not only text.

Read nextCopy Anchor Link

• Kafka wire protocol, the other planned way into streams.
• CloudEvents, the envelope the Fabric receiver emits.
• Elasticsearch API and Query DSL, one of the surfaces logs become searchable through.