Namespaces and tenancy

A namespace is the container for everything else in the Loams engine: its streams, collections, links and, later, tables and graphs. Every API path starts with one (/v1/namespaces/acme/…), every object it holds lives under its own bucket prefix, and placement, caching and (as designed) quotas, roles and encryption keys are all per namespace.

Why we built it this wayCopy Anchor Link

Multi-tenant systems usually bolt tenancy on: a tenant_id column, a filter added to every query, and the hope that nobody forgets it. Then one large tenant shares a cache, a shard and a noisy neighbour with everyone else.

We chose to make the namespace the unit of everything from the first design (principle 5 of the architecture): data layout, metadata, caching, routing, and the usage you could attribute to a team. Two reasons:

• Isolation by construction. Two namespaces never share a manifest, an index, a primary-key index or a cache key. A query in one cannot see the other's data, because it never opens those objects, not because a filter removed them.
• A cold namespace is nearly free. Because the bucket holds the data and nodes hold only caches, an idle namespace costs its bucket bytes and a few kilobytes of metadata. That is what makes one namespace per user, per agent or per customer a reasonable default rather than an extravagance.

Our opinion: if tenants are cheap, give each one its own namespace. Use filters for slicing inside one tenant's data, not for separating tenants.

How it worksCopy Anchor Link

Layout. A namespace's objects live under ns/<namespace_id>/: its stream segments, its collections' Lance datasets, splits, delete bitmaps and manifests, and its primary-key indexes. The one shared prefix is wal/, because one WAL object batches records from many namespaces to keep request counts down (D25); the segmenter moves each namespace's records into its own segments within seconds.

Metadata. The metastore keys namespaces, schemas, streams, collections and links by namespace id. Names are resolved to ids once, so renaming never touches data.

Placement. Collections are placed on query nodes by rendezvous hashing over (namespace, collection). The design adds size classes: small namespaces are placed by namespace alone, so one node warms all of a small tenant's collections together, and very large collections are placed by shard.

Fair share in the hot tier. Each namespace's share of a node's RAM and NVMe budget is the budget divided by the namespaces that hold or want hot structures. A namespace over its share is evicted from first.

Backpressure per collection. Each collection has its own unapplied-data budget; one tenant's bulk load fills its own budget, not a shared queue.

What is designed on top (the M2 milestone, which the release table lists as planned):

• Organizations own namespaces, and API keys, role bindings and quotas live in a separate control store, so that a hosted control plane can serve them for self-hosted clusters.
• Roles grant read, write and admin on namespaces and collections, behind an Authorizer interface, with fine-grained grants through OpenFGA later. Every grant is fenced to its tenant, so no grant can make a subject from another organization effective.
• Quotas on request rate, ingest bytes, concurrent queries, storage bytes and metadata operations, each refused with the protocol's own throttling error.
• Customer-managed keys: a namespace may name a KMS key at creation. Its objects are written under that key, and its chunks inside shared WAL objects are envelope-encrypted with per-chunk data keys. Destroying the key makes the namespace unreadable (crypto-shredding).

What we give upCopy Anchor Link

• Cross-namespace queries are not a thing. A query reads one namespace. If you need to search across tenants, that is one namespace with a filter, by design.
• Coarse authorization today. In Loams 0.1.0, a valid credential can do anything the engine can do. There are no per-namespace roles or quotas yet. Use separate stacks if you need hard separation between people now.
• Metadata grows with namespaces. Every node holds the catalog today. Scaling to millions of namespaces needs nodes to hold only what they own and a metastore sharded by namespace; both are designed and staged, not built.

How other systems do itCopy Anchor Link

• turbopuffer makes the namespace its unit of isolation and cost in the same way; cold namespaces live only in object storage.
• Elasticsearch offers an index per tenant (each with shard overhead) or a shared index with filtered aliases; many tenants per cluster is a known pain point.
• Qdrant recommends one collection with a payload-indexed tenant field for many small tenants, and shard keys for large ones.
• Postgres offers schemas and row-level security; isolation is by policy inside one database.

StatusCopy Anchor Link

Available Namespaces, the per-namespace bucket layout, per-namespace metadata, placement and hot-tier fair share ship in Loams 0.1.0. Organizations, roles, per-namespace authorization, tenant quotas, the namespace router and customer-managed keys are Planned (the release table's "Authorization and tenancy" row).

Go deeperCopy Anchor Link

• Security model: what protects a deployment today.
• Cluster mode: placement in practice.
• Hot tier: budgets and fair share.
• Design: 18 Metastore backends, tenancy and the namespace router (§5 to §7), 10 Operations (§3, multi-tenancy), decision log (D25, D63, D65, D66, D96).
• Source: crates/loams-common (namespace and catalog types), crates/loams-meta.