The S3 API: the bucket contract Loams depends on

Most posts in this series are about protocols Loams serves: Postgres wire, Qdrant, Elasticsearch. This one is about the protocol Loams consumes. The Amazon S3 API is the interface to Loams’ only durable source of truth (D1), so the exact semantics of a few of its operations decide whether an acknowledged write can ever be lost. We call that set of semantics the bucket contract.

Why a bucket at all is covered in Why a retrieval engine should live on object storage, and the default self-hosted store in RustFS: the object store under a self-hosted Loams. This post is about the wire.

What the protocol isCopy Anchor Link

S3 is HTTP. Every operation is a request on a bucket and a key: PUT, GET, HEAD and DELETE on an object, GET /?list-type=2 for a listing, and a family of POST requests for multipart uploads. There is no session and no handshake beyond TLS. Addressing comes in two styles: virtual-hosted (bucket.s3.region.amazonaws.com/key), which AWS prefers, and path-style (host/bucket/key), which most self-hosted stores default to.

Authentication is per request. Signature Version 4 signs a canonical form of the request: the method, the path, the sorted query string, a list of signed headers, and the SHA-256 of the body, carried in x-amz-content-sha256 (or the literal UNSIGNED-PAYLOAD). The signing key is derived by chaining HMAC-SHA256 over the date, region, service and the string aws4_request, so a signature is bound to one request, and the derived key to one day, one region and one service. The result goes in Authorization: AWS4-HMAC-SHA256 Credential=…, SignedHeaders=…, Signature=….

The operations Loams uses, and what it relies on from each:

Consistency. Since December 2020, S3 gives strong read-after-write consistency for every PUT and DELETE, and listings reflect them immediately. Before that, a new object could be missing from a LIST, and an overwrite could return the old bytes for a while. Every bucket-native design that exists today, Loams’ included, assumes the post-2020 model.

Conditional writes. S3 added If-None-Match on PutObject in August 2024 and If-Match later the same year. These two headers are what make a bucket usable as a commit log without a separate lock service. Google Cloud Storage has the same thing as generation preconditions (ifGenerationMatch=0 for create), and Azure Blob Storage as ETag conditions on Put Blob.

Why Loams speaks itCopy Anchor Link

Loams treats compute as a cache and the bucket as the database. The reasoning is in the design overview and the Learn page The bucket is the source of truth. The protocol consequence is simple: S3 is the one API every cloud and every serious self-hosted store already speaks, so speaking it, and only it, is what lets the same binary run on AWS, on Cloudflare R2, on GCS and Azure through their native APIs, and on RustFS in a laptop compose stack.

We chose not to build an S3 service of our own. An earlier draft proposed a Rust S3-compatible server; D381 rejected it. Where Loams runs its own storage it runs RustFS (D61, D178), on Cloudflare it uses R2, and loams-store stays a client. A facade would have duplicated SigV4, multipart and conditional writes for no gain. Tenants are isolated by vended, prefix-scoped credentials from the operator's ObjectStoreProvider, not by a proxy in the data path.

How Loams implements itCopy Anchor Link

All access goes through one crate, loams-store, a thin layer over object_store. Its surface is deliberately small:

• put_if_absent(path, bytes): If-None-Match: *. Returns AlreadyExists when the key is taken.
• put_if_match(path, bytes, version): If-Match on the ETag (or the backend's version marker). Returns PreconditionFailed when the object changed, NotSupported on a backend without conditional updates.
• get, get_range, and get_range_with_info, which returns the bytes of a range and the object's size and ETag from the same response, so one request both reads a footer and learns where the file ends.
• put_multipart, which streams a body in parts of at least 5 MiB, keeps at most four parts in flight, aborts the upload if the body or a part fails, and aborts it in a spawned task if the caller is cancelled. The object appears only on completion.
• list_page(prefix, after, limit), which pages a listing with start-after semantics and never holds more than limit entries.
• copy, head, and a delete that treats a missing key as success.

The file:// backend fsyncs the file and its directory before put returns, so "durable once the call returns" holds on a laptop as it does on S3.

Who uses which guaranteeCopy Anchor Link

That last row is a choice. Collections commit through a pointer in the metastore, not through a conditional PUT on the bucket (One manifest, two formats). Streams and collections generate offset assignments, consumer commits and leases at a rate that a contended conditional PUT, at tens to hundreds of milliseconds per attempt, cannot sustain, so that metadata lives in the metastore and only the bytes live in the bucket.

The fenced appendCopy Anchor Link

The Git WAL shows the bucket contract at its sharpest. A writer that holds the state after segment n commits a batch by creating segment n+1:

Three answers are possible. A 200 means committed. A 412 means another writer owns n+1: fetch it, apply it, re-check the batch, try n+2. A timeout or 5xx after the request left means the outcome is unknown, so the writer reads wal/<n+1> back: its own batch id means committed, another batch id means fenced, and a 404 means resend the same bytes. Nothing in that protocol needs a lock service, and correctness never depends on there being a single writer. It also works on R2, where one key accepts about one write per second: a separately swapped head pointer would have capped a repository at one commit a second, while sequential segment names are each written once.

Two consistency properties carry the whole design. Read-after-write means a 404 on wal/<n+1> proves n is the tail at that instant, so a linearizable read costs one GET. List-after-write means a reader that lists a prefix after a commit sees it.

How we test itCopy Anchor Link

We do not take compatibility matrices on trust. Testing happens at three levels:

1. A fault-injecting store. loams-store's FaultyStore wraps any object_store and injects, per operation class, a plain error, an error after the write was applied (a lost acknowledgement), a failed precondition shaped the way a real store reports one, and random delays. Loams Git's ref-log histories run under it through the linearizability checker. Injecting S3's own throttling (503 SlowDown) is in the M2 test plan, not in the wrapper yet.
2. Every backend-sensitive test on three stores: in memory, through the fault wrapper (whose listing order Store cannot know), and on a real file:// directory.
3. A real S3 server in CI. The Loams Git suites (gt1.yml) start RustFS 1.0.0 and run the store-backed WAL suite against it over SigV4. The connector suite runs against RustFS 1.0.1 and checks that its object sink's If-None-Match: * PUT is refused on overwrite. The Postgres end-to-end workflow runs Loams Postgres's storage on RustFS.

What is not done yet: the provider conformance suite that D61 calls for, on RustFS on every pull request and on real S3, GCS and Azure nightly, with the S3 fault matrix producing the same expected table as the in-memory store. Until it runs, our verified list is the one above, and claims about other stores come from their documentation.

GotchasCopy Anchor Link

A 412 on create is not always reported as 412. S3 answers a failed If-None-Match: * with 412; object_store surfaces it as "already exists", and other stores answer 409. loams-store maps both to AlreadyExists, and the fault wrapper reproduces the store-shaped variant so the code paths are exercised.

A concurrent write can also give 409. S3 documents 409 ConditionalRequestConflict when a conditional write races another operation on the same key. Treat it like an unknown outcome: read the key back, then decide.

Not every bucket lists in order. S3 general-purpose buckets list keys in UTF-8 binary order. S3 Express One Zone directory buckets do not, and S3-compatible stores vary. loams-store assumes an S3 URL lists unordered unless the caller opts in, so list_page stays correct on any store and only reads less on an ordered one. GCS, Azure and the in-memory store are known to be ordered.

Atomic visibility has a size on some stores. RustFS's conditional PUT is atomic for bodies up to 1 MiB (its PR #6798). Loams caps WAL segments at 1 MiB on every store for that reason (D389, D390). Large objects are another question: until a test confirms that a reader can never see part of a RustFS multipart upload, Loams Git refuses content-named blobs over 64 MiB on a store marked non-atomic. The probe for it (Q385) polls an upload of a large blob and fails if a single read returns a partial object; it runs in the Git workflow.

A self-copy is not a refresh. S3 refuses CopyObject onto the same key without a metadata change, and a local directory treats it as a no-op. loams-store turns a self-copy into a read and a write, and documents that it is only safe for content-named objects.

SigV4 details differ between clients. RustFS rejects a SigV4 request that lacks x-amz-content-sha256, which old versions of curl's --aws-sigv4 do not send. Our RustFS bootstrap script adds the empty-body hash by hand. Clients that insist on virtual-hosted addressing need the store told its domain; Loams’ own tests use path-style (aws_virtual_hosted_style_request=false).

Percent signs in keys. object_store's Path::from percent-encodes its input, so a key with a literal % would not round-trip between put and list. loams-store parses paths as already-decoded segments instead, and refuses ., .. and control characters.

An aborted multipart upload can leave parts behind. If the abort itself fails, the parts stay until the bucket's lifecycle rules expire them. Configure an AbortIncompleteMultipartUpload rule on any bucket Loams writes to.

A store without conditional writes is not a Loams store. The operator's provider trait refuses one (D178). An S3-compatible store that ignores If-None-Match will silently let two writers own one segment, which is exactly the failure the contract exists to prevent. Verify it before you point Loams at it.

The next post moves from the bucket's API to Git's: the smart protocol, pkt-lines, and the remote helper that lets stock git push straight into a bucket.