Forking Neon, TiDB and Grafeo: owning your dependencies

"Just fork it" is the cheapest sentence in software and one of the most expensive decisions. A fork is a promise to do, forever, the work an upstream team used to do for you: rebuilding images, merging security releases, porting to new major versions, answering bug reports. Most forks die because nobody costed that promise before making it.

Loams depends on several large open-source systems, and we have forked some of them. Our position is that a fork is a liability you take on only when a stated trigger forces it, and then you run it like a product: a written inventory of every change, a CI check that refuses anything not in the inventory, a test gate, pinned and signed artifacts, and a staffing number written down before the work starts.

This post goes through each dependency, from the ones we left alone to the one we own outright, and says why it sits where it does.

Rung one: do not forkCopy Anchor Link

The default is to run upstream unmodified. Unmodified upstream keeps upgrades and security fixes cheap, and it lets us say "this is TiKV v8.5.8" with no asterisk. That was the first decision we wrote down for PingCAP's stack (D126): PD, TiKV and TiDB from official releases, with fixes we needed offered upstream instead of patched in.

ClickHouse: the fork we chose not to makeCopy Anchor Link

Loams House serves ClickHouse-compatible analytics on chDB, ClickHouse as a library. The House runs SQL from users and agents, and ClickHouse can do a great deal: table functions that read files, URLs and other databases, engines that reach the network, settings that change all of it. The tempting answer is a fork that cuts those features out.

We decided against it. The House's risk is how much ClickHouse can do, and ClickHouse already ships the controls for that. So the layers are, in order (design 49):

1. ClickHouse's own access control first. The users the workers run as have no SOURCES grants and a constrained settings profile, so table functions and engines that reach files, URLs or other databases are refused by ClickHouse itself.
2. Statement checks in the House, as a second line.
3. An OS sandbox for every worker: Landlock and seccomp by default, with no execve, no new processes and TCP only to the worker's read proxy. A worker whose sandbox cannot be put in force refuses to boot.

A fork of ClickHouse would mean merging a large, fast-moving C++ codebase on every release, to remove features its own permission system already turns off. That is all cost and no gain. The House sits on the "not forked" rung, and stays there unless ClickHouse's controls turn out not to hold.

Rung two: unmodified, pinnedCopy Anchor Link

PD and TiKV run exactly as PingCAP releases them, pinned to v8.5.8. Six Loams components keep their state in TiKV (the KV post), which is exactly why we do not touch it: a store that holds everything is the last place we want private changes.

TiDB's next-generation kernel keeps its storage on object storage, and its shared-storage TiKV engine is not public. A Loams fork of TiKV with an S3 tier is designed as later work (design 47 §8.3), and the same section explains why it would have to be a fork and not a plug-in: TiKV chooses its storage engine at compile time. Loams SQL's GA plan ships without it (§8.2), and we have not started it. Pinning and upgrading in PingCAP's order (PD, then TiKV, then TiDB, D933) is our whole relationship with TiKV today.

Rung three: lightly patchedCopy Anchor Link

TiDB: one cherry-pickCopy Anchor Link

Loams SQL runs a stateless TiDB per branch in its own keyspace. That broke on one thing: classic TiDB on one PD elects a single DDL owner across every keyspace, so DDL in all the other keyspaces waited forever. PingCAP had already fixed it on master (pingcap/tidb#60403), but not in the 8.5 release line.

The rules we wrote for "lightly patched" (D721, D722; design 47 §5.4) are narrow on purpose:

• The queue is a branch, loams/v8.5.8-patches on ostrium-labs/tidb: the upstream tag plus cherry-picks made with git cherry-pick -x, and nothing else. No Loams-only code.
• Each image is built from an annotated tag v8.5.8-loams.N by a script in the Loams repository and pinned by digest.
• The list of allowed patches is in the design document. The first entry was "none, if the spike confirms the documented behaviour". The DDL owner fix is the one that the spike proved necessary.

So ghcr.io/ostrium-labs/tidb:v8.5.8-loams.1 is upstream v8.5.8 plus that one commit. When a TiDB release line carries the fix, the patch goes away and so does the fork.

The TiKV Rust clientCopy Anchor Link

The Rust client, tikv-client, is pre-1.0, and its README says so. Our spike found a timestamp stream that never recovered after a PD stall, async-commit locks a reader could not resolve, and private protobuf modules we needed for the change feed. Loams pins ostrium-labs/client-rust by revision. The comment above that line in the root Cargo.toml lists what the fork carries, and says that the pin moves only in a pull request that reruns the loams-tikv, metastore and Live suites. Each fix is drafted as an upstream pull request. The goal of this fork is to stop existing.

Rung four: a patch queue on a young engineCopy Anchor Link

Loams Graph executes GQL with Grafeo, an embedded graph engine written in Rust. We were wary of it from the start, and wrote down why when we first reviewed it (D350): the repository was eight months old, and 90 of its last 100 commits came from one author. A dependency like that can be excellent and can also disappear.

So we arranged things so that losing Grafeo would hurt but not kill us:

• The truth is our log, not Grafeo's files (D742). Every committed graph transaction is an engine-neutral change set in the graph's stream in the bucket, acknowledged after the append. Grafeo's local files are a rebuildable cache. If we had to replace the engine, the data would not move.
• A dependency policy with a fork trigger (D759): an exact pin at least 14 days old, Grafeo's NOTICE in ours, "Loams Graph, built on the Grafeo engine" and never Grafeo as our product name, and a list of events that make us fork: a security issue unfixed for 30 days, no release for 90 days when we need one, a licence change, or an API change we cannot follow in one release.

Then the integration hit correctness bugs. Each one had a workaround in our code, and workarounds pile up: the graph service had code to compensate for a missing commit hook, a statement timeout that did not apply to statements with parameters, ORDER BY answering relationships wrongly, and a DETACH DELETE that could leave a dangling edge. On 2026-10-12 we decided to fix Grafeo in a fork instead of working around it.

ostrium-labs/grafeo starts at upstream v0.5.43, the commit the crates.io release was published from. It keeps a patch queue in PATCHES.md with ids LG-0001 onward, on branch loams/v0.5.43-patches, and 0.1.0 ships v0.5.43-loams.1. The first four patches fix the four bugs above, each replacing a workaround with a test that now passes. One more, LG-0005, is a design note for snapshot isolation, not yet implemented, so a graph read still waits while a write batch commits.

Every LG patch is meant to go upstream. A patch queue on a young engine is a way of contributing with a buffer, not a way of leaving.

Rung five: the hard forkCopy Anchor Link

Loams Postgres runs on Neon's storage: the pageserver, Neon's patched Postgres per major version, and the compute tooling around them. Here we did not have a choice.

Upstream stopped. Neon was acquired by Databricks, and neondatabase/neon went quiet. It had 11 commits in the twelve months to 2026-08-31 and none since (design 51 §2). The last public images we pin carry Postgres 17.5, while upstream Postgres is at 17.11, and they will never be rebuilt. A soft fork that waits for upstream merges would wait for nothing.

So on 2026-10-09 we made it a hard fork with our own name (D800, D820). ostrium-labs/neon became ostrium-labs/loams-postgres. Neon's last public commit is tagged neon-fork-point. We own main. The NOTICE keeps Neon's Apache-2.0 attribution and adds "Based on Neon by Databricks/Neon Inc." If Neon ever publishes again, a commit worth having is cherry-picked by hand as a patch with its own id. Nothing obliges us to follow it.

Names that are stored somewhere stay as they are: the neon extension, neon.* settings, the on-disk layer formats, the WAL record ids, the users' Docker volume names. A user-visible name that lives in someone's saved data is never renamed in place (D822). Branding is cheap; breaking a restore is not.

Four of Neon's upstream repositories turned out to be private: a REST broker, a fleet manager, and two CI repositories. Code we cannot build, audit or redistribute is dropped, not replaced (D816). The REST broker stays an in-tree stub, and the Data API we would have used it for is its own design now.

How we keep forks honestCopy Anchor Link

A fork without rules drifts into a second codebase nobody can explain. These are the rules, shared across the forks and strictest for Loams Postgres:

• Every change has an id and a reason. Each commit after the fork point carries a Loams-Patch: LP-<nnnn> trailer, and LOAMS_PATCHES.md has a row per id: title, area, owner, origin and the condition that drops it (D803). A script in the gate fails a commit without a trailer, a trailer without a row, and a row without a commit. After a hard fork, that file is also the Apache-2.0 statement of changes against Neon.
• History is merged, never rebased (D802, D821). Cargo fetches by revision, so a rebase would orphan revisions that the Loams repository and older releases pin. Every pinned revision carries a loams-* tag, and nothing is force-pushed.
• Upstream Postgres is tracked directly. A nightly job try-merges each supported REL_<N>_STABLE branch of postgres/postgres into ours. A clean merge opens a pull request; a conflict opens an issue (D811 as amended).
• A four-tier gate blocks merges and releases (design 51 §6.4): G0 lint, licences (no CDDL in anything we ship) and the patch check; G1 unit tests per supported major; G2 Neon's own test runner on neon_local, per major; G3 the release candidate's digests run through Loams' own Postgres end-to-end tests. A known flake is skipped only with a listed skip and an issue, never by deleting the test.
• Artifacts are signed and traceable (D809, proposed): each image published by digest with SLSA provenance, an SPDX SBOM and a keyless cosign signature, and Loams CI verifies every pinned digest.
• A quarterly review, on a Postgres minor release day, drops rows that upstream made unnecessary. Dropping is a revert commit, never a history rewrite.
• Security has targets (D805): 72 hours for critical, 7 days for high and for Postgres security minors, 30 days otherwise.

Not all of this is finished. 0.1.0 still runs Neon's last public images by digest, and the patch check is not yet a required status on every branch. The fork's own builds come next (NF1), and the first release that is not a rebuild of Neon's revisions, lp-2026.11.0, has to be out before Postgres 17.12 on 2026-11-12. We are writing that down here so nobody reads the rules above as "done".

What it costsCopy Anchor Link

We costed the Neon fork before committing to it, and re-costed it when it became a hard fork (D824, design 51 §13). These are estimates:

Engineering is the cost that matters, not CI. The hard fork made it larger, not smaller: upstream was dormant, so syncing with it was cheap, but it also means nobody else fixes Neon's storage bugs or reviews its code. We staff it as one full-time maintainer plus a second at about a quarter of their time, and require two named maintainers at all times, because with a hard fork the second maintainer is also the only other person who knows the code.

There are written triggers between the modes. Fewer than two maintainers for more than 30 days drops the fork to minimum maintenance; a security fix that needs more than five engineer-days, or a major that the roadmap needs, brings it back. A missed security target or a storage bug that could lose data is escalated to the project lead, never handled quietly.

And there is an exit. If ownership becomes unaffordable, the fallback is minimum maintenance on Postgres 17 and 18 only, with CloudNativePG for users who need a newer major sooner (design 51 §13.4). That is a product decision, and it is recorded now so it is not discovered late.

What other projects get wrong, and rightCopy Anchor Link

The common failure is the silent fork: a vendored copy with local edits, no list of what changed, rebased now and then by whoever is brave that week. It works until a security release lands in a file you changed and nobody remembers why.

The other failure is the reverse: refusing to fork on principle, and accumulating workarounds in your own code instead. That is where we were with Grafeo for a few weeks. A workaround hides the bug from upstream and from the next reader, and it is untested against the real fix when one arrives. A small patch queue with a test per patch is more honest than either.

Neon itself did this well for Postgres: its patches lived in their own branches per major and were merged with each upstream minor, and its core_changes.md lists every patched area. We kept that discipline and wrote it into CI.

Where we standCopy Anchor Link

Owning a dependency is not a badge. Each rung to the right is work that someone has to do every quarter for years, and we would happily move any of these back to the left. The rules above exist so that, at any moment, we can say exactly what we changed, why, and what it would take to stop.

The next post is the list of what Loams is bad at today, the forks' unfinished parts included.