Links and workers
Declarative materialization with exactly-once apply, run by a stateless worker pool.
Links are Loams’ replacement for connector pipelines: declared, continuously maintained materializations from streams into tables, collections and graphs. Workers are the stateless pool that runs links and every other background task.
Declaring a link
The design uses SQL DDL, with a stateless transform over each batch:
CREATE LINK tickets_search
FROM STREAM support_tickets
INTO COLLECTION tickets
TRANSFORM (
SELECT key AS _pk,
json_get_str(value, '$.subject') AS subject,
json_get_str(value, '$.body') AS body,
timestamp AS ts
)
WITH (batch_interval = '2s', on_error = 'dead_letter');Transforms cover projections, filters, JSON extraction, casts and scalar functions, including an optional embed() that calls an external embedding endpoint. Stateful joins and windowed aggregations are out of scope; run RisingWave alongside Loams for those.
What exists today
The link framework has two target kinds. Every collection is fed by an implicit collection
link from its own stream; the HTTP API creates counter links,
which sum integer values per key and report their lag. SQL DDL and the table and graph targets
come with their milestones.
Exactly-once apply
A link task leases its source partitions in the metastore, with an epoch.
It reads from the last applied offset recorded in the target's latest commit.
It builds the target's files, then commits the target together with the new applied offsets, in one conditional pointer swap.
A zombie task with a stale epoch fails its commit. A restarted task resumes from the committed applied offset, so work repeated after a crash is discarded, never applied twice.
What workers run
| Task | Output |
|---|---|
| Segmenter | Stream segments; WAL objects deleted |
| Retention | Segments and files past their policy deleted |
| Link apply | Target commits |
| Split merge, Lance compaction, Iceberg compaction | Larger, fewer files |
| Hot artifact build | HNSW graphs and projections |
| Graph sidecar build | CSR and CSC sidecars |
| Garbage collection | Unreferenced objects deleted after a grace period |
| Metastore snapshot | A snapshot in the bucket |
Read more in §09 Links and workers.