Run a cluster

How to deploy a Lakefront cluster, N server nodes and a router sharing one catalog Postgres and one object store, plus a local compose demo of the same topology.

A cluster is N server processes, each either a call to serve() or the container image configured by LAKEFRONT_* environment variables, all pointing at the same catalog Postgres and the same S3-compatible object store. A router sits in front of them and picks which node answers each board. None of this is specific to any particular Postgres or object-store host: bring your own catalog Postgres, and any S3-compatible store works.

Every node attaches the same lake, so every node can serve any board. The router just decides which one should, most of the time.

Configure a node

Every node, whether started with serve() directly or via the container entrypoint, needs the same catalog Postgres (LAKEFRONT_POSTGRES / OpenOptions.postgres) and the same object store prefix (LAKEFRONT_DATA / OpenOptions.data, plus LAKEFRONT_S3_* credentials when that prefix is s3://). Every node in one cluster also needs the same LAKEFRONT_NAMESPACE / OpenOptions.namespace, so they attach to one catalog rather than several. The full set of variables and the option each maps to is in the environment variable reference; the container entrypoint reads them, and embedded use passes the same fields as ServeOptions directly.

topology defaults to "clustered", which is what every node in this guide should run: it polls the lake head in the background so cross-board reads stay bounded-fresh. "single-node" asserts this process is the only writer against the lake and turns that polling off; running two "single-node" engines against one lake is a correctness bug, not a performance trade, so it has no place in a multi-node deployment.

Set writePath to wal

writePath: "wal" (the engine’s default; LAKEFRONT_WRITE_PATH=wal in the container) gives every node a shared, ordered log to sync from, and it is what a multi-node deployment requires. Under writePath: "lake", a write commits synchronously to the lake on whichever node accepted it: durable, but a peer node has no WAL tail to read, so it only sees the write once its own head-refresh poll catches the new lake snapshot. Freshness on any node is a WAL property, not a lake one.

Point the router at the nodes

The router needs a node list, LAKEFRONT_NODES (a comma-separated id=url list) or RouteOptions.nodes embedded, plus LAKEFRONT_HEDGE_MS / RouteOptions.hedgeDelayMs for the hedge delay below. It picks a node per board with rendezvous hashing, so the same board lands on the same node call after call: cache affinity, with no central assignment table to keep in sync.

• Hedging: if the primary hasn’t answered within hedgeDelayMs, it fires the same read at the next-ranked node and takes whichever answers first.
• Failover: idempotent procedures, and mutations carrying an idempotencyKey, fail over to the next healthy node on a transport failure, because a WAL-backed retry answers with the original ack rather than double-applying.

The hashing formula, the hedging and failover rules, and why they are safe are covered in full in the routing rules.

Freshness across nodes

Because every node syncs from the same WAL, a read routed to any node, hedged, failed over, or just the primary, can honor a minWatermark from a write that landed on a different node. A write acked through one node is immediately readable at that watermark from every node, including ones the router would never pick for that board. See the consistency model for the watermark model this rests on.

Node loss

Stopping a node does not lose anything, because the node’s local replica file was only ever a cache: the boards it owned re-route to a survivor on the next request, which rebuilds them from the object store plus whatever WAL tail the flusher hadn’t drained yet. Recovery does not depend on the dead node’s disk.

Try it locally

The repo ships docker/compose.yaml as a demo of this same topology: three server nodes, a router, a catalog Postgres, and MinIO standing in for the object store. It exists so the topology above is easy to try locally; bring your own Postgres and S3-compatible store for anything real.

The catalog Postgres is available on host port 55432.

bun run cluster:up    # docker compose up -d --build
bun run cluster:logs  # follow every service's logs
bun run cluster:down  # tear down and remove volumes

Once the cluster reports healthy, run bun run demo to exercise routing and node failover. The demo stops node-b to demonstrate recovery.

Run the demo against a cluster that is already up. It targets http://127.0.0.1:58000 by default, or LAKEFRONT_ROUTER/an argv override for anything else.