Consistency and watermarks

The watermark model behind Lakefront's reads and writes. Bounded staleness by default, read-your-write on request, and what neither buys you.

Every committed change to a board advances a single number: the watermark. A write returns the watermark it committed at; a read reports the watermark it was served at. Consistency in Lakefront is entirely about how those two numbers relate for a given request.

What a watermark is

A watermark is a monotonic sequence over committed changes to a board. Under the default write path (writePath: "wal"), it is the sequence number a WAL append landed at: allocated by the WAL’s Postgres sequence and made safe to read by the same advisory locks that give same-board appends their order (see how the WAL is structured).

Under writePath: "lake", the watermark is a DuckLake snapshot id instead. The two number spaces are not interchangeable: a replica file built under one write path rebuilds from scratch if it is reattached under the other, because its watermark no longer means the same thing.

Which write path an engine runs, and the durability contract behind each, is covered in write paths; this page only needs that the watermark exists and moves forward.

Every mutating call on Engine (insert, insertMany, update, updateMany, delete, deleteMany, upsert, upsertMany, transaction, the lifecycle calls) returns the watermark its commit landed at inside its Result. Every read returns the watermark it was served at, alongside the query result:

const written = await engine.update("tasks", "task-2", { status: "done" });
const watermark = written.unwrap("update"); // the watermark this write committed at

const read = await engine.query("tasks", "SELECT status FROM {{table}} WHERE id = $1", {
  params: ["task-2"],
});
read.unwrap("read").watermark; // the watermark this read was served at

A watermark is only meaningful relative to another watermark, or to a later read of the same board. It is a position in a log, not a timestamp.

Reads are bounded-stale by default

A query does not go to the lake. It goes to the board’s replica, which syncs forward from wherever it last stopped to the node’s current belief about the head, then answers from the now-current file (see replicas for sync-then-query mechanics). Two things can make that answer older than “the very latest committed anywhere”:

• The node’s belief about the head is itself a cached value, refreshed on a timer (headRefreshMs, see below) rather than checked fresh on every request.
• If several nodes are writing, a commit that just landed on a different node has not necessarily reached this node’s belief of the head yet.

Checking the catalog’s authoritative head on every read adds a Postgres round trip. The default read path uses the cached head to avoid that cost. The trade is bounded staleness, not unbounded: a read is never older than the last refresh interval behind the true head, and a write is always durable the instant it acks regardless of when any replica catches up to it.

Read-your-write is opt-in, per read

A caller holding a watermark from its own write can demand a read no older than it. Pass minWatermark to Engine.query:

const watermark = (await engine.update("tasks", "task-2", { status: "done" })).unwrap("update");

const fresh = await engine.query("tasks", "SELECT status FROM {{table}} WHERE id = $1", {
  params: ["task-2"],
  minWatermark: watermark,
});

Through the client, the same thing is a method rather than an option, returning a new client scoped to the constraint so it cannot leak into unrelated queries:

const client = connect({ url }).board("tasks");
const watermark = (await client.update("task-2", { status: "done" })).unwrap("update");

const fresh = await client.atLeast(watermark).selectAll().execute();

When minWatermark is set and the replica has not caught up that far, the read pays the one round-trip bounded-stale reads normally avoid: the node forces its head belief forward to at least that watermark, then syncs the replica to match, before running the query. Reads whose freshness requirement is already satisfied avoid this additional catalog round trip.

Failover safety

Read-your-write is not a property of “the node you happened to write on.” Under writePath: "wal", every server node syncs from the same WAL, so any node can be asked to honor a watermark it did not itself produce. It syncs its own replica forward to that point first.

If the node that acked the write dies, a survivor rebuilds from the lake, replays whatever WAL tail it is missing, and still honors the watermark.

This is why the router’s hedged reads and write failover (see routing) do not weaken consistency. A hedge that lands on the runner-up node, or a failover after the native node dies, can still be asked to prove it read at least as far as a watermark the caller holds: the guarantee travels with the watermark rather than the node.

head() versus headFromCatalog()

Two ways to ask for the current watermark, at very different cost:

headRefreshMs controls how often the cached belief is refreshed in the background, which bounds how stale head() (and therefore a bounded-stale read) can be. The default is 5ms. A commit this node performs is folded into the cache immediately, with no need to wait for the next refresh; the interval only matters for learning about commits made elsewhere.

Under single-node topology

Under topology: "single-node", where this process is asserted to be the lake’s only writer, refreshing is disabled entirely (headRefreshMs behaves as 0): there is no “elsewhere” to poll for, so the cached belief is exact rather than bounded, and the background catalog traffic disappears.

Running two single-node engines against the same lake is a correctness error under this assumption: each would be blind to the other’s writes. The full knob table, including headRefreshMs and topology, is on the engine reference.

Limits

There is no cross-board snapshot isolation on reads. Watermarks are per-board: reading board A at its current watermark and then board B at its current watermark does not mean the two reads reflect one consistent instant across both boards, because each board syncs independently and there is no shared transaction covering the pair.

There is no global serializability either. Writes to a single board are totally ordered: the WAL’s advisory locking guarantees that same-board appends commit in watermark order, and a board’s replica applies them in that same order. But nothing enforces an ordering between writes to different boards beyond what actually happened to run concurrently.

Cross-board atomicity is available, but only through engine.transaction, which commits several writes across multiple boards as one atomic unit and returns a single watermark that is the read-your-write watermark for everything the block wrote. See transactions for how to use it. Outside of an explicit transaction, treat each board’s watermark as meaningful only about that board.