---
title: Routing
description: Weighted rendezvous hashing for node selection, hedged reads, and why writes get exactly one attempt unless they opt in.
---

Once a read's cost depends on a warm local replica file, routing is not load balancing in the usual sense. Sending the same board's traffic to different nodes on different requests means paying a cold-cache rebuild on every single one. The router's job is cache affinity first (keep a board landing on the same node), with load distribution and failure handling built on top of that, not instead of it.

## Weighted rendezvous hashing

Every router instance computes the same node ranking for a given board independently, with no coordinator and no ring to maintain. Each node scores itself against the board:

```
score = -weight / ln(u)      where u = hash(nodeId, boardId) normalised to (0, 1)
```

and the ranking is every eligible node sorted best-first by that score. The head of the ranking is the board's "native" node; the rest is where it lands on failure, in an order every caller agrees on without communicating.

The score gives each node a selection probability proportional to its weight.

Capacity weights are quantised into five levels (`[0.25, 0.5, 1, 1.5, 2]`) before they reach the ranking. A node's raw capacity signal is rounded to the nearest level. This exists so a small drift in observed load does not reshuffle which node a board prefers: a board-to-node mapping that flaps on every minor capacity change would spend its warm caches constantly rebuilding for no benefit.

A node reports weight `0` when it is unhealthy or draining, which removes it from the ranking entirely without renumbering anyone else's scores.

## What moves when membership changes

When a node joins or leaves, only the boards that scored that node highest move. On departure, they redistribute across the survivors according to each survivor's own score, not all onto one neighbor. This is why rendezvous hashing beats a naive scheme: the blast radius of a membership change is exactly the boards native to the node that changed, nothing more.

## Hedged reads

A read is sent to the board's native node. If that node has not answered by its own recently observed p95 latency, the router fires the same read at the runner-up node and takes whichever answers first, aborting the loser:

```ts
const result = await router.forward(boardId, "/rpc/query.execute", body);
// result.hedged: whether a hedge fired
// result.servedBy: which node actually answered
```

The hedge delay is not a fixed timeout. Each node's recent latencies are tracked in a 128-sample rolling window; once a node has at least 8 samples, its own observed p95 becomes the hedge trigger for reads routed to it. Before that much signal exists, a configured `hedgeDelayMs` is used as the fallback.

Hedging at a node's own p95 means roughly 5% of requests get hedged in steady state: a small, constant overhead purchasing a cut tail latency, rather than a fixed timeout that is either too eager or too late to help.

This is also what absorbs a cold cache after a restart: a request that would otherwise wait out a full rebuild on the restarted node instead gets a hedge to a node that may already be warm, so the rebuild costs latency on one request rather than becoming visible to every request that lands during it.

## Writes are not hedged

A bare mutation (no idempotency key) gets exactly one attempt at exactly one node. It is never hedged and never failed over.

Hedging duplicates the request by construction, which is free for a read and corrupting for a write: if a slow insert takes long enough to trigger a hedge, both attempts can land, and because the lake is append-only nothing downstream can undo the duplicate. It survives into every replica rebuilt from that table afterward.

So the default has to be conservative: if a bare write's single attempt fails, the caller is told, because a failed request and a request that landed but lost its response are indistinguishable from the outside, and guessing wrong in the optimistic direction is worse than surfacing the ambiguity.

A mutation that carries an idempotency key is different. It opts in by setting the `x-lakefront-idempotent` header (the client sets this automatically whenever the request body carries an `idempotencyKey`), and the router treats it exactly like a read: eligible for hedging, and eligible for failover down the full node ranking if earlier attempts fail outright.

This is safe because the WAL deduplicates by key: a retried or hedged write that reaches a second node still applies at most once, answered with the original watermark on any repeat (see [the idempotency guide](/guides/idempotency) for the exactly-once contract). Reads are always retryable in this sense; there is no idempotency concern in re-running a `SELECT`.

## Membership and health

The router polls every member's `/health` once a second. A node that fails to answer is marked unhealthy and drops out of the ranking with weight `0`. Routing continues over whoever is left, redistributing that node's boards among the survivors as described above. If every node is unhealthy, a request gets a `503`; if the candidates the ranking offers all fail during a request, a retryable request that exhausts every candidate gets a `502` rather than hanging indefinitely.