Operations

The flusher, manual flush and maintenance, parked boards, engine stats, eviction, and the destructive operations that stay off the Engine surface.

Flusher

Under writePath: "wal", a background flusher drains each board’s pending WAL records into the lake, off the write path. flushIntervalMs controls the cadence, defaulting to 30 s. 0 disables the timer and leaves flushing entirely to explicit calls, which is what the test suite does.

The flusher batches records to reduce the number of serialized catalog commits.

Flush lateness only costs rebuild-replay length on the next cold attach. Durability already lives in the WAL the moment a write acks, regardless of when the flusher gets to it.

Call engine.flush(boardId?) to drain on demand, one board, or every board with pending WAL records when called with no argument:

const stats = await engine.flush("acme_roadmap");
stats.unwrap("flush failed"); // FlushStats[]: from, to, records, skipped

Under writePath: "lake" every write already commits to the lake synchronously, so flush is a no-op that returns an empty list.

Maintenance

engine.maintain(olderThanHours?) runs lake maintenance now: snapshot expiry, file cleanup, and small-file merge. The flusher also runs this on its own clock, once an hour, so under writePath: "wal" you normally never call it yourself. It exists for a writePath: "lake" deployment (which has no flusher) to wire into its own rotation, or for forcing an off-cycle cleanup:

await engine.maintain(0); // expire snapshots older than 0 hours -- i.e., all of them

Parked boards

A board the flusher fails to flush repeatedly gets parked rather than retried in a tight loop against the same failure: backoff starts at 30 s and doubles up to a 10-minute cap.

Parking never touches durability: the board’s WAL records sit untouched, since durability never depended on flushing. They do accumulate, though, so a parked board is surfaced rather than swallowed. engine.stats().flusherParked and the node.stats wire procedure both carry it, as a list of { boardId, failures, since }. One poisoned board parking does not stop any other board’s flush from proceeding on the same pass.

Recovery is fixing whatever made the lake-side apply fail: a dropped table, a permissions change, anything that turns a normal write into a thrown error. Then either wait for the backoff to elapse, or call engine.flush(boardId) directly: an explicit call always retries regardless of the parked state, which is the manual unpark.

Reading EngineStats

engine.stats() (and the node.stats wire procedure, which adds nodeId and the list of currently-cached boards) reports:

• cached / capacity: attached replica files against the configured ceiling
• hits / rebuilds / reattaches: how requests found their replica, already-attached, rebuilt from the lake, or reattached from an existing file
• evictions, connectionsInUse, flusherParked

What healthy looks like:

• traffic dominated by hits, with rebuilds/reattaches showing up only on genuinely cold boards or after a restart
• connectionsInUse well below poolSize, not sitting persistently near it
• flusherParked empty, or clearing within a backoff cycle or two

Investigate sustained pool saturation or boards that remain parked.

Eviction

engine.evict(boardId, deleteFile = false) drops a replica from the attached-file cache. The file stays on disk by default; deleteFile: true also removes it, which is how a test simulates losing the node’s local disk entirely rather than just its warm cache.

Eviction waits out in-flight work on that board before it lets go: it will not detach a file a concurrent request is still using. The cache converges after the call returns, not necessarily inside the same tick.

Do not assert cachedBoards() or stats().cached immediately after an evict (or a dropBoard, which evicts internally) in a test or a script; poll, or accept that convergence is asynchronous.

Replica index upkeep

The engine automatically rebuilds replica ART indexes to reclaim space after updates and deletes. Rebuilds are expected on frequently updated boards and require no operator action.

Compacting after upsert-heavy load

upsert writes a new version of each row rather than issuing an UPDATE, so repeated upserts to the same ids leave superseded versions behind. engine.compact(boardId) collapses them:

await engine.compact("acme_roadmap");

Run it after a board has taken a burst of upsert traffic, or on a schedule if upserts are its normal write shape.

Destructive: resetLake and dropCatalogSchema