---
title: Benchmarking
description: Run the OLTP, engine, and ClickBench suites, and how to read what each one
  reports.
---

See the
[performance numbers and methodology](/concepts/performance) for published
results and how they were measured.

| Command              | What it measures                                                           | Opponent                                |
| -------------------- | -------------------------------------------------------------------------- | --------------------------------------- |
| `bun run oltp`       | pgbench-style OLTP workloads (`select-only`, `tpcb-like`, a bare `UPDATE`) | Real PostgreSQL                         |
| `bun run bench`      | Engine/node/cluster layers, OLAP + OLTP workload mix, over time            | Itself, diffed against a saved baseline |
| `bun run clickbench` | The 43-query ClickBench OLAP suite                                         | Native DuckDB, same machine             |

## bun run oltp

`packages/bench/src/oltp.ts` compares lakefront against a real PostgreSQL,
using pgbench's own builtin workloads: `select-only` (a single point lookup by
primary key), `tpcb-like` (pgbench's default: three `UPDATE`s across three
tables, one `SELECT`, one `INSERT`, all in one transaction), and a bare
single-row `UPDATE` with no transaction wrapper on either side, to isolate raw
write cost from transaction overhead. The three pgbench tables plus a history
table map onto four boards in the same lake.

### Network setup

pgbench reaches Postgres over TCP; Lakefront's HTTP node also includes a
network hop. The harness reports embedded results separately to show the
cost of running the engine without the HTTP serving layer.

### Run pgbench from the host

Run `pgbench` and `psql` **from the host** so both systems are reached through
a Docker port mapping. Without host Postgres client tools, the harness falls
back to `docker exec` and prints a warning.

:::warning[In-container runs use a different network path]
Running pgbench inside the container uses a loopback connection. In the
published campaign, this increased Postgres `select-only` throughput by
7–19×. Install `pgbench` and `psql` on the host to match the network setup.
:::

```bash
bun run oltp                 # writePath "wal" (the default), scale 10, 10s per stage
bun run oltp -- --lake       # pin writePath "lake" instead
bun run oltp -- --scale=50 --time=30
bun run oltp -- --cluster    # also measure through a 2-node cluster + router
```

### Durability pairing rule

Under `writePath: "wal"`, the comparison target is Postgres with
`synchronous_commit=on` (the default). It fsyncs its WAL per commit, the same
durability class a WAL ack lives in.

The harness also prints a `synchronous_commit=off` Postgres line on the write
workloads as a reference with weaker durability. Section headers identify the
durability setting for each comparison.

## bun run bench

`packages/bench/src/main.ts` is the engine benchmark suite: no external
Postgres comparison, just this system measured against itself over time.

It seeds boards at three sizes (1,000 rows, 100,000 rows, and optionally
1,000,000), then runs the OLAP and OLTP workload sets from `workloads.ts`
(group-bys, filters, top-N, wide reads, point lookups, range scans) against
each size.

It also runs dedicated write workloads: single-row insert, batch insert,
single-row update, and a full write-then-read-your-write round trip, plus a
concurrency sweep at 1/8/32 in flight.

```bash
bun run bench                    # engine, node, and cluster layers, default sizes
bun run bench -- --large         # adds the 1M-row board
bun run bench -- --save          # records this run as the new baseline
bun run bench -- --layers=engine # restrict to one layer (engine, node, cluster)
bun run bench -- --single-node   # topology: "single-node" instead of "clustered"
```

Each layer runs the same workloads through a different entry point:

- `engine`: calls `Engine` directly
- `node`: goes through one `serve` node over HTTP
- `cluster`: spins up three nodes and a router and goes through that

Comparing the three layers against each other is how you see what HTTP and
routing cost on top of the bare engine, on the same machine, same workload.

Pass `--save` after a change you want to compare future runs against. It
writes `bench/baseline.json`, and every run without `--save` diffs against
whatever baseline is already there.

## bun run clickbench / clickbench:duckdb

`clickbench` runs the [ClickBench](https://benchmark.clickhouse.com/) suite
against a lakefront board: one flat ~100M-row web-analytics table and 43
analytical queries. It is a genuine fit for the OLAP half of this system (wide
table, no joins, filters/group-bys/aggregations/top-N).

It is a poor fit for everything else about it: one table means one board, so
multi-tenancy, routing, and sync-then-query freshness are all inert for this
run. Reading a ClickBench score as a verdict on the multi-board architecture
would be a mistake; it measures DuckDB's execution plus this system's serving
overhead, nothing more.

```bash
bun run clickbench            # loads the dataset if needed, then runs the 43 queries
bun run clickbench -- --reload
bun run clickbench -- --runs=5
bun run clickbench:duckdb     # native DuckDB, same machine, same dataset, same queries
```

`clickbench` needs `hits.parquet` downloaded locally first (the command prints
the `curl` line if it's missing).

`clickbench:duckdb` runs the identical create/load/query sequence against a
bare `DuckDBInstance` with no lakefront serving layer at all. Same machine,
so hardware is not a confound the way it would be comparing against
ClickHouse's own published numbers on different hardware. The gap between the
two runs is what the serving layer costs over the engine it embeds, and
nothing else.

## How to read the output

`oltp` and `bench` both report `tps` (or ops/sec) alongside `p50`/`p95` latency
per row, one row per system/client-count combination. Read across a
client-count to see how each system holds up under concurrency, and down a
system's rows to see its own latency distribution.

`bench`'s read workloads additionally split lakefront's own latency into
`syncMs` (sync-then-query catch-up cost) and `queryMs` (execution once
synced), which is the number that actually distinguishes a cold rebuild from a
warm cache hit: a workload dominated by `syncMs` on repeat runs means the
replica isn't staying warm, not that queries are slow.

`clickbench` prints cold/warm/hot timings per query (first run, second run, last
run) and a summary of hot p50/p95 and total time across all 43. The hot column
is the number worth comparing across runs or against `clickbench:duckdb`, since
cold includes whatever DuckDB's own plan caching and buffer warmup cost on a
first touch.