How the performance index is calculated
The homepage's Index tab plots one score per release. This note documents exactly how that number is produced, so it can be checked, criticized, or reproduced from the published ledgers.
Mean of median-normalized metrics, ×100. Lower is faster.
For each release we take every core latency metric that was measured, divide it by that product's own median for the same metric across all its releases, average the ratios, and multiply by 100. A score of 100 means "a typical release of this product"; 80 means 20% faster than typical; 130 means something regressed.
the inputs
Four metrics feed the index, all from the same measured session on the benchmark runner: interactive (launch → TUI accepts input; only counted when the session reached "ready"), spawn (bare process launch), response (launch → first real model answer, MCP loaded), and response without MCP (same, MCP disabled — isolating tooling overhead). A metric missing from a run is simply excluded from that release's average rather than imputed.
the computation, verbatim
This mirrors the site's own chart code — the index the homepage renders is computed client-side from the published history:
ratio(metric) = release[metric] / median(product, metric)
index(release) = round( mean(available ratios) × 100 )why median-normalized
Raw milliseconds can't be averaged across metrics (a 200 ms spawn and a 4,000 ms response would let response dominate), and products can't be compared on absolutes (different architectures, different work at launch). Dividing each metric by the product's own median puts everything on the same dimensionless scale: "how does this release compare to what's normal for this tool." Trends and regressions pop; absolute claims don't sneak in.
honest caveats
It's a composite — use the Response, Startup, and Size tabs for
absolute numbers. The median shifts as history grows, so old
scores can drift slightly when new releases land (the shape of
the curve is stable; the baseline is living). And the index says
nothing about model quality — only how fast the CLI gets you to
a working answer. Ledgers with per-run provenance are public in
the repository (bench-data/), so every point on the
chart traces to a named runner and run.