methodology

How OVM verifies, measures, and broadcasts every release

OVM installs and switches versions of Claude Code, Codex, and Pi — but its real job is judgment: deciding which upstream release is safe to run, measuring the ones that are, and publishing the verdict. This is the whole system. The regression testing that gates a release, the benchmark that rides the same pass, and the public feed that pushes the result are one methodology, not three tools.

Verify

Every upstream release is installed and driven through a real tool-use canary — not a --version check — on macOS and Linux. Only releases that provably run are promoted.

Measure

Verified releases are benchmarked in the same pass: launch latency, and — in the next dataset — cold vs warm, memory, and energy.

Broadcast

Every verdict — verified, benchmarked, or withheld — is published to a public Atom feed with WebSub push, so subscribers hear about a new model the moment it clears.

01Installed is not works

The premise of the whole system: a release that installs cleanly can still be dead on arrival. Codex 0.144.0 moved command execution into a sidecar binary. codex --version worked, login worked, and the first real tool call died — no such file, every session, on a code path no smoke test exercised. A version manager that reported "installed, success" would have shipped it to every user. So OVM does not trust installation; it verifies execution, then measures, then broadcasts.

02The regression gate

Every new release is held out of the registry until it proves it runs. The gate is not a fixed check but a growing ledger — each row added in response to a real failure, with the date it landed and the reason it exists. ok on any platform admits a version; an execution-probe fail is a veto that withholds or revokes it even when it installed and launched cleanly.

The gate is proven both directions as a reproducible pair: one tag ships the pre-fix installer and the probe reddens, so the release never promotes; the next ships the fixed installer and the same release passes. Tags, not assertions.

03The version manager's own suite

Before OVM can be trusted to verify anything, its own code has to be correct — it moves symlinks, swaps control planes, and downloads binaries. Roughly 500 unit and integration tests run beside the source, isolated in throwaway temp directories, never the real ~/.ovm. The hardest paths get adversarial coverage: the atomic self-update swap is fault-injected at every mutation stage and must roll back to the exact prior selection; an install owner killed mid-download is recovered by a second process into exactly one complete install; checksum, archive-size, and SSRF-redirect attacks are rejected across every download path; the claudex proxy withholds its key from any listener whose identity it can't confirm. CI enforces the full gate on both platforms:

A macOS Gatekeeper sweeper closes the last gap: a binary that installed cleanly can later be blocked if its Developer ID cert is revoked, so the sweeper asks the live macOS policy engine (spctl) every time rather than caching a verdict — a benchmark never launches something the OS would refuse.

04The benchmark next dataset

Benchmarking is not a separate activity — it is the measurement half of the same per-release pass, run only on versions that cleared the canary. Each version is exercised in the same modes: spawn (--version), interactive (launch → the TUI accepting input), and response (launch → first real model answer, with the real MCP config, again with MCP disabled, and — for products with a faster tier — a fast variant).

The rest of this page describes the next dataset — proposed changes to the benchmark, written for review before the ledgers are reset. The verification system above is what runs today.

05Cold vs warm

The first launch of a freshly installed or re-indexed tool is a different measurement from its steady state; averaging them hides both. Each mode will run four times, with the first kept separate.

run #1      → cold  (first-init)   // recorded as *ColdMs
runs #2–4   → warm                 // *AvgMs (mean) + *StdDevMs

// today, for comparison — the cold run is smeared into the mean:
avg = mean(run#1, run#2, run#3)

This is not cosmetic. Our clean-room harness already showed why: Codex rebuilds a SQLite state index from its rollout files on the first launch after an import or migration — a one-time backfill of ~14 ms per session file, reaching ~25 s at 1,897 files / 6.8 GB. Claude has no separate index, so it is flat even cold. Warm time-to-ready is flat for both regardless of store size (Codex ~570 ms, Claude ~690 ms). Today that finding lives only in the research harness; this puts a cold and a warm number on every release, where a reindex regression would surface first.

06Resource & energy

Latency is half the story. A sampler walks the whole process tree — the tool plus the MCP servers it spawns — during the warm window.

07The record

One append-only JSONL line per product / version / run. Every new field is optional, so older records stay valid, and the platform is read from the line rather than inferred from which folder it sits in.

// timing — split cold vs warm, per mode
spawnColdMs · spawnAvgMs · spawnStdDevMs
interactiveColdMs · interactiveAvgMs · interactiveStdDevMs
responseColdMs · responseAvgMs · responseStdDevMs · responseNoMcp* · responseFast*

// resource + energy (macOS / Apple Silicon)
peakRssBytes · steadyRssBytes · cpuUserMs · cpuSysMs · threadCount · openFds
energyMilliJoules · avgCpuPowerMw · idlePackagePowerMw · powerAvailable

// environment — self-describing platform
os · arch · hostCpuModel · hostCores · hostRamBytes · loadAvg1

08Lanes

The macOS lane runs on the Mac mini against a real, growing session store on a subscription login — the numbers a heavy daily user feels, and the only honest place for the Codex reindex tax. The Linux lane runs in CI: reproducible and clean, strongest for spawn and startup. A combined lane unions both for range.

Because the real store keeps growing, a synthetic fixed fixture gives a frozen baseline that isolates a version's effect from session growth. It is built from open-source conversations — no personal data — and reproduces the shape that actually drives startup: the same file count, total bytes, and median and p90 file size as a real store (Codex ≈ 2,000 files / 7 GB, Claude ≈ 9,000 files / 6 GB). A CI runner with an empty home can then produce numbers comparable to the mini's, on demand.

09The public feed

Every verdict is broadcast, not just stored. OVM publishes a public Atom feed at /versions.atom — "new agent CLI versions, install-verified and benchmarked" — whose entries are the verification detections themselves: a new release, a version withheld from the registry because it hasn't verified on any platform, a promotion once it clears. The feed carries a WebSub hub link, and the pipeline pings the hub after each deploy, so subscribers are pushed the moment a new model clears the gate — no polling.

10One pass, two platforms

None of this is run by hand. When the release watcher detects an upstream version, a single pass executes on the Mac mini (macOS) and in CI (Linux):

1. Install the release through OVM exactly as a user would.
2. Canary — the regression gate above: tool-use probe, strict sidecars, version-swap, schema diff.
3. Gatekeeper-check the macOS binary against the live policy engine.
4. Benchmark the verified version and append its ledger line.
5. Promote & broadcast — only cleared versions enter the registry; the site re-renders and the feed pushes.

The night the canary went live, upstream shipped four releases overnight; the scheduled run installed, probed, and benchmarked all four on both platforms and refreshed the registry autonomously, before anyone was awake.

11Reproducible by design

Nothing here is a private green check. The test suite is public, the canary's proof is a tagged pair, and every benchmark point in bench-data/ carries the runner and run URL that produced it. The benchmark measures how fast and how cheaply a CLI reaches a working answer, not model quality; energy attribution leans on a quiet, serialized machine and an idle baseline; cold numbers depend on their starting state, so each run records the session-store size behind them. When the next dataset ships, this page becomes the as-built record.