methodology

How OVM verifies releases, measures coding agents, and broadcasts verdicts

OVM installs and switches versions of Claude Code, Codex, and Pi — but its real job is judgment: deciding which upstream release is safe to run, measuring the ones that are, and publishing the verdict. This is the whole system. The regression testing that gates a release, the benchmark that rides the same pass, and the public feed that pushes the result are one methodology, not three tools.

Verify

Every upstream release is installed on real hardware before it is promoted, and a release that fails is withheld rather than offered. How far verification goes depends on the product: Claude Code and Codex are driven through real tool-use canaries on macOS and Linux; Pi is install-verified by unpacking completely and reporting its own version through the OVM launcher. Products states the tier for each one.

Measure

Claude Code and Codex are benchmarked in the same pass: launch latency, and — in the next dataset — cold vs warm, memory, and energy. Pi carries no performance numbers; an empty cell there means not measured, never measured and slow.

Broadcast

Release detections and the supported promotion or withholding events are published to a public Atom feed with WebSub push. The feed is an event surface; the registry remains authoritative for what can be installed.

01Installed is not works

The premise of the whole system: a release that installs cleanly can still be dead on arrival. Codex 0.144.0 moved command execution into a sidecar binary. codex --version worked, login worked, and the first real tool call died — no such file, every session, on a code path no smoke test exercised. A version manager that reported "installed, success" would have shipped it to every user. So OVM does not trust installation; it verifies execution, then measures, then broadcasts.

02The regression gate

Every new release is held out of the registry until it proves it runs. The gate is not a fixed check but a growing ledger — each row added in response to a real failure, with the date it landed and the reason it exists. ok on any platform admits a version; an execution-probe fail is a veto that withholds or revokes it even when it installed and launched cleanly.

03The version manager's own suite

Before OVM can be trusted to verify anything, its own code has to be correct — it moves symlinks, swaps control planes, and downloads binaries. Over 700 unit and integration tests run beside the source, isolated in throwaway temp directories, never the real ~/.ovm. The hardest paths get adversarial coverage: the atomic self-update swap is fault-injected at every mutation stage and must roll back to the exact prior selection; an install owner killed mid-download is recovered by a second process into exactly one complete install; checksum, archive-size, and SSRF-redirect attacks are rejected across every download path; the claudex proxy withholds its key from any listener whose identity it can't confirm. CI enforces the full gate on both platforms:

A macOS Gatekeeper sweeper closes the last gap: a binary that installed cleanly can later be blocked if its Developer ID cert is revoked, so the sweeper asks the live macOS policy engine (spctl) every time rather than caching a verdict — a benchmark never launches something the OS would refuse. It inspects with spctl and codesign only, never by launching: running a revoked binary raises a modal block dialog, which on an unattended machine wedges the lane until someone dismisses it.

Revocation has no release attached — nothing upstream changes — so the sweep runs daily across every supported version, not just the newest. That set is derived from the registry and then made resident on the runner, because a signature can only be assessed against a binary that is actually on disk. The distinction matters: when coverage was instead read off whatever happened to be installed, routine pruning shrank the sweep from 24 versions to 5 while every report still read 0 blocked. A scan now fails when any expected version is missing, so an incomplete sweep can never be mistaken for a clean one.

This is the general shape of every check here: absence must never render as a verdict. A monitor that cannot see has to say so, otherwise silence is indistinguishable from success.

04Three states of evidence

Every probe the pipeline runs answers in one of three states, never two: ok, fail, or indeterminate. The third state exists because "the probe could not run" is neither a pass nor a failure, and every attempt to squeeze it into a boolean lost something. An indeterminate blocks a new version's admission on that platform — a swap test that never happened must not count as proof the swap worked — but it never revokes a version that is already published, because the deep lane reports indeterminate for infrastructure gaps, and one broken runner must not pull healthy releases out from under users.

Evidence arrives from several independent lanes — the benchmark's launch verdicts, credential-less install probes, the deep execution probes — and the lanes combine with AND, never last-writer-wins: a platform passes only if every lane with an opinion passes, and a lane with no opinion abstains. This rule is the product of a real bug found twice: a passing install probe used to be able to overwrite a benchmark verdict that had watched the same release crash. A pass is admission evidence; it is never an answer to another lane's recorded failure.

The same honesty applies to what the registry says afterward. The per-platform verified stamp on a version is rewritten from the current run's evidence alone — a platform that stops passing loses its label even when the version itself rightly stays published — and a version nobody re-probed keeps its historical stamp untouched. Absence of evidence neither wipes a past verification nor invents one.

For Codex, a fourth kind of evidence gates stable releases: behavioral downgrade classification. Upstream releases migrate the CLI's local state database; OVM's whole purpose is letting you move backwards, so each new stable is classified — compatible, degraded, broken, or indeterminate — by actually exercising an older binary against state the newer one has touched. Only compatible evidence admits a stable; degraded, broken, or indeterminate results veto it outright — behavioral evidence is held to a stricter standard than the execution probes, whose indeterminate merely withholds, because "we could not prove the downgrade works" is exactly the situation a version manager must not ship through. The one adjudicable case is a pure disagreement between compatible behavioral evidence and the static migration analysis, resolved by an explicitly recorded human entry; genuine behavioral failures are never adjudicable at all.

05The registry defends itself

The registry is the thing everything else trusts, so its refresh pipeline treats its own inputs as hostile. Upstream answers are validated for shape before anything is transformed: a version list must be a list of plausible version strings — an error object that parses as JSON, a bare string, or a list of garbage each used to transform "successfully" into nonsense. The verification gate applies the same discipline to its own baseline: a missing field, a padded or impossible version string, or an unreadable file refuses the run loudly before anything is written, because a damaged baseline that fails open would silently stop gating anything, and one that fails closed would mass-withhold the registry.

Some corruption cannot be detected by shape at all: an upstream API returning a truncated-but-valid page is indistinguishable from the truth. So the pipeline bounds the damage instead — a mass-retirement circuit breaker refuses any single refresh that would retire more than a small fraction of the previously published versions, and the override that permits a genuine mass removal must name the product it applies to. The one truncated listing the breaker was built against would have retired 471 of 472 published versions in one write.

Finally, nothing the registry advertises may dangle: every dist tag must name a version the file actually publishes, checked against what survives the gate — not just against what this run removed — so a tag left stale by an earlier run heals instead of persisting.

06Trusting bytes, not names

Every install is a transaction: a per-version lock, a quarantined rebuild of the source tree, an .installing marker, and a .complete marker written only after every check passes. Downloads verify whatever the strongest available evidence is — publisher checksums where upstream ships them, npm's subresource integrity, declared byte lengths, and on macOS the publisher's code signature — and a published checksum that cannot be fetched refuses the install rather than quietly downgrading to weaker checks.

Importing a binary that is already on the machine (ovm adopt) is held to the strongest checks an import can have — it cannot reconstruct a publisher checksum or npm's integrity metadata for bytes it did not download, so what remains is built on one principle: the bytes verified must be the bytes published. The source file is resolved once, opened without following a final symlink, and proven — by file identity, not by path — to be the file its name claimed; every later step reads that open descriptor, so no rename, retarget, or link swap after the fact can change what is copied. The staged copy is then verified (signature and a re-run version check) before a same-directory rename publishes it. Every file the transaction writes into its freshly created tree is opened create-only, so a link planted at a destination fails the install instead of truncating whatever it points at.

The limits are documented as deliberately as the guarantees: this defends the version label and the transaction's own tree, not a machine an attacker already controls — someone who owns the disk can hand OVM a matching binary and OVM will believe it. Honest edges, stated in the code, are part of the methodology.

07The benchmark next dataset

Benchmarking is not a separate activity — it is the measurement half of the same per-release pass, run only on versions that cleared the canary. Each version is exercised in the same modes: spawn (--version), interactive (launch → the TUI accepting input), and response (launch → first real model answer, with the real MCP config, again with MCP disabled, and — for products with a faster tier — a fast variant).

A requested model is also a compatibility probe. Model identifiers and protocol expectations can move ahead of an older CLI harness, so a current flagship is not assumed to run on every historical version. Explicit unsupported-model responses are published as incompatible, kept out of latency aggregates, and shown as gaps in the model timeline. Unknown execution errors remain failures; neither class is converted into a successful benchmark sample.

Observed example: Codex 0.138.0 rejects gpt-5.6-sol because that model requires a newer Codex harness. OVM records the pair as incompatible; it is not evidence that the model itself failed and it is not plotted as a zero or slow response.

Sections 08–11, marked below, describe the next dataset — proposed changes to the benchmark, written for review before the ledgers are reset. Everything outside that band — the verification system above, the feed, and the pipeline below — runs today.

run #1      → cold  (first-init)   // recorded as *ColdMs
runs #2–4   → warm                 // *AvgMs (mean) + *StdDevMs

// today, for comparison — the cold run is smeared into the mean:
avg = mean(run#1, run#2, run#3)

// timing — split cold vs warm, per mode
spawnColdMs · spawnAvgMs · spawnStdDevMs
interactiveColdMs · interactiveAvgMs · interactiveStdDevMs
responseColdMs · responseAvgMs · responseStdDevMs · responseNoMcp* · responseFast*

// resource + energy (macOS / Apple Silicon)
peakRssBytes · steadyRssBytes · cpuUserMs · cpuSysMs · threadCount · openFds
energyMilliJoules · avgCpuPowerMw · idlePackagePowerMw · powerAvailable

// environment — self-describing platform
os · arch · hostCpuModel · hostCores · hostRamBytes · loadAvg1

12The public feed

Every verdict is broadcast, not just stored. OVM publishes a public Atom feed at /versions.atom for managed-product release verdicts. Its entries are the verification detections themselves: a new release, a version withheld from the registry because it hasn't verified on any platform, a promotion once it clears. The feed carries a WebSub hub link, and the pipeline pings the hub after each deploy, so subscribers are pushed the moment a new model clears the gate — no polling.

13One pass, two platforms

The scheduled and release-triggered lanes execute on the Mac mini (macOS) and in CI (Linux). The checks in a pass depend on the product's verification tier:

1. Install the release through OVM exactly as a user would.
2. Canary coding agents with the applicable tool-use, sidecar, version-swap, and schema checks.
3. Gatekeeper-check applicable signed macOS binaries against the live policy engine.
4. Benchmark Claude Code and Codex and append their ledger lines; Pi deliberately has no performance row.
5. Promote & broadcast — only cleared versions enter the registry; the site re-renders and the feed pushes.

The night the canary went live, upstream shipped four releases overnight; the scheduled run installed, probed, and benchmarked all four on both platforms and refreshed the registry autonomously, before anyone was awake.

14Reproducible by design

Nothing here is a private green check. The test suite is public, the canary's proof is a tagged pair, and every benchmark point in bench-data/ carries the runner and run URL that produced it. The benchmark measures how fast and how cheaply a CLI reaches a working answer, not model quality; energy attribution leans on a quiet, serialized machine and an idle baseline; cold numbers depend on their starting state, so each run records the session-store size behind them. When the next dataset ships, this page becomes the as-built record.