infrastructure
The benchmark machine: everything that runs on it
The methodology describes how OVM verifies upstream releases and measures the benchmarked coding agents. This page documents where: one always-on Apple Silicon Mac mini, operated by the maintainer, that carries the authenticated lane of the pipeline. Every job that touches it is listed here — what it does, when it fires, and the trust rules that decide what is allowed near it.
- One machine, every authenticated verdict
- Real subscription auth, resident caches, and a contemporaneous macOS + Linux pass can't live on a throwaway CI runner. The Mac mini is where the pipeline touches reality.
- Schedule and dispatch only
- Nothing event-driven by untrusted code ever runs on it. No pull-request trigger targets the machine — from the private pipeline repo or the public source repo, which uses only GitHub-hosted runners.
- Fail closed, alert loud
- A leg that can't run produces an alert and a withheld verdict — never a silent pass. A truncated view judges nothing.
01Why a physical machine at all
Three things the pipeline needs cannot be faked on ephemeral CI:
Authenticated sessions. The deep benchmark drives each CLI against real, signed-in accounts — actual model responses, not mocks. Hosted runners are wiped between jobs; holding a live login there is both impractical and a bad idea. The mini holds the sessions, so the numbers describe what a subscriber actually experiences.
A lived-in home directory. Cold-start measurements on a runner with an empty home flatter every tool (methodology §11). The mini's benchmark user has a realistic, persistent environment, and the ledgers record the machine state alongside every number.
macOS reality. Gatekeeper verdicts, signature revocations, and notarization behavior only exist on real macOS with a real security assessment policy. The signature-drift watch below is only meaningful because the same physical machine asks the same question every day.
02The inventory
Everything OVM runs on the machine, exhaustively. If a job isn't in this table, it doesn't run there.
| job | fires | what it does |
|---|---|---|
| version watch | every 5 min | A resident daemon (not a CI job) polls upstream registries for new Claude Code, Codex, and Pi releases. On a hit it dispatches the deep pass. Every tick it also measures the machine — container VM, services, free disk — and publishes both the pulse and those readings as one heartbeat (see §04). |
| benchmark deep | new release · nightly | The authenticated lane. A native macOS leg and an arm64 Linux container leg install each candidate and run the product's verification tier. Coding agents get the applicable execution probes and benchmarks. Probe verdicts feed the registry gate; ledgers are committed; the site data republishes. |
| gatekeeper watch | daily | Apple can revoke a signing certificate without the binary
changing. This job re-inspects every installed macOS Codex binary
with spctl + codesign — it never launches
them — and raises an issue if a formerly-runnable version has
flipped to blocked. Found the real revocation that moved the
supported floor. |
| alpha canary | each OVM release | Every OVM release candidate is installed and exercised on real hardware, in a throwaway home directory, before it can be promoted. The verdict lands as a commit status the release workflow requires (see §05 for the trust design). |
| status / prune | manual | Housekeeping, dispatch-only. A read-only disk diagnostic, and a retention prune that uninstalls old benchmarked versions on both platforms — benchmark ledgers are never touched, so trend history survives pruning. |
| vm maintenance | manual | Resizes or rebuilds the Linux container VM when disk pressure demands it. The Linux ledgers live in git, so the VM is disposable by design. |
What does not run on the machine: CI for pushes and pull requests, release builds, and the unauthenticated benchmark lane — all of those run on GitHub-hosted runners.
03The path of a new upstream release
One pass, two platforms, one verdict
- Detection. The resident watcher checks Claude Code, Codex, and Pi every five minutes.
- Verification. Both legs — macOS native, Linux container — install the release and run its verification tier of execution probes.
- Measurement. The same pass benchmarks Claude Code and Codex under real authentication. Pi intentionally carries no performance measurements.
- Verdict. Probe results update the registry gate — a release that fails execution is withheld from promotion, per platform. Ledgers commit, the feed and site update.
04Watching the watcher
A machine that quietly dies would look identical to a world with no new releases — and per the methodology's rule, silence must never read as success. So the mini's watcher pushes a heartbeat commit on every poll, and an hourly job on GitHub-hosted infrastructure — deliberately not on the mini — checks that the heartbeat is fresh. If the machine goes dark, the hosted canary raises the alarm. The failure mode of the failure detector is covered by putting it somewhere else.
A pulse only proves the machine is ticking, though — not that it can still do the work. So the heartbeat carries the machine's vitals as well: whether the Linux container VM is up, whether the machine is still reachable, and how much disk is left on each volume — OVM's own dependencies and nothing else, since those readings travel into an alert and an alert is no place to enumerate a shared machine. The same hosted canary judges those against thresholds and opens an issue when one is breached. A machine that is up but degraded now reports itself, instead of looking healthy until the next nightly run discovers a leg it cannot execute.
That consolidation was earned the hard way. A separate health check used to run on the mini, writing its findings to a local log file that nothing ever read — a watcher with no consumer, which is indistinguishable from no watcher at all. Merging its questions into the heartbeat means there is exactly one signal to emit and one place that judges it.
One heartbeat, two questions — judged off the machine
The same principle applies inside a run: when one leg of a deep pass cannot execute (say, the container VM is unavailable), the run fails loudly and alerts. It does not publish a half-view, and a missing verdict never regresses a previously-verified release.
05The trust model
A self-hosted runner is a standing computer that executes what CI tells it to — which makes what may reach it the entire security question. The rules:
What may reach the machine — and what may not
Release candidates are the exception that proves the rule: they do run on the mini, in a throwaway home directory, while the token-bearing job that reports their verdict runs elsewhere.
No untrusted triggers. Every job on the machine fires
from a schedule or an explicit maintainer dispatch. No
pull_request workflow targets it, and the public source
repository's workflows use GitHub-hosted runners exclusively — outside
code has no path onto the machine.
Untrusted artifacts are quarantined from credentials. The alpha canary exists to run candidate binaries — by definition not yet trusted. It is split into separate jobs so that the token-bearing step that reports the verdict never runs on the same machine as, or after, the candidate artifacts. The candidate runs in a throwaway home directory; the runner's real state is never read or written.
Inspection over execution. The signature-drift watch
never launches the binaries it audits — spctl and
codesign only. Nothing runs just to be checked.
Honest limits. The deep lane's job is to execute real upstream releases with real authentication — that is the product, and it is an irreducible exposure: a malicious upstream release would run with the benchmark user's privileges. The mitigations are scoped accounts, macOS signature assessment before anything launches, the container boundary on the Linux leg, and the fact that every binary comes from the vendor's official distribution channel — the same artifacts every user runs.
06An honest machine, measured honestly
The mini is a real computer, not a cleanroom: it is also the maintainer's always-on worker and hosts unrelated private workloads. Rather than pretend otherwise, the pipeline records the machine state with every measurement — host CPU model, core count, memory, and load average ride along in each ledger row (methodology §10), and every published number carries the runner identity and the exact CI run that produced it. A noisy run is visible as a noisy run, and anyone can check the provenance of any figure on the benchmark page.
That is the deal this page offers: the code that orchestrates the machine stays private — publishing runbooks for a standing computer is a gift to nobody but attackers — but every job that runs on it, every trigger that can reach it, and every number that leaves it are public and accounted for.