The canary that catches what --version can't
Codex 0.144.0 installed cleanly, printed the right version string, and could not execute a single shell command. Every check a version manager traditionally runs passed. Here is what the release actually broke, and the three layers built so the next one is caught before anyone types it.
1The incident
On 2026-07-09, Codex 0.144.0 shipped a new runtime architecture: shell
commands now route through a sidecar binary,
codex-code-mode-host, which the CLI spawns from its own
install directory. OVM's installer downloaded the main codex
binary — as it always had — and reported success.
codex --version worked. codex login worked. The
install looked healthy by every measure. Then the first real session
tried to run a command:
failed to spawn code-mode host ~/.ovm/products/codex/versions/rust-v0.144.0/release/bin/codex-code-mode-host: No such file or directory
Every tool call, in every session, dead on arrival — and nothing in the install pipeline had any idea. The breakage only surfaces after authentication, at tool-execution time, on a code path no smoke test was exercising.
2Why this class of bug is invisible
A version manager's usual health checks live at the shallow end: the
binary exists, it is executable, --version prints the right
string. Codex 0.144.0 passed all of them while being completely
unusable.
The deep end — can this install actually do work — requires either knowing every runtime dependency a release ships (a moving target; Codex releases carry a dozen companion binaries, most of which are not runtime requirements), or observing real behaviour: a genuine session, authenticated, executing a genuine command.
The premise the verification lane is built on: installed is not works. Everything below follows from refusing to treat a clean install as evidence.
3Three layers
The release-radar canary now runs three checks, each catching what the previous one cannot.
| Layer | What it does | What it catches |
|---|---|---|
| asset-manifest diff | Every new release's asset list is reduced to version- and platform-independent stems and diffed against the previous release. | Packaging changes, early. codex-code-mode-host first appeared in rust-v0.143.0-alpha.38, two days before it broke stable installs. |
| execution probe | A real command driven through the OVM launcher in an isolated, credential-less home. It separately verifies that every sidecar the release ships is present next to the installed binary and spawnable. | Missing or unspawnable helper binaries. A spawn failure is never excused as "just needs login." |
| authenticated tool-use probe | With credentials present, a real billed session must execute echo <nonce> | rev and return the reversed nonce. |
The whole class, behaviourally. A transcript echoing the prompt back cannot fake a reversed nonce — only a real shell execution produces it. |
The canary runs the full matrix on macOS and Linux every four hours, installs anything new upstream, probes it, and refreshes the version registry — before a human would have noticed the release exists.
4Proving it both directions
Claiming a canary would have caught an incident is cheap. So it
was proved in both directions against the real upstream release
(rust-v0.144.1), using two OVM builds published as a pair.
Down path — the broken installer must fail loudly
OVM v0.0.1 is the last build before the fix. Installing
codex 0.144.1 with it reproduces the incident exactly: codex
present, sidecar absent.
codex rust-v0.144.1: broken — sidecar codex-code-mode-host missing: release ships codex-code-mode-host but .../release/bin/codex-code-mode-host is absent exit code 1
The canary reddens, the release is never promoted, and no user sees the spawn failure.
Up path — the fixed installer must pass quietly
OVM v0.0.2 contains the sidecar-aware installer. Same
release, same probe: the sidecar is present and spawnable, the execution
probe stops cleanly at the auth boundary (proof the launch path works),
and with credentials the tool-use probe round-trips the nonce through a
real shell. Exit code 0.
Both runs are reproducible from the published tags. That is why
v0.0.1 exists as a prerelease with a warning label: it
intentionally ships the broken installer, as the demo's "before".
5The part that matters
The night after the canary went live, upstream shipped four releases — a Claude stable promotion, a Codex stable patch (0.144.1, on the heels of the incident), a Codex alpha, and a Pi patch. The scheduled run caught all four, installed them, probed them on both platforms, and refreshed the registry autonomously, hours before anyone was awake.
That is the whole promise. When the next 0.144.0-class release lands, the canary meets it first. How the rest of the verification lane fits together is on how it works.