ovm.sh / research
research note · 2026-07-10 · incident write-up

The canary that catches what --version can't

On 2026-07-09, Codex 0.144.0 passed every check a version manager traditionally runs — binary present, executable, right version string — and could not execute a single shell command. This is the incident, the class of bug behind it, and the three-layer canary built so it never reaches a user again.

verdict

Health checks at the shallow end miss releases that drown at the deep end.

Codex 0.144.0 moved shell execution into a sidecar binary, codex-code-mode-host, spawned from the install directory. OVM installed the main binary — as it always had — and reported success. codex --version worked. codex login worked. The first real tool call died: failed to spawn code-mode host … No such file or directory. Every session, dead on arrival, on a code path no smoke test exercised.

three layers, each catching what the previous can't

01

Asset-manifest diff

Every new release's asset list is reduced to platform-independent stems and diffed against the previous release. codex-code-mode-host first appeared in rust-v0.143.0-alpha.38 — two days before it broke stable installs. Packaging early warning.

02

Execution probe + sidecar check

A real command driven through the OVM launcher in an isolated, credential-less home. It stops at the login boundary by design — so it separately verifies every sidecar the release ships is present and spawnable. A spawn failure is never excused as "just needs login."

03

Authenticated tool-use probe

A real session that must execute echo <nonce> | rev and produce the reversed nonce. A transcript echoing the prompt can never fake it. Any missing sidecar, any broken spawn path, under any future name, fails here exactly the way it would fail a user.

04

Proof, both directions

Published as a reproducible pair: v0.0.1 intentionally ships the pre-fix installer — the probe reddens and the release never promotes. v0.0.2 ships the sidecar-aware installer — same release, same probe, exit 0. Claiming a canary would have caught an incident is cheap; tags are not.

the part that matters

The night after the canary went live, upstream shipped four releases — a Claude stable promotion, the Codex 0.144.1 patch, a Codex alpha, and a Pi patch. The scheduled run caught all four, installed them, probed them on both platforms, and refreshed the registry autonomously, hours before anyone was awake. When the next 0.144.0-class release lands, the canary meets it first.