ovm.sh / dev log

Fifty-nine hours of a Codex that would not start

From the morning of 25 September until we noticed on the afternoon of the 27th, ovm install codex gave you the latest Codex in a form its terminal UI could not start — on macOS until OVM 0.1.10 that evening, and on Linux, which we missed, until 0.1.11 the next morning. Our verification gate looked at both affected versions and admitted them. This is what happened, with the clock on it, and what we changed so the gate says no next time.

Published · Last updated · codex · incident · release

If you installed Codex 0.157.0 or 0.157.1 through OVM and got this CLI has no complete local package, we are sorry. The break was ours, not Codex's, and it took us two and a half days to notice something our own pipeline should have caught within hours. OVM 0.1.11 fixes new installs and repairs existing ones, on macOS and on Linux; ovm self update then ovm install codex is the whole recovery. Until you update, codex --no-daemon works.

Linux was broken too, and our first fix missed it. An earlier version of this post said only that Linux "still installs the bare binary", which read as if Linux were fine. It was not, and Linux users stayed broken for eight and a half hours longer than macOS users. Section 5 says how that happened.

1What broke

From 0.157, the Codex terminal UI runs on a background app server — a daemon it starts on first launch. It will only start that server from a complete package: the binary, a manifest (codex-package.json), a bundled rg, and a codex-resources/ folder. OpenAI publishes that package as its own release asset, codex-package-<target>.tar.gz, alongside the bare binary it has always shipped.

OVM 0.1.9 installs the bare binary. It always has, and until 0.156.1 that was the whole program. On 0.157 the bare binary reports its version, runs codex exec, and does everything except the one thing most people launch it for: the interactive UI fails at startup with 

this CLI has no complete local package; install a packaged Codex CLI
or use the standalone installer … rerun with --no-daemon

The message is Codex's own. It does not say that the installer at fault was OVM, and there was nothing in our output to say it either.

One qualification. Codex installs its daemon into your Codex home the first time a complete package starts, and any Codex after that finds it there. If another install on the machine — the desktop app, an npm or standalone Codex — had already done that, the bare OVM install started fine. The break was for people whose Codex came from OVM alone, which is what OVM is for.

2Timeline

All times UTC.

59 hours from the first broken version to detection. Six minutes from detection to a fix on main, which says the fix was easy and the finding was the whole problem. Eight and a half hours from there to a release, most of it the gates a stable release goes through — CI, a rehearsal cut, the private canary, the public rebuild with signing, notarisation and attestation — plus two of our own release rules that had never met a hotfix branch. For Linux the whole span, from the first broken version to 0.1.11, was 77 hours.

3Why the gate said yes

Every Codex release goes through a verification gate before the registry vouches for it: state-database downgrade and recovery contracts on both platforms, a launch of the interactive UI in a real terminal, and a signature assessment of the binary. A version that fails is withheld and pages us. This one passed, eight times.

The launch check decided ready when it had seen two terminal control sequences — the ones that turn on bracketed paste and focus reporting — and the process was still alive half a second later. That heuristic was written for the failure it had seen: a UI that never comes up, or exits at once. The bare 0.157 binary emits both sequences as its very first output, then goes looking for its package, and fails. By then the check had already recorded a pass.

The gate was measuring the terminal being prepared, and calling that the program working. It is the same shape as the npm that answered from the past: a signal that looks like success, read at the same confidence as success.

The other checks could not have caught it either. The state contracts test the database, which this bug does not touch; codex exec does not need the daemon. And the credential-less probe we run on GitHub-hosted runners — the lane closest to a new user's machine, because it installs the released OVM into an empty home — only asked codex --version, which a bare binary answers perfectly. That probe did fail on macOS for 0.157.1, but for an unrelated reason: the runner's shared anonymous GitHub API quota was exhausted and the download got a 403. Had it downloaded, it would have passed.

4What changed

5How we missed Linux

We found the break on a Mac, fixed it on macOS, and deferred Linux: its package puts the sandbox helper somewhere else, and we wanted to check that before switching. That was a reasonable thing to check. What we did not do was ask the prior question — whether Linux users were broken in the meantime — and the post said "Linux still installs the bare binary" as if that were a neutral fact. A bare binary was the whole bug. Every check we had on Linux was the same kind as the ones that missed it on macOS, so nothing was going to tell us.

It surfaced the next morning while chasing a different open question. Checking before switching turned out to matter as well: the first Linux fix would have been refused by our own post-install check, which looked for the sandbox helper in the old place. Code review caught that, and a real Ubuntu machine confirmed it, before anything shipped. The rule we are keeping: when a fix is scoped to one platform, the others get checked for the same break the same day, not deferred along with the fix.

6What we are still doing

7References