What a synthetic session store does and doesn't reproduce
These CLIs get slower as your session history grows, so a startup benchmark run against a real home is really measuring one machine's accumulated history. A synthetic store fixes that. It also turns out to reproduce Claude Code's response latency far better than Codex's — and the gap is the useful part.
1The problem with your own home directory
Both Claude Code and Codex keep session transcripts on disk and read them at startup. That makes the benchmark's most obvious setup — run it against the machine you actually work on — the least useful one. The number you get describes your history, not the release. Nobody can reproduce it, and it drifts every day you keep working.
Deleting the history is worse. An empty store measures a state almost no real user is in, and it hides exactly the regressions that only appear at scale.
2Matching shape, not just size
So the fixture generates a store designed to resemble a real one. The naive version — N files of the average size — gets the total bytes right and the behaviour wrong, because real session stores are heavy-tailed: a few enormous transcripts alongside many small ones.
The generator measures the real store first, recording the JSONL session
count, total bytes, mean size, and the sorted list of individual
file sizes, then reproduces that distribution rather than its
average. Only .jsonl sessions are matched — Claude also
keeps tool-result sidecar files in the same tree, and counting those as
sessions would distort the shape it is trying to copy.
3The measured comparison
On 2026-06-30 the same no-MCP response benchmark
(prompt: Reply with exactly: hi) was run against a real
store and a matched synthetic one, three samples per cell, at full scale
(20 sessions, matchReal on).
| Product | Real median | Synthetic median | Difference |
|---|---|---|---|
| Claude Code | 3,905 ms | 3,533 ms | −10% |
| Codex | 5,635 ms | 3,426 ms | −39% |
Claude Code lands within about 10% — close enough that a synthetic store is a reasonable stand-in. Codex does not: the synthetic run is nearly two seconds faster than the real one, which is far too large a gap to publish as equivalent.
Read the honest conclusion: a matched synthetic store is a good proxy for Claude Code's response latency and not a good proxy for Codex's. Whatever dominates Codex's real-home response time is something the fixture does not currently reproduce.
4Caveats worth stating
- Three samples per cell. Claude Code's real samples span 2,729–4,635 ms. At that spread, a 10% median difference is suggestive, not settled.
- One date, one machine. These are single-session numbers from 2026-06-30, not a tracked series.
- Response only. This comparison covers no-MCP response latency. It is not evidence about spawn or interactive startup, which are measured separately.
- The synthetic run is warmed. One warmup sample runs before the measured ones so first-run setup in a fresh synthetic home is not counted in the median — real homes are already warm.
- The response run mutates the store it measures, by adding its own benchmark sessions.
The raw numbers behind the table, including every individual sample, are published at /response-real-vs-synthetic.json.