We fixed the adapter that runs Claude Code in our agent eval and the score jumped +5.1 points. Then we found out the judge had been reading 433 characters of answers that were 2,102 characters long (medians). Up to half of that win was our own bug.
The result itself was solid on paper: 200 tasks on BU Bench v2.1, paired 95% CI [+1.0, +9.2], 97 wins, 67 losses, 36 ties. That's the kind of number you put in a launch post.
The judge saw a fifth of the answer
The adapter only handed the judge the text after FINAL ANSWER:. On 188 of 200 tasks, the actual deliverable sat above that marker. On most tasks the judge never saw the full answer.
Our fix changed the agent and also stopped cutting the answer. So the +5.1 was two changes stacked on top of each other, and we couldn't tell them apart from the score alone.
How we split it
We rejudged the 200 old baseline runs with the whole final message. No agent rerun, just the saved evidence and the full answer. That alone moved the baseline +2.6. A rejudge with nothing changed also moves things, by +0.8, so against that control the bug is worth +1.8.
The bug was worth 1.8 to 2.6 points, a third to a half of the headline. The agent change earned +2.5, and its interval includes zero.
If you take one thing from this post: rejudge your baseline before you believe a win.
Keep the evidence, or you can't rejudge
Every task in our eval writes its own result.json (answer, trajectory, artifacts) next to its score. The replay harness sends that file to the judge again and refuses to run unless the sha256, model, judge, dataset hash and target SHA all match the original. Since mid-September we've run 92 rejudge-only workflows.
It also handles judge crashes. A crash means no score, not a zero, because a zero charges the agent for our failure. It happened once in 1,400 judgments in our harness bake-off (one screenshot needed 30,456 image patches against a limit of 30,000) and twice in 200 on a browser-use 1.0.4rc1 run (empty JSON). All of them were rejudged.
Rejudging is noisy too. On 200 unchanged runs, the average task moved 6.0 points on identical evidence. The total barely moved. Don't read a rejudge of ten tasks.
The v2.1 judge has no human-label calibration yet, so I won't quote an accuracy. Our earlier rubric judge agreed with settled human labels on 81.7% of 104 hard internal tasks, and the humans disagreed with each other on 37% of them before the label was settled.
One config, one hash
Two runs only compare if they're the same experiment. Every dispatch resolves its git refs to SHAs once, and every task VM checks out exactly those.
The task subset and max_parallel are recorded but not hashed, so a 2-task smoke test and a 200-task run can share a hash. Compare runs only when the hash, task set and runner all match.
Tools lie about what they ran. The browser-use adapter reads options with defaults, so a typo quietly runs the default. One harness CLI ignored an unsupported reasoning_effort without a word, so its setup now fails if the installed binary doesn't report the requested version.
In July we replaced the database and MCP server from our February eval system with one 814-line eval.py and a 270-line workflow file.
One task tells you nothing
We ran one configuration 11 times: same hash, GPT-6 Luna at low effort, full BU Bench v2.1. The run means landed between 35.3 and 38.1, about one point of standard deviation.
Per task it's a mess. Of the 189 tasks scored in all 11 runs, only 6 got the same score every time. 105 crossed the 50 line at least once. The median task's score ranged over 36 points.
So we compare paired per-task deltas on the tasks scored in both arms, with a bootstrap CI, and check wins, losses and ties to see whether a delta comes from many tasks or three. Unscored tasks get dropped and listed, never zeroed. Our public leaderboard uses the same rule, and The Benchmark Behind the Benchmark goes deeper.
The task that cost $26.57
browser-use 1.0.3 had one task that ran 705 steps and cost $26.57, with a prompt-cache hit share of 0.057. Screenshot pruning rewrote the message history on every request, so the provider's cache never hit. 1.0.4rc1 prunes in batches.
Mean cost per task fell from $0.434 to $0.208 and the score stayed at 76. Both runs used max_parallel 200, above our own cap, but the pair is matched, so the delta holds. Report median and max cost, not just the mean.
Our own browser tool didn't win
Same agent loop (Codex 0.166.0), same model (GPT-6.1 Terra, medium), same judge. Only the browser tool changes. Seven arms of 200 tasks, one run each. We build Browser Harness and Browser Use MCP, and this is our benchmark and judge, so weigh it accordingly.
Playwright MCP scored 3.0 points above ours, inside the noise. The top five are a tie. Stagehand and OpenCode's browse tool lose clearly.
And yes, this harness had the FINAL ANSWER: clipping too. We found out after the bake-off. All seven arms went through it, so the comparison is fair between them, but we haven't rejudged, so the absolute scores may be low.
Then we tried to fix each harness from its traces, with a paired rerun against its own baseline. OpenClaw with the Browser Use CLI gained +7.1 [+3.2, +11.0]: its process supervisor was killing the browser daemon after every call. OpenClaw with native CDP moved -0.9 [-4.2, +2.4]. Hermes Agent stayed flat at -0.04 [-3.4, +3.3]; we blamed hidden background tabs for its timeouts, kept tabs in front, and the timeout rate didn't change (6.3% vs 5.9%). Wrong guess. Four of the attempts were nulls, and we report them as nulls.
The 8 rules
- Freeze code, data, model, judge and limits into one config hash. Match the task set and runner too.
- Smoke test with 2 exact task ids, never the full set.
- A judge error is missing data, not a zero. Rejudge from saved evidence.
- Same tasks on both arms. Paired deltas, bootstrap CI, wins, losses and ties.
- Rerun infra failures by exact task id, on both arms. On one baseline, rerunning 5 failures moved the mean from 72.83 to 73.34. Rerun one arm only and you get half a point for free.
- Cap concurrency at the provider's rate limit (12 for OpenAI, 20 for Anthropic). Throttling looks exactly like a regression.
- One versioned price table, and report median and max cost. Ours was wrong twice: cache reads priced at $0.20 per million instead of $0.10 turned a $48.86 run into $61.74.
- Before you believe an agent win, rejudge the baseline.
The BU Bench v2.1 tasks and judge are public in the benchmark repo.
