Engineering

Our Eval Said +5.1. Up to Half of It Was Our Own Bug.

A fix scored +5.1 on our agent eval. Then we rejudged the saved runs and found our own harness had been feeding the judge a fifth of each answer.

Magnus Müller

Co-Founder, CEO

We fixed the adapter that runs Claude Code in our agent eval and the score jumped +5.1 points. Then we found out the judge had been reading 433 characters of answers that were 2,102 characters long (medians). Up to half of that win was our own bug.

The result itself was solid on paper: 200 tasks on BU Bench v2.1, paired 95% CI [+1.0, +9.2], 97 wins, 67 losses, 36 ties. That's the kind of number you put in a launch post.

The judge saw a fifth of the answer

The adapter only handed the judge the text after FINAL ANSWER:. On 188 of 200 tasks, the actual deliverable sat above that marker. On most tasks the judge never saw the full answer.

Terminal output: on 188 of 200 tasks the deliverable sits above the FINAL ANSWER marker; median 2,102 characters written, 433 judged.
What the judge read on the baseline arm.

Our fix changed the agent and also stopped cutting the answer. So the +5.1 was two changes stacked on top of each other, and we couldn't tell them apart from the score alone.

How we split it

We rejudged the 200 old baseline runs with the whole final message. No agent rerun, just the saved evidence and the full answer. That alone moved the baseline +2.6. A rejudge with nothing changed also moves things, by +0.8, so against that control the bug is worth +1.8.

Terminal output splitting the reported +5.1 [+1.0, +9.2]: harness bug +2.6 [+0.7, +4.6] against original judgments or +1.8 [-0.1, +3.7] against a rejudge control; agent change +2.5 [-1.4, +6.4]; rejudge with nothing changed +0.8 [-0.8, +2.4].
200 tasks, Claude Sonnet 5.5, paired bootstrap with 10,000 resamples.

The bug was worth 1.8 to 2.6 points, a third to a half of the headline. The agent change earned +2.5, and its interval includes zero.

If you take one thing from this post: rejudge your baseline before you believe a win.

Keep the evidence, or you can't rejudge

Every task in our eval writes its own result.json (answer, trajectory, artifacts) next to its score. The replay harness sends that file to the judge again and refuses to run unless the sha256, model, judge, dataset hash and target SHA all match the original. Since mid-September we've run 92 rejudge-only workflows.

It also handles judge crashes. A crash means no score, not a zero, because a zero charges the agent for our failure. It happened once in 1,400 judgments in our harness bake-off (one screenshot needed 30,456 image patches against a limit of 30,000) and twice in 200 on a browser-use 1.0.4rc1 run (empty JSON). All of them were rejudged.

Rejudging is noisy too. On 200 unchanged runs, the average task moved 6.0 points on identical evidence. The total barely moved. Don't read a rejudge of ten tasks.

The v2.1 judge has no human-label calibration yet, so I won't quote an accuracy. Our earlier rubric judge agreed with settled human labels on 81.7% of 104 hard internal tasks, and the humans disagreed with each other on 37% of them before the label was settled.

One config, one hash

Two runs only compare if they're the same experiment. Every dispatch resolves its git refs to SHAs once, and every task VM checks out exactly those.

Terminal output of the resolved config: platform and target SHAs, dataset hash, harness, model, options, judge and limits are hashed; message, actor and run id are not. The config_hash lands on the eval, every datapoint and the trace span.
Same hash, same configuration, whatever someone named the run.

The task subset and max_parallel are recorded but not hashed, so a 2-task smoke test and a 200-task run can share a hash. Compare runs only when the hash, task set and runner all match.

Tools lie about what they ran. The browser-use adapter reads options with defaults, so a typo quietly runs the default. One harness CLI ignored an unsupported reasoning_effort without a word, so its setup now fails if the installed binary doesn't report the requested version.

In July we replaced the database and MCP server from our February eval system with one 814-line eval.py and a 270-line workflow file.

One task tells you nothing

We ran one configuration 11 times: same hash, GPT-6 Luna at low effort, full BU Bench v2.1. The run means landed between 35.3 and 38.1, about one point of standard deviation.

Per task it's a mess. Of the 189 tasks scored in all 11 runs, only 6 got the same score every time. 105 crossed the 50 line at least once. The median task's score ranged over 36 points.

So we compare paired per-task deltas on the tasks scored in both arms, with a bootstrap CI, and check wins, losses and ties to see whether a delta comes from many tasks or three. Unscored tasks get dropped and listed, never zeroed. Our public leaderboard uses the same rule, and The Benchmark Behind the Benchmark goes deeper.

The task that cost $26.57

browser-use 1.0.3 had one task that ran 705 steps and cost $26.57, with a prompt-cache hit share of 0.057. Screenshot pruning rewrote the message history on every request, so the provider's cache never hit. 1.0.4rc1 prunes in batches.

Terminal output comparing the worst task: browser-use 1.0.3 took 705 steps, cost 26.57 dollars, cache hit share 0.057, score 57; 1.0.4rc1 took 48 steps, cost 0.26 dollars, score 100. Whole run score 76.0 to 76.1, mean cost 0.434 to 0.208 dollars.
Same 200 tasks, same model (GPT-6.1 Terra, medium).

Mean cost per task fell from $0.434 to $0.208 and the score stayed at 76. Both runs used max_parallel 200, above our own cap, but the pair is matched, so the delta holds. Report median and max cost, not just the mean.

Our own browser tool didn't win

Same agent loop (Codex 0.166.0), same model (GPT-6.1 Terra, medium), same judge. Only the browser tool changes. Seven arms of 200 tasks, one run each. We build Browser Harness and Browser Use MCP, and this is our benchmark and judge, so weigh it accordingly.

Terminal table versus Browser Harness at 70.5: Playwright MCP 73.5 (+3.0, -0.4 to +6.6), Browser Use MCP 71.7, Chrome DevTools 70.3, agent-browser 69.6, Stagehand 62.8 (-7.6, -11.6 to -3.9), OpenCode browse 62.4 (-8.0, -11.8 to -4.4).
BU Bench v2.1, 200 tasks per arm, paired 95% CI against Browser Harness.

Playwright MCP scored 3.0 points above ours, inside the noise. The top five are a tie. Stagehand and OpenCode's browse tool lose clearly.

And yes, this harness had the FINAL ANSWER: clipping too. We found out after the bake-off. All seven arms went through it, so the comparison is fair between them, but we haven't rejudged, so the absolute scores may be low.

Then we tried to fix each harness from its traces, with a paired rerun against its own baseline. OpenClaw with the Browser Use CLI gained +7.1 [+3.2, +11.0]: its process supervisor was killing the browser daemon after every call. OpenClaw with native CDP moved -0.9 [-4.2, +2.4]. Hermes Agent stayed flat at -0.04 [-3.4, +3.3]; we blamed hidden background tabs for its timeouts, kept tabs in front, and the timeout rate didn't change (6.3% vs 5.9%). Wrong guess. Four of the attempts were nulls, and we report them as nulls.

The 8 rules

  1. Freeze code, data, model, judge and limits into one config hash. Match the task set and runner too.
  2. Smoke test with 2 exact task ids, never the full set.
  3. A judge error is missing data, not a zero. Rejudge from saved evidence.
  4. Same tasks on both arms. Paired deltas, bootstrap CI, wins, losses and ties.
  5. Rerun infra failures by exact task id, on both arms. On one baseline, rerunning 5 failures moved the mean from 72.83 to 73.34. Rerun one arm only and you get half a point for free.
  6. Cap concurrency at the provider's rate limit (12 for OpenAI, 20 for Anthropic). Throttling looks exactly like a regression.
  7. One versioned price table, and report median and max cost. Ours was wrong twice: cache reads priced at $0.20 per million instead of $0.10 turned a $48.86 run into $61.74.
  8. Before you believe an agent win, rejudge the baseline.

The BU Bench v2.1 tasks and judge are public in the benchmark repo.

Published