Engineering

LLM-as-a-Judge Bug: Our Judge Read a Fifth of Each Answer

A fix scored +5.1 points on our agent eval. Up to half of that came from a bug: our LLM judge only saw the end of each answer. How rejudging saved runs caught it.

Magnus Müller

Co-Founder, CEO

TL;DR: A change to how we run Claude Code in our agent eval scored +5.1 points. Our LLM judge had been reading only the text after a FINAL ANSWER: marker, a median of 433 characters out of 2,102, and the change also stopped that cut. Rejudging the saved baseline runs with the full answer showed the bug was worth 1.8 to 2.6 points, a third to a half of the win. Before you believe an agent eval win, rejudge your baseline with the same judge.

We fixed the adapter that runs Claude Code in our agent eval, and the score jumped +5.1 points. Then we found out the judge had been reading 433 characters of answers that were 2,102 characters long (medians). Up to half of that win was our own bug.

On paper the result was solid: 200 tasks on BU Bench v2.1, a paired 95% CI of [+1.0, +9.2], 97 wins, 67 losses and 36 ties. That's the kind of number you put in a launch post.

The LLM judge saw a fifth of the answer

The adapter only handed the judge the text after FINAL ANSWER:. On 188 of 200 tasks, the actual deliverable sat above that marker, so on most tasks the judge never saw the full answer.

Terminal output: on 188 of 200 tasks the deliverable sits above the FINAL ANSWER marker; median 2,102 characters written, 433 judged.
What the judge read on the baseline arm.

Our fix changed the agent and also stopped cutting the answer. The +5.1 was two changes stacked on top of each other, and the score alone couldn't tell them apart.

How rejudging split the score

We rejudged the 200 old baseline runs with the whole final message. We didn't rerun the agent, only sent the saved evidence and the full answer to the judge again. That alone moved the baseline up 2.6 points. A rejudge with nothing changed also moves things, by 0.8 here, so against that control the bug is worth 1.8.

Terminal output splitting the reported +5.1 [+1.0, +9.2]: harness bug +2.6 [+0.7, +4.6] against original judgments or +1.8 [-0.1, +3.7] against a rejudge control; agent change +2.5 [-1.4, +6.4]; rejudge with nothing changed +0.8 [-0.8, +2.4].
200 tasks, Claude Sonnet 5.5, paired bootstrap with 10,000 resamples.

So the bug was worth 1.8 to 2.6 points, a third to a half of the headline. The agent change earned +2.5, and its interval includes zero.

Keep the evidence so you can rejudge

None of this works unless the old runs are still there. Every task in our eval writes its own result.json (answer, trajectory, artifacts) next to its score. The replay harness sends that file to the judge again, and it refuses to run unless the sha256, model, judge, dataset hash and target SHA all match the original. Since mid-September we've run 92 rejudge-only workflows.

Each run's config is resolved to exact git SHAs and hashed once, so two runs only count as the same experiment if their hashes match. The task subset and max_parallel are recorded but not hashed, so we also check that the task set and runner match before comparing.

The same setup handles judge crashes. A crash means no score, not a zero, because a zero charges the agent for our failure. It happened once in 1,400 judgments in one bake-off (one screenshot needed 30,456 image patches against a limit of 30,000) and twice in 200 on a browser-use 1.0.4rc1 run (empty JSON). All of them were rejudged.

How noisy an LLM judge is

Rejudging is noisy too. On 200 unchanged runs, the average task moved 6.0 points on identical evidence, even though the total barely moved. Don't read anything into a rejudge of ten tasks.

The agent is noisy on top of that. We ran one configuration 11 times (same hash, GPT-6 Luna at low effort, full BU Bench v2.1). The run means landed between 35.3 and 38.1, about one point of standard deviation. Per task it was a mess: of the 189 tasks scored in all 11 runs, only 6 got the same score every time, and 105 crossed the 50 line at least once. The median task's score ranged over 36 points.

So we compare paired per-task deltas on the tasks scored in both arms, with a bootstrap CI, and look at wins, losses and ties to see whether a delta comes from many tasks or three. Unscored tasks get dropped and listed, never zeroed. Our public leaderboard uses the same rule, and The Benchmark Behind the Benchmark goes deeper.

The v2.1 judge has no human-label calibration yet, so I won't quote an accuracy for it. Our earlier rubric judge agreed with settled human labels on 81.7% of 104 hard internal tasks, and the humans disagreed with each other on 37% of them before the label was settled.

Our rules for LLM-as-a-judge evals

  1. Freeze code, data, model, judge and limits into one config hash, and match the task set and runner too.
  2. Smoke test with 2 exact task ids, never the full set.
  3. A judge error is missing data, not a zero. Rejudge it from the saved evidence.
  4. Run the same tasks on both arms and report paired deltas, a bootstrap CI, and wins, losses and ties.
  5. Rerun infra failures by exact task id on both arms. On one baseline, rerunning 5 failures moved the mean from 72.83 to 73.34, so rerunning one arm only gives you half a point for free.
  6. Cap concurrency at the provider's rate limit (12 for OpenAI, 20 for Anthropic). Throttling looks exactly like a regression.
  7. Before you believe an agent win, rejudge the baseline.

The BU Bench v2.1 tasks and judge are public in the benchmark repo, if you want to try the last rule on your own agent.

Published