Browser UseTalk to sales
All articles

Odysseys2: a long-horizon web agent benchmark

100 long tasks on the live web, from Carnegie Mellon University and Browser Use. The best agent fully completes 47 of 100.

  • Lawrence Jang (CMU)
  • Alexander Yue (Browser Use)
  • Magnus Müller (Browser Use)
  • Gregor Zunic (Browser Use)
  • Andrew Keunwoo Jang (CMU)
  • Younghan Park (CMU)
  • Mareks Woodside (CMU)
  • Daniel Fried (CMU)
  • Jing Yu Koh (CMU)
  • Ruslan Salakhutdinov (CMU)

Research

Odysseys2: a collaboration between Carnegie Mellon University and Browser Use
Odysseys2: a collaboration between Carnegie Mellon University and Browser Use

Odysseys2 is a long-horizon web agent benchmark from Carnegie Mellon University and Browser Use: 100 long tasks on the live web, each graded on 8 to 12 pass/fail checks. It builds on Carnegie Mellon University's Odysseys. The full paper-style write-up, with interactive charts and all 100 tasks, is at odysseysbench.com/v2. This post is the short version.

The result

The best agent fully completes 47 of 100 tasks. GPT-6 Astra (default effort) has a mean rubric score of 84.6, yet falls short on the other 53.

Fig. 1Tasks fully completed, out of 1001 task fully completednot fully completed

GPT-6 Astra fully completes 47 of 100 tasks. No other agent reaches 20, and 5 complete none.

Fully completed = every rubric check met. 15 agents, 1 attempt per task, BrowserCode harness, Odysseys2 judge, default effort unless marked high. Mean = mean rubric score, 0 to 100.

Sorted by tasks fully completed, then mean score. Opus 5.5 is second with 18. Fable 5: content filters blocked 2 tasks, so its mean uses 98. Scores come from an LLM judge that has not been validated against a human reference.

15 agents, one 60-minute attempt per task, all in BrowserCode. Opus 5.5 is second with 18 fully completed. Five agents complete none. Full leaderboard.

Fast is not good

Fig. 2Score banked over time

At 15 minutes GPT-5.6 Luna has 37 points and Opus 5.5 has 18. By 60 minutes, Opus 5.5 and Grok 4.7 finish well above Luna.

Mean rubric score (0 to 100) from attempts finished by each minute, over all 100 attempts. BrowserCode harness, high effort, 1 dot per minute.

Grok 4.7 catches Luna at about 30 minutes and Opus 5.5 at about 36. An attempt adds its score when it finishes, not while it runs. Luna's line ends at 58.9, below its 59.7 mean, because one attempt has no recorded time.

At 15 minutes GPT-5.6 Luna leads with 37 points to Opus 5.5's 18. Opus catches up at about 36 minutes and ends at 71.2. More.

The harness matters

Fig. 3Same model, three harnesses

In the Python library Luna stays near 30. In BrowserCode it climbs to 63.0, in Pi to 71.9.

Mean rubric score, 0 to 100, for GPT-5.6 Luna on all 100 tasks, by reasoning effort. One run per setup. Python library has no max setting.

Harnesses hand the judge different records of the run, so these gaps mix differences in execution with differences in the evidence available for grading. Pi was tested with Luna only.

Same model, GPT-5.6 Luna: about 33 in the Browser Use Python library, 59 to 72 in BrowserCode and Pi at high to max effort. More.

Grading changes scores

Fig. 4Same attempts, two judgesOdysseys judgeOdysseys2 judge

Under the original judge every agent scores 77 to 96. Under the Odysseys2 judge the same attempts spread from 2 to 85.

Mean rubric score, 0 to 100, for 14 agents, each graded twice on the same saved attempts. Gemini 3.7 Flash: 94.9 vs 5.8.

The original Odysseys judge grades one check at a time from the action history and screenshots and does not see delivered files. The Odysseys2 judge also reads the files and zeroes flagged attempts. These differences measure disagreement between the judges; lower scores alone do not establish greater accuracy.

On the same attempts, the original Odysseys judge and the stricter Odysseys2 judge, which also reads the delivered files, disagree a lot: Gemini 3.7 Flash scores 94.9 under one and 5.8 under the other. That is disagreement, not proof of accuracy. How tasks are built, how grading works, the two judges.

Limitations

The judge is not yet validated against a reliable human reference, reward-hacking flags are model judgments, and the live web changes. See validating our judge for the full list.

Get the data

To cite Odysseys2:

@misc{jang2026odysseys2,
  title        = {Odysseys2: A Web Agent Benchmark Based on Real User Requests},
  author       = {Jang, Lawrence and Yue, Alexander and M{\"u}ller, Magnus and
                  Zunic, Gregor and Jang, Andrew Keunwoo and Park, Younghan and
                  Woodside, Mareks and Fried, Daniel and Koh, Jing Yu and Salakhutdinov, Ruslan},
  year         = {2026},
  howpublished = {\url{https://odysseysbench.com/v2/}}
}

Thanks to our co-authors at Carnegie Mellon University: Lawrence Jang, Andrew Keunwoo Jang, Younghan Park, Mareks Woodside, Daniel Fried, Jing Yu Koh and Ruslan Salakhutdinov.

Published

Keep reading

All articles

Comparisons

Browser Use vs Browserless: Pricing, Stealth & Features

Compare Browser Use and Browserless on pricing, stealth, and performance. Find the right browser infrastructure for your automation workflows.

Nik Tomazic

Roundups

11 Best Browser Automation Tools: Agents, Scripts, and Cloud Browsers

Compare 11 browser automation options across AI agents, scripts, cloud browsers, and visual workflows, with pricing, capabilities, and practical tradeoffs.

Nik Tomazic

Roundups

The Best Browser Agent MCP Servers (2026)

MCP servers that take a goal and run a whole browser agent: Browser Use Cloud, Browser Use open source, Hyperbrowser and agent-browser. Models, cost, live view.

Gregor Zunic