The benchmarks that measure AI agents doing real tasks on websites come in three kinds. Sandboxed sites that the benchmark controls (WebArena, VisualWebArena, WorkArena, REAL) give reproducible scores, mostly checked by code. Live-web benchmarks (Online-Mind2Web, WebVoyager, WebBench, AssistantBench, BrowseComp) run on real sites and are graded by humans, an LLM judge or an answer key, while offline datasets like Mind2Web score each predicted step against recorded human actions.
We make Browser Use and run our own benchmark, BU Bench V2.1. It's one row in the table, marked as ours. Everything about the other benchmarks comes from their own papers, repos and sites, checked October 25, 2026.
Three kinds of benchmark
- Sandbox. The sites run in your own containers, on hosted instances or as hosted replicas, all under the benchmark's control. A script checks the final page, database or answer, so a score reproduces. A few checks still ask an LLM. The catch is coverage: the Online-Mind2Web paper notes that site diversity in sandboxes "is inherently limited due to the sheer difficulty of creating full replica of modern websites."
- Live web. The agent uses the real site, with its pop-ups, logins and bot checks. That's closest to what a user sees, but sites change, so tasks break and scores drift over time.
- Offline. The agent sees saved snapshots of pages and predicts the next action. Mind2Web evaluates each step "independently with the ground truth action history provided", so it measures picking the right element, not finishing a task.
Comparison
| Benchmark | What it measures | Tasks | Environment | Grading | Source |
|---|---|---|---|---|---|
| WebArena, 2023 | Long tasks on working copies of an online store, a forum, GitLab and a content management system, plus a map and manuals | 812 | Self-hosted sandbox | Programmatic checks; some answers matched by an LLM (GPT-4) | Paper, repo |
| VisualWebArena, 2024 | Tasks that need the agent to understand images on the page | 910 | Self-hosted sandbox (classifieds, shopping, Reddit) | Programmatic checks; some ask a vision model about an image | Paper, repo |
| WorkArena / WorkArena++, 2024 | Knowledge-work tasks in ServiceNow: lists, forms, knowledge bases, catalogs, dashboards, and multi-step workflows | 33 tasks (19,912 instances); 682 in WorkArena++ | Hosted ServiceNow instances (gated access) | Programmatic validation of the instance | Paper, ++ paper, repo |
| REAL (AGI Inc), 2025 | Retrieval and state-changing tasks on replicas of Amazon, Airbnb, Gmail, Uber and others | 112 on 11 sites (paper) | Hosted deterministic replicas | Programmatic state checks; rubric LLM judge for retrieval | Paper, site |
| WebVoyager, 2024 | End-to-end tasks on 15 popular sites | 643 | Live web | Human evaluation; GPT-4V judge agrees with humans 85.3% of the time | Paper, repo |
| Online-Mind2Web, 2025 | Realistic user tasks across many sites, in easy, medium and hard | 300 on 136 sites | Live web | Separate human-eval and WebJudge (o4-mini) auto-eval leaderboards; WebJudge about 85% agreement with humans | Paper, repo |
| WebBench (Halluminate), 2025 | Read tasks plus create, update, delete and file tasks, including logins | 2,454 public on 452 sites (of about 5,750) | Live web | Human annotators | Repo, report |
| AssistantBench, 2024 | Realistic, time-consuming information tasks with a checkable answer | 214 | Live web | Automatic match against a gold answer (word F1, partial credit for numbers) | Paper, site |
| BrowseComp (OpenAI), 2025 | Persistent search for hard-to-find facts | 1,266 | Live web, any site | LLM grader compares a short answer to the reference | Paper, repo |
| Mind2Web 2, 2025 | Long research tasks that need cited, synthesized answers | 130 | Live web | Agent-as-a-Judge with per-task rubrics, checks answers and sources | Paper |
| Mind2Web, 2023 | Predicting the next action on real sites | 2,350 from 137 sites | Offline snapshots | Each step compared to recorded human actions | Paper, site |
| BU Bench V2.1 (ours), 2026 | Long tasks with deliverables, scored with partial credit; model cost and time reported per task | 200 (the leaderboard scores every model on the same 180) | Live web | LLM judge (GPT-5.6 Luna) against a weighted rubric, 0 to 100 | Repo, scores |
| BrowserGym, 2024 | A framework, not a benchmark: one harness that runs MiniWoB, WebArena, WebArena-Verified, VisualWebArena, WorkArena, AssistantBench, WebLINX and others | Inherits each benchmark's tasks | Whatever each benchmark uses | Each benchmark's own grader | Paper, repo |
| OSWorld, 2024 | Full computer, not web-only: tasks across web and desktop apps, files and multi-app workflows | 369, plus 43 on Windows | Ubuntu virtual machine | Execution-based check script per task | Paper, repo |
Task counts are as each source states them. A few have newer versions: WebArena-Verified re-audits all 812 WebArena tasks, drops the LLM judging for deterministic checks and adds a 258-task hard subset. OSWorld became OSWorld-Verified in July 2025, and REAL's site now lists a v2.
How to choose
- Reproducible research or training. WebArena, or WebArena-Verified if you want deterministic scoring. Add VisualWebArena if your agent works from screenshots, or REAL for replicas of consumer apps. Run WebArena and VisualWebArena through BrowserGym so every model gets the same harness; REAL ships its own harness.
- Reliability on the live web. Online-Mind2Web, which is maintained and has a human-evaluated leaderboard alongside its auto-eval one. WebBench if you care about write tasks and logins, not just reading. Our BU Bench V2.1 leaderboard if you want long tasks with model cost and time next to the score.
- Deep research and browsing. BrowseComp for short answers that are hard to find, Mind2Web 2 for long answers with citations, AssistantBench for time-consuming information tasks.
- Enterprise workflows. WorkArena and WorkArena++ on ServiceNow. If the work leaves the browser, OSWorld.
- Your own product. None of these is your site. A few dozen of your real tasks, run more than once, tell you more than any public score.
Why scores don't compare across papers
Two numbers on the same benchmark often aren't the same measurement.
- Harness, model and judge differ. The BrowserGym ecosystem paper says many benchmarks "suffer from fragmentation and inconsistent evaluation methodologies", which is why it runs six models on six benchmarks in one harness. The judge alone moves a lot: when we swapped only the judge model on the same 104 finished runs, the score ranged from 62.5% to 83.7% (the benchmark behind the benchmark).
- The metric matters. Mind2Web's repo warns that its paper reports macro-average accuracy and that the micro average "may bias towards tasks with larger number of steps."
- Live sites drift. WebVoyager's repo says dates in its time-sensitive tasks, mostly Booking and Google Flights, have to be updated by hand before running. Online-Mind2Web replaced 36 tasks in November 2025 that had gone invalid or hit CAPTCHAs, and logged more task updates through May 2026. Halluminate cut WebBench to 2,454 tasks after removing ones that "became infeasible due to website changes".
- Self-reported isn't verified. OSWorld only lists results on its verified leaderboard after the maintainers run your agent themselves. Our own numbers are self-reported too: 89.1% on WebVoyager in December 2024, on 586 tasks, after removing tasks we judged impossible and re-checking failed and unclear grades by hand (report), and 97% on Online-Mind2Web in March 2026, graded by our own judge (write-up). The Online-Mind2Web authors cite near-90% WebVoyager claims, ours included, as over-optimistic, and found a simple search agent solved 51% of a 100-task WebVoyager sample.
Our benchmark
BU Bench V2.1 is 200 tasks on live websites. Each is scored 0 to 100 against a weighted rubric by an LLM judge, so partial work gets partial credit, and we report model cost and minutes per task next to the score. The tasks and rubrics are encrypted in the repo to keep them out of training data. The live leaderboard scores every model on the same 180 of them. Our first version, BU Bench V1, drew 100 hard tasks from WebBench, Mind2Web 2, GAIA, BrowseComp and our own set; its results are in this post.
