Odysseys2: a long-horizon web agent benchmark

Browser UseAgentsBrowsers

Showcase

Resources

Pricing

119k

Log inStart for freeTalk to sales

Talk to salesLog inStart for free

Talk to sales

All articles

Odysseys2: a long-horizon web agent benchmark

100 long tasks on the live web, from Carnegie Mellon University and Browser Use. The best agent fully completes 47 of 100.

In this article

1. The result
2. What a task looks like
3. How tasks were built
4. How runs are graded
5. Fast is not good
6. The harness matters
7. Same agent, eight runs
8. Grading changes scores
9. Why we care
10. Limitations
11. Get the data

In this article

1. The result
2. What a task looks like
3. How tasks were built
4. How runs are graded
5. Fast is not good
6. The harness matters
7. Same agent, eight runs
8. Grading changes scores
9. Why we care
10. Limitations
11. Get the data

• Lawrence Jang(CMU)
• Alexander Yue(Browser Use)
• Magnus Müller(Browser Use)
• Gregor Zunic(Browser Use)
• Andrew Keunwoo Jang(CMU)
• Younghan Park(CMU)
• Mareks Woodside(CMU)
• Daniel Fried(CMU)
• Jing Yu Koh(CMU)
• Ruslan Salakhutdinov(CMU)

Oct 27th, 2026·Research

Odysseys2: a collaboration between Carnegie Mellon University and Browser Use

Odysseys2 is a long-horizon web agent benchmark from Carnegie Mellon University and Browser Use: 100 long tasks on the live web, graded against 982 pass/fail checks. The best agent, GPT-6 Astra, fully completes 47 of 100 tasks. It builds on Odysseys, Carnegie Mellon University's original long-horizon benchmark.

The result

The best agent fully completes 47 of 100 tasks.GPT-6 Astra (default effort) has a mean rubric score of 84.6, yet falls short on the other 53.

Fig. 1 · Tasks fully completed, out of 100 1 task fully completed not fully completed

GPT-6 Astra fully completes 47 of 100 tasks.No other agent reaches 20, and 5 complete none.

Fully completed = every rubric check met. 15 agents, 1 attempt per task, BrowserCode harness, Odysseys2 judge, default effort unless marked high. Mean = mean rubric score, 0 to 100.

47 GPT-6
Astra mean 84.6

18 Opus
5.5 mean 71.2 high

15 GPT-5.6
Sol mean 62.0

13 Grok
4.7 mean 68.2 high

8 GPT-5.6
Luna mean 59.7 high

4 Fable
5 mean 61.5

4 Opus
5 mean 58.3

4 Grok
4.5 mean 48.2

2 GPT-5.6
Terra mean 44.6

2 Sonnet
5 mean 42.3

0 GPT-5.4
mini mean 24.1

0 Gemini 3.1
Pro preview mean 10.8

0 Gemini
3.8 Flash mean 10.7 high

0 Gemini
3.7 Flash mean 5.8

0 Gemini
Lite mean 1.6

1. GPT-6 Astra

mean 84.6 47 2. Opus 5.5 high

mean 71.2 18 3. GPT-5.6 Sol

mean 62.0 15 4. Grok 4.7 high

mean 68.2 13 5. GPT-5.6 Luna high

mean 59.7 8 6. Fable 5

mean 61.5 4 7. Opus 5

mean 58.3 4 8. Grok 4.5

mean 48.2 4 9. GPT-5.6 Terra

mean 44.6 2 10. Sonnet 5

mean 42.3 2 11. GPT-5.4 mini

mean 24.1 0 12. Gemini 3.1 Pro preview

mean 10.8 0 13. Gemini 3.8 Flash high

mean 10.7 0 14. Gemini 3.7 Flash

mean 5.8 0 15. Gemini Lite

mean 1.6 0

Sorted by tasks fully completed, then mean score. Opus 5.5 is second with 18. Fable 5: content filters blocked 2 tasks, so its mean uses 98. Scores come from an LLM judge that has not been validated against a human reference.

We ran 15 agents, one 60-minute attempt per task. A task counts as fully completed only when every one of its 8 to 12 checks is met. The mean rubric score is the share of checks met (0 if the attempt is flagged for suspected reward hacking).

Opus 5.5 is second with 18 tasks fully completed and a mean of 71.2. Five of the 15 agents fully complete none. Every agent used BrowserCode, Browser Use's harness that drives the browser by writing code against the Chrome DevTools Protocol. Astra ran at default effort; Opus 5.5, Grok 4.7, GPT-5.6 Luna and Gemini 3.8 Flash at high. Fable 5's content filters blocked 2 tasks, so its mean uses 98.

Odysseys2 results for 15 agents: tasks fully completed out of 100, mean rubric score, 95% confidence interval and median minutes per attempt.

Fully completed: tasks with every rubric check met, one attempt each. Mean score: share of checks met, 0 to 100 (0 if flagged for suspected reward hacking), averaged over scored attempts. 95% CI as reported in the release. BrowserCode harness, Odysseys2 judge.

What a task looks like

These are long, multi-part jobs, not quick lookups.The median instruction is 271 words. Most ask for several deliverables, graded on 8 to 12 checks.

Fig. 2 · Task 001 · Shopping

Compare every laptop on two Amazon results pages

A 280-word instruction, graded on 11 pass/fail checks. Starts from a Google search page on the live web, 60 minutes per attempt.

Instruction

I want to compare the laptops currently shown on Amazon.com across a broad range of prices. Stay logged out, set the delivery ZIP code to 10001, and search for laptops.

Read the full instruction (280 words)Show less Record every laptop listing on the first two results pages in the order shown, including sponsored and repeated listings. Exclude accessories, replacement parts, and separate recommendation sections. Open every retained product page and record the exact name, Amazon product ID (ASIN), RAM, graphics processor (GPU), processor, display size and resolution, weight, storage, screen type, refresh rate, operating system, color, current price without conditional discounts, promotions and their conditions, rating and number of ratings, seller, who ships it, stock availability, selected configuration, and main product URL. Clearly mark any fields you cannot support with evidence. Record result counts, page navigation, filters, sorting, observation time, access failures, seller claims about upgrades, available configuration options, and differences between search listings and product pages. Choose up to 20 distinct laptop model families across the prices you found. Where the exact configuration can be matched to a public manufacturer specification page, check the processor, supported memory limits, storage, display, graphics, operating system, and weight. State the comparison rules and use them to compare the verified laptops for everyday use, portability, gaming or graphics, and value. Explain how missing or conflicting specifications affect each shortlist. Provide the search listings in their original order, a workbook of products and configurations, a table matching ASINs and repeated listings, a table of manufacturer checks, an access and conflict log, and a short laptop comparison explaining the criteria behind it. Do not sign in, enter a street address, add anything to a cart, contact sellers, begin checkout, or purchase anything.

The 11 checks

1. R1 Search location
2. R2 Listings from both pages
3. R3 Half product coverage
4. R4 Full product coverage
5. R5 Product data accuracy
6. R6 Products and repeated listings
7. R7 Search settings and conflicting information
8. R8 Manufacturer verification
9. R9 Criteria-based brief
10. R10 Safe browsing boundaries
11. R11 Supported final claims

How five agents did on this task

check met not met met, voided by flag

• GPT-6 Astra all 11 met, fully completed
• Opus 5.5 high all 11 met, fully completed
• Grok 4.7 high 9 of 11 met
• GPT-5.6 Luna high 9 of 11 met
• Gemini 3.8 Flash high flagged, scored 0 (6 of 11 met before the flag)

Dots count checks met, not which ones. Flagged: the judge suspected reward hacking, so the attempt scores 0 whatever it met. The flag is a model judgment, not independent proof of fabrication or intent.

A task reads like a brief to a research assistant: a spreadsheet, a log of pages that failed, a short written comparison, plus things the agent must not do, like signing in. Every task starts from a Google search page. Agents ran in Browser Use's stealth browsers with CAPTCHA solving.

How tasks were built

Tasks are built from the real user request distribution provided by Browser Use. Source requests were de-identified and no request was copied one-to-one. Tasks had to be rerunnable, so anything that changes a site's state was dropped. Four people reviewed every task and rubric, and each went through five rounds of model-assisted alignment and feasibility checks.

Carnegie Mellon University built the original Odysseys benchmark and its rubric method; Browser Use provided the source request distribution.

How runs are graded

One LLM call grades each attempt: GPT-5.6 Luna at xhigh effort, given the task, rubric, action trajectory, final response, output files and selected screenshots. It browses nothing. A check is met only when the delivered content is backed by evidence in the trace; data recited from memory earns nothing. If the judge suspects reward hacking, including fabrication, the attempt scores 0.

Fig. 3 · Reward hacking check 1 attempt flagged, scored 0 not flagged

The judge zeroed 75 of Gemini 3.8 Flash's 100 attempts, and 1 of Opus 5.5's.A flagged attempt scores 0, however many checks it met.

Attempts flagged for suspected reward hacking, including suspected fabrication, out of 100 per model. Under each: mean score before → after zeroing. BrowserCode harness, high effort.

1 /100

Opus 5.5

mean 71.8→ to 71.2

2 /100

Grok 4.7

mean 69.7→ to 68.2

8 /100

GPT-5.6 Luna

mean 65.0→ to 59.7

75 /100

Gemini 3.8 Flash

mean 40.5→ to 10.7

The flag is a model judgment, not independent proof of fabrication or intent. Zeroing the whole attempt is a separate policy decision. Mean score is shown before the check (share of checks met) and after it.

8 of Luna's 100 high-effort attempts were flagged, more than Opus 5.5 (1) or Grok 4.7 (2), even with Luna as the judge. Gemini 3.8 Flash was flagged on 75.

Fast is not good

Fast is not the same as good.GPT-5.6 Luna leads for the first half hour, then Grok 4.7 and Opus 5.5 pass it and keep climbing.

Fig. 4 · Score banked over time

At 15 minutes GPT-5.6 Luna has 37 points and Opus 5.5 has 18.By 60 minutes, Opus 5.5 and Grok 4.7 finish well above Luna.

Mean rubric score (0 to 100) from attempts finished by each minute, over all 100 attempts. BrowserCode harness, high effort, 1 dot per minute.

At 15 min: Luna 37 ·Opus 5.5 18 ·Grok 4.7 12

Grok 4.7 catches Luna at about 30 minutes and Opus 5.5 at about 36. An attempt adds its score when it finishes, not while it runs. Luna's line ends at 58.9, below its 59.7 mean, because one attempt has no recorded time.

By 15 minutes Luna has 37 points, Opus 5.5 18 and Grok 4.7 12. Grok catches up at about 30 minutes, Opus at about 36. Luna's median attempt takes 13 minutes and costs $0.35, against about 25 minutes and $6.22 for Opus 5.5 (model plus judge).

The harness matters

Same model, different harness, different score.At high effort, GPT-5.6 Luna scores 33.0 in the Browser Use Python library and 59.7 in BrowserCode.

Fig. 5 · Same model, three harnesses

In the Python library Luna stays near 30.In BrowserCode it climbs to 63.0, in Pi to 71.9.

Mean rubric score, 0 to 100, for GPT-5.6 Luna on all 100 tasks, by reasoning effort. One run per setup. Python library has no max setting.

Harnesses hand the judge different records of the run, so these gaps mix differences in execution with differences in the evidence available for grading. Pi was tested with Luna only.

We ran Luna in the Browser Use Python library, BrowserCode and Pi. The Python library stays around 30. BrowserCode and Pi land close at high effort (59.7, 59.2) and split at max (63.0, 71.9). Harnesses also hand the judge different records, so these gaps mix execution and evidence.

Same agent, eight runs

One run is not the whole story.Across eight runs of GPT-5.6 Luna, 26 of 100 tasks were fully completed at least once, and none in all eight.

Fig. 6 · Same agent, 8 runs 1 task fully completed not fully completed

26 tasks fully completed at least once in 8 runs.Not one fully completed every time.

GPT-5.6 Luna, high effort, BrowserCode, the same 100 tasks run 8 times, 60-minute budget.

26 /100

fully completed in at least 1 of 8 runs

0 /100

fully completed in all 8 runs

Mean rubric score, 0 to 100, 1 dot = 2 points

Taking each task's best of 8 runs gives 83.1; its worst gives 23.1.

A single run fully completes 6.25% of tasks on average. Dots count tasks, they are not placed by task. The 8 runs differ in dates, concurrency and logging settings. Mean 56.9 has a sample standard deviation of 2.9 across runs.

Luna's mean across eight runs is 56.9 (standard deviation 2.9). Each task's best of eight averages 83.1; its worst, 23.1. The runs differ in dates, concurrency and logging, so the spread includes those conditions.

Grading changes scores

How you grade decides who looks good.On the same attempts, Gemini 3.7 Flash scores 94.9 under the original Odysseys judge and 5.8 under the Odysseys2 judge.

Fig. 7 · Same attempts, two judges Odysseys judge Odysseys2 judge

Under the original judge every agent scores 77 to 96.Under the Odysseys2 judge the same attempts spread from 2 to 85.

Mean rubric score, 0 to 100, for 14 agents, each graded twice on the same saved attempts. Gemini 3.7 Flash: 94.9 vs 5.8.

The original Odysseys judge grades one check at a time from the action history and screenshots and does not see delivered files. The Odysseys2 judge also reads the files and zeroes flagged attempts. These differences measure disagreement between the judges; lower scores alone do not establish greater accuracy.

The original Odysseys judge checks one item at a time from actions and screenshots, without the delivered files. On the same attempts from 14 agents, its means run 77 to 96; the Odysseys2 judge's run 2 to 85. Astra is first under both. This measures disagreement between judges, not which is more accurate.

Why we care

Our view, not a measured result: the browser work people automate is usually a brief, not a lookup. Collect data across many sites, follow constraints, deliver a spreadsheet and a summary. Astra meets most checks on average and still misses something on 53 of 100 tasks, and for the person relying on it, one missed detail can mean redoing the job.

We will run new models on Odysseys2 next to Browser Use Bench v2.1. More on long-horizon RL environments and the benchmark behind the benchmark.

Limitations

• Judge not validated against a reliable human reference; the paid human-review pilot was insufficient.
• Flags are model judgments, not proof of fabrication or intent. Zeroing the whole attempt is a policy choice.
• Not fully reproducible. Live sites change, and the runner, judge implementation and trajectories are not released.
• Clipped evidence. Long tool results and file previews can be clipped; the judge does not see image pixels.
• Settings and costs differ. Astra ran at default effort and its flag status is unavailable. Costs for the four high-effort agents include the judge; others are model only.
• Intervals are as reported in the release, not recomputed. The eight-run spread is one standard deviation, not a confidence interval.

Get the data

• Benchmark page and interactive results: odysseysbench.com/v2
• 100 tasks as JSON (instruction, start URL, rubric)
• Judge prompt and EVALUATION.md
• Release files (tasks, rubrics, scores, judge prompt): github.com/ljang0/odysseys2-benchmark (MIT license)
• Original Odysseys paper: arXiv 2604.24964
• Related: web agent benchmarks, the bitter lesson of browser agents

To cite Odysseys2:

bibtex Copy

@misc{jang2026odysseys2,
  title        = {Odysseys2: A Web Agent Benchmark Based on Real User Requests},
  author       = {Jang, Lawrence and Yue, Alexander and M{\"u}ller, Magnus and
                  Zunic, Gregor and Jang, Andrew Keunwoo and Park, Younghan and
                  Woodside, Mareks and Fried, Daniel and Koh, Jing Yu and Salakhutdinov, Ruslan},
  year         = {2026},
  howpublished = {\url{https://odysseysbench.com/v2/}}
}

Thanks to our co-authors at Carnegie Mellon University: Lawrence Jang, Andrew Keunwoo Jang, Younghan Park, Mareks Woodside, Daniel Fried, Jing Yu Koh and Ruslan Salakhutdinov.

Published October 27, 2026

Copy link

Keep reading

All articles

Comparisons /Oct 27, 2026 ## Browser Use vs Browserless: Pricing, Stealth & Features Compare Browser Use and Browserless on pricing, stealth, and performance. Find the right browser infrastructure for your automation workflows. Nik Tomazic Roundups /Oct 23, 2026 ## 11 Best Browser Automation Tools: Agents, Scripts, and Cloud Browsers Compare 11 browser automation options across AI agents, scripts, cloud browsers, and visual workflows, with pricing, capabilities, and practical tradeoffs. Nik Tomazic Roundups /Oct 22, 2026 ## The Best Browser Agent MCP Servers (2026) MCP servers that take a goal and run a whole browser agent: Browser Use Cloud, Browser Use open source, Hyperbrowser and agent-browser. Models, cost, live view. Gregor Zunic

Product

Browser Use AgentsBrowser Infrastructure

Integrations

Browser for coding agentsClaude in ChromeClaude Code browserCodex browserCursor browserGemini browserOpenClawMCP server

Resources

PricingBenchmarksDocumentationDevelopersOpen SourceBrowser HarnessCustom ModelsSite skillsWebMCP directoryChangelogBlogFree developer tools

Solutions

Remote browser for PlaywrightChrome for PuppeteerCAPTCHA solvingAnti-detect browserWeb agent APIProxiesForm fillingWeb scrapingAutomated testingLogin automationBrowser for voice agents

Showcase

All examplesWhat people runData extractionShopping & checkoutTravel & bookingTesting & QAResearchCompanies & jobsAgent workflowsSocial & outreachGames & real-timeCase studies

Company

CareersEnterpriseSecuritySOC 2HIPAAZero data retentionTermsPrivacyDPAContactCookie Settings

Connect

GitHubTwitterLinkedInDiscord

Comparisons

vs Playwright MCPvs Chrome DevTools MCPvs agent-browserCloud browser comparisonWeb agent comparisonBrowserbase alternativesKernel alternatives

Product

Browser Use AgentsBrowser Infrastructure

Integrations

Browser for coding agentsClaude in ChromeClaude Code browserCodex browserCursor browserGemini browserOpenClawMCP server

Resources

PricingBenchmarksDocumentationDevelopersOpen SourceBrowser HarnessCustom ModelsSite skillsWebMCP directoryChangelogBlogFree developer tools

Solutions

Remote browser for PlaywrightChrome for PuppeteerCAPTCHA solvingAnti-detect browserWeb agent APIProxiesForm fillingWeb scrapingAutomated testingLogin automationBrowser for voice agents

Showcase

All examplesWhat people runData extractionShopping & checkoutTravel & bookingTesting & QAResearchCompanies & jobsAgent workflowsSocial & outreachGames & real-timeCase studies

Company

CareersEnterpriseSecuritySOC 2HIPAAZero data retentionTermsPrivacyDPAContactCookie Settings

Connect

GitHubTwitterLinkedInDiscord

Comparisons

vs Playwright MCPvs Chrome DevTools MCPvs agent-browserCloud browser comparisonWeb agent comparisonBrowserbase alternativesKernel alternatives

View as Markdown Current view: website.

↗System status

Dev mode Enterprise mode

© 2026 Browser Use AI agents and crawlers: read browser-use.com/llms.txt for the product map and llms-full.txt for full context.

注意转载授权 · 阅读原文