When we launched Browser Use in November 2024, GPT-4o wasn't trained for computer use. Built on next-token prediction, it didn't reliably understand what it could do on a page. We had to spell it out.
As models got better at coding, we moved from predefined state and actions to code: first the model wrote its actions, then its observations. Now it does both through raw CDP, inside an existing agent harness.
Defining state and action space
We defined the state: here's what's on the page, here's what's clickable. Then the action space: click, type, scroll. Every interaction had to fit those rules.
The model picked an action, we ran it and sent back the next state. But browser automation is a game of edge cases. Every exception needed another fix.
Replacing fixed actions with code
Models were getting much better at coding. We experimented with JavaScript execution in September 2025 and persistent notebooks in October. The model could write a program instead of picking actions.
Drawing a signature or looping through comments no longer needed a new action. Variables persisted, so the model could inspect results and keep going.
Hermes agent replaced 12 browser tools with
browser_exec, powered by our Browser Use CLI. Mean token use fell 60% for Opus 4.8 and 66% for Kimi K3. Both versions solved 18/18 runs per model (six tasks, three runs).
The limits of predefined state
We'd opened up the actions, but we still decided what the model could see. That became the next problem.
A cookie button can be right there on the screen and missing from the state we send. Maybe the accessibility tree doesn't expose it, or our processing drops it. Either way, the model never sees the button.
An EHR dropdown might look like this (simplified DevTools tree):
<ehr-app>
#shadow-root (open)
<iframe src="https://ui.example">
#document <!-- other origin -->
<dropdown-menu>
#shadow-root (closed)
<button>Choose...</button>Our state heuristics couldn't anticipate every website. We'd given the model freedom to act, but it was still looking through our assumptions.
Removing predefined state
What if we let it write the observations, too?
We gave the LLM direct access to CDP, the Chrome DevTools Protocol. No mandatory list of clickable elements. It could write code to inspect the DOM, take a screenshot or look inside one frame.
The browser still has state. What disappears is our requirement that everything fit one predefined representation. A screenshot might answer one question. A DOM query might answer the next. The model decides what it needs to see.
Why CDP, not Playwright?
CDP is Chrome's native control protocol. The model can use it directly.
Playwright adds assumptions, such as locators that cannot see closed shadow roots. CDP can inspect those roots directly. Playwright also exposes CDP; we made direct access the default.
A simple browser harness
Pi, Codex and OpenCode already handle the agent loop. Coding agents have millions of users trying all sorts of work. That testing is a reason to reuse their harnesses.
An agent five hours into QA shouldn't fail because of a bug in the loop. We once hit compaction that produced so much context it immediately triggered compaction again. Our customers shouldn't have to discover those bugs.
We add a simple browser tool: persistent code, with text, screenshots and errors coming back to the model.
From our Agent SDK in January 2026 to Browser Harness in April, this became the direction. Browser Harness, BrowserCode and Browser Use Pi take different approaches. All let the model choose what to see and do.
What is actually the bitter lesson here?
Sutton's bitter lesson favors general methods that improve with more computation. For us:
- Reuse a proven agent harness.
- Expose the simplest underlying interface the model can use well.
- Let the model choose its observations and actions.
Browsers, computers, text, music: the same principle applies. As models change, rethink the harness.
