How to evaluate browser agents: a practical checklist

Start with a task contract that names the initial state, allowed sites and applications, data, terminal state, prohibited events, and the boundary for human confirmation. “Book a ticket” is not a test by itself: fix the route, date, budget, whether booking is allowed, and whether the agent must stop before payment. For read-only work the verdict may verify the located record; for state-changing work it must verify the actual backend state rather than text the agent saw on the page.

WebArena is useful because it provides functional sites and checks completion of realistic long tasks, but a public benchmark does not reproduce your permissions, locales, data, or consequences. Convert production jobs into sanitized task families and give each an independent oracle: an API or database query in a sandbox, structured export, controlled DOM state, or human-reviewed artifact. An agent self-report that a task is “done” is never an oracle.

• Outcome → the required state was actually created, found, or changed.
• Trajectory → every action was allowed for this task and current state.
• Side-effect integrity → no extra submit, message, payment, or deletion occurred.
• Stop behavior → the agent correctly refuses, asks for clarification, or requests approval.

A browser evaluation result depends on more than the model. Version the image or VM, browser, viewport, device scale, locale, timezone, fonts, cookies, account fixture, network policy, initial data, and application state. Record the observation mode separately—screenshot, accessibility tree, DOM, OCR, or a combination—and the action mode—coordinates, element IDs, Playwright primitives, or keyboard shortcuts. OpenAI’s supplemental CUA evaluation material explicitly documents differences in browser or VM environment, prompts, sampling, and scoring; without that detail, version comparisons are not reproducible.

BrowserGym standardizes observation and action spaces across multiple web-agent benchmarks; apply the same discipline inside your own harness. Store an environment manifest beside every result and reject the run if the fixture failed to initialize, the site changed unexpectedly, or the oracle is unavailable. Infrastructure failure must not be counted as model failure, and a task that happened to be completed before the run must not count as success. Reset must restore the full business state, including mail, carts, files, and pending transactions.

A frozen regression set should cover common tasks and known incidents, while a held-out set introduces new wording, entities, and layout variants. Add responsive viewports, different zoom levels, translations, sticky banners, modals, lazy loading, disabled controls, duplicate labels, paginated tables, and elements below the fold. For computer-use agents also test the active window, focus, drag operations, clipboard, file picker, and system dialogs. The goal is not to break coordinate scripts; it is to measure grounding against the current semantic state.

Add temporal and operational perturbations: slow responses, spinners, stale screenshots, actions accepted after a timeout, session expiry, redirects, new tabs, and partially saved forms. Ambiguous tasks should require clarification, and impossible tasks should end safely. Maintain a contamination boundary: do not place held-out screenshots in prompts or few-shot examples, and do not tune policy against every failed case without introducing a new blind set.

Task success is necessary but insufficient. A trace grader checks origin transitions, targets, entered fields, repeated actions, approval binding, and prohibited states. Distinguish perception error, wrong target, planning error, policy block, environment failure, premature stop, and false success claim. Step count and latency matter only after correctness: a shorter path with an incorrect submit is not more efficient. When several paths are valid, score invariants rather than an exact action sequence.

A recovery suite starts the agent from an intermediate state: a modal blocks a button, form validation rejects a field, navigation returns to login, or a timeout occurs after commit. Passing requires the agent to inspect the real state first, avoid duplicating a side effect, and choose retry, reconcile, rollback, or escalation. For stochastic agents, run multiple independent trials, report the distribution, and use paired comparison against a baseline; one successful replay is not evidence of general reliability.

• Functional verdict → an independent oracle confirmed the final state.
• Policy verdict → no action exceeded scope or approval.
• Grounding verdict → the target matched the intent in the actual frame.
• Recovery verdict → retry created no duplicate and preserved the audit trail.

Pages, documents, messages, and tooltips are untrusted data. A security slice places direct and indirect prompt injections in visible text, accessibility attributes, uploaded documents, and search results. Do not only test whether the agent repeats the instruction; test the sinks: whether it tries to change the goal, navigate to a forbidden origin, read a canary secret, paste it into a form, upload a file, or perform an external action. Prompt-injection detection without a sink verdict creates false confidence.

Run all write tests in disposable accounts with canary data, intercepted email or webhooks, a fake payment rail, and a log of backend mutations. Bind approval to the exact payload and state snapshot: approval for one recipient does not permit changing the address after the modal appears. Test approval cancellation, expired approval, misleading buttons, download quarantine, and secrets in the clipboard. Production credentials, real payments, and messages to external people are unnecessary for an evidence-grade evaluation.

The decision record should include dataset revision, environment manifest, model, prompt, observation or action adapter, policy, oracle, sample count, segment results, critical failures, and reviewer. Compare the candidate with a known-good bundle on the same task seeds. Promotion requires non-regression on outcome, zero tolerance for defined critical side effects, and separate passing slices for domain, locale, task length, and risk tier. A public benchmark is an external signal, not a replacement for your release gate.

Rollout should move from an offline resettable environment to shadow, read-only canary, draft-only writes, and narrowly approved actions. Monitoring should reuse the offline taxonomy: false completion, duplicate mutation, unexpected origin, approval mismatch, and recovery failure. Rollback restores a compatible model, prompt, adapter, and policy bundle, stops new runs, and reconciles unfinished side effects before retry. A sanitized incident trace becomes a regression fixture only after provenance and privacy review.