AI agent benchmarks: GAIA, WebArena, OSWorld, and SWE-bench

GAIA tests answers to realistic questions that combine reasoning, multimodality, web browsing, and tool use. WebArena tests long tasks in reproducible functional websites. OSWorld adds desktop applications, file I/O, and multi-app workflows in a real computer environment. SWE-bench starts from a GitHub issue and a repository snapshot and checks whether a patch resolves the software-engineering problem. These are not four interchangeable rankings: each measures a different task distribution, observation/action space, and definition of success.

Name the production decision first: select a scaffold, enable a browser workflow, change a coding agent, or expand an autonomy tier. Then map the real work to the benchmark contract. An external result provides a prior about capability, but it does not prove quality on your data, permissions, locale, UI, repository, or cost of failure. Release still requires an internal eval slice with the same task shape and an independent oracle.

GAIA fits when the agent gathers facts, works with files and multimodal inputs, uses the web or tools, and returns a verifiable short answer. Its strength is composing basic assistant abilities; its production limitation is that it does not contain your application state or consequential side effects. Add your own knowledge-work tasks, citation rules, freshness cutoff, and abstention cases.

Choose WebArena for a browser agent that navigates e-commerce, forums, collaborative-development, or content-management interfaces and must reach a functional state. OSWorld better matches a computer-use agent that moves across browser, office, operating-system, and file workflows. For a coding agent, SWE-bench provides an issue-to-patch signal on real repositories, but your internal replay must reproduce your toolchain, instructions, hidden checks, and review policy.

• GAIA → general-assistant questions, browsing, tools, and multimodal evidence.
• WebArena → functional web tasks and execution-based outcomes.
• OSWorld → desktop, files, and workflows across applications.
• SWE-bench → repository issue, code change, and test-based resolution.

A model name plus one percentage is not a reproducible result. Record the benchmark revision or split, agent scaffold, prompt, tools, observation mode, action adapter, budget, retry policy, sample count, environment image, dependency state, and grader version. For web and desktop environments, also record viewport, locale, account fixtures, reset status, and the share of infrastructure failures. For coding evals, pin the base commit, test command, patch application, network policy, and contamination controls.

Compare only runs with compatible rules. Pass@k with multiple attempts is not first-run reliability; success with a browser is not success without web access; a different scaffold may explain the gap better than model capability. Do not carry historical baseline numbers into the current product: they belong to a specific paper and harness version. Store a link to the primary result artifact in the decision record instead of copying a leaderboard number without provenance.

Construct validity asks whether the task and grader measure the capability you actually need. Exact-answer grading is useful for a GAIA-like question but says nothing about trajectory safety. A passing test in SWE-bench can confirm patch behavior without proving maintainability or compliance with internal policy. A web or desktop grader may observe the correct final state while missing an unnecessary message, data leak, or repeated side effect. Add separate outcome, trajectory, policy, and side-effect verdicts.

Check whether tasks, solutions, screenshots, or repositories may have appeared in training data, examples, or prompt tuning. A held-out internal set should not be visible to the team optimizing the agent. Label environment failures separately: a site that failed to start, an unavailable dependency, or a missing oracle is neither a model failure nor a success. Replay a known-good baseline regularly so agent regression can be separated from harness drift.

A bridge set is a small versioned dataset that preserves the shape of the external benchmark while replacing the domain with yours. For a research assistant, use multi-source questions with a freshness date and citation oracle; for a browser agent, use a resettable copy of a key workflow; for a desktop agent, use synthetic files and intercepted outputs; for a coding agent, use sanitized historical issues on pinned commits. Every task needs an owner, initial state, authority, terminal state, prohibited events, and an independent grader.

Run the public signal, bridge set, and production regression as three separate layers. The first supports comparison with the research ecosystem, the second tests capability transfer, and the third protects local invariants. Promotion requires acceptable results across risk slices, no defined critical failures, and a tested rollback for the model, prompt, tools, and policy bundle. An average score cannot offset an unauthorized payment, secret leak, or destructive patch.

The evidence pack contains the decision question, benchmark-to-workflow mapping, manifest for every run, raw outcomes, grader evidence, failure taxonomy, segmented results, reviewer sign-off, and known limitations. Record separately what was not tested: a new locale, a longer task, another permission tier, production latency, or an external side effect. This is not bureaucracy; it defines the boundary of what the evidence actually supports.

After a model, scaffold, environment, or dataset update, the previous verdict becomes historical evidence. First replay the frozen slice, then held-out tasks, and only then a narrow canary. If the candidate loses to the local baseline or creates a critical failure, rollback restores a compatible bundle and stops new runs; incomplete side effects must be reconciled separately. A public leaderboard never overrides this release gate.