Grafy News
Try Grafy free

WebArena‑Pro Introduces a Heterogeneous, Multimodal, Reproducible Benchmark for Web Agents

Grafy Research Lab releases WebArena‑Pro, a benchmark that expands web‑agent evaluation to audio, video and stateful workflows while preserving reproducibility.

Illustration for the news story: WebArena‑Pro Introduces a Heterogeneous, Multimodal, Reproducible Benchmark for Web AgentsSource: Grafy Research Lab

WebArena‑Pro, a new benchmark released by the Grafy Research Lab, aims to broaden the evaluation of AI web agents by including heterogeneous applications, multimodal content, and stateful workflows while keeping experiments reproducible. The announcement matters because it addresses long‑standing gaps in how the research community measures the real‑world capabilities of browser‑based agents.

What is WebArena‑Pro and why does it matter?

WebArena‑Pro is positioned as a benchmark suite for web agents that can interact with realistic browser environments. According to the lab’s pre‑print, current agents are increasingly deployed on tasks that involve not only text and images but also audio, video, and complex navigation patterns. Existing reproducible benchmarks, however, cover only a narrow set of web applications and restrict modality to text and vision. By expanding the scope, WebArena‑Pro seeks to provide a more faithful testbed for the next generation of agents.

How does it differ from prior web‑agent benchmarks?

The paper notes three main shortcomings of earlier benchmarks:

  1. Limited application coverage – only a handful of sites from a few software categories are represented.
  2. Modality restriction – evaluations focus on textual and visual inputs, ignoring audio and video.
  3. Reproducibility trade‑offs – live benchmarks that add more sites suffer from page‑content drift, making results hard to compare over time. WebArena‑Pro tackles each issue by curating a diverse set of heterogeneous web apps, adding audio and video streams to the task list, and ensuring that every page snapshot is version‑controlled so that runs can be reproduced exactly.

What modalities and tasks does the benchmark cover?

While the announcement does not enumerate a full task matrix, it emphasizes that the benchmark includes:

  • Textual interactions such as form filling and search queries.
  • Visual content like image selection, OCR, and screen‑region analysis.
  • Audio handling – agents must detect, play, and possibly transcribe embedded sound clips.
  • Video manipulation – tasks may require pausing, seeking, or extracting frames from streamed video.
  • Stateful workflows – multi‑step processes that depend on previous actions, such as shopping carts or multi‑page authentication flows. By mixing these modalities, the benchmark forces agents to coordinate language, vision, and auditory reasoning within a single browsing session.

How will researchers and developers use it?

The release invites the community to benchmark their models against the curated suite and report performance across each modality. Because the benchmark is reproducible, researchers can compare results across papers without worrying about site changes. The paper also suggests that the benchmark can serve as a training ground for multimodal agents, enabling curriculum learning that progresses from simple text tasks to richer audio‑video scenarios. Developers building commercial assistants, automated testing tools, or data‑extraction pipelines can use WebArena‑Pro to validate that their systems handle the full spectrum of web content they will encounter in production.

Applying the benchmark with Grafy

Grafy users can engage with WebArena‑Pro without leaving the platform. The Operator app (/operator/operator) provides a browsing‑agent interface that can run scripted interactions against any web page. By loading the benchmark’s version‑controlled URLs into Operator, creators can:

  1. Run baseline agents – use built‑in LLM or vision‑language models to execute the prescribed tasks.
  2. Compare custom agents – plug in your own model via the multi‑model selector in Operator and record success metrics.
  3. Capture provenance – Operator logs each action, which can be visualized in the Graph app (/graph/graph) to trace decision paths. If you simply want to study the benchmark paper, the Reader app (/reader/reader) lets you upload the PDF and ask natural‑language questions about methodology, task definitions, or results. This workflow is helpful for quickly extracting evaluation details before implementing an agent.

Frequently asked questions

How does WebArena‑Pro ensure reproducibility across runs?

The benchmark stores every web page snapshot in a version‑controlled repository, so the exact HTML, CSS, and media assets are served to agents each time. This eliminates drift caused by live site updates and lets researchers obtain identical test conditions.

Does the benchmark evaluate agents on audio and video understanding?

Yes. Unlike prior suites that focus only on text and images, WebArena‑Pro includes tasks that require agents to play, pause, seek, and interpret audio clips and video streams, thereby testing multimodal reasoning capabilities.

Which Grafy app is best for experimenting with WebArena‑Pro?

The Operator app is the most direct fit because it is designed for AI‑driven web navigation. It can execute the benchmark’s scripted interactions and log results, while Reader offers a convenient way to explore the accompanying research paper.


Source: Grafy Research Lab — WebArena-Pro: A Heterogeneous, Multimodal, Reproducible Benchmark for Web Agents