Grafy News
Try Grafy free

GDP.xlsx Benchmark Shows Frontier Models Still Struggle with Real‑World Spreadsheets

A new arXiv benchmark of 70 professional spreadsheet tasks reveals that even top models like Gemini 4 Argon pass only 38% of requests.

Illustration for the news story: GDP.xlsx Benchmark Shows Frontier Models Still Struggle with Real‑World SpreadsheetsSource: Grafy Research Lab

What is GDP.xlsx and why it matters?

The research team behind Surge AI released a benchmark called GDP.xlsx that bundles 70 realistic workplace requests drawn from 13 different professional domains. Each request is paired with a full Excel or CSV file and an expert‑crafted rubric containing binary criteria that the answer must satisfy. According to the arXiv preprint (Garre, Mehta, Chen, 2026), the benchmark totals 2.18 million populated cells and 791 rubric criteria, making it one of the most extensive evaluations of AI agents interacting with semi‑structured data.

The significance lies in its focus on end‑to‑end workbook comprehension: an AI must locate evidence across multiple tabs, interpret formatting cues, resolve hidden filters, and return set‑valued answers without extra items. This mirrors everyday office scenarios where a new analyst inherits a spreadsheet and must answer questions like “Which leases will increase rent next year?” without explicit navigation instructions.

How was the benchmark constructed?

The authors collected requests directly from domain professionals, ensuring each task reflected a genuine pain point where prior AI attempts failed. After three rounds of expert review, the tasks were packaged with their corresponding files and a rubric that grades both completeness (all required items present) and exclusivity (no forbidden items). The benchmark also includes contamination controls: only three tasks are publicly released, while the remaining 67 are kept hidden to protect the leaderboard.

Evaluation runs inside a container that provides standard spreadsheet tools. An agentic judge—a model that can open the workbook and apply the rubric—automatically scores each attempt. The primary metric is strict pass@1, the proportion of attempts that satisfy every rubric criterion in a single try.

What were the key findings?

Across 20 model configurations from Google, Anthropic, OpenAI, and others, the top performer was Gemini 4 Argon, which achieved a pass@1 of 38.3 %. The median score across all configurations was 19.3 %, indicating that even the most advanced frontier models leave a large portion of the tasks unanswered correctly.

Failure analysis highlighted recurring error modes:

  • Missing set members – models often omitted required items, especially when those items were hidden behind filters or stored in comments.
  • Evidence in another location – some answers required cross‑tab lookups that the model failed to trace.
  • Formatting cues ignored – fill colors, cell comments, and embedded images that signaled validity were frequently overlooked.

These patterns suggest that current LLMs excel at language understanding but still lack robust spreadsheet‑specific reasoning and tool use.

How does it compare to existing spreadsheet benchmarks?

Prior benchmarks such as SpreadsheetBench, SheetCopilot, and FRTR‑Bench focus on formula prediction, workflow execution, or single‑table question answering. GDP.xlsx extends the scope by demanding full‑workbook navigation, handling set‑valued answers, and enforcing strict binary rubrics that penalize any extra or missing element. Table 1 in the paper situates GDP.xlsx alongside related suites, noting that it uniquely combines real‑world professional requests, large‑scale cell counts, and agentic judging.

The benchmark also aligns with the broader trend of agent‑as‑judge evaluations, where a model both produces an answer and verifies it against the environment. For further reading on this approach, see the Grafy News article on the Agent‑as‑a‑Judge initiative.

How creators can explore the benchmark with Grafy

While Grafy does not host a dedicated spreadsheet‑analysis app, creators can still experiment with GDP.xlsx using Grafy Chat. By uploading a workbook to the chat interface and selecting a multi‑model configuration (e.g., Gemini 4 Argon, Claude 3.5, or GPT‑4o), users can pose the same professional request and observe the model’s step‑by‑step reasoning. The chat’s built‑in tool usage lets the model open sheets, apply filters, and retrieve values, closely mirroring the benchmark’s container setup.

For deeper inspection of the rubric logic, the Reader app can ingest the benchmark’s PDF and allow keyword search across the 791 criteria, helping developers understand where models tend to trip. Additionally, the publicly released three tasks on Hugging Face can be downloaded and fed into Grafy Chat for quick prototyping.

Frequently asked questions

Which model achieved the highest pass@1 score on GDP.xlsx?

The Gemini 4 Argon model from Google topped the leaderboard with a strict pass@1 of 38.3 %.

How many domains are covered by the benchmark?

The benchmark spans 13 domains, including manufacturing, finance, healthcare, legal, and engineering, among others.

Can I run the full 70‑task suite locally?

The authors provide an evaluation harness on GitHub and three public tasks on Hugging Face; the remaining 67 tasks are kept private to prevent leaderboard leakage, but the harness allows local execution of any released task.


Source: Grafy Research Lab — GDP.xlsx: Benchmarking Frontier Models on Professional Workbooks