Grafy News
Try Grafy free

STEMBench Introduces Postdoc‑Level Science Benchmark for LLMs

A new benchmark called STEMBench evaluates large language models on twelve advanced scientific tasks.

Illustration for the news story: STEMBench Introduces Postdoc‑Level Science Benchmark for LLMsSource: Grafy Research Lab

Researchers released STEMBench, a new benchmark that evaluates large language models on twelve postdoc‑level problems drawn from high‑energy physics, astrophysics and neuroscience. By scoring each task with the actual figure of merit used by the respective scientific community, the benchmark aims to measure how well AI agents can produce genuinely useful scientific solutions rather than just matching reference answers.

What is STEMBench and how does it differ from existing benchmarks?

STEMBench was presented by Avanesyan, Boldyrev, Deng and colleagues in a pre‑print on arXiv (2026) as a way to align AI evaluation with the criteria that scientists actually use to compare methods. Most current benchmarks rely on a fixed answer key, unit tests, rubrics, or leaderboard positions, which are convenient for automation but do not reflect the continuous metrics (e.g., background‑rejection efficiency, interval coverage) that decide which paper advances a field. STEMBench instead adopts those community‑specific figures of merit directly, treating them as the gold standard for scoring AI‑generated solutions.

Figure 2: Raw task scores of eight frontier models using the basic agent at high reasoning effort. Each marker is the mean of the three retained scored runs, including ze…
Figure 2. Raw task scores of eight frontier models using the basic agent at high reasoning effort. Each marker is the mean of the three retained scored runs, including zeros. Vertical bars show the minimum and maximum of these three scores, not standard deviations or confidence intervals. Horizontal lines mark the baseline b (dashed) and expert reference e (solid). A numerical zero does not by itself identi… Figure 2 of “Benchmarking LLM abilities on postdoc-level natural science problems” by Alina Avanesyan, Alexey Boldyrev, Liqun Deng et al. (arXiv.org), CC BY 4.0.

Which scientific domains and tasks are included?

The benchmark covers three research areas:

  • High‑energy physics: new‑physics anomaly detection (lhco_anomaly), Higgs uncertainty quantification (higgsml_uncertainty), top‑quark tagging (top_quark_tagging), and dark‑machine anomaly searches (darkmachines_anomaly).
  • Astrophysics: simulated and real Stokes inversion (stokes_inversion, hinode_inversion), and Type Ia supernova light‑curve classification (lightcurve_classification).
  • Neuroscience: autism spectrum disorder classification (abide_classification), neural mass model (NMM) family identification, digital twin generation, parameter recovery, and resting‑state BOLD deconvolution. Each task comes with a uniform contract that specifies input data, allowed retrievals, and the exact metric used by the field.

How are the scores calculated and what are the anchors?

For every task, the authors ship two reference submissions that can be run on a single workstation. The baseline anchor is a naïve solution that deliberately avoids any domain‑specific knowledge, establishing a floor performance. The expert anchor reproduces a published state‑of‑the‑art method, providing a ceiling. Both are executed on hidden test splits, and their scores are reported alongside any AI agent’s result. This interval‑based design makes it possible to see how much of the performance gap is bridged by scientific insight versus engineering tricks.

What did the evaluation of eight frontier models reveal?

The paper reports a high‑effort cross‑model comparison in which eight leading LLMs were run with substantial reasoning effort on each task. As shown in Figure 2, raw scores vary widely across tasks. A model that topped five of the twelve tasks did not lead the average score in any domain, highlighting a dissociation between task‑level wins and domain‑level leadership. Importantly, at least one model surpassed the expert anchor on seven tasks, indicating that current LLMs can already outperform reproduced published methods on a majority of the benchmark. The remaining five tasks—mostly involving uncertainty estimation, latent‑parameter inference, or multi‑site clinical classification—still favor the expert reference, suggesting areas where AI reasoning still lags.

How can creators, developers and researchers use this benchmark with Grafy?

Grafy’s Code studio lets users run the benchmark’s reference implementations and plug in their own LLM‑driven agents without leaving the platform. By importing the task contracts from the STEMBench repository, developers can experiment with prompt engineering, tool use, or custom model wrappers, then submit their outputs to the hosted grader for an immediate score. Researchers who prefer to explore the scientific literature behind each task can load the accompanying PDFs into Reader, where the AI can answer questions about methodology or metric definitions. This workflow lets users iterate quickly between code, evaluation, and insight while staying within Grafy’s multimodal environment.

Source: https://arxiv.org/abs/2611.24321

Frequently asked questions

How does STEMBench define a "baseline" and why is it important?

The baseline is a deliberately simple submission that solves the task without using any domain‑specific knowledge. It establishes the minimum performance any agent can achieve through generic engineering, allowing researchers to measure how much of the total progress is due to scientific insight rather than code optimisation.

Which metric is used for the high‑energy physics anomaly‑detection tasks?

Each HEP anomaly task is scored by the background‑rejection rate at a fixed signal‑efficiency, which is the same figure of merit that particle‑physics collaborations use to compare new search strategies.

Can I run the benchmark on my own hardware, or does it require cloud resources?

The benchmark’s code and data are sized between 0.1 GB and 12 GB per task and are designed to run offline on a single workstation. Grafy Code provides a convenient sandbox, but the same environment can be reproduced locally on any machine that meets the modest storage and CPU requirements.


Source: Grafy Research Lab — Benchmarking LLM abilities on postdoc-level natural science problems