Infrathon Benchmark Finds Long‑Horizon AI Agents Keep Improving Infrastructure

What is the Infrathon benchmark and why was it created?

The Grafy Research Lab introduced Infrathon as a systematic way to assess how frontier AI agents handle real‑world infrastructure tasks when given extended periods to act. Traditional agent evaluations often run for a few minutes or hours, which can hide the ability of a model to discover deeper, cost‑saving strategies that only emerge after many iterations. By framing the problem as a "long‑horizon" optimization challenge, the authors aimed to surface performance differences that only become apparent when agents are allowed to explore, experiment, and refine over a full 24‑hour budget.

How were the experiments run and what were the headline findings?

The study ran six distinct language‑based agents across 150 separate runs, each run limited to a 24‑hour wall‑clock budget. In total the experiments consumed more than 10,000 GPU‑hours, making it one of the most compute‑intensive agent benchmarks to date. The tasks spanned multiple layers of AI infrastructure, from model selection and hyper‑parameter tuning to resource allocation and cost‑optimization across cloud providers.

Key observations included:

• Model rankings shifted depending on whether performance was measured after the first half (12 hours) or the full 24 hours. A model that appeared strongest early on could be overtaken later.
• 135 out of the 150 runs (90 %) recorded additional improvements in the second half of the budget, indicating that most agents kept learning or adapting rather than plateauing.
• Shorter evaluation windows risk under‑reporting both absolute gains and relative ordering of agents, which could mislead developers when choosing a model for production use.

These results were reported by the authors Zhengyu Hu, Zheyuan Xiao, Yike Wang, and colleagues in a pre‑print posted on 10 November 2026.

Why do the results matter for developers and researchers?

Long‑horizon behavior is critical for any organization that relies on AI to manage expensive compute resources. If a model appears efficient after a brief test but later stalls, the hidden costs can add up quickly. Conversely, an agent that discovers subtle scheduling tricks or better hardware‑mixes after several hours can deliver sizable savings that short tests would miss.

The benchmark also highlights the importance of evaluation methodology. Many existing leaderboards rank agents based on a single snapshot, which may favor models that excel at rapid, shallow reasoning. Infrathon suggests that a more nuanced, time‑aware metric better reflects real‑world utility, especially for tasks like auto‑scaling clusters, continuous model deployment, or dynamic pricing of cloud instances.

For researchers, the dataset of 150 runs and the associated logs become a valuable resource for studying agent learning dynamics, such as how exploration strategies evolve, when agents switch from trial‑and‑error to exploitation, and how they cope with stochastic cloud‑service responses.

How can practitioners incorporate these insights into their own AI workflows?

1. Allocate longer evaluation windows when testing new agent‑based automation. Even a modest extension from 4 to 12 hours can reveal different convergence patterns.
2. Track performance over time rather than relying on a single final score. Plotting cost or latency curves helps spot late‑stage improvements.
3. Use multi‑model ensembles. Since rankings shift, rotating between agents at different phases of a long‑run may capture the strengths of each.
4. Integrate budget‑aware stopping criteria. The Infrathon study showed that most agents keep improving up to the budget limit, suggesting that adaptive budgets (e.g., stop when marginal gain falls below a threshold) can be more efficient than fixed‑time cuts.
5. Leverage open‑source tools for logging and analysis. The benchmark’s authors released their run logs, which can be replayed to benchmark custom agents against the same tasks.

By redesigning evaluation pipelines to respect long‑horizon dynamics, teams can avoid premature model selection and unlock hidden optimization potential in their AI‑driven infrastructure stacks.

How can Grafy users explore or apply the Infrathon benchmark?

Grafy provides several apps that make it easy to dig into the Infrathon paper and experiment with similar workflows:

• Grafy Reader lets you upload the PDF, ask natural‑language questions, and retrieve specific sections without leaving the platform.
• Grafy Graph can map the branching decisions an agent makes during a run, letting you visualize how prompts, outputs, and model selections evolve over time.
• Grafy Chat gives you instant access to the same frontier models evaluated in the study, so you can run short pilot experiments and compare their early‑stage behavior.

While Grafy does not yet host a built‑in long‑horizon benchmark runner, the combination of Reader, Graph, and Chat lets creators prototype the kind of iterative optimization loops Infrathon examines. For example, you could set up a 24‑hour loop that repeatedly queries a model to suggest cloud‑cost reductions, records each suggestion in Graph, and monitors cumulative savings.

---

Source: Grafy Research Lab, https://arxiv.org/abs/2611.11995

Frequently asked questions

What does "long‑horizon" mean in the context of AI agent evaluation?

It refers to giving an agent a prolonged, continuous time budget—typically many hours—so it can explore, adapt, and refine its actions beyond the quick, shallow decisions captured in short tests.

Did all six models improve during the second half of their runs?

No, but a large majority did: 135 of the 150 runs (90 %) showed additional improvement after the first 12 hours, indicating that most agents kept learning or finding better solutions.

Can Grafy run a full 24‑hour Infrathon‑style benchmark directly?

Grafy does not currently provide a dedicated long‑run automation engine, but you can use Reader to study the benchmark, Graph to track iterative prompts, and Chat to experiment with the same models on a smaller scale.

---

Source: Grafy Research Lab — Infrathon: Benchmarking Language Agents for Real-World, Long-Horizon AI Infra Optimization