Grafy News
Try Grafy free

Accelerate Bench evaluates AI coding agents on program optimization

Accelerate Bench, a new benchmark of 32 CPU‑level optimization tasks from scientific code.

Illustration for the news story: Accelerate Bench evaluates AI coding agents on program optimizationSource: Grafy Research Lab

What is Accelerate Bench and why it matters?

Accelerate Bench is a benchmark introduced by the Grafy Research Lab to measure how well autonomous coding agents can improve real‑world scientific programs. It contains 32 distinct CPU optimization tasks drawn from existing supercomputing competitions and research code bases. By providing a full program, a defined workload, correctness criteria, timing limits, and a list of permissible edits, the benchmark forces agents to reason about where to change code, what transformations are safe, and how to coordinate multiple edits across a code base. The ability to automate such program‑wide improvements could shorten development cycles for high‑performance computing and make AI‑driven performance tuning more reliable.

How were the 32 tasks designed and what constraints are imposed?

Each task in Accelerate Bench starts from a working implementation of a scientific kernel or application. The authors extracted the computational core, documented the expected input size, and measured a baseline execution time on a reference CPU. For every task they specified:

  • Correctness requirements – the output must match a reference result within a tolerance defined by the original application.
  • Timing boundaries – a maximum runtime that the optimized version must stay under, typically a fraction of the baseline.
  • Permitted edits – a whitelist of code transformations such as loop unrolling, vectorization, memory layout changes, or library substitution. Arbitrary rewrites that alter algorithmic behavior are disallowed. The selection of tasks spans linear algebra, stencil computations, particle simulations, and data‑intensive reductions, mirroring the diversity of workloads that appear in scientific and engineering pipelines. The benchmark’s 49‑page paper (including eight figures) details the provenance of each task and the rationale behind the edit constraints.

What evaluation process does the benchmark use?

Accelerate Bench adopts a two‑stage evaluation. First, participants submit iterative versions of their agent’s edits through a public testing interface that returns immediate feedback on correctness and runtime. This stage encourages rapid prototyping and lets developers refine strategies without penalty. After a predefined number of iterations, a final submission is sent to an isolated verifier that runs the program on a hidden hardware configuration, checks that all correctness checks still pass, and records the wall‑clock time. The verifier then assigns a score based on how far the runtime falls below the prescribed boundary, rewarding larger speed‑ups while penalizing violations. The public leaderboard ranks agents by their final scores, making it easy to compare different approaches.

Early insights from the benchmark’s initial results

The authors report that baseline agents—simple heuristic‑driven scripts—often achieve modest speed‑ups (10‑15 %) but struggle to respect the edit whitelist on more complex kernels. More sophisticated agents that combine static analysis with reinforcement‑learning‑based edit selection manage to cross the timing thresholds on roughly half of the tasks, sometimes delivering improvements of 30 % or more. A recurring difficulty is coordinating edits that interact, such as changing a data layout while also applying loop tiling; agents that treat each edit in isolation frequently break correctness. The benchmark therefore highlights the need for global reasoning and for mechanisms that can backtrack when an edit leads to a failed verification.

How Grafy users can explore Accelerate Bench insights

Developers interested in building or testing coding agents can leverage Grafy Code, the AI‑powered coding studio that runs code, installs dependencies, and provides instant execution feedback. By importing a task’s source files into Grafy Code, a user can experiment with the same edit constraints described in the benchmark and use the built‑in terminal to measure runtimes. For visualizing the sequence of edits and their impact, Grafy Graph offers a branching view where each node can represent a code version, the applied transformation, and the resulting performance metric. Researchers who want to read the full paper and query specific sections can do so with Grafy Reader, which lets them ask natural‑language questions about the methodology while the model references the PDF directly. Links to these apps are provided here: /code, /graph, and /reader.

Frequently asked questions

What types of program changes are allowed by Accelerate Bench?

The benchmark defines a whitelist for each task that may include transformations such as loop unrolling, vectorization, memory layout adjustments, and replacing library calls with faster alternatives. Any change that alters the algorithmic semantics or introduces new functional behavior is prohibited, ensuring that agents focus on performance‑only edits while preserving the original computation.

How does the two‑stage evaluation differ from a single‑run test?

In the first stage, agents receive immediate feedback on correctness and runtime after each iteration, enabling rapid debugging and iterative improvement. The second stage submits a final version to a hidden verifier that runs the program on a standardized hardware platform, checks that all constraints are still met, and assigns a final score. This separation discourages over‑fitting to the public test environment and mirrors real‑world deployment conditions.

Can I use Accelerate Bench without building a full AI agent?

Yes. Researchers can treat each task as a sandbox for manual optimization: apply permitted edits, run the code in Grafy Code, and compare the measured runtime against the benchmark’s baseline. This approach lets developers evaluate individual techniques—such as a new vectorization strategy—before integrating them into an autonomous agent.

Source: https://arxiv.org/abs/2611.20882


Source: Grafy Research Lab — Accelerate Bench: Evaluating Agents for Application-Level Program Optimization