Grafy News
Try Grafy free

New benchmark tests AI agents on building RAG systems

Researchers have introduced AutoRAGBench to evaluate how well AI agents can autonomously design and refine RAG pipelines in open coding environments.

Illustration for the news story: New benchmark tests AI agents on building RAG systemsSource: Grafy Research Lab

Researchers have introduced AutoRAGBench, a new benchmark designed to evaluate the ability of Large Language Model (LLM) agents to autonomously develop Retrieval-Augmented Generation (RAG) systems. This development matters because while RAG is a standard method for grounding AI answers in external data, building an effective pipeline currently requires significant manual engineering and constant iterative testing by human developers.

What is AutoRAGBench and how does it work?

According to the researchers, AutoRAGBench is the first benchmark to assess how well AI agents can build RAG systems from scratch within an open code space. Unlike previous evaluations that focus on how well a specific RAG configuration performs, this benchmark measures the agent's ability to act as a developer.

In the experimental setup, agents are given a task involving specific corpora—covering domains like medicine and news—and are tasked with building a RAG pipeline. They are provided with a six-hour time limit and a single H800 GPU. Within this isolated environment, agents must write their own code, install necessary libraries, and download auxiliary models to solve the task. The benchmark includes nine distinct development tasks to ensure the agents are tested across various requirements, such as multi-hop reasoning or fact verification.

How well can AI agents automate RAG development?

The study evaluated nine different agent configurations. The results indicated that agents are capable of automating effective RAG development. Specifically, the researchers found that the GPT-6.1 Sol configuration using Codex CLI achieved the highest average score of 56.1 points. This performance exceeded the composite Human-SOTA (State-of-the-Art) baseline, which averaged 44.4 points under the same protocol.

Furthermore, the researchers noted that these agents do not just build static systems; they can improve upon existing ones. When the agents were given human-designed systems to work with across seven different tasks, they improved the scores by an average of 16.6 points. They achieved these gains while maintaining the core mechanisms of the original human-designed methods, suggesting that agents can refine and optimize rather than simply replacing logic with random configurations.

How is the performance of these agents measured?

To ensure a rigorous evaluation, the researchers employed a two-part assessment: RAG system evaluation and development process evaluation.

For the system evaluation, the researchers used five different held-out answering models to test the accuracy of the retrieved evidence. By averaging the scores across these models, they could isolate the quality of the RAG system itself from the potential biases of a single answering model.

For the development process, the researchers analyzed the "trajectories" of the agents. This involved looking at how the agents prepared for a task, how they implemented their retrieval methods, and how they used experimental feedback to guide their revisions. This allowed the team to see if the agents were actually learning from their mistakes or simply iterating blindly. The researchers observed that in 72 out of 77 builds with comparable scores, the agents reached a final version that was at least one point higher than their initial attempt.

What are the limitations of automated RAG development?

While the results are promising, the researchers identified certain complexities in the development cycle. They noted that larger gains during the development phase do not always translate to higher accuracy on hidden test sets. Additionally, the study found that in 20 builds, the final observed development score was at least one point lower than the best score achieved during the process, indicating that agents can sometimes regress during iteration.

This highlights the difficulty of the task: choosing the right retrieval strategy, context construction, and evidence selection is highly dependent on the specific corpus and the types of questions being asked. The ability of an agent to navigate these design choices autonomously is the primary frontier being tested by AutoRAGBench.

How can researchers apply these findings?

For researchers and developers working with complex documents, understanding the nuances of RAG is essential. While AutoRAGBench focuses on the automation of the pipeline itself, the underlying goal is to make AI more reliable when interacting with large datasets.

If you are currently conducting research on specialized papers or large PDF collections, you can explore how different RAG-like interactions feel by using Reader. While you are not building the pipeline from scratch, you can observe how AI models navigate, search, and question dense academic content, which is the ultimate goal of the automated systems described in this research.

Frequently asked questions

What is the main goal of AutoRAGBench?

AutoRAGBench aims to evaluate how effectively LLM agents can autonomously design, implement, and refine RAG systems from scratch. It moves beyond testing the accuracy of a single RAG setup to testing the agent's ability to function as a software engineer in an open coding environment.

Which agent performed best in the study?

The GPT-6.1 Sol configuration paired with Codex CLI was identified as the top performer. It achieved an average score of 56.1, outperforming both the human-designed baseline and existing automated RAG optimization methods that operate within predefined configuration spaces.

Can AI agents improve human-made RAG systems?

Yes, the research demonstrates that agents can enhance existing systems. When applied to seven different human-designed tasks, the agents improved the average score by 16.6 points. Crucially, they did this while preserving the core mechanisms of the original human-designed methods.

Source: Hang Zhang et al., AutoRAGBench: A Benchmark for LLM Agents Automating RAG System Development (arXiv:2611.2558)


Source: Grafy Research Lab — AutoRAGBench: A Benchmark for LLM Agents Automating RAG System Development