Survey launches Agent-as-a-Judge to make AI evaluation active and reliable
A new arXiv survey defines Agent-as-a-Judge, an active evaluator that gathers missing evidence.
Source: Grafy Research LabA new arXiv survey titled "Agent-as-a-Judge" maps the emerging shift from passive LLM judges to active, tool‑enabled evaluators. The authors argue that active judges are essential for reliable AI evaluation when evidence lies outside the supplied context.
What is an Agent-as-a-Judge?
The paper defines an Agent-as-a-Judge as an evaluator that maintains an evaluation state, performs targeted checks, and incorporates the results before issuing a verdict (according to the arXiv paper). Unlike a fixed‑context judge that only sees the provided evaluand and its immediate context, an agentic judge can plan actions, invoke external tools, and iteratively refine its understanding. The authors call this "evaluation‑directed checking" and position it as the core of the agentic approach.
Why passive LLM‑as‑a‑Judge falls short
Traditional LLM‑as‑a‑Judge systems operate in a static packet: the model reads the output to be judged, any reference material, and then returns a score or critique. This works for well‑specified tasks, but the survey highlights two major failure modes. First, critical environmental states—such as the result of a code execution, the current state of a workspace, or a live web page—are often absent from the static log. Second, factual claims that are controversial or outside the model’s internal knowledge need external verification. In both cases a passive judge cannot retrieve the missing evidence, leading to unreliable or incomplete verdicts (source).
How Agent-as-a-Judge works: the four key mechanisms
The authors organize the design space around four mechanisms that together enable active evaluation:
- Multi‑agent collaboration – Separate sub‑agents specialize in adversarial probing, perspective taking, analytical reasoning, or verdict control, and their outputs are aggregated to reduce bias (see Figure 2).
- Tool integration – The judge can call APIs, run code, browse the web, or query databases to obtain fresh evidence.
- Memory – Findings, intermediate artifacts, and prior evaluation experience are stored across steps, allowing the judge to reference earlier checks.
- Planning – A planning module decides which claims to verify, selects appropriate tools, and sequences actions until sufficient confidence is reached.
Figure 1 contrasts a passive LLM judge (left) with an active Agent-as-a-Judge (right), illustrating how the latter plans, gathers evidence, and iteratively verifies its judgment.
Survey findings and performance gaps
The survey consolidates published scores for five representative agentic judges across three unified benchmarks. The meta‑evaluation shows that while some agentic judges improve preference‑oriented metrics, the gains are uneven and often modest. In particular, evidence‑seeking reliability—how often the judge successfully locates missing facts—varies widely, indicating that current implementations are not yet uniformly robust. The authors therefore call for dedicated benchmarks that measure the completeness and traceability of the evidence‑gathering process.
Practical implications for creators, developers, and researchers
For anyone building or evaluating AI systems, the shift toward active judges suggests two immediate actions. First, incorporate verification steps into your own evaluation pipelines rather than relying solely on static LLM scores. Second, leverage tools that can automate evidence collection. Grafy provides several ways to experiment with these ideas:
- Grafy Chat (/chat) lets you interact with multiple LLMs side‑by‑side, making it easy to prototype a simple "judge" that asks follow‑up questions or calls external APIs via prompts.
- Grafy Graph (/graph) enables you to build a branching workflow where each node represents a check (e.g., web search, code execution) and the edges capture the plan. This mirrors the multi‑step reasoning described in the survey.
- For more complex agents, Grafy Code (/code) can host Python scripts that invoke the Operator browsing agent or other cloud services, turning a prototype into a reproducible evaluation tool.
By structuring your evaluation as a graph of checks, you can capture the memory of earlier findings and reuse them across multiple evaluands, aligning with the memory mechanism highlighted in the paper.
How to try Agent-as-a-Judge concepts in Grafy
- Define the evaluand – Upload the model output you want to assess (e.g., a generated summary) using the Files app.
- Create a check graph – In Graph, add a node that prompts a LLM to list any factual claims that need verification.
- Integrate tools – Attach a sub‑node that calls the Operator to fetch the latest web information or runs a code snippet in Code.
- Store findings – Use Graph’s built‑in memory to attach the retrieved evidence to the original claim node.
- Plan the next step – Add a planning node that decides whether additional checks are needed based on confidence thresholds.
- Produce the final verdict – A final LLM node reads the accumulated evidence and outputs a grounded score or critique.
This workflow reproduces the core ideas of the Agent-as-a-Judge taxonomy without requiring custom model training. It also lets you experiment with different collaboration patterns—for example, by adding a second LLM node that plays the adversarial role and challenges the first judge’s conclusions.
Source: https://arxiv.org/abs/2611.18067
Frequently asked questions
How does an Agent-as-a-Judge obtain evidence that is not in the original evaluation package?
It uses tool integration to call external services such as web browsers, code execution environments, or databases. The planner decides which claim to verify, invokes the appropriate tool, and stores the result for later reasoning.
Can I use Grafy to build a multi‑agent judging system without writing code?
Yes. Grafy Graph lets you visually chain together LLM prompts, tool calls, and memory nodes, effectively creating a multi‑agent workflow that mirrors the collaboration mechanism described in the survey.
What are the current limitations of Agent-as-a-Judge approaches?
The survey notes uneven performance across benchmarks, high computational cost for extensive tool use, and open challenges in safety, reliability, and governance when judges autonomously retrieve and act on external information.


