New Study Shows Limits of Agreement and Acceptance in Multi‑Agent Screening

Researchers at the University of Cambridge and XSci have released a new arXiv paper analyzing the limits of evidence in multi‑agent screening systems. The work clarifies what agreement, acceptance, overlap and audit data can actually tell us about recall, a key metric for e‑discovery, AI‑assisted literature review and other high‑cost human‑in‑the‑loop pipelines.

What problem does multi‑agent screening address?

In many enterprise and research workflows, cheap AI agents first filter a large collection—documents, code changes, or scholarly papers—before a costly human assessor reviews the selected items. This "incomplete‑judgment" setting means the assessor only sees what the agents forward, leaving a large unknown portion of the collection unexamined. The paper frames four practical questions operators face: (1) Does agreement among agents imply correctness? (2) How many relevant items are being missed? (3) Is adding another agent worth the expense? and (4) How much budget should be reserved for a post‑hoc audit?

Which types of evidence can be gathered from the system?

The authors distinguish four observable signals:

• Agreement – the joint output distribution of the $k$ agents.
• Acceptance – the human assessor’s accept/reject decisions on the pooled items.
• Overlap – knowledge of which agent(s) nominated each accepted item.
• Audit – a random sample of items that were not forwarded, judged separately. Each signal provides a different statistical handle on the unknown recall, defined as the proportion of all relevant items that eventually reach the assessor.

What can agreement among agents reveal?

The paper proves that consensus is not a guarantee of correctness. Modeling agents as sharing a latent representation $Z$ that influences both their outputs $O$ and the true relevance $Y$, the authors show two key points (Theorem 3.1):

1. The error rate of any decision rule that combines the agents is bounded below by $\mathbb{E}\min\{\eta(Z),1-\eta(Z)\}$, where $\eta(Z)=P(Y=1\mid Z)$. This floor is positive unless $Z$ fully determines relevance.
2. The distribution of the agents’ outputs does not depend on $\eta$. Consequently, the same pattern of agreement can correspond to any accuracy level between the extremes $\mathbb{E}\min\{g(Z),1-g(Z)\}$ and $\mathbb{E}\max\{g(Z),1-g(Z)\}$, where $g(Z)=P(A=1\mid Z)$. In short, agreement alone cannot distinguish a correct majority from a collectively wrong one.

How does assessor acceptance inform precision and recall?

When the assessor labels the pooled items, the observable acceptance rate $p = P(Y=1\mid A=1)$ directly estimates precision—the fraction of passed items that are truly relevant. However, as Proposition 1 states, this gives only a lower bound on recall because the unknown proportion $H = P(Y=1, A=0)$ of missed relevant items can vary anywhere from 0 to $1-a$ (with $a = P(A=1)$ the pool share). The resulting recall interval is $[ap/(ap+1-a),\,1]$, showing that acceptance cannot certify how many relevant items remain hidden.

When does overlap between agents bound recall?

If the system records who found each accepted item, the overlap statistics can tighten the recall upper bound—provided the agents’ finds are not negatively correlated. By splitting agents into two non‑empty groups and measuring the joint probabilities $h_{10}, h_{01}, h_{11}$ (items found by one group only, the other only, or both), the authors define a ratio $L = \max \frac{h_{10}h_{01}}{h_{11}}$. Under a mild positive‑dependence assumption, Theorem 3.2 shows that the missed‑relevant proportion $H$ must satisfy $H \ge L$. This yields a tighter recall interval $[ap/(ap+1-a),\,ap/(ap+L)]$ when $L \le 1-a$. In practice, low overlap (small $h_{11}$) signals many missed items, while high overlap caps the potential gain from additional agents.

How can audits close the recall gap?

The only way to identify $H$ without assumptions is to audit items outside the pool. Proposition 3 demonstrates that sampling $n_0$ items uniformly from the un‑pooled set and measuring the relevant fraction yields an estimator $\hat H$ with Hoeffding‑style confidence: with probability $1-\delta$, $|\hat H - H| \le \epsilon$ provided

For a concrete scenario—1,000 items, pool share $a=0.2$, tolerance $\epsilon=0.05$, and confidence $\delta=0.05$—the formula demands about 473 audit judgments. This sample size is independent of the number of agents, emphasizing that a modest audit budget can dramatically sharpen recall estimates.

Using Grafy to explore multi‑agent screening

Grafy’s suite offers practical ways to apply these insights:

• Grafy Reader lets you upload the arXiv PDF, ask the AI to summarize each section, and extract the formal statements (theorems, propositions) for quick reference.
• Grafy Graph can be used to model the workflow: nodes for each agent’s output, edges for the pooling rule, and a branch for the audit sample. Visualizing the overlap statistics as a bipartite graph helps spot low‑coverage areas.
• If you need to prototype a multi‑agent pipeline, the Chat app (linked via /chat) lets you experiment with different combination rules (e.g., majority vote, max‑score) and instantly see how the theoretical bounds shift.

These tools let developers and researchers iterate on screening designs while staying aware of the statistical limits outlined in the paper.

Source: https://arxiv.org/abs/2611.12705

Frequently asked questions

Does higher agreement among AI agents guarantee higher recall?

No. The study shows that agreement only indicates consensus, not correctness. Even a perfect agreement can correspond to any recall level because the agents may share a bias that systematically misses relevant items.

Can I estimate recall without auditing the un‑pooled items?

Only a lower bound (from precision) and, under strong positive‑dependence assumptions, an upper bound (from overlap) are available. Exact recall requires an audit of a random sample from the items the agents did not forward.

How much audit effort is needed to achieve a 5% error margin?

Using the Hoeffding bound from Proposition 3, the required sample size depends on the pool share $a$. For a collection of 1,000 items with $a=0.2$, achieving ±5% error at 95% confidence needs roughly 473 audited items.

---

Source: Grafy Research Lab — Multi-Agent Screening: What Agreement and Acceptance Cannot Tell You