Study finds multi-agent LLMs lose factual conflicts during delegation
Researchers introduced a benchmark that injects controlled factual conflicts into document‑review tasks and found that delegating work to multiple LLM agents.
Source: Grafy Research LabMulti‑agent language‑model systems are increasingly used to split large document‑review tasks across specialized workers. A new arXiv paper shows that while delegation improves raw evidence collection, it also creates a hidden consistency problem: factual conflicts retrieved by workers often disappear before the lead agent can see them, lowering overall conflict detection.
What is multi‑agent memory inconsistency?
Multi‑agent memory inconsistency refers to the situation where facts gathered by individual worker agents are not fully preserved in the shared context that the lead agent uses to form a final answer. In a lead–worker architecture, the lead assigns subtasks, the workers read separate document partitions, and then each worker sends a summary back. If a worker omits a claim that contradicts another worker’s finding, the lead can miss the conflict entirely. The authors illustrate this with a simple example: two workers retrieve different revenue figures for CorpX (A and B) but only one of them is reported to the lead, so the lead sees a single figure and does not flag the inconsistency. (Figure 1 shows this flow.)
How did the researchers measure the problem?
The team built a benchmark on top of the Harvey LAB document‑review suite. They injected 73 controlled numerical conflicts across 11 tasks, then ran three language‑model families under three delegation policies:
- None – a single agent processes the whole collection.
- Optional – the lead may delegate but is not forced to.
- Required – the lead must split the collection and assign workers.
During each run, an "evidence‑flow profiler" recorded four stages: (1) Source Reads – which facts each worker actually read, (2) Worker Reports – what each worker sent back, (3) Lead Evidence – what the lead retained after aggregating reports, and (4) Final Report – the completed document. Two metrics were extracted:
- Pair Recall – the proportion of conflict pairs that appear in a given stage.
- Judge Pass – the proportion of pairs that the system correctly flags as contradictory.
The authors used the same per‑task dollar budget for all models to keep generation spending comparable.
What were the key findings?
Across the three model families, requiring delegation increased the raw retrieval of both conflicting claims from 50.2 % (None) to 68.7 % (Required). However, the ability to detect the conflict dropped from 16.2 % to 13.2 %. Optional delegation performed best for detection, achieving 24.2 %.
A deeper look revealed that the biggest loss occurs at the worker‑to‑lead handoff. Under the Required policy, workers collectively retrieved both claims for a pair 68.7 % of the time, but the lead only saw both claims for 35.2 % of those pairs. In other words, more than half of the retrieved conflicts never made it into the lead’s evidence pool.
When the two conflicting claims were read by the same worker (the "Same" condition), detection was higher (34.2 % vs 22.5 % for single‑agent). When the claims were split across workers (the "Split" condition), detection fell (10.0 % vs 18.9 % for single‑agent). This asymmetry is visualised in Figure 3, which summarises how delegation improves collection but harms synthesis.
Why does delegation increase retrieval but decrease detection?
Delegation forces workers to explore separate document partitions in parallel, which naturally raises the chance that at least one worker encounters each side of a conflict. Yet each worker operates with a limited context window and is instructed to produce a concise report. The study found that workers often omit facts that are not central to their assigned subtask, even when those facts are needed to resolve a conflict later on. The lead, receiving only the trimmed reports, lacks the missing piece and therefore cannot flag the inconsistency.
The authors identify the worker‑to‑lead handoff as the primary target for improvement. Potential remedies include richer summary protocols, explicit prompting to surface contradictory evidence, or a second‑stage verification pass where the lead asks workers to re‑examine their sources for missing links.
How can creators and developers address this in their own workflows?
If you are building a multi‑agent pipeline—whether for research synthesis, legal document review, or software documentation—consider the following practical steps:
- Enforce evidence‑preserving summaries. Prompt workers to include any claim that differs from previously seen information, even if it seems peripheral.
- Add a reconciliation stage. After the lead aggregates reports, run a lightweight verification agent that cross‑checks the combined evidence for overlapping entities and numerical values.
- Track provenance explicitly. Store which worker supplied each fact and surface that provenance in the lead’s context, making it easier to spot gaps.
- Use a profiler. The benchmark’s evidence‑flow profiler can be mimicked by logging each API call and the returned snippets, allowing you to measure Pair Recall at each stage of your own system.
These tactics help preserve the “missing” facts that the paper shows are most often lost when delegation is mandatory.
Grafy perspective
Grafy does not yet provide a built‑in multi‑agent orchestration engine, but the problem can be explored with existing tools. Use Grafy Chat to prototype a lead‑worker conversation: assign a sub‑prompt to a separate chat instance, retrieve its summary, and then manually combine the outputs in a new session. To examine the benchmark itself, upload the PDF to Grafy Reader and ask the model to highlight where each conflict appears and whether it was reported. This hands‑on approach lets developers experience the evidence‑loss pattern and experiment with richer prompting strategies before committing to a custom orchestration layer.
Source: https://arxiv.org/abs/2611.22245


