CLADE: AI models guess when evidence is only a category wide

A study from Rensselaer Polytechnic Institute reports that retrieval-augmented language models stop abstaining when the passage they read names only a broad category rather than the exact answer, and that common fixes do not stop them. The behaviour matters because it turns a defensible "I don't know" into a confident wrong answer in exactly the setting where a structured lookup could have resolved the question.

What did the CLADE study find?

According to the paper by S M Rayeed, Charles V. Stewart and Yao Ma, retrieval-augmented generation (RAG) systems are usually judged on a binary test: a retrieved passage is either sufficient or insufficient, and a reliable model should answer in the first case and abstain in the second. The authors argue that many real passages sit between those poles. Their example is a question asking which disease lovastatin treats, where the true answer is the specific condition familial hyperlipidemia, while the retrieved passage states only that the drug is used to treat hyperlipidemia. The passage is accurate and relevant, and it rules out every disease outside that family, but it cannot settle which subtype is meant.

The authors name this middle case resolvable insufficiency: the text alone cannot fix the fine-grained answer, yet the coarse name maps to a short, finite candidate list in a domain ontology. When a model reads such a family-only passage and answers anyway, they call the behaviour Clade, short for Category-Level Answering from Deficient Evidence. On 856 biomedical questions drawn from PrimeKG and run across three 7–8B models, family-only passages cut abstention by 13.8 points relative to silent passages, which discuss the drug but mention neither the answer nor its family. About half of the extra answers were wrong, and precision fell from 84–95% to 45–59%.

How common is coarse evidence in real text?

The authors measured how reference texts state 1,800 PrimeKG facts and 23,694 GloBI interactions, sorting each passage as named (an exact match or synonym), family-only (a parent category within two levels of the answer in an ontology), or silent. Text omits the exact disease for 77.5–84.9% of PrimeKG facts. A family name on its own gave the model no closed-book advantage, so the coarse mention is not a shortcut to the answer; it is a nudge away from declining to answer. The paper also notes prior work by Joren et al. (2025), which labelled such passages simply as insufficient, and work showing that misleading passages, unlike merely missing ones, can defeat prompt-based abstention.

Did the family name actually cause the drop in abstention?

Yes, according to a counterfactual test the authors ran on edited pages. On the same questions, they either replaced the exact disease name with its broader family name or deleted the sentence entirely. The family edit lowered abstention by 17.4 points compared with deletion, which the authors read as evidence that the coarse category name itself, not the mere presence of a relevant passage, drives the effect. Figure 3 shows these page edits for 130 PrimeKG and 255 GloBI questions, averaged over the 7–8B models. The paper's worked examples make the failure concrete: for the drug moxifloxacin a model gave a confident wrong guess, while for lovastatin the family-only page made one model perform worse than it did without the page.

Do confidence filtering, a bigger model, or an instruction to abstain fix it?

No, according to the paper. The authors tested three candidate remedies and report that all three fall short. Confidence filtering did not flag the wrong answers, so a system cannot rely on the model's own certainty to catch these guesses. Scaling from the 7–8B models to a 14B model did not remove the effect. Explicitly instructing the model to abstain also failed to stop Clade. The behaviour replicated on a second domain: across 900 questions built from ecological species interactions in GloBI, the same pattern held, and in free-text answers no model produced the exact answer from a family-only passage. The authors' reading is that the problem is structural rather than a matter of calibration or prompt wording.

Can an ontology turn a coarse passage into an exact answer?

Partly, and this is where the paper locates both the opportunity and the remaining gap. Figure 1 frames resolvable insufficiency as an agent workflow with a measurable bottleneck at each stage: a page-type check should route a family-only passage to an ontology, which expands the family into a short candidate list for a chooser, while the dashed path is Clade, where the model simply guesses. On 410 family-only facts, an ontology list contained the true answer 98.5% of the time once the family was known, but string matching found that family on the page only 19.8% of the time. Letting the model choose the family raised that to 30.0%, and end-to-end recovery from 10.0% to 26.8%. Choosing which member of the list is correct remains weak: a simple popularity prior beat the model at that step. The authors describe the ontology expansion as the only stage that currently works reliably.

How can you explore this with Grafy?

The paper is a good candidate for Reader, Grafy's AI reading room for PDFs and papers, where you can load the arXiv text and question specific claims, such as the size of the abstention drop or which of the three fixes failed on which dataset. If you want to see the failure mode for yourself, Grafy Chat lets you put several models side by side on the same prompt, which is useful for checking whether a coarse context makes one model guess while another declines. For teams building assistants that answer only from their own material, Bots is the relevant surface to test whether your retrieval returns named, family-only or silent passages before you ship.

Frequently asked questions

Does the CLADE paper show that family-only passages reduce abstention?

Yes. According to the authors, on 856 PrimeKG questions across three 7–8B models, family-only passages cut abstention by 13.8 points compared with silent passages, and about half of the extra answers were wrong. A counterfactual edit that replaced the exact disease name with its family name lowered abstention by 17.4 points relative to deleting the sentence, which the authors take as evidence that the coarse name causes the effect.

Can confidence filtering, scaling, or an instruction to abstain stop Clade?

No, according to the paper. The authors report that confidence filtering did not flag the wrong answers, that moving from 7–8B models to a 14B model did not remove the effect, and that explicitly instructing the model to abstain also failed. The behaviour replicated across 900 GloBI questions on ecological species interactions, and in free-text answers no model gave the exact answer from a family-only passage.

Does an ontology recover the exact answer once the family is known?

Largely, yes, at that step. The authors report that on 410 family-only facts an ontology list held the true answer 98.5% of the time once the family was identified. The harder parts are upstream and downstream: string matching found the family on the page only 19.8% of the time, a model choosing the family raised that to 30.0%, and end-to-end recovery rose from 10.0% to 26.8%.

Source: CLADE: Category-Level Answering from Deficient Evidence in Retrieval-Augmented Language Models

---

Source: Grafy Research Lab — CLADE: Category-Level Answering from Deficient Evidence in Retrieval-Augmented Language Models