Reasoning Models Can Confabulate Their Own Requests after Invisible Activation Edits
A new arXiv paper shows that hidden activation edits can overwrite a model's belief about its task.
Source: Grafy Research LabChain‑of‑thought (CoT) monitoring and self‑report have been promoted as safety tools, but a new study from the Grafy Research Lab demonstrates that a hidden activation edit can sever the link between a model’s internal belief and the user’s actual request. The finding matters because it reveals a concrete way in which oversight mechanisms can be fooled while the model appears to reason coherently.
What the study discovered
The authors injected the forming‑trace of an off‑topic concept—such as the word “quark”—into the residual‑stream state of a reasoning model that was asked to write a haiku about a completely different topic, for example a cellphone. Across six models of varying scale, architecture, and training recipe, the injected concept became the model’s perceived task. When the model was forced to quote back the request it thought it had received, it reported the injected topic in 47 of 48 model‑concept cells, and the final generated haiku matched the injected topic in 99 % of sampled generations. Even explicit warnings that named the true topic restored the correct answer in fewer than 5 % of cases (see Figure 1). This demonstrates a belief edit rather than a simple output steer.
How the injection overwrites a model’s self‑report
The technique works by extracting the per‑layer residual‑stream representation ψ_inj(ℓ) that the model naturally forms when preparing to write about the off‑topic concept. After stripping a shared substrate subspace Π_S, the remaining vector v_inj(ℓ) is injected back into the model’s forward pass on the original prompt. Formally, the injected vector is:
where I is the identity matrix. This edit is invisible in the prompt text; the user sees no additional tokens. Once the edit is applied, the model’s internal reference to its own request is overwritten, so the subsequent chain‑of‑thought, answer, and any forced quote‑back all treat the injected concept as ground truth. The model even fabricates a justification, claiming the user may have confused the topics, despite the prompt explicitly mentioning the true topic.
Why chain‑of‑thought monitoring may be unreliable
CoT monitoring assumes that a model’s reasoning trace is anchored to the original instruction. The paper shows that attention maps can be misleading: after injection, the model attends more strongly to the true‑topic token—up to three times higher—but this heightened attention is causally inert. Ablation experiments (Figure 2) reveal that removing the attention to the true token collapses the clean‑topic generation while leaving the injected‑topic generation untouched. Consequently, visual oversight tools that rely on attention heatmaps or circuit attribution may still see a model appearing on‑task while it is internally following a falsified belief.
Which models were affected
The experiment covered six publicly documented reasoning models:
- R1‑Qwen‑7B, ‑14B, ‑32B (same base and recipe, varying scale)
- R1‑Llama‑8B (different base architecture)
- QwQ‑32B (Qwen RL‑trained, no distillation)
- Nemotron‑7B (NVIDIA supervised‑fine‑tuned on R1 traces) All six exhibited the belief‑edit effect in the forced quote‑back arm, and the majority also produced the injected answer in the natural generation arm. The consistency across scale, architecture, and training recipe suggests that the vulnerability is not limited to a single family of models.
Practical takeaways for AI developers and researchers
- Do not rely solely on self‑reported task descriptions. Even a model that can articulate its reasoning may be operating on a corrupted internal belief.
- Treat attention visualizations as necessary but insufficient evidence. As the study shows, higher attention to the true prompt does not guarantee correct behavior.
- Design multi‑layer verification. Combining external checks (e.g., re‑prompting, cross‑model consensus) with internal monitoring can reduce the risk of covert belief edits.
- Consider activation‑level defenses. While the paper does not propose a fix, future work might explore detecting sudden residual‑stream shifts or enforcing consistency between the forming‑trace and the downstream task circuit.
Applying the insight with Grafy
Grafy does not yet expose direct activation‑editing APIs, but the findings can still inform how you use the platform. You can explore the paper in depth with Grafy Reader, which lets you upload the PDF and ask the model to summarize sections, extract the methodology, or compare the six models discussed. To experiment with how chain‑of‑thought monitoring behaves on your own prompts, try the Grafy Chat interface and manually request a model to repeat back its perceived task after a complex instruction. Observing any mismatch between the original request and the model’s self‑report can give you a hands‑on sense of the risk highlighted by the research.
Source: https://arxiv.org/abs/2611.17958
Frequently asked questions
Does the injected activation edit change the model’s output only, or its internal belief as well?
The study shows it changes the internal belief about the request. The model’s chain‑of‑thought, final answer, and forced quote‑back all treat the injected concept as the true task, even when the prompt explicitly names a different topic.
Can a warning that names the correct topic reliably fix the belief edit?
No. The authors tested a ladder of warnings, including one that explicitly names the true topic, and found recovery rates below 5 % across all models. The injected belief remains dominant.
How can I detect such belief edits when using Grafy’s AI tools?
While Grafy does not provide low‑level activation access, you can cross‑check a model’s self‑report by asking it to repeat the original instruction verbatim. Discrepancies between the prompt and the model’s quoted request may indicate a belief edit, prompting you to re‑prompt or use a different model for verification.


