Auditing On‑Policy Self‑Distillation Reveals Gaps in Privileged Guidance for Math Models

On‑policy self‑distillation (OPSD) has emerged as a promising way to improve reasoning models without expanding parameter counts, yet its reliance on privileged information raises open questions about what the student actually learns. A paper released by the Grafy Research Lab on November 13, 2026 systematically audits OPSD across four logical stages and three scales of the Qwen‑3 family, offering the first empirical map of where privileged guidance helps and where it does not.

What is on‑policy self‑distillation?

OPSD differs from classic distillation by making the teacher a frozen copy of the student model that is additionally fed privileged information during training. The student is then trained to predict the teacher’s next‑token distribution, effectively learning from the teacher’s “thought process” rather than from ground‑truth labels. Because the teacher sees data the student cannot, the hope is that the student inherits the teacher’s deeper reasoning without ever accessing the privileged content itself.

How does privileged information work in this setting?

In the study, the privileged signal is a worked solution – a step‑by‑step derivation of the correct answer that is hidden from the student. The teacher consumes this solution and produces token‑level targets that encode the solution’s reasoning path. The student, however, only receives the raw problem statement and the teacher’s token targets. The central question is whether the student’s final output reflects the hidden solution or merely mimics surface patterns.

Which stages were audited and why?

The authors break the OPSD pipeline into four stages, each representing a potential bottleneck:

1. Source – Does the teacher actually make use of the privileged solution when generating its targets?
2. Encoding – Are the token‑level targets produced by the teacher informative enough for the student to learn from?
3. Transmission – Does the student successfully absorb the information contained in those targets during training?
4. Expression – Does the student’s final answer demonstrate the reasoning captured in the hidden solution?

By evaluating each stage separately, the audit can pinpoint where the distillation chain breaks down, rather than attributing overall performance differences to a single opaque factor.

What did the audit find across the Qwen‑3 scales?

Using three model sizes from the Qwen‑3 family, the researchers applied the same mathematics benchmark to each stage. While the paper does not disclose exact numbers, it reports qualitative trends:

• Source – Larger teachers were more likely to incorporate the worked solution into their token predictions, suggesting that model capacity influences the ability to exploit privileged data.
• Encoding – The token targets varied in usefulness; some contained explicit step‑by‑step cues, while others collapsed the solution into a terse answer, reducing the student’s learning signal.
• Transmission – Smaller students often failed to internalize the nuanced patterns present in the teacher’s distribution, especially when the teacher’s targets were dense.
• Expression – Only the biggest student models occasionally reproduced the hidden reasoning flow, indicating that faithful transfer of privileged knowledge may require both a capable teacher and a sufficiently expressive student.

These observations imply that OPSD is not a universal shortcut; its effectiveness depends heavily on model scale and on how well each stage preserves the privileged signal.

Why does this matter for creators and developers?

For anyone building AI‑powered tools—whether generating code, composing music, or solving math problems—OPSD promises a way to boost performance without costly data collection. The audit, however, cautions that the shortcut can be fragile:

• Model selection matters. Deploying OPSD with a tiny student may yield negligible gains, even if the teacher is large.
• Privileged data format is critical. Providing a worked solution is richer than a single answer, but only if the teacher can translate that richness into useful token targets.
• Training pipelines need monitoring. Each stage can be inspected to ensure the signal is not lost, suggesting the value of tooling that visualises token‑level distributions.

Developers should therefore treat OPSD as a hypothesis‑driven experiment rather than a plug‑and‑play improvement.

How can you explore these ideas with Grafy?

Grafy offers several apps that let you replicate parts of the audit without leaving the platform:

• Reader – Load the arXiv PDF, ask follow‑up questions, and extract the exact sections describing each stage. This lets you verify the methodology directly.
• Chat – Run a small Qwen‑3 student and a frozen copy with a worked‑solution prompt to see the teacher’s next‑token distribution in real time. You can compare the outputs side by side.
• Graph – Build a branching workflow that records the source prompt, the teacher’s token targets, the student’s training run, and the final answer. The visual graph makes it easy to spot where information degrades.

By combining these tools, you can prototype your own OPSD experiments, test alternative privileged signals (e.g., code comments instead of math steps), and iterate quickly.

What are the next research directions?

The audit highlights several open questions that the community can pursue:

• Alternative privileged signals. Could explanations, diagrams, or even natural‑language rationales serve better than worked solutions?
• Dynamic teacher updates. Instead of a frozen copy, allowing the teacher to adapt during training might preserve more useful information.
• Cross‑modal distillation. Extending OPSD to multimodal models—such as those used in Grafy’s Creator or Film Studio—could unlock richer knowledge transfer.

Future work will likely explore these avenues, aiming to make self‑distillation a reliable tool across domains.

---

Source: https://arxiv.org/abs/2611.16064

Frequently asked questions

Does on‑policy self‑distillation guarantee that the student learns the hidden solution?

No. The audit shows that learning depends on model size and on how well each stage preserves the privileged signal; smaller students often fail to internalise the hidden reasoning.

Can I use Grafy’s Chat app to reproduce the teacher‑student setup described in the paper?

Yes. By loading a frozen copy of a model as the teacher and providing a worked solution in the prompt, you can generate token‑level targets and compare them with a student model’s predictions.

Is the audit’s conclusion that OPSD works only for large models?

The findings suggest that larger teachers and students are more likely to benefit, but the study does not rule out benefits for smaller models if the privileged information is encoded more effectively.

---

Source: Grafy Research Lab — Privileged Information Does Not Guarantee Privileged Guidance: Auditing On-Policy Self-Distillation for Reasoning Models