Sparse Teacher Support Cuts LLM Distillation Cost Without Losing Accuracy
Researchers show that keeping only 0.01% of a language model's vocabulary during forward‑KL distillation preserves MMLU‑Pro performance while slashing training.
Try in Readerdistillationllmteacher supportresearch