Spotlight

Papers

Field

Top tools here

Open problems in Teacher policy distillation

Teacher recoverability on incorrect student rollouts1 paper raises it · 1 addresses it
Teachers often follow student errors instead of correcting them, degrading supervision quality in on-policy distillation.
- Recovering Reliable Supervision via Critique-Guided On-Policy Distillationraises · addresses
Gradient conflicts in multi-teacher on-policy distillation1 paper raises it · 2 address it
Divergent teacher styles and expertise cause conflicting gradients, harming joint optimization and student performance.
- Beyond Domain-Routed Supervision: Cross-Teacher Guarding for Multi-Teacher On-Policy Distillationraises · addresses
- Latent-MOPD: Latent Multi-Teacher On-Policy Distillationaddresses
Policy shift opacity in teacher distillation1 paper raises it · 1 addresses it
Endpoint policy shifts hide whether local action preferences evolved consistently during teacher training, limiting selective transfer.
- Waypoint-OPD: Transferring RL-Induced Policy Shifts via Gated On-Policy Distillationraises · addresses
Cross-tokenizer supervision reliability1 paper raises it · 1 addresses it
Vocabulary mismatch and alignment coverage issues reduce the effectiveness of token-level supervision in on-policy distillation.
- Rethinking Cross-Tokenizer On-Policy Distillation: From Alignment Coverage to Supervision Reliabilityraises · addresses
Collapse in iterative agent self-distillation1 paper raises it · 1 addresses it
Successive distillation cycles degrade deployment performance and privileged-information-conditioned behavior, hindering recursive self-improvement.
- ReSAIL: Mitigating Collapse in Iterative Agent Self-Distillationraises · addresses

Named by an AI from these papers' abstracts.

Open problems

Recently funded

Topics in this field

teacher policy distillation · policy distillation opd · teacher supervision309 papers46 this week
ai tutors · ai tutor · tutoring54 papers9 this week

Topics in this field

Active contributors · 90 days

Hao LiReTeach: Building a Self-Teacher through Multi-Round Reflection and Retry4
Houcheng JiangReCal: Calibrating Structured Pruning for On-Policy Distillation Recovery4
Junfeng FangReCal: Calibrating Structured Pruning for On-Policy Distillation Recovery4
Mao ZhengReCal: Calibrating Structured Pruning for On-Policy Distillation Recovery4
Mingyang SongReCal: Calibrating Structured Pruning for On-Policy Distillation Recovery4
Kaixiang ZhaoRecovering Reliable Supervision via Critique-Guided On-Policy Distillation3

Active contributors

Activity · 12 weeks

Coming with more sources: repos

Activity

★ Spotlight this month

arXiv cs.LGOct 15

Training-Free On-Policy Distillation

Bangji Yang, Jiajun Fan, Zhaoyu Zhu, Hongbo Ma, Ge Liu

On-policy distillation consolidates specialist models into one student at the price of rollouts, teacher passes and gradient steps for every teacher. We show that the consolidation does not need training: with forward passes only, the capabilities of separately tuned specialists can be aggregated into one model that is, on average, more accurate than either specialist alone. When specialists share a base model, distillation has an exact layer-wise solution: each linear layer of the fused model must reproduce every specialist's output on the inputs that specialist receives while reading its own answers, a least-squares problem whose solution needs only the second moments of those inputs. One forward pass over a few hundred rollouts provides them. We call this training-free on-policy distillation (\method). What the rollouts contain matters more than the solver: answers truncated in repetition loops are few among a mathematics specialist's rollouts but carry a large share of their tokens, and removing them lifts the fused model on AIME25 from task arithmetic's level to the specialist's at matched precision. On Olmo~3 7B, a few GPU-hours of forward computation per specialist yield a…

PDF arXiv page Details

Spotlight

Recent papers

All in Research →

arXiv cs.LGCodeOct 15▲ 5
Predicting Privileged Information Effectiveness for On-Policy Self-Distillation
Kaijian Zou, Emiliano Penaloza, MohammadHossein Rezaei, Anisha Gunjal, Mohit Raghavendra +3
On-policy self-distillation (OPSD) enables large language models (LLMs) to learn from privileged information (PI) without a separate teacher model. However, prior work largely treats PI as a fixed design choice, leaving the effects of its content and construction poorly understood. This challenge is particularly important in multi-turn agentic settings, where trajectories contain actions, observations, and reasoning that can be transformed into PI in many ways. We systematically evaluate 16 PI configurations across three multi-turn agentic benchmarks and three student models that vary in scale and reasoning mode. We find that PI effectiveness depends on its content, generation source, and the student model. Information-dense trajectories and PI produced by more capable generators generally yield larger gains, whereas thinking models benefit less consistently. To assess PI before training, we compare PI utility with two KL-based metrics: sampled-token reverse KL ($D$) and error-step KL contrast ($Δd^{-}_{\mathrm{err}}$). The KL-based metrics track post-training performance more consistently than PI utility, which can be distorted by information leakage. Empirically, selecting each…
PDF arXiv Code
arXiv cs.CVCodeOct 15▲ 3
Skill-Decoupled On-Policy Expert Distillation for Unified Vision-Language Models
Yuqi Xiong, Chunyi Peng, Zhipeng Xu, Qing Yang, Zhenghao Liu +3
Reinforcement learning (RL) can elicit specialized visual capabilities in vision-language models (VLMs), yet jointly optimizing heterogeneous objectives often introduces cross-capability gradient conflicts. Training experts independently mitigates this interference, but direct reward optimization may entangle specialized knowledge with reward-hacking biases. This paper proposes Skill-Decoupled On-Policy Expert Distillation (SCOPED), which trains capability-specific experts and extracts explicit skills through reward-guided reflection. During on-policy distillation, these skills guide expert supervision while remaining unavailable to the unified student, encouraging it to internalize procedural knowledge without explicit skills at inference time. Across eight multimodal benchmarks, SCOPED consistently outperforms the backbone and mixed-capability RL, demonstrating the effectiveness of skill-decoupled on-policy expert distillation for integrating independently acquired capabilities. Further analyses reveal interference during joint capability optimization and suggest that skill augmentation promotes more consistent expert trajectories, greater attention to task-relevant visual…
PDF arXiv Code
arXiv cs.LGCodeOct 15▲ 668
Waypoint-OPD: Transferring RL-Induced Policy Shifts via Gated On-Policy Distillation
Bingxi Hou, Guochao Jiang, Guofeng Quan, Weiqing Li, Wenfeng Feng +2
Policy-shift distillation reuses the difference between a teacher's log probabilities before and after reinforcement learning (RL) as supervision on student-generated trajectories. However, this endpoint policy shift cannot reveal whether local action preferences evolved consistently during teacher training. The same endpoint shift can result from early and late shifts that reinforce or partially cancel each other. We investigate whether agreement between early and late teacher policy shifts can guide selective transfer. We introduce Waypoint-guided On-Policy Distillation (Waypoint-OPD), which uses an intermediate teacher checkpoint from the same RL run to compare early and late shifts in relative action preferences at student-visited prefixes. We gate the endpoint shift according to the directional agreement between the early and late shifts, suppressing it when they point in opposing directions. Compared with endpoint-only transfer, our method requires only one additional frozen-teacher scoring pass. Using RL-trained Qwen3-4B teachers, we evaluate our method with Qwen3-8B, Qwen3-14B, and Qwen3-30B-A3B as student models in mathematical and code reasoning. In all six…
PDF arXiv Code
arXiv cs.LGCodeOct 14▲ 79
Recovering Reliable Supervision via Critique-Guided On-Policy Distillation
Kaixiang Zhao, Tianrun Yu, Shangzhe Li, Yuxiao Yang, Amanda Hughes +2
On-policy distillation (OPD) trains a student model on its own rollouts with token-level supervision from a teacher, assuming that the teacher can reach a correct answer from any prefix the student writes. We call this property the recoverability of the teacher and show it fails on incorrect rollouts: the teacher follows student errors, although it recognizes them. We propose CriticOPD, using error recognition to restore recoverability. The teacher critiques an incorrect rollout, allowing the student to rewrite its solution following the error. The resulting output, including the critique as privileged information, is scored by the teacher while the student uses the revision without the critique for training. On two Qwen3 students and six mathematical reasoning benchmarks, CriticOPD improves OPD, with the largest gains on competition problems.
PDF arXiv Code
arXiv cs.CLCodeOct 14▲ 8
VASPO: Self-Distillation with Privileged Value Estimation
Fei Wu, Zhenrong Zhang, Jianshu Zhang, Jun Du
On-policy self-distillation provides dense supervision for reasoning models via a privileged self-teacher. However, this supervision reflects local token preferences rather than the expected return of an unguided student continuation. Recent methods combine this signal with GRPO. They use GRPO for update directions and the teacher for signal scaling. However, we demonstrate a more effective integration within PPO. Specifically, we identify complementary positional dynamics: privileged information enhances a critic's value-prediction accuracy toward the response's end, whereas self-teacher guidance is strong initially, drops mid-generation, and rebounds finally. These insights motivate Value-Augmented Self-Distillation Policy Optimization (VASPO), wherein a self-teacher and critic evaluate identical privileged information at student-generated prefixes. The critic computes token-level PPO advantages from predicted continuation returns, while the self-teacher modulates selected positive advantages via bounded probability ratios. Across six mathematical reasoning benchmarks, VASPO improves mean accuracy over PPO by 2.59 and 1.81 percentage points on Qwen3-4B and OLMo3-7B variants,…
PDF arXiv Code
arXiv cs.AICodeOct 14▲ 57
ActionOPD: Distilling Decisions, Not Tokens
Huining Yuan, Xiangmin Yi, Zelai Xu, Zixu Wang, Xinlei Chen +3
Distilling capable language-model agents into smaller models requires aligning the student's behavior with the teacher's. This raises a fundamental question: at which states and at what granularity should this alignment be performed? Offline imitation provides dense but token-specific supervision and can suffer from state-distribution mismatch. Token-level on-policy distillation (OPD) addresses this mismatch but typically requires teacher token probabilities and compatible tokenization. We argue that semantic actions provide a natural unit of supervision for agent distillation. We introduce ActionOPD, an action-level on-policy distillation method motivated by maximizing expected teacher-action likelihood at student-encountered states. At each such state, ActionOPD samples actions from both the student and a black-box teacher, rewarding student actions according to their semantic agreement with the sampled teacher actions. Through policy optimization, ActionOPD aligns the student's action-level behavior with the teacher's decisions without requiring reproduction of teacher-written reasoning or access to teacher token probabilities. Training Qwen3-4B and Qwen3-8B with two teachers…
PDF arXiv Code

Papers

Topics in this field

teacher policy distillation · policy distillation opd · teacher supervision309 papers46 this week
ai tutors · ai tutor · tutoring54 papers9 this week

★ Spotlight this month

arXiv cs.LGOct 15

Training-Free On-Policy Distillation

Bangji Yang, Jiajun Fan, Zhaoyu Zhu, Hongbo Ma, Ge Liu

On-policy distillation consolidates specialist models into one student at the price of rollouts, teacher passes and gradient steps for every teacher. We show that the consolidation does not need training: with forward passes only, the capabilities of separately tuned specialists can be aggregated into one model that is, on average, more accurate than either specialist alone. When specialists share a base model, distillation has an exact layer-wise solution: each linear layer of the fused model must reproduce every specialist's output on the inputs that specialist receives while reading its own answers, a least-squares problem whose solution needs only the second moments of those inputs. One forward pass over a few hundred rollouts provides them. We call this training-free on-policy distillation (\method). What the rollouts contain matters more than the solver: answers truncated in repetition loops are few among a mathematics specialist's rollouts but carry a large share of their tokens, and removing them lifts the fused model on AIME25 from task arithmetic's level to the specialist's at matched precision. On Olmo~3 7B, a few GPU-hours of forward computation per specialist yield a…

PDF arXiv page Details

Active contributors · 90 days

Hao LiReTeach: Building a Self-Teacher through Multi-Round Reflection and Retry4
Houcheng JiangReCal: Calibrating Structured Pruning for On-Policy Distillation Recovery4
Junfeng FangReCal: Calibrating Structured Pruning for On-Policy Distillation Recovery4
Mao ZhengReCal: Calibrating Structured Pruning for On-Policy Distillation Recovery4
Mingyang SongReCal: Calibrating Structured Pruning for On-Policy Distillation Recovery4
Kaixiang ZhaoRecovering Reliable Supervision via Critique-Guided On-Policy Distillation3

Recent papers

All in Research →

arXiv cs.LGCodeOct 15▲ 5
Predicting Privileged Information Effectiveness for On-Policy Self-Distillation
Kaijian Zou, Emiliano Penaloza, MohammadHossein Rezaei, Anisha Gunjal, Mohit Raghavendra +3
On-policy self-distillation (OPSD) enables large language models (LLMs) to learn from privileged information (PI) without a separate teacher model. However, prior work largely treats PI as a fixed design choice, leaving the effects of its content and construction poorly understood. This challenge is particularly important in multi-turn agentic settings, where trajectories contain actions, observations, and reasoning that can be transformed into PI in many ways. We systematically evaluate 16 PI configurations across three multi-turn agentic benchmarks and three student models that vary in scale and reasoning mode. We find that PI effectiveness depends on its content, generation source, and the student model. Information-dense trajectories and PI produced by more capable generators generally yield larger gains, whereas thinking models benefit less consistently. To assess PI before training, we compare PI utility with two KL-based metrics: sampled-token reverse KL ($D$) and error-step KL contrast ($Δd^{-}_{\mathrm{err}}$). The KL-based metrics track post-training performance more consistently than PI utility, which can be distorted by information leakage. Empirically, selecting each…
PDF arXiv Code
arXiv cs.CVCodeOct 15▲ 3
Skill-Decoupled On-Policy Expert Distillation for Unified Vision-Language Models
Yuqi Xiong, Chunyi Peng, Zhipeng Xu, Qing Yang, Zhenghao Liu +3
Reinforcement learning (RL) can elicit specialized visual capabilities in vision-language models (VLMs), yet jointly optimizing heterogeneous objectives often introduces cross-capability gradient conflicts. Training experts independently mitigates this interference, but direct reward optimization may entangle specialized knowledge with reward-hacking biases. This paper proposes Skill-Decoupled On-Policy Expert Distillation (SCOPED), which trains capability-specific experts and extracts explicit skills through reward-guided reflection. During on-policy distillation, these skills guide expert supervision while remaining unavailable to the unified student, encouraging it to internalize procedural knowledge without explicit skills at inference time. Across eight multimodal benchmarks, SCOPED consistently outperforms the backbone and mixed-capability RL, demonstrating the effectiveness of skill-decoupled on-policy expert distillation for integrating independently acquired capabilities. Further analyses reveal interference during joint capability optimization and suggest that skill augmentation promotes more consistent expert trajectories, greater attention to task-relevant visual…
PDF arXiv Code
arXiv cs.LGCodeOct 15▲ 668
Waypoint-OPD: Transferring RL-Induced Policy Shifts via Gated On-Policy Distillation
Bingxi Hou, Guochao Jiang, Guofeng Quan, Weiqing Li, Wenfeng Feng +2
Policy-shift distillation reuses the difference between a teacher's log probabilities before and after reinforcement learning (RL) as supervision on student-generated trajectories. However, this endpoint policy shift cannot reveal whether local action preferences evolved consistently during teacher training. The same endpoint shift can result from early and late shifts that reinforce or partially cancel each other. We investigate whether agreement between early and late teacher policy shifts can guide selective transfer. We introduce Waypoint-guided On-Policy Distillation (Waypoint-OPD), which uses an intermediate teacher checkpoint from the same RL run to compare early and late shifts in relative action preferences at student-visited prefixes. We gate the endpoint shift according to the directional agreement between the early and late shifts, suppressing it when they point in opposing directions. Compared with endpoint-only transfer, our method requires only one additional frozen-teacher scoring pass. Using RL-trained Qwen3-4B teachers, we evaluate our method with Qwen3-8B, Qwen3-14B, and Qwen3-30B-A3B as student models in mathematical and code reasoning. In all six…
PDF arXiv Code
arXiv cs.LGCodeOct 14▲ 79
Recovering Reliable Supervision via Critique-Guided On-Policy Distillation
Kaixiang Zhao, Tianrun Yu, Shangzhe Li, Yuxiao Yang, Amanda Hughes +2
On-policy distillation (OPD) trains a student model on its own rollouts with token-level supervision from a teacher, assuming that the teacher can reach a correct answer from any prefix the student writes. We call this property the recoverability of the teacher and show it fails on incorrect rollouts: the teacher follows student errors, although it recognizes them. We propose CriticOPD, using error recognition to restore recoverability. The teacher critiques an incorrect rollout, allowing the student to rewrite its solution following the error. The resulting output, including the critique as privileged information, is scored by the teacher while the student uses the revision without the critique for training. On two Qwen3 students and six mathematical reasoning benchmarks, CriticOPD improves OPD, with the largest gains on competition problems.
PDF arXiv Code
arXiv cs.CLCodeOct 14▲ 8
VASPO: Self-Distillation with Privileged Value Estimation
Fei Wu, Zhenrong Zhang, Jianshu Zhang, Jun Du
On-policy self-distillation provides dense supervision for reasoning models via a privileged self-teacher. However, this supervision reflects local token preferences rather than the expected return of an unguided student continuation. Recent methods combine this signal with GRPO. They use GRPO for update directions and the teacher for signal scaling. However, we demonstrate a more effective integration within PPO. Specifically, we identify complementary positional dynamics: privileged information enhances a critic's value-prediction accuracy toward the response's end, whereas self-teacher guidance is strong initially, drops mid-generation, and rebounds finally. These insights motivate Value-Augmented Self-Distillation Policy Optimization (VASPO), wherein a self-teacher and critic evaluate identical privileged information at student-generated prefixes. The critic computes token-level PPO advantages from predicted continuation returns, while the self-teacher modulates selected positive advantages via bounded probability ratios. Across six mathematical reasoning benchmarks, VASPO improves mean accuracy over PPO by 2.59 and 1.81 percentage points on Qwen3-4B and OLMo3-7B variants,…
PDF arXiv Code
arXiv cs.AICodeOct 14▲ 57
ActionOPD: Distilling Decisions, Not Tokens
Huining Yuan, Xiangmin Yi, Zelai Xu, Zixu Wang, Xinlei Chen +3
Distilling capable language-model agents into smaller models requires aligning the student's behavior with the teacher's. This raises a fundamental question: at which states and at what granularity should this alignment be performed? Offline imitation provides dense but token-specific supervision and can suffer from state-distribution mismatch. Token-level on-policy distillation (OPD) addresses this mismatch but typically requires teacher token probabilities and compatible tokenization. We argue that semantic actions provide a natural unit of supervision for agent distillation. We introduce ActionOPD, an action-level on-policy distillation method motivated by maximizing expected teacher-action likelihood at student-encountered states. At each such state, ActionOPD samples actions from both the student and a black-box teacher, rewarding student actions according to their semantic agreement with the sampled teacher actions. Through policy optimization, ActionOPD aligns the student's action-level behavior with the teacher's decisions without requiring reproduction of teacher-written reasoning or access to teacher token probabilities. Training Qwen3-4B and Qwen3-8B with two teachers…
PDF arXiv Code

Activity · 12 weeks

Coming with more sources: repos

Fields · AI · updated 6m ago

Teacher policy distillation

Found by the topic model: teacher policy distillation, ai tutors.

55papers this week−25.7% vs last week

All papers · 363 With code · 20

Newest Most relevant

350 papers this month

Open problems in Teacher policy distillation

Teacher recoverability on incorrect student rollouts1 paper raises it · 1 addresses it
Teachers often follow student errors instead of correcting them, degrading supervision quality in on-policy distillation.
- Recovering Reliable Supervision via Critique-Guided On-Policy Distillationraises · addresses
Gradient conflicts in multi-teacher on-policy distillation1 paper raises it · 2 address it
Divergent teacher styles and expertise cause conflicting gradients, harming joint optimization and student performance.
- Beyond Domain-Routed Supervision: Cross-Teacher Guarding for Multi-Teacher On-Policy Distillationraises · addresses
- Latent-MOPD: Latent Multi-Teacher On-Policy Distillationaddresses
Policy shift opacity in teacher distillation1 paper raises it · 1 addresses it
Endpoint policy shifts hide whether local action preferences evolved consistently during teacher training, limiting selective transfer.
- Waypoint-OPD: Transferring RL-Induced Policy Shifts via Gated On-Policy Distillationraises · addresses
Cross-tokenizer supervision reliability1 paper raises it · 1 addresses it
Vocabulary mismatch and alignment coverage issues reduce the effectiveness of token-level supervision in on-policy distillation.
- Rethinking Cross-Tokenizer On-Policy Distillation: From Alignment Coverage to Supervision Reliabilityraises · addresses
Collapse in iterative agent self-distillation1 paper raises it · 1 addresses it
Successive distillation cycles degrade deployment performance and privileged-information-conditioned behavior, hindering recursive self-improvement.
- ReSAIL: Mitigating Collapse in Iterative Agent Self-Distillationraises · addresses

Named by an AI from these papers' abstracts.