Papers

Field

Top tools here

Open problems in Teacher policy distillation

Teacher recoverability on incorrect student rollouts1 paper raises it · 1 addresses it
Teachers often follow student errors instead of correcting them, reducing the quality of on-policy distillation supervision.
- Recovering Reliable Supervision via Critique-Guided On-Policy Distillationraises · addresses
Gradient conflicts in multi-teacher on-policy distillation1 paper raises it · 2 address it
Divergent teacher supervision can cause parameter updates to interfere, degrading student capabilities during joint optimization.
- Beyond Domain-Routed Supervision: Cross-Teacher Guarding for Multi-Teacher On-Policy Distillationraises · addresses
- Latent-MOPD: Latent Multi-Teacher On-Policy Distillationaddresses
Tokenization mismatch in cross-tokenizer distillation1 paper raises it · 2 address it
Different tokenizers between teacher and student complicate alignment, making supervision unreliable or incomplete.
- Rethinking Cross-Tokenizer On-Policy Distillation: From Alignment Coverage to Supervision Reliabilityraises · addresses
- ActionOPD: Distilling Decisions, Not Tokensaddresses
Policy shift opacity in teacher distillation1 paper raises it · 1 addresses it
Endpoint policy shifts do not reveal whether local action preferences evolved consistently during teacher training, limiting selective transfer.
- Waypoint-OPD: Transferring RL-Induced Policy Shifts via Gated On-Policy Distillationraises · addresses
Collapse in iterative agent self-distillation1 paper raises it · 1 addresses it
Successive distillation cycles can degrade deployment performance and privileged-information-conditioned behavior, hindering recursive self-improvement.
- ReSAIL: Mitigating Collapse in Iterative Agent Self-Distillationraises · addresses

Named by an AI from these papers' abstracts.

Open problems

Recently funded

Topics in this field

teacher policy distillation · policy distillation opd · teacher supervision318 papers43 this week
ai tutors · ai tutor · tutoring55 papers7 this week

Topics in this field

Active contributors · 90 days

Active contributors

Activity · 12 weeks

Coming with more sources: repos

Activity

Recent papers

All in Research →

arXiv cs.LGSep 28
d-OPD: Future-Aware On-Policy Distillation for Block Diffusion Language Models
Ruitao Liu, Qinghao Hu, Song Han
Large language models (LLMs) typically generate text autoregressively (AR), predicting one token at a time. Block diffusion language models (dLLMs) instead generate blocks sequentially while denoising multiple tokens in parallel within each block, offering a promising way to accelerate generation. Rather than training such models from scratch, recent work adapts strong pretrained AR models into block dLLMs through distillation. On-policy distillation (OPD) has been widely used for LLM training because it supervises the student on states generated by its current policy, rather than only on fixed offline trajectories. By training on the states the student actually visits, it reduces the mismatch between training and generation and can provide more relevant supervision as the student evolves. Recent work has extended this idea to AR-to-block-diffusion conversion. However, this setting introduces a fundamental mismatch in supervision: the block-diffusion student and the causal AR teacher condition on different information at the same training state. The student predicts from the entire partially denoised block, including visible future context, whereas the standard AR teacher target…
PDF arXiv
arXiv cs.LGSep 28
Beyond Teacher Assignment: Domain-Normalized Multi-Teacher On-Policy Distillation
Xin Li, Hao Jiang, Xin Gao, Annan Wang, Yuchen Xie +4
Reinforcement learning can turn one language model into several specialists, each excellent at a single skill such as mathematics, coding or following instructions, but users need one model with all of these skills. Multi-teacher on-policy distillation (MOPD) merges them by letting the specialists teach one student: the student answers each prompt, and the specialist for that prompt's domain gives feedback on every token. This routing decides which specialist teaches, but not how strongly its feedback moves the shared student. In Qwen3.5 models at three sizes, we find that MOPD's student does not beat one taught by the best single specialist and gains little of the mathematics specialist's advantage. The feedback is unbalanced: instruction-following feedback is several times more spread out than mathematics feedback and dominates the student's updates. We propose Domain-Normalized MOPD (DN-MOPD), which keeps the routing and rescales each domain's feedback by its measured spread. On six public benchmarks, DN-MOPD improves the average score over MOPD at every size, across three random seeds and under two answer-length limits, and recovers most of the lost mathematics gain. Controls…
PDF arXiv
arXiv cs.LGSep 28
Teacher-Student Gaps Are Not Enough: Outcome-Guided On-Policy Distillation for Multi-Turn Autonomous Agents
Tong Zhang, Zhou Liu, Yihao Liu, Jiahua Bao, Xuchen Li +6
On-policy distillation (OPD) trains a student on its own trajectories with dense teacher supervision. Recent work on OPD for multi-turn autonomous agents often treats large teacher-student token-level distributional gaps as promising intervention points, linking larger gaps to a greater need for correction. Yet, our empirical analysis reveals a supervision-benefit mismatch: large gaps can be benign, while small gaps can be outcome-critical. Teacher-student gaps capture differences at the current turn, whereas the benefit of teacher guidance depends on how the current student interacts with the environment afterward. The student may still succeed despite choosing an action that differs from the teacher's, while a teacher-preferred action may lead to a state from which the student cannot complete the task. Local gaps alone are therefore not enough to determine whether teacher guidance benefits the current student. Effective supervision should instead emphasize guidance that the current student can translate into better final task outcomes. Accordingly, we propose Outcome-Guided On-Policy Distillation (OG-OPD), which applies trajectory-relative weighting to teacher supervision and…
PDF arXiv
arXiv cs.CLSep 28
Rubric-Aware On-Policy Self-Distillation for LLM Personalization
Yilun Qiu, Xiaoyan Zhao, Chengbing Wang, Cilin Yan, Rui Zu +4
LLM personalization aims to generate responses aligned with individual users' preferences and needs. User-specific rubrics make these expectations explicit, providing direct supervision on what a satisfactory answer should cover. Existing rubric-guided approaches, however, exploit such guidance only at a coarse granularity, either by using rubrics to supervise the prediction of relevant aspects for subsequent generation or by reducing aspect coverage to a single response-level reward for reinforcement learning. This leaves a gap between specifying what a personalized answer should contain and teaching the model how to generate it. To bridge this gap, we propose GRASP, a rubric-aware on-policy self-distillation framework for LLM personalization that turns user-specific rubric aspects into fine-grained, token-level supervision. Specifically, GRASP pairs a rubric-free student with a rubric-informed teacher that additionally receives the target user-specific rubrics. By aligning their next-token distributions along on-policy trajectories generated by the student, GRASP transfers the teacher's rubric-conditioned guidance into the student, translating user-specific semantic…
PDF arXiv
arXiv cs.LGSep 28
On-Policy or Off-Policy Learning? A Systematic Study of Distillation Dynamics
Julianna Piskorz, Antonin Berthon, Mihaela van der Schaar
On-policy learning has been argued to reduce catastrophic forgetting, produce sparser parameter updates, and improve generalisation. However, existing comparisons between supervised fine-tuning and reinforcement learning vary many factors simultaneously, making the contribution of rollout policy difficult to isolate. We study the effect of rollout policy in a controlled strong-to-weak distillation setting, by independently varying rollout policy, token-level KL direction, and learning rate across the Llama3 and Qwen2.5 model families and reasoning tasks spanning scientific, medical, and arithmetic domains. Our analysis reveals a nuanced picture of distillation dynamics in which rollout policy does not necessarily play a central role. Instead, token-level KL direction more clearly shapes task performance and output coverage, while learning rate governs forgetting and update sparsity. Analysis of KL gradients and experiments along a continuous student-teacher rollout-policy spectrum explain this pattern: forward KL is remarkably robust to rollout policy, with its performance stable and strong despite changes to the rollout policy, whereas reverse KL is substantially more sensitive…
PDF arXiv
arXiv cs.CLSep 28
Understanding On-Policy Distillation: A Mechanistic Interpretability Perspective via Sparse Crosscoders
Zichao Yu, Qianshuo Ye, Xu Wang, Difan Zou
On-policy distillation (OPD) is a widely adopted post-training technique for LLM reasoning. It is commonly believed to transfer knowledge from a stronger teacher, yet what OPD actually distills into the student's internal representations remains unclear. We study this question with sparse crosscoders, which learn one feature dictionary shared by the student before and after OPD and the teacher. Standard crosscoder analyses, however, identify model-specific features but cannot tell how a model's use of its features changes, since all models are encoded into one set of feature activations. We therefore propose the swap readout, which reads each student checkpoint's feature activations on its own, measuring how training changes the student's use of each feature, even for checkpoints unseen by the crosscoder. Across three OPD settings, we find that OPD neither creates features nor passes on the teacher's own, and leaves the firing rates of over 98% of the student's frequently used features within 20%. We further examine the SFT warm-up on the teacher's rollouts that commonly precedes OPD and makes it more effective. Rather than adding features, the warm-up reweights the shared ones…
PDF arXiv

Papers

Topics in this field

teacher policy distillation · policy distillation opd · teacher supervision318 papers43 this week
ai tutors · ai tutor · tutoring55 papers7 this week

Recent papers

All in Research →

arXiv cs.LGSep 28
d-OPD: Future-Aware On-Policy Distillation for Block Diffusion Language Models
Ruitao Liu, Qinghao Hu, Song Han
Large language models (LLMs) typically generate text autoregressively (AR), predicting one token at a time. Block diffusion language models (dLLMs) instead generate blocks sequentially while denoising multiple tokens in parallel within each block, offering a promising way to accelerate generation. Rather than training such models from scratch, recent work adapts strong pretrained AR models into block dLLMs through distillation. On-policy distillation (OPD) has been widely used for LLM training because it supervises the student on states generated by its current policy, rather than only on fixed offline trajectories. By training on the states the student actually visits, it reduces the mismatch between training and generation and can provide more relevant supervision as the student evolves. Recent work has extended this idea to AR-to-block-diffusion conversion. However, this setting introduces a fundamental mismatch in supervision: the block-diffusion student and the causal AR teacher condition on different information at the same training state. The student predicts from the entire partially denoised block, including visible future context, whereas the standard AR teacher target…
PDF arXiv
arXiv cs.LGSep 28
Beyond Teacher Assignment: Domain-Normalized Multi-Teacher On-Policy Distillation
Xin Li, Hao Jiang, Xin Gao, Annan Wang, Yuchen Xie +4
Reinforcement learning can turn one language model into several specialists, each excellent at a single skill such as mathematics, coding or following instructions, but users need one model with all of these skills. Multi-teacher on-policy distillation (MOPD) merges them by letting the specialists teach one student: the student answers each prompt, and the specialist for that prompt's domain gives feedback on every token. This routing decides which specialist teaches, but not how strongly its feedback moves the shared student. In Qwen3.5 models at three sizes, we find that MOPD's student does not beat one taught by the best single specialist and gains little of the mathematics specialist's advantage. The feedback is unbalanced: instruction-following feedback is several times more spread out than mathematics feedback and dominates the student's updates. We propose Domain-Normalized MOPD (DN-MOPD), which keeps the routing and rescales each domain's feedback by its measured spread. On six public benchmarks, DN-MOPD improves the average score over MOPD at every size, across three random seeds and under two answer-length limits, and recovers most of the lost mathematics gain. Controls…
PDF arXiv
arXiv cs.LGSep 28
Teacher-Student Gaps Are Not Enough: Outcome-Guided On-Policy Distillation for Multi-Turn Autonomous Agents
Tong Zhang, Zhou Liu, Yihao Liu, Jiahua Bao, Xuchen Li +6
On-policy distillation (OPD) trains a student on its own trajectories with dense teacher supervision. Recent work on OPD for multi-turn autonomous agents often treats large teacher-student token-level distributional gaps as promising intervention points, linking larger gaps to a greater need for correction. Yet, our empirical analysis reveals a supervision-benefit mismatch: large gaps can be benign, while small gaps can be outcome-critical. Teacher-student gaps capture differences at the current turn, whereas the benefit of teacher guidance depends on how the current student interacts with the environment afterward. The student may still succeed despite choosing an action that differs from the teacher's, while a teacher-preferred action may lead to a state from which the student cannot complete the task. Local gaps alone are therefore not enough to determine whether teacher guidance benefits the current student. Effective supervision should instead emphasize guidance that the current student can translate into better final task outcomes. Accordingly, we propose Outcome-Guided On-Policy Distillation (OG-OPD), which applies trajectory-relative weighting to teacher supervision and…
PDF arXiv
arXiv cs.CLSep 28
Rubric-Aware On-Policy Self-Distillation for LLM Personalization
Yilun Qiu, Xiaoyan Zhao, Chengbing Wang, Cilin Yan, Rui Zu +4
LLM personalization aims to generate responses aligned with individual users' preferences and needs. User-specific rubrics make these expectations explicit, providing direct supervision on what a satisfactory answer should cover. Existing rubric-guided approaches, however, exploit such guidance only at a coarse granularity, either by using rubrics to supervise the prediction of relevant aspects for subsequent generation or by reducing aspect coverage to a single response-level reward for reinforcement learning. This leaves a gap between specifying what a personalized answer should contain and teaching the model how to generate it. To bridge this gap, we propose GRASP, a rubric-aware on-policy self-distillation framework for LLM personalization that turns user-specific rubric aspects into fine-grained, token-level supervision. Specifically, GRASP pairs a rubric-free student with a rubric-informed teacher that additionally receives the target user-specific rubrics. By aligning their next-token distributions along on-policy trajectories generated by the student, GRASP transfers the teacher's rubric-conditioned guidance into the student, translating user-specific semantic…
PDF arXiv
arXiv cs.LGSep 28
On-Policy or Off-Policy Learning? A Systematic Study of Distillation Dynamics
Julianna Piskorz, Antonin Berthon, Mihaela van der Schaar
On-policy learning has been argued to reduce catastrophic forgetting, produce sparser parameter updates, and improve generalisation. However, existing comparisons between supervised fine-tuning and reinforcement learning vary many factors simultaneously, making the contribution of rollout policy difficult to isolate. We study the effect of rollout policy in a controlled strong-to-weak distillation setting, by independently varying rollout policy, token-level KL direction, and learning rate across the Llama3 and Qwen2.5 model families and reasoning tasks spanning scientific, medical, and arithmetic domains. Our analysis reveals a nuanced picture of distillation dynamics in which rollout policy does not necessarily play a central role. Instead, token-level KL direction more clearly shapes task performance and output coverage, while learning rate governs forgetting and update sparsity. Analysis of KL gradients and experiments along a continuous student-teacher rollout-policy spectrum explain this pattern: forward KL is remarkably robust to rollout policy, with its performance stable and strong despite changes to the rollout policy, whereas reverse KL is substantially more sensitive…
PDF arXiv
arXiv cs.CLSep 28
Understanding On-Policy Distillation: A Mechanistic Interpretability Perspective via Sparse Crosscoders
Zichao Yu, Qianshuo Ye, Xu Wang, Difan Zou
On-policy distillation (OPD) is a widely adopted post-training technique for LLM reasoning. It is commonly believed to transfer knowledge from a stronger teacher, yet what OPD actually distills into the student's internal representations remains unclear. We study this question with sparse crosscoders, which learn one feature dictionary shared by the student before and after OPD and the teacher. Standard crosscoder analyses, however, identify model-specific features but cannot tell how a model's use of its features changes, since all models are encoded into one set of feature activations. We therefore propose the swap readout, which reads each student checkpoint's feature activations on its own, measuring how training changes the student's use of each feature, even for checkpoints unseen by the crosscoder. Across three OPD settings, we find that OPD neither creates features nor passes on the teacher's own, and leaves the firing rates of over 98% of the student's frequently used features within 20%. We further examine the SFT warm-up on the teacher's rollouts that commonly precedes OPD and makes it more effective. Rather than adding features, the warm-up reweights the shared ones…
PDF arXiv

Active contributors · 90 days

Activity · 12 weeks

Coming with more sources: repos

Fields · AI · updated 22m ago

Teacher policy distillation

Found by the topic model: teacher policy distillation, ai tutors.

50papers this week−35.1% vs last week

All papers · 373 With code · 20

Newest Most relevant

356 papers this month

Open problems in Teacher policy distillation

Teacher recoverability on incorrect student rollouts1 paper raises it · 1 addresses it
Teachers often follow student errors instead of correcting them, reducing the quality of on-policy distillation supervision.
- Recovering Reliable Supervision via Critique-Guided On-Policy Distillationraises · addresses
Gradient conflicts in multi-teacher on-policy distillation1 paper raises it · 2 address it
Divergent teacher supervision can cause parameter updates to interfere, degrading student capabilities during joint optimization.
- Beyond Domain-Routed Supervision: Cross-Teacher Guarding for Multi-Teacher On-Policy Distillationraises · addresses
- Latent-MOPD: Latent Multi-Teacher On-Policy Distillationaddresses
Tokenization mismatch in cross-tokenizer distillation1 paper raises it · 2 address it
Different tokenizers between teacher and student complicate alignment, making supervision unreliable or incomplete.
- Rethinking Cross-Tokenizer On-Policy Distillation: From Alignment Coverage to Supervision Reliabilityraises · addresses
- ActionOPD: Distilling Decisions, Not Tokensaddresses
Policy shift opacity in teacher distillation1 paper raises it · 1 addresses it
Endpoint policy shifts do not reveal whether local action preferences evolved consistently during teacher training, limiting selective transfer.
- Waypoint-OPD: Transferring RL-Induced Policy Shifts via Gated On-Policy Distillationraises · addresses
Collapse in iterative agent self-distillation1 paper raises it · 1 addresses it
Successive distillation cycles can degrade deployment performance and privileged-information-conditioned behavior, hindering recursive self-improvement.
- ReSAIL: Mitigating Collapse in Iterative Agent Self-Distillationraises · addresses

Named by an AI from these papers' abstracts.