Papers

Field

Top tools here

Open problems in Teacher policy distillation

Teacher recoverability on incorrect student rollouts1 paper raises it · 1 addresses it
Teachers often follow student errors instead of correcting them, reducing the quality of on-policy distillation supervision.
- Recovering Reliable Supervision via Critique-Guided On-Policy Distillationraises · addresses
Gradient conflicts in multi-teacher on-policy distillation1 paper raises it · 2 address it
Divergent teacher supervision can cause parameter updates to interfere, degrading student capabilities during joint optimization.
- Beyond Domain-Routed Supervision: Cross-Teacher Guarding for Multi-Teacher On-Policy Distillationraises · addresses
- Latent-MOPD: Latent Multi-Teacher On-Policy Distillationaddresses
Tokenization mismatch in cross-tokenizer distillation1 paper raises it · 2 address it
Different tokenizers between teacher and student complicate alignment, making supervision unreliable or incomplete.
- Rethinking Cross-Tokenizer On-Policy Distillation: From Alignment Coverage to Supervision Reliabilityraises · addresses
- ActionOPD: Distilling Decisions, Not Tokensaddresses
Policy shift opacity in teacher distillation1 paper raises it · 1 addresses it
Endpoint policy shifts do not reveal whether local action preferences evolved consistently during teacher training, limiting selective transfer.
- Waypoint-OPD: Transferring RL-Induced Policy Shifts via Gated On-Policy Distillationraises · addresses
Collapse in iterative agent self-distillation1 paper raises it · 1 addresses it
Successive distillation cycles can degrade deployment performance and privileged-information-conditioned behavior, hindering recursive self-improvement.
- ReSAIL: Mitigating Collapse in Iterative Agent Self-Distillationraises · addresses

Named by an AI from these papers' abstracts.

Open problems

Recently funded

Topics in this field

teacher policy distillation · policy distillation opd · teacher supervision318 papers42 this week
ai tutors · ai tutor · tutoring55 papers7 this week

Topics in this field

Active contributors · 90 days

Active contributors

Activity · 12 weeks

Coming with more sources: repos

Activity

Recent papers

All in Research →

arXiv cs.LGSep 28
TIDE: Teacher-Student Transition via Informative Distillation and Exploration for Agentic RL
Yibin Huang, Xinming Xu, Conghui Zhu
Effective multi-turn agents require interaction strategies that coordinate information gathering, actions, and feedback over long horizons. GRPO is a reinforcement learning algorithm used to train these agents, but sparse trajectory-level rewards limit early exploration in small models. Recent methods augment RL with on-policy distillation (OPD) from a stronger teacher. However, a fixed mixture assumes that teacher guidance and reward optimization should retain a constant relative role throughout training and across interaction turns. This assumption can fail at two scales. Globally, as training progresses, maintaining strong distillation pressure can constrain the model from moving beyond the teacher's capabilities. Locally, teacher--student disagreement identifies where the student departs from the teacher, but cannot tell whether that departure is exploration supported by better outcomes or low-quality policy drift. Our methodological insight is that teacher guidance and reward optimization should be dynamically rebalanced over training and jointly allocated across turns. We instantiate this insight in \tide. Globally, \tide uses the measured disagreement trend as a practical…
PDF arXiv
arXiv cs.LGSep 28
When Sparse Reward Meets Dense Distillation: Training Dynamics of On-Policy Distillation
Xinke Jiang, Tao Feng, Zhibang Yang, Zhixin Zhang, Weixuan Xu +2
Reinforcement learning with verifiable rewards provides a sparse post-training signal: a single binary outcome evaluates the entire rollout, and every token receives the same sequence-level advantage regardless of its individual contribution. To complement this sparse supervision, a growing family of methods adds a scalar-weighted teacher KL term to the policy-gradient objective, providing dense token-level guidance that may be unreliable at some positions. Despite the benefits of combining these signals, their interaction during optimization can destabilize joint training. To understand how this instability develops, we study the learning dynamics of hybrid reward--distillation training through a neural tangent kernel (NTK) analysis. We introduce the cross-signal NTK $K_{DR}(n)$, a token-level statistic that measures the alignment between reward and distillation gradients at position n. Through this analysis, we identify two failure modes: 1 Magnitude drowning, where the reward gradient exceeds the distillation gradient by orders of magnitude, so that even weak directional conflict can cause the distillation loss to rise despite its explicit inclusion in the training objective;…
PDF arXiv
arXiv cs.AISep 28
Can We Trust the Teacher? Decoupled Credit Direction-Magnitude for Self-Distillation
Yugu Li, Zehong Cao, Peizhen Li, Yang Zhang, Siyi Hu +1
RLVR provides reliable trajectory-level credit, while OPSD offers dense supervision for token-level credit. This exposes a fundamental coupling when updating step-level credit direction and magnitude with teacher supervision, preventing steps from receiving reliable credit directions and contribution magnitudes, while making both vulnerable to teacher judgment errors and preference variance, as supported by our theoretical analysis. To separate credit direction from its contribution magnitude, we introduce \textit{Decoupled Credit Self-Distillation (DCSD)}, which theoretically decouples credit direction and magnitude into two reliable signals and uses them to calibrate privileged teacher supervision. Specifically, we design belief-margin probing to determine credit direction and marginal information gain to quantify credit magnitude, enabling step-to-token credit assignment for policy optimization. Across 11 benchmarks, DCSD achieves the best overall scores against GRPO, OPSD, RLSD, and RLCSD. Compared with base models, DCSD improves the overall score by 8.45 points on mathematical reasoning and 7.01 points on multimodal reasoning, while correcting the credit direction for 6\% of…
PDF arXiv
arXiv cs.LGSep 28
DivOPD: Spread Wide, Look Close for Asynchronous On-Policy Distillation of Multi-turn Agents
Hanyang Wang, Zeyuan Liu, Zhengyu Chen, Jingqing Ruan, Chaoxu Pang +5
On-policy distillation (OPD) trains student agents through teacher supervision on their own interactions with an environment. However, in asynchronous multi-turn training, arrival-order batching can allow a few early or long rollouts to dominate learner updates while other valid rollouts become stale before being used, wasting already-generated experience. To address this problem, we introduce DivOPD, a simple learner-side batch-selection method that spreads a fixed turn budget across more rollouts and, within each rollout, prioritizes turns with larger cumulative teacher-student disagreement. Turns without usable teacher feedback are excluded. The per-turn loss and optimizer remain fixed; selection only changes which student-visited turns receive training weight. For no-progress rollouts, an optional extension briefly hands control to the teacher before returning it to the student. Across six teacher-student settings on the simulated ALFWorld, ScienceWorld, and WebShop benchmarks, with 1.5B-7B students, DivOPD raises cross-setting mean peak success rate from 77.4 to 84.4 and mean success over the last five evaluations from 71.5 to 78.6. It reaches all reported setting-specific…
PDF arXiv
arXiv cs.LGSep 28
No Pain, More Gain: Iterative Merging for Effective Multi-Teacher On-Policy Distillation
Seonghyeon Kim, Chaeyun Jang, Noah Lee, Boseop Kim, Juho Lee
Multi-teacher on-policy distillation (MOPD) combines independently developed domain teachers into a single student by distilling their predictions on student-generated samples. We study a setting where teachers share a reference model but undergo different post-training procedures, and find that MOPD can struggle to recover some teacher capabilities. Because distillation occurs on student-generated prefixes, the student initialization can strongly affect subsequent recovery. However, initial benchmark performance is not a reliable predictor of a good MOPD initialization. For example, merge initialization can start below SFT warm-up yet finish higher after MOPD. We further find that effective merging depends on both the relative teacher contributions and the overall merge scale, with some strong configurations lying outside the simplex of convex parameter averaging. Thus, selecting a good merge initialization requires evaluating not only its immediate performance but also the learning it enables under MOPD, making one-shot coefficient search difficult. We propose Iterative Merging for MOPD (IM-MOPD), which starts from a uniform merge and progressively adds task-vector increments…
PDF arXiv
arXiv cs.CLSep 28
Beyond Token Alignment: Event Completion for Cross-Tokenizer On-Policy Distillation
Jiacheng Liu, Jingwei Song, Qituan Zhang, Siheng Chen, Linfeng Zhang
On-policy distillation (OPD) transfers knowledge between language models through teacher supervision on student-generated trajectories. With different tokenizers, a single teacher token may require multiple student tokens to generate, creating intermediate states where the event is entered but not yet completed. Existing cross-tokenizer methods align tokens or text spans to construct comparable prediction targets. We study a complementary problem after partial generation: once the student produces a prefix of a teacher token, multiple next tokens may complete the same remaining bytes, but the teacher only specifies the required completion rather than how probability should be divided among these valid continuations. We introduce Event-Set Completion Distillation (ESCD), which complements cross-tokenizer probability alignment with completion-set supervision. ESCD aggregates prefix-related teacher events and supervises the total probability of byte-compatible one-step student completions, avoiding tokenizer-dependent probability splits among individual tokens. The method reuses student trajectories and predictions, requiring neither additional rollouts nor changes to the student…
PDF arXiv

Papers

Topics in this field

teacher policy distillation · policy distillation opd · teacher supervision318 papers42 this week
ai tutors · ai tutor · tutoring55 papers7 this week

Recent papers

All in Research →

arXiv cs.LGSep 28
TIDE: Teacher-Student Transition via Informative Distillation and Exploration for Agentic RL
Yibin Huang, Xinming Xu, Conghui Zhu
Effective multi-turn agents require interaction strategies that coordinate information gathering, actions, and feedback over long horizons. GRPO is a reinforcement learning algorithm used to train these agents, but sparse trajectory-level rewards limit early exploration in small models. Recent methods augment RL with on-policy distillation (OPD) from a stronger teacher. However, a fixed mixture assumes that teacher guidance and reward optimization should retain a constant relative role throughout training and across interaction turns. This assumption can fail at two scales. Globally, as training progresses, maintaining strong distillation pressure can constrain the model from moving beyond the teacher's capabilities. Locally, teacher--student disagreement identifies where the student departs from the teacher, but cannot tell whether that departure is exploration supported by better outcomes or low-quality policy drift. Our methodological insight is that teacher guidance and reward optimization should be dynamically rebalanced over training and jointly allocated across turns. We instantiate this insight in \tide. Globally, \tide uses the measured disagreement trend as a practical…
PDF arXiv
arXiv cs.LGSep 28
When Sparse Reward Meets Dense Distillation: Training Dynamics of On-Policy Distillation
Xinke Jiang, Tao Feng, Zhibang Yang, Zhixin Zhang, Weixuan Xu +2
Reinforcement learning with verifiable rewards provides a sparse post-training signal: a single binary outcome evaluates the entire rollout, and every token receives the same sequence-level advantage regardless of its individual contribution. To complement this sparse supervision, a growing family of methods adds a scalar-weighted teacher KL term to the policy-gradient objective, providing dense token-level guidance that may be unreliable at some positions. Despite the benefits of combining these signals, their interaction during optimization can destabilize joint training. To understand how this instability develops, we study the learning dynamics of hybrid reward--distillation training through a neural tangent kernel (NTK) analysis. We introduce the cross-signal NTK $K_{DR}(n)$, a token-level statistic that measures the alignment between reward and distillation gradients at position n. Through this analysis, we identify two failure modes: 1 Magnitude drowning, where the reward gradient exceeds the distillation gradient by orders of magnitude, so that even weak directional conflict can cause the distillation loss to rise despite its explicit inclusion in the training objective;…
PDF arXiv
arXiv cs.AISep 28
Can We Trust the Teacher? Decoupled Credit Direction-Magnitude for Self-Distillation
Yugu Li, Zehong Cao, Peizhen Li, Yang Zhang, Siyi Hu +1
RLVR provides reliable trajectory-level credit, while OPSD offers dense supervision for token-level credit. This exposes a fundamental coupling when updating step-level credit direction and magnitude with teacher supervision, preventing steps from receiving reliable credit directions and contribution magnitudes, while making both vulnerable to teacher judgment errors and preference variance, as supported by our theoretical analysis. To separate credit direction from its contribution magnitude, we introduce \textit{Decoupled Credit Self-Distillation (DCSD)}, which theoretically decouples credit direction and magnitude into two reliable signals and uses them to calibrate privileged teacher supervision. Specifically, we design belief-margin probing to determine credit direction and marginal information gain to quantify credit magnitude, enabling step-to-token credit assignment for policy optimization. Across 11 benchmarks, DCSD achieves the best overall scores against GRPO, OPSD, RLSD, and RLCSD. Compared with base models, DCSD improves the overall score by 8.45 points on mathematical reasoning and 7.01 points on multimodal reasoning, while correcting the credit direction for 6\% of…
PDF arXiv
arXiv cs.LGSep 28
DivOPD: Spread Wide, Look Close for Asynchronous On-Policy Distillation of Multi-turn Agents
Hanyang Wang, Zeyuan Liu, Zhengyu Chen, Jingqing Ruan, Chaoxu Pang +5
On-policy distillation (OPD) trains student agents through teacher supervision on their own interactions with an environment. However, in asynchronous multi-turn training, arrival-order batching can allow a few early or long rollouts to dominate learner updates while other valid rollouts become stale before being used, wasting already-generated experience. To address this problem, we introduce DivOPD, a simple learner-side batch-selection method that spreads a fixed turn budget across more rollouts and, within each rollout, prioritizes turns with larger cumulative teacher-student disagreement. Turns without usable teacher feedback are excluded. The per-turn loss and optimizer remain fixed; selection only changes which student-visited turns receive training weight. For no-progress rollouts, an optional extension briefly hands control to the teacher before returning it to the student. Across six teacher-student settings on the simulated ALFWorld, ScienceWorld, and WebShop benchmarks, with 1.5B-7B students, DivOPD raises cross-setting mean peak success rate from 77.4 to 84.4 and mean success over the last five evaluations from 71.5 to 78.6. It reaches all reported setting-specific…
PDF arXiv
arXiv cs.LGSep 28
No Pain, More Gain: Iterative Merging for Effective Multi-Teacher On-Policy Distillation
Seonghyeon Kim, Chaeyun Jang, Noah Lee, Boseop Kim, Juho Lee
Multi-teacher on-policy distillation (MOPD) combines independently developed domain teachers into a single student by distilling their predictions on student-generated samples. We study a setting where teachers share a reference model but undergo different post-training procedures, and find that MOPD can struggle to recover some teacher capabilities. Because distillation occurs on student-generated prefixes, the student initialization can strongly affect subsequent recovery. However, initial benchmark performance is not a reliable predictor of a good MOPD initialization. For example, merge initialization can start below SFT warm-up yet finish higher after MOPD. We further find that effective merging depends on both the relative teacher contributions and the overall merge scale, with some strong configurations lying outside the simplex of convex parameter averaging. Thus, selecting a good merge initialization requires evaluating not only its immediate performance but also the learning it enables under MOPD, making one-shot coefficient search difficult. We propose Iterative Merging for MOPD (IM-MOPD), which starts from a uniform merge and progressively adds task-vector increments…
PDF arXiv
arXiv cs.CLSep 28
Beyond Token Alignment: Event Completion for Cross-Tokenizer On-Policy Distillation
Jiacheng Liu, Jingwei Song, Qituan Zhang, Siheng Chen, Linfeng Zhang
On-policy distillation (OPD) transfers knowledge between language models through teacher supervision on student-generated trajectories. With different tokenizers, a single teacher token may require multiple student tokens to generate, creating intermediate states where the event is entered but not yet completed. Existing cross-tokenizer methods align tokens or text spans to construct comparable prediction targets. We study a complementary problem after partial generation: once the student produces a prefix of a teacher token, multiple next tokens may complete the same remaining bytes, but the teacher only specifies the required completion rather than how probability should be divided among these valid continuations. We introduce Event-Set Completion Distillation (ESCD), which complements cross-tokenizer probability alignment with completion-set supervision. ESCD aggregates prefix-related teacher events and supervises the total probability of byte-compatible one-step student completions, avoiding tokenizer-dependent probability splits among individual tokens. The method reuses student trajectories and predictions, requiring neither additional rollouts nor changes to the student…
PDF arXiv

Active contributors · 90 days

Activity · 12 weeks

Coming with more sources: repos

Fields · AI · updated 38m ago

Teacher policy distillation

Found by the topic model: teacher policy distillation, ai tutors.

49papers this week−36.4% vs last week

All papers · 373 With code · 20

Newest Most relevant

356 papers this month

Open problems in Teacher policy distillation

Teacher recoverability on incorrect student rollouts1 paper raises it · 1 addresses it
Teachers often follow student errors instead of correcting them, reducing the quality of on-policy distillation supervision.
- Recovering Reliable Supervision via Critique-Guided On-Policy Distillationraises · addresses
Gradient conflicts in multi-teacher on-policy distillation1 paper raises it · 2 address it
Divergent teacher supervision can cause parameter updates to interfere, degrading student capabilities during joint optimization.
- Beyond Domain-Routed Supervision: Cross-Teacher Guarding for Multi-Teacher On-Policy Distillationraises · addresses
- Latent-MOPD: Latent Multi-Teacher On-Policy Distillationaddresses
Tokenization mismatch in cross-tokenizer distillation1 paper raises it · 2 address it
Different tokenizers between teacher and student complicate alignment, making supervision unreliable or incomplete.
- Rethinking Cross-Tokenizer On-Policy Distillation: From Alignment Coverage to Supervision Reliabilityraises · addresses
- ActionOPD: Distilling Decisions, Not Tokensaddresses
Policy shift opacity in teacher distillation1 paper raises it · 1 addresses it
Endpoint policy shifts do not reveal whether local action preferences evolved consistently during teacher training, limiting selective transfer.
- Waypoint-OPD: Transferring RL-Induced Policy Shifts via Gated On-Policy Distillationraises · addresses
Collapse in iterative agent self-distillation1 paper raises it · 1 addresses it
Successive distillation cycles can degrade deployment performance and privileged-information-conditioned behavior, hindering recursive self-improvement.
- ReSAIL: Mitigating Collapse in Iterative Agent Self-Distillationraises · addresses

Named by an AI from these papers' abstracts.