Papers

Field

Top tools here

Open problems in Teacher policy distillation

Teacher recoverability on incorrect student rollouts1 paper raises it · 1 addresses it
Teachers often follow student errors instead of correcting them, reducing the quality of on-policy distillation supervision.
- Recovering Reliable Supervision via Critique-Guided On-Policy Distillationraises · addresses
Gradient conflicts in multi-teacher on-policy distillation1 paper raises it · 2 address it
Divergent teacher supervision can cause parameter updates to interfere, degrading student capabilities during joint optimization.
- Beyond Domain-Routed Supervision: Cross-Teacher Guarding for Multi-Teacher On-Policy Distillationraises · addresses
- Latent-MOPD: Latent Multi-Teacher On-Policy Distillationaddresses
Tokenization mismatch in cross-tokenizer distillation1 paper raises it · 2 address it
Different tokenizers between teacher and student complicate alignment, making supervision unreliable or incomplete.
- Rethinking Cross-Tokenizer On-Policy Distillation: From Alignment Coverage to Supervision Reliabilityraises · addresses
- ActionOPD: Distilling Decisions, Not Tokensaddresses
Policy shift opacity in teacher distillation1 paper raises it · 1 addresses it
Endpoint policy shifts do not reveal whether local action preferences evolved consistently during teacher training, limiting selective transfer.
- Waypoint-OPD: Transferring RL-Induced Policy Shifts via Gated On-Policy Distillationraises · addresses
Collapse in iterative agent self-distillation1 paper raises it · 1 addresses it
Successive distillation cycles can degrade deployment performance and privileged-information-conditioned behavior, hindering recursive self-improvement.
- ReSAIL: Mitigating Collapse in Iterative Agent Self-Distillationraises · addresses

Named by an AI from these papers' abstracts.

Open problems

Recently funded

Topics in this field

teacher policy distillation · policy distillation opd · teacher supervision318 papers48 this week
ai tutors · ai tutor · tutoring55 papers7 this week

Topics in this field

Active contributors · 90 days

Active contributors

Activity · 12 weeks

Coming with more sources: repos

Activity

Recent papers

All in Research →

arXiv cs.LGOct 14
CARD: Closing the Teacher-Student Loop for Adversarial Robustness Distillation
Zhi Li, Haowei Liu, Hongchen Yang, Xiaoxuan Wang, Song Gao +3
Adversarial robustness distillation typically treats a robust teacher as a fixed knowledge source and transfers robustness to a compact student in a one-way manner. This paradigm terminates teacher-student interaction once the student is distilled, thereby discarding potentially useful feedback from the distilled student. However, exploiting such feedback is nontrivial because indiscriminate weak-to-strong transfer may propagate student errors to the teacher. To address this issue, we propose Cyclic Adversarial Robustness Distillation (CARD), a closed-loop framework that turns the distilled student into a feedback source for refining the robust teacher and subsequently transfers the teacher gains back to the same student. CARD first performs strong-to-weak adversarial distillation and then uses reliability-enhanced student feedback to refine the robust teacher. The refinement combines inverse adversarial student feedback with teacher-student discrepancy maximization, after which the refined teacher is used to continue training the same student under the unchanged base distillation objective. We further establish a sample-level sufficient condition for reliable weak-to-strong…
PDF arXiv
arXiv cs.AIOct 14
Teaching Agents in their Own Style: Blackbox Teacher Distillation for Web Agents
Md Farhan Ishmam, Kenneth Marino
Frontier web agents using closed models have high inference and computational costs. Smaller open models can be an alternative, but to be performant, they require training supervision from a human or teacher model. Token-level teacher distillation is not an option for closed teachers as it needs access to the model's logits. The standard approach is to imitate teacher demonstrations offline, but it trains the student on states it has never visited, and in a style it does not write in. We introduce WebKD, where the teacher is the editor of the student's failures. Given a failed episode, the teacher finds the first point of mistake, and rewrites the student's reasoning traces, harness states, and actions from that point onward. The editing reveals that the teacher's corrections can be more surprising to the student than the student's own text. WebKD instead aims to edit minimally. In the data space, the teacher writes each correction in the student's style. In the objective space, the reasoning is weighted by its contribution to the action. A role-asymmetric KL anchor allows the corrected action to move freely while keeping the reasoning near the student's prior. On the TimeWarp…
PDF arXiv
arXiv cs.AICodeOct 14▲ 57
ActionOPD: Distilling Decisions, Not Tokens
Huining Yuan, Xiangmin Yi, Zelai Xu, Zixu Wang, Xinlei Chen +3
Distilling capable language-model agents into smaller models requires aligning the student's behavior with the teacher's. This raises a fundamental question: at which states and at what granularity should this alignment be performed? Offline imitation provides dense but token-specific supervision and can suffer from state-distribution mismatch. Token-level on-policy distillation (OPD) addresses this mismatch but typically requires teacher token probabilities and compatible tokenization. We argue that semantic actions provide a natural unit of supervision for agent distillation. We introduce ActionOPD, an action-level on-policy distillation method motivated by maximizing expected teacher-action likelihood at student-encountered states. At each such state, ActionOPD samples actions from both the student and a black-box teacher, rewarding student actions according to their semantic agreement with the sampled teacher actions. Through policy optimization, ActionOPD aligns the student's action-level behavior with the teacher's decisions without requiring reproduction of teacher-written reasoning or access to teacher token probabilities. Training Qwen3-4B and Qwen3-8B with two teachers…
PDF arXiv Code
arXiv cs.LGOct 14
Function-Space Temporal Self-Distillation for Policy Optimization
Aryan Das, Arnab Bhattacharya, Vinay Kumar Verma
Reinforcement Learning with Verifiable Rewards (RLVR) improves reasoning in language models, but scalar outcomes provide coarse supervision over response trajectories. Self-distillation offers denser token-level guidance by comparing an unassisted policy with a feedback-informed self-teacher, yet the teacher evolves with the student, creating unstable targets and motivating additional regularization. We introduce Function-Space Policy Optimization (FSPO), a single-policy framework that combines current privileged adaptation with function-space temporal consolidation. FSPO stores sparse historical next-token distributions as fixed temporal targets, providing temporal separation without maintaining a persistent teacher model. It further introduces privileged disagreement correction to explicitly suppress locally overvalued actions, reliability-adaptive grounding that strengthens reference supervision when self-generated evidence repeatedly fails, and quality and coverage-aware temporal memory that retains verified behaviors and improves privileged demonstrations. Across reasoning and tool-use benchmarks, FSPO achieves the highest average accuracy under a 5-hour training budget,…
PDF arXiv
arXiv cs.LGOct 14
Beyond Distribution Matching: Selective Margin Preservation for Supervised Fine-Tuning
Yuxuan Wang, Guangxiang Zhao, Xinyu Tan, Yahui Han, Xinxiang Yin +4
Supervised fine-tuning (SFT) changes logit margins among non-target tokens at teacher-forced prefixes. Matching a local reference distribution keeps those probabilities and also penalizes margin expansions. We ask whether a soft auxiliary can preserve selected local non-target margin relations without freezing that distribution. We introduce \textbf{Selective Rank-Preserving SFT (SRP-SFT)}, a soft margin target: a hinge on reference-scaled lower bounds for pairs from the reference's top-$K$ non-target tokens, applied at fixed teacher-forced prefixes. The hinge stops once the requested bound is met. We derive the KL cost of violating a margin, characterize the feasible region and its conditional utility, and give a finite-step condition for reducing aggregate violations. In one-epoch experiments on Qwen3-4B-Base and UltraFeedback, this target raises MATH-500 and code-generation scores over CE and LP-SFT at all three training seeds. At seed 1234, MATH-500 pass@1 rises from 49.56\% to 54.56\% and the four-metric code mean rises from 65.91\% to 69.50\% relative to LP-SFT. Ranking controls, held-out cache diagnostics, sensitivity sweeps, and six additional backbones show where these…
PDF arXiv
arXiv cs.CLOct 14
HACHIMI-II: From Role-Play to Learner Behavior Modeling for Faithful Student Simulation
Yilin Jiang, Fei Tan, Xuanyu Yin, Jing Leng, Aimin Zhou
Simulated students let researchers test tutoring systems and teaching strategies before real learners are available, but only if the simulation reproduces where a particular learner succeeds and fails. The common approach asks a large language model to role-play a student, but a role does not model what this learner knows or will do next, so the model's own competence fills the gap. Simulated students perform at a level set by the model, differ little from one another, and seldom make the real learners' errors. We model the learner's behavior and recast student simulation as third-person prediction of the learner's next action. HACHIMI-II builds a student model from the learner's history, infers the learner's state for the current item or tutor turn, and compiles it into a behavior harness that constrains the next action. On three item-response datasets, the Wasserstein-1 distance between simulated and real student ability is 2.4--8.4$\times$ smaller than that of the strongest baseline. On two dialogue datasets, HACHIMI-II reproduces real student errors with the same knowledge type, locus, and mechanism 1.8--5.1$\times$ more often than the strongest baseline, and ablations trace…
PDF arXiv

Papers

Topics in this field

teacher policy distillation · policy distillation opd · teacher supervision318 papers48 this week
ai tutors · ai tutor · tutoring55 papers7 this week

Active contributors · 90 days

Recent papers

All in Research →

arXiv cs.LGOct 14
CARD: Closing the Teacher-Student Loop for Adversarial Robustness Distillation
Zhi Li, Haowei Liu, Hongchen Yang, Xiaoxuan Wang, Song Gao +3
Adversarial robustness distillation typically treats a robust teacher as a fixed knowledge source and transfers robustness to a compact student in a one-way manner. This paradigm terminates teacher-student interaction once the student is distilled, thereby discarding potentially useful feedback from the distilled student. However, exploiting such feedback is nontrivial because indiscriminate weak-to-strong transfer may propagate student errors to the teacher. To address this issue, we propose Cyclic Adversarial Robustness Distillation (CARD), a closed-loop framework that turns the distilled student into a feedback source for refining the robust teacher and subsequently transfers the teacher gains back to the same student. CARD first performs strong-to-weak adversarial distillation and then uses reliability-enhanced student feedback to refine the robust teacher. The refinement combines inverse adversarial student feedback with teacher-student discrepancy maximization, after which the refined teacher is used to continue training the same student under the unchanged base distillation objective. We further establish a sample-level sufficient condition for reliable weak-to-strong…
PDF arXiv
arXiv cs.AIOct 14
Teaching Agents in their Own Style: Blackbox Teacher Distillation for Web Agents
Md Farhan Ishmam, Kenneth Marino
Frontier web agents using closed models have high inference and computational costs. Smaller open models can be an alternative, but to be performant, they require training supervision from a human or teacher model. Token-level teacher distillation is not an option for closed teachers as it needs access to the model's logits. The standard approach is to imitate teacher demonstrations offline, but it trains the student on states it has never visited, and in a style it does not write in. We introduce WebKD, where the teacher is the editor of the student's failures. Given a failed episode, the teacher finds the first point of mistake, and rewrites the student's reasoning traces, harness states, and actions from that point onward. The editing reveals that the teacher's corrections can be more surprising to the student than the student's own text. WebKD instead aims to edit minimally. In the data space, the teacher writes each correction in the student's style. In the objective space, the reasoning is weighted by its contribution to the action. A role-asymmetric KL anchor allows the corrected action to move freely while keeping the reasoning near the student's prior. On the TimeWarp…
PDF arXiv
arXiv cs.AICodeOct 14▲ 57
ActionOPD: Distilling Decisions, Not Tokens
Huining Yuan, Xiangmin Yi, Zelai Xu, Zixu Wang, Xinlei Chen +3
Distilling capable language-model agents into smaller models requires aligning the student's behavior with the teacher's. This raises a fundamental question: at which states and at what granularity should this alignment be performed? Offline imitation provides dense but token-specific supervision and can suffer from state-distribution mismatch. Token-level on-policy distillation (OPD) addresses this mismatch but typically requires teacher token probabilities and compatible tokenization. We argue that semantic actions provide a natural unit of supervision for agent distillation. We introduce ActionOPD, an action-level on-policy distillation method motivated by maximizing expected teacher-action likelihood at student-encountered states. At each such state, ActionOPD samples actions from both the student and a black-box teacher, rewarding student actions according to their semantic agreement with the sampled teacher actions. Through policy optimization, ActionOPD aligns the student's action-level behavior with the teacher's decisions without requiring reproduction of teacher-written reasoning or access to teacher token probabilities. Training Qwen3-4B and Qwen3-8B with two teachers…
PDF arXiv Code
arXiv cs.LGOct 14
Function-Space Temporal Self-Distillation for Policy Optimization
Aryan Das, Arnab Bhattacharya, Vinay Kumar Verma
Reinforcement Learning with Verifiable Rewards (RLVR) improves reasoning in language models, but scalar outcomes provide coarse supervision over response trajectories. Self-distillation offers denser token-level guidance by comparing an unassisted policy with a feedback-informed self-teacher, yet the teacher evolves with the student, creating unstable targets and motivating additional regularization. We introduce Function-Space Policy Optimization (FSPO), a single-policy framework that combines current privileged adaptation with function-space temporal consolidation. FSPO stores sparse historical next-token distributions as fixed temporal targets, providing temporal separation without maintaining a persistent teacher model. It further introduces privileged disagreement correction to explicitly suppress locally overvalued actions, reliability-adaptive grounding that strengthens reference supervision when self-generated evidence repeatedly fails, and quality and coverage-aware temporal memory that retains verified behaviors and improves privileged demonstrations. Across reasoning and tool-use benchmarks, FSPO achieves the highest average accuracy under a 5-hour training budget,…
PDF arXiv
arXiv cs.LGOct 14
Beyond Distribution Matching: Selective Margin Preservation for Supervised Fine-Tuning
Yuxuan Wang, Guangxiang Zhao, Xinyu Tan, Yahui Han, Xinxiang Yin +4
Supervised fine-tuning (SFT) changes logit margins among non-target tokens at teacher-forced prefixes. Matching a local reference distribution keeps those probabilities and also penalizes margin expansions. We ask whether a soft auxiliary can preserve selected local non-target margin relations without freezing that distribution. We introduce \textbf{Selective Rank-Preserving SFT (SRP-SFT)}, a soft margin target: a hinge on reference-scaled lower bounds for pairs from the reference's top-$K$ non-target tokens, applied at fixed teacher-forced prefixes. The hinge stops once the requested bound is met. We derive the KL cost of violating a margin, characterize the feasible region and its conditional utility, and give a finite-step condition for reducing aggregate violations. In one-epoch experiments on Qwen3-4B-Base and UltraFeedback, this target raises MATH-500 and code-generation scores over CE and LP-SFT at all three training seeds. At seed 1234, MATH-500 pass@1 rises from 49.56\% to 54.56\% and the four-metric code mean rises from 65.91\% to 69.50\% relative to LP-SFT. Ranking controls, held-out cache diagnostics, sensitivity sweeps, and six additional backbones show where these…
PDF arXiv
arXiv cs.CLOct 14
HACHIMI-II: From Role-Play to Learner Behavior Modeling for Faithful Student Simulation
Yilin Jiang, Fei Tan, Xuanyu Yin, Jing Leng, Aimin Zhou
Simulated students let researchers test tutoring systems and teaching strategies before real learners are available, but only if the simulation reproduces where a particular learner succeeds and fails. The common approach asks a large language model to role-play a student, but a role does not model what this learner knows or will do next, so the model's own competence fills the gap. Simulated students perform at a level set by the model, differ little from one another, and seldom make the real learners' errors. We model the learner's behavior and recast student simulation as third-person prediction of the learner's next action. HACHIMI-II builds a student model from the learner's history, infers the learner's state for the current item or tutor turn, and compiles it into a behavior harness that constrains the next action. On three item-response datasets, the Wasserstein-1 distance between simulated and real student ability is 2.4--8.4$\times$ smaller than that of the strongest baseline. On two dialogue datasets, HACHIMI-II reproduces real student errors with the same knowledge type, locus, and mechanism 1.8--5.1$\times$ more often than the strongest baseline, and ablations trace…
PDF arXiv

Activity · 12 weeks

Coming with more sources: repos

Fields · AI · updated 2h ago

Teacher policy distillation

Found by the topic model: teacher policy distillation, ai tutors.

55papers this week−32.1% vs last week

All papers · 373 With code · 20

Newest Most relevant

358 papers this month

Open problems in Teacher policy distillation

Teacher recoverability on incorrect student rollouts1 paper raises it · 1 addresses it
Teachers often follow student errors instead of correcting them, reducing the quality of on-policy distillation supervision.
- Recovering Reliable Supervision via Critique-Guided On-Policy Distillationraises · addresses
Gradient conflicts in multi-teacher on-policy distillation1 paper raises it · 2 address it
Divergent teacher supervision can cause parameter updates to interfere, degrading student capabilities during joint optimization.
- Beyond Domain-Routed Supervision: Cross-Teacher Guarding for Multi-Teacher On-Policy Distillationraises · addresses
- Latent-MOPD: Latent Multi-Teacher On-Policy Distillationaddresses
Tokenization mismatch in cross-tokenizer distillation1 paper raises it · 2 address it
Different tokenizers between teacher and student complicate alignment, making supervision unreliable or incomplete.
- Rethinking Cross-Tokenizer On-Policy Distillation: From Alignment Coverage to Supervision Reliabilityraises · addresses
- ActionOPD: Distilling Decisions, Not Tokensaddresses
Policy shift opacity in teacher distillation1 paper raises it · 1 addresses it
Endpoint policy shifts do not reveal whether local action preferences evolved consistently during teacher training, limiting selective transfer.
- Waypoint-OPD: Transferring RL-Induced Policy Shifts via Gated On-Policy Distillationraises · addresses
Collapse in iterative agent self-distillation1 paper raises it · 1 addresses it
Successive distillation cycles can degrade deployment performance and privileged-information-conditioned behavior, hindering recursive self-improvement.
- ReSAIL: Mitigating Collapse in Iterative Agent Self-Distillationraises · addresses

Named by an AI from these papers' abstracts.