Papers

Field

Top tools here

Open problems in Teacher policy distillation

Teacher recoverability on incorrect student rollouts1 paper raises it · 1 addresses it
Teachers often follow student errors instead of correcting them, reducing the quality of on-policy distillation supervision.
- Recovering Reliable Supervision via Critique-Guided On-Policy Distillationraises · addresses
Gradient conflicts in multi-teacher on-policy distillation1 paper raises it · 2 address it
Divergent teacher supervision can cause parameter updates to interfere, degrading student capabilities during joint optimization.
- Beyond Domain-Routed Supervision: Cross-Teacher Guarding for Multi-Teacher On-Policy Distillationraises · addresses
- Latent-MOPD: Latent Multi-Teacher On-Policy Distillationaddresses
Tokenization mismatch in cross-tokenizer distillation1 paper raises it · 2 address it
Different tokenizers between teacher and student complicate alignment, making supervision unreliable or incomplete.
- Rethinking Cross-Tokenizer On-Policy Distillation: From Alignment Coverage to Supervision Reliabilityraises · addresses
- ActionOPD: Distilling Decisions, Not Tokensaddresses
Policy shift opacity in teacher distillation1 paper raises it · 1 addresses it
Endpoint policy shifts do not reveal whether local action preferences evolved consistently during teacher training, limiting selective transfer.
- Waypoint-OPD: Transferring RL-Induced Policy Shifts via Gated On-Policy Distillationraises · addresses
Collapse in iterative agent self-distillation1 paper raises it · 1 addresses it
Successive distillation cycles can degrade deployment performance and privileged-information-conditioned behavior, hindering recursive self-improvement.
- ReSAIL: Mitigating Collapse in Iterative Agent Self-Distillationraises · addresses

Named by an AI from these papers' abstracts.

Open problems

Recently funded

Topics in this field

teacher policy distillation · policy distillation opd · teacher supervision318 papers40 this week
ai tutors · ai tutor · tutoring55 papers7 this week

Topics in this field

Active contributors · 90 days

Active contributors

Activity · 12 weeks

Activity

Open problems in Teacher policy distillation

Teacher recoverability on incorrect student rollouts1 paper raises it · 1 addresses it
Teachers often follow student errors instead of correcting them, reducing the quality of on-policy distillation supervision.
- Recovering Reliable Supervision via Critique-Guided On-Policy Distillationraises · addresses
Gradient conflicts in multi-teacher on-policy distillation1 paper raises it · 2 address it
Divergent teacher supervision can cause parameter updates to interfere, degrading student capabilities during joint optimization.
- Beyond Domain-Routed Supervision: Cross-Teacher Guarding for Multi-Teacher On-Policy Distillationraises · addresses
- Latent-MOPD: Latent Multi-Teacher On-Policy Distillationaddresses
Tokenization mismatch in cross-tokenizer distillation1 paper raises it · 2 address it
Different tokenizers between teacher and student complicate alignment, making supervision unreliable or incomplete.
- Rethinking Cross-Tokenizer On-Policy Distillation: From Alignment Coverage to Supervision Reliabilityraises · addresses
- ActionOPD: Distilling Decisions, Not Tokensaddresses
Policy shift opacity in teacher distillation1 paper raises it · 1 addresses it
Endpoint policy shifts do not reveal whether local action preferences evolved consistently during teacher training, limiting selective transfer.
- Waypoint-OPD: Transferring RL-Induced Policy Shifts via Gated On-Policy Distillationraises · addresses
Collapse in iterative agent self-distillation1 paper raises it · 1 addresses it
Successive distillation cycles can degrade deployment performance and privileged-information-conditioned behavior, hindering recursive self-improvement.
- ReSAIL: Mitigating Collapse in Iterative Agent Self-Distillationraises · addresses

Named by an AI from these papers' abstracts.

Activity · 12 weeks

Recent papers

All papers in Research

arXiv cs.LGSep 22
Slow Decay and Silenced Expression: Iterated Subliminal Trait Transfer in Language-Model Lineages
Ryan Vo, Duc-Vu Nguyen, Matt Kretchmar, Ngan Luu-Thuy Nguyen
Language models are increasingly trained on the outputs of other models, forming chains that we call lineages, in which a trait present in one generation can pass to the next. Prior work on subliminal learning has shown that a teacher's trait can transmit to a student through filtered data carrying none of the trait's content. However, the evidence covers only a single training step. We study whether such a trait holds or fades across lineages. We instill the trait into three copies of Qwen2.5-7B-Instruct and iterate the training step to depth ten from each, reading every generation two ways on the same held-out prompts: a keyword screen that looks for expressions of the trait in the model's output, and an activation probe that projects each model's displacement from the base onto a direction built from the other lineages' teachers. We report two findings. First, the trait persists through ten generations across three lineages. The instilled models express it on every completion; the keyword-screen rate falls to 55.6% after the first step and to 21.1% by generation ten. The base itself matches the screen on none of its 300 completions. Second, the trait can be present internally…
PDF arXiv
arXiv cs.LGSep 22
What Should a Self-Teacher See? Privileged Context Design for On-Policy Self-Distillation
Kanghui Tian, Siyuan Liu, Tianxiang Jiang, Shuai Dong, Yizhuo Li +6
More privileged information does not always make a better teacher. We study this tension in on-policy self-distillation (OPSD), where a self-teacher scores the student's own rollouts under privileged context, conventionally a complete reference solution that bundles the final answer with one particular reasoning path. Holding the student view and training fixed within each scale, we compare that default against three abstractions compiled offline, a named strategy, a method-independent framing, and a problem category, and against an answer-only control that keeps the destination but removes the path. In the primary runs on competition mathematics, the best intermediate contexts improve the in-domain peak mean over the full solution by 1.4 points at 4B and 1.6 at 8B, while storing an order of magnitude fewer hint tokens. Comparisons across three seeds also show positive mean gains for the framing and category contexts at both scales. Answer-only conditioning remains competitive in the primary runs, within 0.2 points of the full solution at these scales. The preferred context varies with student scale and task. Initial teacher-student KL does not order downstream performance. What…
PDF arXiv
arXiv cs.AISep 21
ShowTellArena: Evaluating Business Workflow Understanding from Demonstrations
David Garg, Ritobrata Sarkar, Ehsan Azarnasab, Siddhartha Borah
We often teach a colleague by showing the work and explaining the decisions as we go. How can we check what an agent understood from the same lesson? We introduce ShowTellArena, a benchmark protocol and public dataset for comprehension after narrated business demonstrations. The v1.0 release contains 50 business workflow tasks, with recordings, screenshots, narration, fixture seeds, and 502 questions. Tasks span finance, hiring, procurement, customer decisions, inventory, and logistics. The protocol holds the business scenario and quiz fixed while allowing each product to capture the lesson through its own teaching interface. Questions test operational rules, boundaries, exceptions, and errors in proposed automations. We analyze 218 selected pilot attempts across 39 workflow cases, including 28 cases attempted by all three evaluated systems. These exploratory results expose both answer errors and failures to complete the teaching experience. We describe the release's verification gaps and the pilot's uneven coverage, exclusions, and grading provenance. The contribution is an inspectable dataset and assessment workflow that others can extend; the selected pilot is not a controlled…
PDF arXiv
arXiv cs.AISep 21
Potential for Enhanced Learning in Machine Learning Classes by Using Wiki LLM Indexing
Brian Wright
Large language models are increasingly deployed as course-specific tutors, but their usefulness depends on grounding in vetted instructional materials that are often revised mid-semester. Our prior work built a multimodal retrieval-augmented generation (RAG) system over an authentic machine learning course corpus (Foundations of Machine Learning) and found that retrieval improved contextual grounding, but that fixed retrieval strategies were suboptimal. That motivates a different question: whether how a corpus is structured at ingest time matters more than how much is retrieved at query time. We present a controlled head-to-head comparison of two knowledge representations over an identical classroom corpus: (A) vector RAG, replicating the best-performing configuration from our prior study, and (B) an LLM-compiled wiki (Karpathy framework), in which the corpus is synthesized at ingest into linked concept pages with explicit cross-references and citations back to source materials. We evaluate 59 questions spanning single-fact recall, cross-unit concept linking, synthesis and explanation, and currency after a syllabus revision, scored by an LLM judge against a human-authored rubric.…
PDF arXiv
arXiv cs.LGSep 21
iSDFT: Information-Proximal Self-Distillation for Continual Learning in LLMs
Ahmed Khaled Khamis, Xiaotong Ji, Hassan Jaber, Rasul Tutunov, Matthieu Zimmer +2
On-policy self-distillation fine-tuning (SDFT) learns new skills from demonstrations while reducing forgetting, but it always distils toward the full demonstration-conditioned teacher. This fixes teacher influence at the full-teacher endpoint, providing no control over how much demonstration information should be transferred at each prediction state. We introduce Information-Proximal SDFT (iSDFT), which instead treats the teacher as a budgeted source of information. At each token, iSDFT selects the distribution closest to the current student that satisfies a prescribed teacher-information constraint, yielding a closed-form exponential target with a locally determined tilt. To control cumulative drift, we further anchor the student to its frozen base policy. Across four heterogeneous LLM backbones and two specialisation tasks, iSDFT improves vanilla SDFT in 7 of 8 model-task settings and matches it in the remaining one. It also provides tighter retention on the original SDFT benchmark suite, with 73% of evaluations remaining within 0.5 points of the base model versus 52% for the strongest baseline, while achieving the largest mean improvement on all ten additional mathematics,…
PDF arXiv
arXiv cs.LGSep 21
1% of Tokens Can Be Enough: On Gradient Estimation in On-Policy Distillation
Huanxin Sheng, Zhiling Ye, Haonan Wang, Jian Wang, Jinjie Gu +1
Sparse on-policy distillation (OPD) allocates teacher supervision to a small subset of tokens in student-generated trajectories. However, useful teacher guidance can yield a noisy update when its gradient is estimated from a sampled next token. We study this estimation problem at a fixed prefix in information geometry and propose an information-efficiency ratio (IER) based on a signal-to-noise decomposition. IER characterizes relative gradient estimation error under an optimal scalar baseline. A candidate-set approximation enables token selection based on IER and its combination with existing usefulness scores, while retaining the sampled reverse-KL training objective. On mathematical and medical reasoning tasks, adding IER improves existing selectors in multiple settings, with sparse configurations matching or exceeding full OPD without token selection at small token budgets of 0.1\%--1\%. These results support accounting for both usefulness and gradient-estimation reliability when allocating sparse supervision. Our code is available at https://github.com/BruceSheng1202/IER-OPD.
PDF arXiv

Papers

Topics in this field

teacher policy distillation · policy distillation opd · teacher supervision318 papers40 this week
ai tutors · ai tutor · tutoring55 papers7 this week

Recent papers

All papers in Research

arXiv cs.LGSep 22
Slow Decay and Silenced Expression: Iterated Subliminal Trait Transfer in Language-Model Lineages
Ryan Vo, Duc-Vu Nguyen, Matt Kretchmar, Ngan Luu-Thuy Nguyen
Language models are increasingly trained on the outputs of other models, forming chains that we call lineages, in which a trait present in one generation can pass to the next. Prior work on subliminal learning has shown that a teacher's trait can transmit to a student through filtered data carrying none of the trait's content. However, the evidence covers only a single training step. We study whether such a trait holds or fades across lineages. We instill the trait into three copies of Qwen2.5-7B-Instruct and iterate the training step to depth ten from each, reading every generation two ways on the same held-out prompts: a keyword screen that looks for expressions of the trait in the model's output, and an activation probe that projects each model's displacement from the base onto a direction built from the other lineages' teachers. We report two findings. First, the trait persists through ten generations across three lineages. The instilled models express it on every completion; the keyword-screen rate falls to 55.6% after the first step and to 21.1% by generation ten. The base itself matches the screen on none of its 300 completions. Second, the trait can be present internally…
PDF arXiv
arXiv cs.LGSep 22
What Should a Self-Teacher See? Privileged Context Design for On-Policy Self-Distillation
Kanghui Tian, Siyuan Liu, Tianxiang Jiang, Shuai Dong, Yizhuo Li +6
More privileged information does not always make a better teacher. We study this tension in on-policy self-distillation (OPSD), where a self-teacher scores the student's own rollouts under privileged context, conventionally a complete reference solution that bundles the final answer with one particular reasoning path. Holding the student view and training fixed within each scale, we compare that default against three abstractions compiled offline, a named strategy, a method-independent framing, and a problem category, and against an answer-only control that keeps the destination but removes the path. In the primary runs on competition mathematics, the best intermediate contexts improve the in-domain peak mean over the full solution by 1.4 points at 4B and 1.6 at 8B, while storing an order of magnitude fewer hint tokens. Comparisons across three seeds also show positive mean gains for the framing and category contexts at both scales. Answer-only conditioning remains competitive in the primary runs, within 0.2 points of the full solution at these scales. The preferred context varies with student scale and task. Initial teacher-student KL does not order downstream performance. What…
PDF arXiv
arXiv cs.AISep 21
ShowTellArena: Evaluating Business Workflow Understanding from Demonstrations
David Garg, Ritobrata Sarkar, Ehsan Azarnasab, Siddhartha Borah
We often teach a colleague by showing the work and explaining the decisions as we go. How can we check what an agent understood from the same lesson? We introduce ShowTellArena, a benchmark protocol and public dataset for comprehension after narrated business demonstrations. The v1.0 release contains 50 business workflow tasks, with recordings, screenshots, narration, fixture seeds, and 502 questions. Tasks span finance, hiring, procurement, customer decisions, inventory, and logistics. The protocol holds the business scenario and quiz fixed while allowing each product to capture the lesson through its own teaching interface. Questions test operational rules, boundaries, exceptions, and errors in proposed automations. We analyze 218 selected pilot attempts across 39 workflow cases, including 28 cases attempted by all three evaluated systems. These exploratory results expose both answer errors and failures to complete the teaching experience. We describe the release's verification gaps and the pilot's uneven coverage, exclusions, and grading provenance. The contribution is an inspectable dataset and assessment workflow that others can extend; the selected pilot is not a controlled…
PDF arXiv
arXiv cs.AISep 21
Potential for Enhanced Learning in Machine Learning Classes by Using Wiki LLM Indexing
Brian Wright
Large language models are increasingly deployed as course-specific tutors, but their usefulness depends on grounding in vetted instructional materials that are often revised mid-semester. Our prior work built a multimodal retrieval-augmented generation (RAG) system over an authentic machine learning course corpus (Foundations of Machine Learning) and found that retrieval improved contextual grounding, but that fixed retrieval strategies were suboptimal. That motivates a different question: whether how a corpus is structured at ingest time matters more than how much is retrieved at query time. We present a controlled head-to-head comparison of two knowledge representations over an identical classroom corpus: (A) vector RAG, replicating the best-performing configuration from our prior study, and (B) an LLM-compiled wiki (Karpathy framework), in which the corpus is synthesized at ingest into linked concept pages with explicit cross-references and citations back to source materials. We evaluate 59 questions spanning single-fact recall, cross-unit concept linking, synthesis and explanation, and currency after a syllabus revision, scored by an LLM judge against a human-authored rubric.…
PDF arXiv
arXiv cs.LGSep 21
iSDFT: Information-Proximal Self-Distillation for Continual Learning in LLMs
Ahmed Khaled Khamis, Xiaotong Ji, Hassan Jaber, Rasul Tutunov, Matthieu Zimmer +2
On-policy self-distillation fine-tuning (SDFT) learns new skills from demonstrations while reducing forgetting, but it always distils toward the full demonstration-conditioned teacher. This fixes teacher influence at the full-teacher endpoint, providing no control over how much demonstration information should be transferred at each prediction state. We introduce Information-Proximal SDFT (iSDFT), which instead treats the teacher as a budgeted source of information. At each token, iSDFT selects the distribution closest to the current student that satisfies a prescribed teacher-information constraint, yielding a closed-form exponential target with a locally determined tilt. To control cumulative drift, we further anchor the student to its frozen base policy. Across four heterogeneous LLM backbones and two specialisation tasks, iSDFT improves vanilla SDFT in 7 of 8 model-task settings and matches it in the remaining one. It also provides tighter retention on the original SDFT benchmark suite, with 73% of evaluations remaining within 0.5 points of the base model versus 52% for the strongest baseline, while achieving the largest mean improvement on all ten additional mathematics,…
PDF arXiv
arXiv cs.LGSep 21
1% of Tokens Can Be Enough: On Gradient Estimation in On-Policy Distillation
Huanxin Sheng, Zhiling Ye, Haonan Wang, Jian Wang, Jinjie Gu +1
Sparse on-policy distillation (OPD) allocates teacher supervision to a small subset of tokens in student-generated trajectories. However, useful teacher guidance can yield a noisy update when its gradient is estimated from a sampled next token. We study this estimation problem at a fixed prefix in information geometry and propose an information-efficiency ratio (IER) based on a signal-to-noise decomposition. IER characterizes relative gradient estimation error under an optimal scalar baseline. A candidate-set approximation enables token selection based on IER and its combination with existing usefulness scores, while retaining the sampled reverse-KL training objective. On mathematical and medical reasoning tasks, adding IER improves existing selectors in multiple settings, with sparse configurations matching or exceeding full OPD without token selection at small token budgets of 0.1\%--1\%. These results support accounting for both usefulness and gradient-estimation reliability when allocating sparse supervision. Our code is available at https://github.com/BruceSheng1202/IER-OPD.
PDF arXiv

Fields · AI · updated 18m ago

Teacher policy distillation

Found by the topic model: teacher policy distillation, ai tutors.

47papers this week−38.2% vs last week

All papers · 373 With code · 20

Newest Most relevant

355 papers this month