Papers

Field

Top tools here

Open problems in Recursive self improvement

Gradient conflicts in multi-teacher distillation1 paper raises it · 1 addresses it
Conflicting teacher signals degrade joint optimization and cause capability loss when training a single student from multiple teachers.
- Beyond Domain-Routed Supervision: Cross-Teacher Guarding for Multi-Teacher On-Policy Distillationraises · addresses
Unreliable supervision in cross-tokenizer distillation1 paper raises it · 1 addresses it
Vocabulary mismatch and alignment coverage issues make teacher feedback noisy or harmful for student training.
- Rethinking Cross-Tokenizer On-Policy Distillation: From Alignment Coverage to Supervision Reliabilityraises · addresses
Inconsistent policy shifts during teacher training1 paper raises it · 1 addresses it
Endpoint policy shifts may hide early and late preference changes that cancel out, leading to poor transfer.
- Waypoint-OPD: Transferring RL-Induced Policy Shifts via Gated On-Policy Distillationraises · addresses
Lack of decision-grounded trajectory supervision1 paper raises it · 1 addresses it
Successful trajectories often include actions unsupported by the agent's visible history, reducing learning effectiveness.
- Trajectory2Gym: Decision-Grounded Compositional Interaction Synthesis for Tool-Use Agentsraises · addresses
Credit assignment in long-horizon agent training1 paper raises it · 1 addresses it
Critic values vary across turns, making fine-grained credit allocation and policy updates difficult.
- Multiscale Policy Optimization for Long-Horizon LLM Agentsraises · addresses

Named by an AI from these papers' abstracts.

Open problems

Recently funded

Topics in this field

recursive self improvement · ai · agent520 papers34 this week
teacher policy distillation · policy distillation opd · teacher supervision354 papers18 this week
multi agent debate · ai agents · agent debate231 papers18 this week
agent collaboration · multi agent · based multi agent206 papers18 this week
conversational agents · user simulators · user simulation173 papers18 this week
scientific agents · research agents · ai agents196 papers16 this week

Topics in this field

Active contributors · 90 days

Active contributors

Activity · 12 weeks

Activity

Recent papers

All papers in Research

arXiv cs.CLCodeOct 5▲ 47
HuatuoGPT-3: RL-Only Domain Adaptation from Base Models
Junying Chen, Xinyuan Xie, Ziniu Li, Wenyuan Gu, Jianquan Li +5
Domain adaptation aims to turn a general-purpose large language model (LLM) into an expert for a target domain. While the dominant SFT+RL pipeline offers a convenient cold start, it may reduce exploration diversity and introduces additional complexity through multi-stage optimization. These limitations motivate RL-only adaptation. However, pure on-policy RL suffers from a cold-start problem, while mixed-policy RL still falls short: informative tokens in teacher outputs are learned too slowly in early training, and stale teacher outputs can hinder later improvement. We identify these two failure modes as Gradient Starvation and Teacher-Distribution Anchoring. To address them, we propose One-stage Policy Optimization (OnePO), which treats teacher outputs as transient guidance for policy improvement. OnePO combines Adaptive Objective Evolution to strengthen learning on informative low-probability teacher tokens and Teacher Retirement to discard teacher outputs once the current policy can surpass them. On medical adaptation, OnePO achieves 67.2 on HealthBench (Total) with only 20K training samples, outperforming SFT+RL and pure RL by 2.7 and 7.4 points, respectively. We further scale…
PDF arXiv Code
arXiv cs.AICodeOct 5
VERA: Scaling Verifiable Environments for Agentic co-Evolution
Junqi Liu, Yongyang Pan, Zhuosong Jiang, Dongbai Li, Bo Zhang +12
Competent agents need precise and verifiable environments, such as sandboxes that are resumable at any stage and evolve from observable evidence. However, most long-horizon work exposes how rare these are: for example, an agent in medical research must ground a finding, classify it, and write a report over dozens of dependent steps, yet recent environments score only the outcome. To address the challenges in stable training, we present VERA, which builds such environments at scale and lets agents evolve on them. VERA builds these environments from initial trajectories: an agent writes rubrics, executable checks, a judge verifies each sandbox, and only those that pass enter the training bank. On these environments, VERA alternates between two updates: train the model with rubric rewards, or edit the harness skills. We also create a verifier which gates model checkpoints and harness edits using explicit development-set acceptance criteria. This attribution distinguishes VERA's co-evolution from single-axis baselines: its updates target not only the cause but the outcome. With an open-source corpus of 9,000+ long-horizon verifiable environments, a 9B model paired with its co-evolved…
PDF arXiv Code
arXiv cs.LGCodeOct 5▲ 47
TRIAGE: Direction-Aware Mismatch Stabilization of Native NVFP4 Reinforcement Learning
Zhen Li, Shuai Zhang, Yanggan Gu, Yiming Zhang, Yang Yu +5
Low-precision execution can substantially accelerate reinforcement learning (RL) for large language models, but discrepancies between learner and sampler execution can destabilize policy optimization. In this paper, we characterize the interaction between mismatch and the policy-gradient direction, distinguishing locally amplifying from contracting update contributions that mismatch magnitude alone cannot identify. In native NVFP4 runs, we observe an early imbalance between the two amplifying regions, favoring negative-advantage, negative-gap updates. Their tail tokens become concentrated in a small fraction of response segments before mismatch spreads globally. Motivated by these findings, we introduce TRIAGE, a direction-aware stabilization method that uses segment-level diagnosis to selectively rebalance policy-gradient updates and applies bounded repair to residual severe mismatch. TRIAGE modifies the optimization objective while retaining native NVFP4 weight-and activation 4-bit (W4A4) forward execution on both the sampler and learner. Experiments on Qwen3-4B and Qwen3-30B-A3B show stable optimization throughout the evaluated training horizon and achieve full precision level…
PDF arXiv Code
arXiv cs.AICodeOct 4
MESH-Harness: Self-Improving Agent Harnesses via Bandit-Guided Compositional Evolution
Zhiwei Shang, Yu Huo, Mingrong Gong, An Yan, Zikun Qu +4
An agent harness is the code that organizes context, maintains state, and coordinates tool calls for a language model. We study how to improve the harness under a limited evaluation budget while keeping model weights fixed. Our method, MESH-Harness, organizes each harness into functional modules with explicit role-specific interfaces, allowing alternative implementations of each module to be substituted and recombined. It uses shared module representations and full-covariance LinUCB to score candidate combinations based on predicted performance and exploration value. Mixed-start coordinate ascent selects complete configurations for evaluation without enumerating the combinatorial space. Validation traces then guide local code edits, and the resulting candidates are incorporated into fixed-capacity role-specific pools for subsequent recombination. On text tasks, retrieval-augmented mathematical reasoning, code generation, and interactive scientific tasks, MESH-Harness outperforms Meta-Harness by 5.70, 7.01, 2.00, and 5.00 points, respectively, under matched candidate-evaluation budgets. Iterative harness optimization improves MESH-Harness by 5.63-7.79 points over its first-round…
PDF arXiv Code
arXiv cs.AICodeOct 4▲ 55
EVISKILL: Grounding Skill Evolution in Replayable Evidence
Yan Zhou, Yili Wang, Yiwei Dai, Qinggang Zhang, Xin Wang
Continual skill evolution enables LLM agents to accumulate and refine reusable procedural knowledge from interaction experience without updating model parameters. Its effectiveness depends on determining not only what to change, but also why a change is justified and when it should become persistent guidance. However, existing experience-driven methods can lose the behavioral evidence and task contexts supporting edits. Moreover, a global validation outcome provides an incomplete judgment of its constituent changes: locally supported corrections may be discarded with a rejected revision, while evidence may require further experience to inform useful updates. To this end, we introduce EVISKILL, an evidence-driven framework that organizes execution observations into Replayable Evidence Cards and synthesizes edits with explicit links to their supporting contexts. Targeted replay verifies these edits through re-execution and provides feedback for correction. Across epochs, EVISKILL preserves evidence and provisionally retains supported edits for further refinement, while global validation governs their incorporation into the final skill. Experiments on three interactive benchmarks…
PDF arXiv Code
arXiv cs.CLCodeOct 4
TrajLong: Co-Designing Agentic and Long-Context Supervision for Mid-Training
Miao Peng, Qintong Zhang, Nuo Chen, Yuhan Li, Guochen Yan +6
LLM agents for coding, search, and workplace tasks increasingly rely on long-context capabilities to effectively aggregate and reason over extended interaction histories. Recent work has incorporated agent trajectories into mid-training stage, drawing on their naturally long and interaction-rich structure. Yet how to organize the information within these trajectories into effective mid-training supervision remains underexplored. In this work, we investigate the relationship between long-context and agent atomic capabilities and introduce TrajLong, a novel framework that compiles trajectories into long-context training tasks with dense supervision, targeting three representative atomic capabilities: evidence grounding, cross-evidence aggregation, and temporal state maintenance. We mid-train Qwen3-14B-Base and Qwen3-30B-A3B-Base with data compiled by TrajLong, followed by supervised fine-tuning. Experiments on 6 long-context and 12 agent benchmarks demonstrate broad performance gains, with controlled ablations showing improvements over raw and masked trajectory baselines. Capability-level analyses further reveal task-dependent associations between long-context and agent atomic…
PDF arXiv Code

Papers

Topics in this field

recursive self improvement · ai · agent520 papers34 this week
teacher policy distillation · policy distillation opd · teacher supervision354 papers18 this week
multi agent debate · ai agents · agent debate231 papers18 this week
agent collaboration · multi agent · based multi agent206 papers18 this week
conversational agents · user simulators · user simulation173 papers18 this week
scientific agents · research agents · ai agents196 papers16 this week

Active contributors · 90 days

Recent papers

All papers in Research

arXiv cs.CLCodeOct 5▲ 47
HuatuoGPT-3: RL-Only Domain Adaptation from Base Models
Junying Chen, Xinyuan Xie, Ziniu Li, Wenyuan Gu, Jianquan Li +5
Domain adaptation aims to turn a general-purpose large language model (LLM) into an expert for a target domain. While the dominant SFT+RL pipeline offers a convenient cold start, it may reduce exploration diversity and introduces additional complexity through multi-stage optimization. These limitations motivate RL-only adaptation. However, pure on-policy RL suffers from a cold-start problem, while mixed-policy RL still falls short: informative tokens in teacher outputs are learned too slowly in early training, and stale teacher outputs can hinder later improvement. We identify these two failure modes as Gradient Starvation and Teacher-Distribution Anchoring. To address them, we propose One-stage Policy Optimization (OnePO), which treats teacher outputs as transient guidance for policy improvement. OnePO combines Adaptive Objective Evolution to strengthen learning on informative low-probability teacher tokens and Teacher Retirement to discard teacher outputs once the current policy can surpass them. On medical adaptation, OnePO achieves 67.2 on HealthBench (Total) with only 20K training samples, outperforming SFT+RL and pure RL by 2.7 and 7.4 points, respectively. We further scale…
PDF arXiv Code
arXiv cs.AICodeOct 5
VERA: Scaling Verifiable Environments for Agentic co-Evolution
Junqi Liu, Yongyang Pan, Zhuosong Jiang, Dongbai Li, Bo Zhang +12
Competent agents need precise and verifiable environments, such as sandboxes that are resumable at any stage and evolve from observable evidence. However, most long-horizon work exposes how rare these are: for example, an agent in medical research must ground a finding, classify it, and write a report over dozens of dependent steps, yet recent environments score only the outcome. To address the challenges in stable training, we present VERA, which builds such environments at scale and lets agents evolve on them. VERA builds these environments from initial trajectories: an agent writes rubrics, executable checks, a judge verifies each sandbox, and only those that pass enter the training bank. On these environments, VERA alternates between two updates: train the model with rubric rewards, or edit the harness skills. We also create a verifier which gates model checkpoints and harness edits using explicit development-set acceptance criteria. This attribution distinguishes VERA's co-evolution from single-axis baselines: its updates target not only the cause but the outcome. With an open-source corpus of 9,000+ long-horizon verifiable environments, a 9B model paired with its co-evolved…
PDF arXiv Code
arXiv cs.LGCodeOct 5▲ 47
TRIAGE: Direction-Aware Mismatch Stabilization of Native NVFP4 Reinforcement Learning
Zhen Li, Shuai Zhang, Yanggan Gu, Yiming Zhang, Yang Yu +5
Low-precision execution can substantially accelerate reinforcement learning (RL) for large language models, but discrepancies between learner and sampler execution can destabilize policy optimization. In this paper, we characterize the interaction between mismatch and the policy-gradient direction, distinguishing locally amplifying from contracting update contributions that mismatch magnitude alone cannot identify. In native NVFP4 runs, we observe an early imbalance between the two amplifying regions, favoring negative-advantage, negative-gap updates. Their tail tokens become concentrated in a small fraction of response segments before mismatch spreads globally. Motivated by these findings, we introduce TRIAGE, a direction-aware stabilization method that uses segment-level diagnosis to selectively rebalance policy-gradient updates and applies bounded repair to residual severe mismatch. TRIAGE modifies the optimization objective while retaining native NVFP4 weight-and activation 4-bit (W4A4) forward execution on both the sampler and learner. Experiments on Qwen3-4B and Qwen3-30B-A3B show stable optimization throughout the evaluated training horizon and achieve full precision level…
PDF arXiv Code
arXiv cs.AICodeOct 4
MESH-Harness: Self-Improving Agent Harnesses via Bandit-Guided Compositional Evolution
Zhiwei Shang, Yu Huo, Mingrong Gong, An Yan, Zikun Qu +4
An agent harness is the code that organizes context, maintains state, and coordinates tool calls for a language model. We study how to improve the harness under a limited evaluation budget while keeping model weights fixed. Our method, MESH-Harness, organizes each harness into functional modules with explicit role-specific interfaces, allowing alternative implementations of each module to be substituted and recombined. It uses shared module representations and full-covariance LinUCB to score candidate combinations based on predicted performance and exploration value. Mixed-start coordinate ascent selects complete configurations for evaluation without enumerating the combinatorial space. Validation traces then guide local code edits, and the resulting candidates are incorporated into fixed-capacity role-specific pools for subsequent recombination. On text tasks, retrieval-augmented mathematical reasoning, code generation, and interactive scientific tasks, MESH-Harness outperforms Meta-Harness by 5.70, 7.01, 2.00, and 5.00 points, respectively, under matched candidate-evaluation budgets. Iterative harness optimization improves MESH-Harness by 5.63-7.79 points over its first-round…
PDF arXiv Code
arXiv cs.AICodeOct 4▲ 55
EVISKILL: Grounding Skill Evolution in Replayable Evidence
Yan Zhou, Yili Wang, Yiwei Dai, Qinggang Zhang, Xin Wang
Continual skill evolution enables LLM agents to accumulate and refine reusable procedural knowledge from interaction experience without updating model parameters. Its effectiveness depends on determining not only what to change, but also why a change is justified and when it should become persistent guidance. However, existing experience-driven methods can lose the behavioral evidence and task contexts supporting edits. Moreover, a global validation outcome provides an incomplete judgment of its constituent changes: locally supported corrections may be discarded with a rejected revision, while evidence may require further experience to inform useful updates. To this end, we introduce EVISKILL, an evidence-driven framework that organizes execution observations into Replayable Evidence Cards and synthesizes edits with explicit links to their supporting contexts. Targeted replay verifies these edits through re-execution and provides feedback for correction. Across epochs, EVISKILL preserves evidence and provisionally retains supported edits for further refinement, while global validation governs their incorporation into the final skill. Experiments on three interactive benchmarks…
PDF arXiv Code
arXiv cs.CLCodeOct 4
TrajLong: Co-Designing Agentic and Long-Context Supervision for Mid-Training
Miao Peng, Qintong Zhang, Nuo Chen, Yuhan Li, Guochen Yan +6
LLM agents for coding, search, and workplace tasks increasingly rely on long-context capabilities to effectively aggregate and reason over extended interaction histories. Recent work has incorporated agent trajectories into mid-training stage, drawing on their naturally long and interaction-rich structure. Yet how to organize the information within these trajectories into effective mid-training supervision remains underexplored. In this work, we investigate the relationship between long-context and agent atomic capabilities and introduce TrajLong, a novel framework that compiles trajectories into long-context training tasks with dense supervision, targeting three representative atomic capabilities: evidence grounding, cross-evidence aggregation, and temporal state maintenance. We mid-train Qwen3-14B-Base and Qwen3-30B-A3B-Base with data compiled by TrajLong, followed by supervised fine-tuning. Experiments on 6 long-context and 12 agent benchmarks demonstrate broad performance gains, with controlled ablations showing improvements over raw and masked trajectory baselines. Capability-level analyses further reveal task-dependent associations between long-context and agent atomic…
PDF arXiv Code

Activity · 12 weeks

Fields · AI · updated 26m ago

Recursive self improvement

Found by the topic model: recursive self improvement, teacher policy distillation, multi agent debate.

210papers this week−44.7% vs last week

All papers · 2.7k With code · 207

Newest Most relevant

1.7k papers this month

Open problems in Recursive self improvement

Gradient conflicts in multi-teacher distillation1 paper raises it · 1 addresses it
Conflicting teacher signals degrade joint optimization and cause capability loss when training a single student from multiple teachers.
- Beyond Domain-Routed Supervision: Cross-Teacher Guarding for Multi-Teacher On-Policy Distillationraises · addresses
Unreliable supervision in cross-tokenizer distillation1 paper raises it · 1 addresses it
Vocabulary mismatch and alignment coverage issues make teacher feedback noisy or harmful for student training.
- Rethinking Cross-Tokenizer On-Policy Distillation: From Alignment Coverage to Supervision Reliabilityraises · addresses
Inconsistent policy shifts during teacher training1 paper raises it · 1 addresses it
Endpoint policy shifts may hide early and late preference changes that cancel out, leading to poor transfer.
- Waypoint-OPD: Transferring RL-Induced Policy Shifts via Gated On-Policy Distillationraises · addresses
Lack of decision-grounded trajectory supervision1 paper raises it · 1 addresses it
Successful trajectories often include actions unsupported by the agent's visible history, reducing learning effectiveness.
- Trajectory2Gym: Decision-Grounded Compositional Interaction Synthesis for Tool-Use Agentsraises · addresses
Credit assignment in long-horizon agent training1 paper raises it · 1 addresses it
Critic values vary across turns, making fine-grained credit allocation and policy updates difficult.
- Multiscale Policy Optimization for Long-Horizon LLM Agentsraises · addresses

Named by an AI from these papers' abstracts.