Papers

Field

Top tools here

Open problems in Recursive self improvement

Gradient conflicts in multi-teacher distillation1 paper raises it · 1 addresses it
Conflicting teacher signals degrade joint optimization and cause capability loss when training a single student from multiple teachers.
- Beyond Domain-Routed Supervision: Cross-Teacher Guarding for Multi-Teacher On-Policy Distillationraises · addresses
Unreliable supervision in cross-tokenizer distillation1 paper raises it · 1 addresses it
Vocabulary mismatch and alignment coverage issues make teacher feedback noisy or harmful for student training.
- Rethinking Cross-Tokenizer On-Policy Distillation: From Alignment Coverage to Supervision Reliabilityraises · addresses
Inconsistent policy shifts during teacher training1 paper raises it · 1 addresses it
Endpoint policy shifts may hide early and late preference changes that cancel out, leading to poor transfer.
- Waypoint-OPD: Transferring RL-Induced Policy Shifts via Gated On-Policy Distillationraises · addresses
Lack of decision-grounded trajectory supervision1 paper raises it · 1 addresses it
Successful trajectories often include actions unsupported by the agent's visible history, reducing learning effectiveness.
- Trajectory2Gym: Decision-Grounded Compositional Interaction Synthesis for Tool-Use Agentsraises · addresses
Credit assignment in long-horizon agent training1 paper raises it · 1 addresses it
Critic values vary across turns, making fine-grained credit allocation and policy updates difficult.
- Multiscale Policy Optimization for Long-Horizon LLM Agentsraises · addresses

Named by an AI from these papers' abstracts.

Open problems

Recently funded

Topics in this field

recursive self improvement · ai · agent518 papers41 this week
multi agent debate · ai agents · agent debate230 papers22 this week
teacher policy distillation · policy distillation opd · teacher supervision353 papers19 this week
conversational agents · user simulators · user simulation171 papers19 this week
agent collaboration · multi agent · based multi agent204 papers17 this week
scientific agents · research agents · ai agents193 papers17 this week

Topics in this field

Active contributors · 90 days

Active contributors

Activity · 12 weeks

Activity

Recent papers

All papers in Research

arXiv cs.AICodeOct 24▲ 26
Build, Don't Play: Self-Improving LLM Research Agents Develop a Hearthstone Battlegrounds AI That Rivals Top Players
Changlian Ma, Limin Wang
Can LLM agents carry a hard real-world problem from raw data to a system that holds its own against people? We study this in Hearthstone Battlegrounds, an eight-player auto-battler with stochastic shops and battles, hidden information and nearly a thousand interacting cards. We present AutoTavern, a self-improving research system: specialised LLM research agents share a typed research memory, every claim passes verification gates, and live games close the loop, so the agents improve the simulator, opponents and policy and, from the same feedback, their own research process; a human sets goals and rules on ambiguous mechanics. In 20 days AutoTavern built TavernGym, the first faithful, full-content reproduction of the live game, and delivered 23 versions of TavernAgent, trained with PPO against opponents calibrated to real games. TavernAgent plays complete games on the live ladder by itself: its rating rose from 6,000 to about 15,000 and, on the public leaderboard's measure, its last 20 records average a placement of 1.60 with 80% first places, better than the server's ten highest-rated players on every statistic (2.34, 59%). TavernBench, 960 tasks in 40 scenarios, shows that the…
PDF arXiv Code
arXiv cs.AICodeOct 24▲ 6
EnvLunch: The Environment as a Free Lunch for Learning to Interact
Wei Liu, Min Gao, Yali Du, Yulan He
For agentic data analysis, language-model agents often require a large number of annotated tasks to learn how to interact effectively with executable environments (i.e., databases). While high-quality annotated tasks require heavy manual effort, an executable environment already provides rich interaction structure and verifiable outcomes. In this sense, the environment itself offers a free lunch, an additional set of verifiable tasks that require no human annotation. We propose EnvLunch, a warm-up stage that turns this environmental signal into agent training without human-written or model-generated task descriptions, before downstream agentic reinforcement learning. EnvLunch samples valid action trajectories, executes them to obtain final observations, and trains an agent to recover a functionally equivalent trajectory given only its final observation. This label-free warm-up enables the model to outperform the baseline by over 9 points on average across all downstream training budgets, and with few labels, the baseline needs 8 to 32 times as many annotated examples to match the accuracy that the EnvLunch model reaches at its smallest training budget. Letting the environment…
PDF arXiv Code
arXiv cs.CLCodeOct 24▲ 367
Trajectory2Gym: Decision-Grounded Compositional Interaction Synthesis for Tool-Use Agents
Yihao Hu, Zhihao Wen, Yitong Lin, Siyuan Lu, Pan Wang +3
Successful tool-use trajectories are not necessarily useful supervision. Even successful interactions may underrepresent multi-step dependencies or contain actions that are unjustified by the information visible to the agent at decision time. We identify two coupled requirements for effective trajectory supervision: compositional coverage, so that important observation-action dependencies are represented, and decision grounding, so that each supervised action is supported by the learner-visible interaction history. We introduce Trajectory2Gym, which constructs executable interactions around under-supported compositional structures and derives supervision only from grounded decisions in their realized executions. Structural-support selection allocates the resulting experience, while parent-normalized supervised fine-tuning preserves the semantic interaction as the unit of training weight. This reframes trajectory generation as the deliberate construction of learnable interaction experience, rather than the collection of successful trajectories. Across five Qwen3 scales, Trajectory2Gym improves Tau1, Tau2, and BFCL at every tested scale, increasing the equal-weight three-benchmark…
PDF arXiv Code
arXiv cs.CLCodeOct 24▲ 48
Eat Your Precision: Precision Objective Mitigates Reward Hacking in Rubric-based RL
Hyun Ryu, Jong-Heon Kim, Yiqing Xie, Seungone Kim, Carolyn Rose +1
Rubric-based rewards enable effective reinforcement learning on non-verifiable tasks. However, because these rewards are typically defined as a weighted measure of rubric coverage, models are trained primarily to maximize a recall objective. As a result, RL-trained models produce responses that are more complete but also more verbose and contain more unnecessary or factually incorrect content, leading them to be less preferred than responses from the base model. To address this issue, we present the FORK reward, which jointly optimizes recall and a novel precision objective, which requires each unit of a model response to be both necessary and factually correct, as measured by rubric-driven necessity credits and a factual-correctness grader, respectively. Our experimental results show that FORK outperforms existing rubric-based RL methods and achieves Pareto-optimal performance that balances rubric coverage and overall response quality. Compared to the original recall-based reward, FORK substantially improves precision and reduces response length while achieving similar rubric coverage, which leads to significantly higher win rate compared to the base policy (e.g., from 10.6% to…
PDF arXiv Code
arXiv cs.AICodeOct 23▲ 48
RSIHub: A Composable Framework for Agent-Evolution Methods
Zimu Wang, Xiaobo Wang, Siyu Ye, Jiaqi Li, Silin Chen +9
Agent-evolution methods use execution feedback to improve prompts, skills, tools, and orchestration, but their implementations often rebuild the same experimental machinery. We present RSIHub, an open-source framework for building, comparing, and extending these methods on shared infrastructure. Recipes compose reusable operators and declare mutable resources; RSIHub supplies execution, evaluation, recovery, and lineage. Researchers can focus on evolution policy while evaluation stays outside the mutable surface and results remain tied to candidates and evaluation conditions. We instantiate AHE, Hyperagents, A-Evolve, and GEPA without modifying the framework core, keeping method-specific logic in recipes and operators. We evaluate them with MiniSWE and Codex on Terminal Bench 2 and $τ^3$ Banking under fixed model and evaluator configurations. The largest observed overall pass@1 gain is 13.5 percentage points, while method rankings vary across settings and optimization gains do not always transfer to held-out tasks. RSIHub provides a common protocol for comparing evolution policies, inspectable candidate histories, and reusable components for developing new methods.
PDF arXiv Code
arXiv cs.AICodeOct 23▲ 124
Can Test-Time Memory Evolution Outperform Test-Time Training for Scientific Discovery?
Yepeng Liu, Shengmai Chen, Xuandong Zhao, Chengzhi Liu, Dawn Song +2
Recent \emph{test-time training} (TTT) frameworks have enabled open-weight large language models (LLMs) to outperform inference-only frameworks powered by frontier models on scientific discovery tasks. However, TTT introduces additional computational cost for online weight updates during the discovery process, making computational efficiency an important consideration when scaling to larger models. This motivates us to explore an efficient \emph{test-time adaptation} method for scientific discovery. We introduce \emph{Test-Time Memory Evolution} (TTME), a plug-and-play test-time adaptation method, as an alternative to TTT. TTME continually updates the external memory by distilling the experience to guide subsequent discovery without updating model weights. To have a clean comparison, we evaluate TTME and TTT under the same discovery framework, replacing only the test-time adaptation method. Our evaluation shows that TTME outperforms TTT on all five mathematical tasks, while reducing computational cost by $82.8$-$84.6\%$ in FLOPs and $47.7$-$58.4\%$ in runtime. These results establish TTME as an effective and computationally efficient alternative to TTT in the evaluated discovery…
PDF arXiv Code

Papers

Topics in this field

recursive self improvement · ai · agent518 papers41 this week
multi agent debate · ai agents · agent debate230 papers22 this week
teacher policy distillation · policy distillation opd · teacher supervision353 papers19 this week
conversational agents · user simulators · user simulation171 papers19 this week
agent collaboration · multi agent · based multi agent204 papers17 this week
scientific agents · research agents · ai agents193 papers17 this week

Active contributors · 90 days

Recent papers

All papers in Research

arXiv cs.AICodeOct 24▲ 26
Build, Don't Play: Self-Improving LLM Research Agents Develop a Hearthstone Battlegrounds AI That Rivals Top Players
Changlian Ma, Limin Wang
Can LLM agents carry a hard real-world problem from raw data to a system that holds its own against people? We study this in Hearthstone Battlegrounds, an eight-player auto-battler with stochastic shops and battles, hidden information and nearly a thousand interacting cards. We present AutoTavern, a self-improving research system: specialised LLM research agents share a typed research memory, every claim passes verification gates, and live games close the loop, so the agents improve the simulator, opponents and policy and, from the same feedback, their own research process; a human sets goals and rules on ambiguous mechanics. In 20 days AutoTavern built TavernGym, the first faithful, full-content reproduction of the live game, and delivered 23 versions of TavernAgent, trained with PPO against opponents calibrated to real games. TavernAgent plays complete games on the live ladder by itself: its rating rose from 6,000 to about 15,000 and, on the public leaderboard's measure, its last 20 records average a placement of 1.60 with 80% first places, better than the server's ten highest-rated players on every statistic (2.34, 59%). TavernBench, 960 tasks in 40 scenarios, shows that the…
PDF arXiv Code
arXiv cs.AICodeOct 24▲ 6
EnvLunch: The Environment as a Free Lunch for Learning to Interact
Wei Liu, Min Gao, Yali Du, Yulan He
For agentic data analysis, language-model agents often require a large number of annotated tasks to learn how to interact effectively with executable environments (i.e., databases). While high-quality annotated tasks require heavy manual effort, an executable environment already provides rich interaction structure and verifiable outcomes. In this sense, the environment itself offers a free lunch, an additional set of verifiable tasks that require no human annotation. We propose EnvLunch, a warm-up stage that turns this environmental signal into agent training without human-written or model-generated task descriptions, before downstream agentic reinforcement learning. EnvLunch samples valid action trajectories, executes them to obtain final observations, and trains an agent to recover a functionally equivalent trajectory given only its final observation. This label-free warm-up enables the model to outperform the baseline by over 9 points on average across all downstream training budgets, and with few labels, the baseline needs 8 to 32 times as many annotated examples to match the accuracy that the EnvLunch model reaches at its smallest training budget. Letting the environment…
PDF arXiv Code
arXiv cs.CLCodeOct 24▲ 367
Trajectory2Gym: Decision-Grounded Compositional Interaction Synthesis for Tool-Use Agents
Yihao Hu, Zhihao Wen, Yitong Lin, Siyuan Lu, Pan Wang +3
Successful tool-use trajectories are not necessarily useful supervision. Even successful interactions may underrepresent multi-step dependencies or contain actions that are unjustified by the information visible to the agent at decision time. We identify two coupled requirements for effective trajectory supervision: compositional coverage, so that important observation-action dependencies are represented, and decision grounding, so that each supervised action is supported by the learner-visible interaction history. We introduce Trajectory2Gym, which constructs executable interactions around under-supported compositional structures and derives supervision only from grounded decisions in their realized executions. Structural-support selection allocates the resulting experience, while parent-normalized supervised fine-tuning preserves the semantic interaction as the unit of training weight. This reframes trajectory generation as the deliberate construction of learnable interaction experience, rather than the collection of successful trajectories. Across five Qwen3 scales, Trajectory2Gym improves Tau1, Tau2, and BFCL at every tested scale, increasing the equal-weight three-benchmark…
PDF arXiv Code
arXiv cs.CLCodeOct 24▲ 48
Eat Your Precision: Precision Objective Mitigates Reward Hacking in Rubric-based RL
Hyun Ryu, Jong-Heon Kim, Yiqing Xie, Seungone Kim, Carolyn Rose +1
Rubric-based rewards enable effective reinforcement learning on non-verifiable tasks. However, because these rewards are typically defined as a weighted measure of rubric coverage, models are trained primarily to maximize a recall objective. As a result, RL-trained models produce responses that are more complete but also more verbose and contain more unnecessary or factually incorrect content, leading them to be less preferred than responses from the base model. To address this issue, we present the FORK reward, which jointly optimizes recall and a novel precision objective, which requires each unit of a model response to be both necessary and factually correct, as measured by rubric-driven necessity credits and a factual-correctness grader, respectively. Our experimental results show that FORK outperforms existing rubric-based RL methods and achieves Pareto-optimal performance that balances rubric coverage and overall response quality. Compared to the original recall-based reward, FORK substantially improves precision and reduces response length while achieving similar rubric coverage, which leads to significantly higher win rate compared to the base policy (e.g., from 10.6% to…
PDF arXiv Code
arXiv cs.AICodeOct 23▲ 48
RSIHub: A Composable Framework for Agent-Evolution Methods
Zimu Wang, Xiaobo Wang, Siyu Ye, Jiaqi Li, Silin Chen +9
Agent-evolution methods use execution feedback to improve prompts, skills, tools, and orchestration, but their implementations often rebuild the same experimental machinery. We present RSIHub, an open-source framework for building, comparing, and extending these methods on shared infrastructure. Recipes compose reusable operators and declare mutable resources; RSIHub supplies execution, evaluation, recovery, and lineage. Researchers can focus on evolution policy while evaluation stays outside the mutable surface and results remain tied to candidates and evaluation conditions. We instantiate AHE, Hyperagents, A-Evolve, and GEPA without modifying the framework core, keeping method-specific logic in recipes and operators. We evaluate them with MiniSWE and Codex on Terminal Bench 2 and $τ^3$ Banking under fixed model and evaluator configurations. The largest observed overall pass@1 gain is 13.5 percentage points, while method rankings vary across settings and optimization gains do not always transfer to held-out tasks. RSIHub provides a common protocol for comparing evolution policies, inspectable candidate histories, and reusable components for developing new methods.
PDF arXiv Code
arXiv cs.AICodeOct 23▲ 124
Can Test-Time Memory Evolution Outperform Test-Time Training for Scientific Discovery?
Yepeng Liu, Shengmai Chen, Xuandong Zhao, Chengzhi Liu, Dawn Song +2
Recent \emph{test-time training} (TTT) frameworks have enabled open-weight large language models (LLMs) to outperform inference-only frameworks powered by frontier models on scientific discovery tasks. However, TTT introduces additional computational cost for online weight updates during the discovery process, making computational efficiency an important consideration when scaling to larger models. This motivates us to explore an efficient \emph{test-time adaptation} method for scientific discovery. We introduce \emph{Test-Time Memory Evolution} (TTME), a plug-and-play test-time adaptation method, as an alternative to TTT. TTME continually updates the external memory by distilling the experience to guide subsequent discovery without updating model weights. To have a clean comparison, we evaluate TTME and TTT under the same discovery framework, replacing only the test-time adaptation method. Our evaluation shows that TTME outperforms TTT on all five mathematical tasks, while reducing computational cost by $82.8$-$84.6\%$ in FLOPs and $47.7$-$58.4\%$ in runtime. These results establish TTME as an effective and computationally efficient alternative to TTT in the evaluated discovery…
PDF arXiv Code

Activity · 12 weeks

Fields · AI · updated 1h ago

Recursive self improvement

Found by the topic model: recursive self improvement, teacher policy distillation, multi agent debate.

238papers this week−39.9% vs last week

All papers · 2.7k With code · 204

Newest Most relevant

1.7k papers this month

Open problems in Recursive self improvement

Gradient conflicts in multi-teacher distillation1 paper raises it · 1 addresses it
Conflicting teacher signals degrade joint optimization and cause capability loss when training a single student from multiple teachers.
- Beyond Domain-Routed Supervision: Cross-Teacher Guarding for Multi-Teacher On-Policy Distillationraises · addresses
Unreliable supervision in cross-tokenizer distillation1 paper raises it · 1 addresses it
Vocabulary mismatch and alignment coverage issues make teacher feedback noisy or harmful for student training.
- Rethinking Cross-Tokenizer On-Policy Distillation: From Alignment Coverage to Supervision Reliabilityraises · addresses
Inconsistent policy shifts during teacher training1 paper raises it · 1 addresses it
Endpoint policy shifts may hide early and late preference changes that cancel out, leading to poor transfer.
- Waypoint-OPD: Transferring RL-Induced Policy Shifts via Gated On-Policy Distillationraises · addresses
Lack of decision-grounded trajectory supervision1 paper raises it · 1 addresses it
Successful trajectories often include actions unsupported by the agent's visible history, reducing learning effectiveness.
- Trajectory2Gym: Decision-Grounded Compositional Interaction Synthesis for Tool-Use Agentsraises · addresses
Credit assignment in long-horizon agent training1 paper raises it · 1 addresses it
Critic values vary across turns, making fine-grained credit allocation and policy updates difficult.
- Multiscale Policy Optimization for Long-Horizon LLM Agentsraises · addresses

Named by an AI from these papers' abstracts.