Spotlight

Papers

Field

Top tools here

Coming with more sources: stars history

All tools

Agno+17% downloads · 519.1k/wk+16.8%
n8n+25% downloads · 148.3k/wk+25.0%
LangGraph7 releases this week
DeepEval+15% downloads · 769.3k/wk+14.7%
OpenAI Python SDK7 releases this week

Top tools here

Open problems in Agents

Evaluating long-horizon agentic behavior in real-world tasks4 papers raise it · 3 address it
Current benchmarks often simplify or omit the exploration, strategy adjustment, and verification needed for sustained autonomous work.
- Scaling Agentic Data for Long-Horizon Terminal Intelligenceraises · addresses
- FDE-Gym: The Last Mile of AI Deliveryraises · addresses
- AgentSWE: Can Coding Agents Build the Agent You Actually Want?raises · addresses
- Into the Unknown: Benchmarking AI agents on open scientific research problemsraises
Faithful agent development from natural-language requirements1 paper raises it · 1 addresses it
Agents often fail to deliver systems that truly meet the specified requirements, not just appear to.
- AgentSWE: Can Coding Agents Build the Agent You Actually Want?raises · addresses
Learning from experience in unfamiliar environments1 paper raises it · 1 addresses it
Agents struggle to acquire and retain new knowledge through interaction rather than relying on pretrained knowledge.
- Learn2Play Bench: How Well Do LLM Agents Learn from Experience in Unfamiliar Environments?raises · addresses
Detecting when past context matters in long conversations1 paper raises it · 0 address it
AI systems cannot reliably identify when historical information is relevant to the current interaction.
- RealCompanion: Benchmarking Human Understanding from Reasoning over Longitudinal Real-World Conversationsraises
Open and reproducible agent stacks for computer use1 paper raises it · 2 address it
Frontier computer-use agents are often closed, limiting systematic study and community collaboration.
- OpenGCUA: Toward a Fully Open Stack for Generalist Computer Useraises · addresses
- MAI-Gym: An Open All-in-One Platform for GUI Agent Evaluation and RL Training across Mobile and Computer Environmentsaddresses

Named by an AI from these papers' abstracts.

Open problems

Recently funded

Market

GreptileSeries B+ · Oct 19$50M
VesslaPre-seed · Oct 19$3.3M
SupaHumanSeries A · Oct 19$1.5M
BoxLiteSeed · Oct 19undisclosed
RapidCanvasSeries B+ · Oct 16$29M

Recently funded

Topics in this field

coding agents · code generation · coding agent270 papers78 this week
skills · tasks · ai297 papers61 this week
recursive self improvement · harness evolution · adaptive252 papers48 this week
tool using agents · agent systems · multi agent198 papers45 this week
ai agents · indirect prompt injection · adversarial300 papers41 this week
agent · agentic · llm agents155 papers32 this week

Topics in this field

Active contributors · 90 days

Active contributors

Activity · 12 weeks

Coming with more sources: repos

Activity

★ Spotlight this month

arXiv cs.AICodeOct 15▲ 25

MLExpert: Towards a Generalist Agentic Model for Autonomous Machine Learning Engineering

Shangheng Du, Jinxin Shi, Shuaiyu Zhang, Yiming Wu, Junpeng Zhao +5

Large language models (LLMs) are expanding from assisting scientific research and engineering to autonomously developing and improving AI systems, advancing AI for AI (AI4AI) as a pathway toward recursive self-improvement (RSI). Workflow-driven machine learning engineering (MLE) systems provide a practical path toward this pursuit through complex orchestration that manages long-horizon experimentation, while recent MLE systems adopt model-directed interaction, allowing models to autonomously organize experiments and adapt optimization strategies based on execution feedback. However, feedback from each experiment reflects several preceding decisions, and a long MLE run contains many such costly experiments. Training must therefore select useful supervision from long interaction trajectories and learn from intermediate outcomes. We propose a post-training method for autonomous long-horizon MLE optimization and apply it to train MLExpert, a 35B Mixture-of-Experts agentic model. We design the Kaggle-MLE Factory to construct executable tasks and an interaction harness for model-directed experimentation. Hierarchical agentic SFT transfers general tool-use and coding capabilities to MLE…

PDF arXiv page Details & code

Spotlight

Recent papers

All in Research →

arXiv cs.LGCodeOct 19▲ 25
HarnessPro: Recursive Harness Self-Improvement with Pareto Optimization for Performance-Cost Trade-offs
Zhuoshi Pan, ZhiYuan Feng, Yukuo Cen, Honglin Lin, Junru Lu +4
Harness development plays a crucial role in building capable AI agents. Recent approaches improve harnesses through recursive self-improvement (RSI), but their performance gains often come with higher execution costs. We introduce HarnessPro, a label-free method for evolving harnesses to improve task performance while reducing execution cost. In this setting, candidate harnesses are evaluated using LLM-generated tests without access to ground-truth solutions or official tests. Specialized agents diagnose execution trajectories, plan updates with different performance-cost priorities, and implement them as candidate harnesses. Rather than returning a single harness, HarnessPro uses normalized weighted squared-distance selection to provide cost-focused (lite), balanced (medium), and performance-focused (heavy) harnesses for different deployment budgets. Selected harnesses enter the next iteration, where execution feedback guides further updates. Our experiments cover Terminal-Bench 2, SWE-Bench Pro, and ProgramBench, using GPT-5.5 with Codex and GLM-5.2 with Claude Code. Across these benchmarks using GPT-5.5 with Codex, the heavy mode of HarnessPro achieves an average pass-rate…
PDF arXiv Code
arXiv cs.CLCodeOct 19▲ 27
AdaExploreBench: Benchmarking LLM Agents for Proactive Exploration and Test-time Adaptation in Evolving Environments
Rui Wang, Zishen Lai, Jiayu Liu, Cheng Qian, Jeonghwan Kim +13
Large Language Model (LLM) agents are increasingly deployed in unfamiliar and evolving environments, requiring them to proactively explore, understand, and adapt to environmental dynamics. However, existing benchmarks rarely test whether agents can proactively discover how an environment works and adapt as it evolves. To address this gap, we introduce AdaExploreBench, a benchmark for proactive exploration and test-time adaptation in evolving environments. By instantiating environmental dynamics as hidden rules, AdaExploreBench requires agents to actively acquire evidence and infer the rules. Under implicit rule drift, the latent rule changes without announcement, requiring agents to detect from degraded performance that prior knowledge is stale and re-explore. Experiments on eight leading LLMs show that proactive and adaptive exploration remain challenging: GPT-6 Sol achieves only 61.9% calibrated accuracy, while providing sufficient evidence or removing rule drift improves average performance significantly by close to 10%. These results reveal a critical bottleneck in current LLM agents: they can often reason effectively from informative evidence once it is available, but…
PDF arXiv Code
arXiv cs.AICodeOct 18▲ 6
StopLoss: Teaching Agents to Predict Failure and Stop Early
Jiarong Liang, Xueguang Ma, Dongfu Jiang, Yuxuan Zhang, Ping Nie +1
Coding agents keep working on runs that will fail verification, and asking a frontier model to watch for this is expensive in its own right. We introduce StopLoss, a small external monitor that learns failure recognition from completed trajectories and their verifier outcomes. The name is inspired by the stop-loss order in finance: StopLoss ends a run early when the run is likely to fail, which saves resources. Final outcomes label earlier local windows, so a fine-tuned language model can score the most recent tool steps in a single forward pass, and a threshold calibrated on successful runs turns that score into an abort decision without changing the executor. Trained on GPT-family SWE-rebench trajectories and evaluated on SWE-bench Verified and a SWE-bench Pro subset with GPT and Claude executors, StopLoss is compared against six general-purpose judges reading identical windows. Outcome supervision lifts failure prediction from near chance to a useful signal at the first check, at one to two orders of magnitude lower cost per check than any tested judge. This translates into savings: relative to completing every task, StopLoss lowers cost per retained solve by 4.5\% on Verified…
PDF arXiv Code
arXiv cs.AICodeOct 17▲ 5
HSPR: Recovering Software-Agent Benchmark Scores via Hierarchical Posterior Inference
Wanyi Chen, Wanqin Shen, Yuanchen Shi, Zhensu Sun, Fang Kong
Evaluating software agents on a full benchmark is expensive. A small task subset can preserve agent rankings while yielding inaccurate scores. Historical responses provide useful structure, but tasks with identical source responses can have different outcomes for a new target agent. We introduce Hierarchical Score Posterior Recovery (HSPR), which combines source-fitted response factors with uncertain group success rates. Observed target outcomes update the posteriors over the target latent state and group rates; posterior-guided acquisition selects subsequent tasks. Its score-based criterion equals the expected one-step reduction in full-score posterior variance. Observed successes and posterior-predictive counts for unqueried tasks yield a full-score posterior. The final estimate uses its median with source-selected calibration. At a 10% task budget on four task banks from the SWE-bench family, HSPR achieves the lowest mean absolute error (MAE) on three banks and the lowest maximum absolute error (MaxErr) on all four among the methods compared. Removing group uncertainty or replacing adaptive queries increases macro MAE in the respective controls. Macro MAE decreases across…
PDF arXiv Code
arXiv cs.CLCodeOct 16▲ 122
OR-License: Evaluating LLM Agents as Real-World Operations Research Practitioners
Peisong Wang, Feixue Liu, Yize Meng, Bowen Liu, Ruoyu Chen +14
Can large language model (LLM) agents deliver expert-level operations research (OR) solutions directly from original client materials, without prior expert reformulation? Existing OR benchmarks typically begin with self-contained descriptions or predefined optimization tasks, underrepresenting the translation of original business materials into professionally validated solutions. We introduce OR-License, an end-to-end benchmark of 31 instances across 10 industrial scenarios from paid OR engagements totaling USD 3.68 million. It preserves original requirements, input data, deliverable formats, and information boundaries after de-identification. To study information acquisition, we introduce ORSim, a controlled client simulator that discloses business facts without supplying OR expertise. Hidden evaluation checks feasibility and compares objective quality against original expert solutions accepted by clients before benchmark construction. Deployment-Adjusted Quality (DAQ) averages expert-relative quality across instances, assigning zero to infeasible deliveries. Across 11 frontier LLMs using coding-agent harnesses, several exceed the expert reference in conditional objective…
PDF arXiv Code
arXiv cs.CLCodeOct 16▲ 17
DiscoveryRSI: Recursively Learning to Discover Valuable Hypotheses
Jianyuan Zhong, Chenwei Wu, Zhaochen Yu, Xixian Jiang, Xuewei Li +19
Scientific discovery requires developing worthwhile hypotheses into practical investigations, yet learning from this evolving research process remains challenging. We introduce an operational definition of valuable and testable research hypotheses: evidence-grounded claims about consequential uncertainties, coupled with feasible investigations, explicit decision criteria, and alternative explanations. Our DiscoveryRSI framework develops hypothesis trees through multi-agent evolutionary search, compiles selected investigations into executable Harbor tasks, and recursively learns through three mechanisms: hypothesis credit optimization attributes scientific assessments to documented research contributions; harness replay identifies productive strategies and recurring failures to refine research workflows; and rediscovery applies accepted improvements to new investigations, generating further experience for subsequent learning. We also introduce DFM-Bench, which provides comprehensive assessment of hypothesis quality and cumulative research progress at the levels of hypotheses, revisions, development paths and research runs. Across four life-science domains, automated evaluation…
PDF arXiv Code

Papers

Top tools here

Coming with more sources: stars history

All tools

Agno+17% downloads · 519.1k/wk+16.8%
n8n+25% downloads · 148.3k/wk+25.0%
LangGraph7 releases this week
DeepEval+15% downloads · 769.3k/wk+14.7%
OpenAI Python SDK7 releases this week

Topics in this field

coding agents · code generation · coding agent270 papers78 this week
skills · tasks · ai297 papers61 this week
recursive self improvement · harness evolution · adaptive252 papers48 this week
tool using agents · agent systems · multi agent198 papers45 this week
ai agents · indirect prompt injection · adversarial300 papers41 this week
agent · agentic · llm agents155 papers32 this week

Recently funded

Market

GreptileSeries B+ · Oct 19$50M
VesslaPre-seed · Oct 19$3.3M
SupaHumanSeries A · Oct 19$1.5M
BoxLiteSeed · Oct 19undisclosed
RapidCanvasSeries B+ · Oct 16$29M

★ Spotlight this month

arXiv cs.AICodeOct 15▲ 25

MLExpert: Towards a Generalist Agentic Model for Autonomous Machine Learning Engineering

Shangheng Du, Jinxin Shi, Shuaiyu Zhang, Yiming Wu, Junpeng Zhao +5

Large language models (LLMs) are expanding from assisting scientific research and engineering to autonomously developing and improving AI systems, advancing AI for AI (AI4AI) as a pathway toward recursive self-improvement (RSI). Workflow-driven machine learning engineering (MLE) systems provide a practical path toward this pursuit through complex orchestration that manages long-horizon experimentation, while recent MLE systems adopt model-directed interaction, allowing models to autonomously organize experiments and adapt optimization strategies based on execution feedback. However, feedback from each experiment reflects several preceding decisions, and a long MLE run contains many such costly experiments. Training must therefore select useful supervision from long interaction trajectories and learn from intermediate outcomes. We propose a post-training method for autonomous long-horizon MLE optimization and apply it to train MLExpert, a 35B Mixture-of-Experts agentic model. We design the Kaggle-MLE Factory to construct executable tasks and an interaction harness for model-directed experimentation. Hierarchical agentic SFT transfers general tool-use and coding capabilities to MLE…

PDF arXiv page Details & code

Active contributors · 90 days

Recent papers

All in Research →

arXiv cs.LGCodeOct 19▲ 25
HarnessPro: Recursive Harness Self-Improvement with Pareto Optimization for Performance-Cost Trade-offs
Zhuoshi Pan, ZhiYuan Feng, Yukuo Cen, Honglin Lin, Junru Lu +4
Harness development plays a crucial role in building capable AI agents. Recent approaches improve harnesses through recursive self-improvement (RSI), but their performance gains often come with higher execution costs. We introduce HarnessPro, a label-free method for evolving harnesses to improve task performance while reducing execution cost. In this setting, candidate harnesses are evaluated using LLM-generated tests without access to ground-truth solutions or official tests. Specialized agents diagnose execution trajectories, plan updates with different performance-cost priorities, and implement them as candidate harnesses. Rather than returning a single harness, HarnessPro uses normalized weighted squared-distance selection to provide cost-focused (lite), balanced (medium), and performance-focused (heavy) harnesses for different deployment budgets. Selected harnesses enter the next iteration, where execution feedback guides further updates. Our experiments cover Terminal-Bench 2, SWE-Bench Pro, and ProgramBench, using GPT-5.5 with Codex and GLM-5.2 with Claude Code. Across these benchmarks using GPT-5.5 with Codex, the heavy mode of HarnessPro achieves an average pass-rate…
PDF arXiv Code
arXiv cs.CLCodeOct 19▲ 27
AdaExploreBench: Benchmarking LLM Agents for Proactive Exploration and Test-time Adaptation in Evolving Environments
Rui Wang, Zishen Lai, Jiayu Liu, Cheng Qian, Jeonghwan Kim +13
Large Language Model (LLM) agents are increasingly deployed in unfamiliar and evolving environments, requiring them to proactively explore, understand, and adapt to environmental dynamics. However, existing benchmarks rarely test whether agents can proactively discover how an environment works and adapt as it evolves. To address this gap, we introduce AdaExploreBench, a benchmark for proactive exploration and test-time adaptation in evolving environments. By instantiating environmental dynamics as hidden rules, AdaExploreBench requires agents to actively acquire evidence and infer the rules. Under implicit rule drift, the latent rule changes without announcement, requiring agents to detect from degraded performance that prior knowledge is stale and re-explore. Experiments on eight leading LLMs show that proactive and adaptive exploration remain challenging: GPT-6 Sol achieves only 61.9% calibrated accuracy, while providing sufficient evidence or removing rule drift improves average performance significantly by close to 10%. These results reveal a critical bottleneck in current LLM agents: they can often reason effectively from informative evidence once it is available, but…
PDF arXiv Code
arXiv cs.AICodeOct 18▲ 6
StopLoss: Teaching Agents to Predict Failure and Stop Early
Jiarong Liang, Xueguang Ma, Dongfu Jiang, Yuxuan Zhang, Ping Nie +1
Coding agents keep working on runs that will fail verification, and asking a frontier model to watch for this is expensive in its own right. We introduce StopLoss, a small external monitor that learns failure recognition from completed trajectories and their verifier outcomes. The name is inspired by the stop-loss order in finance: StopLoss ends a run early when the run is likely to fail, which saves resources. Final outcomes label earlier local windows, so a fine-tuned language model can score the most recent tool steps in a single forward pass, and a threshold calibrated on successful runs turns that score into an abort decision without changing the executor. Trained on GPT-family SWE-rebench trajectories and evaluated on SWE-bench Verified and a SWE-bench Pro subset with GPT and Claude executors, StopLoss is compared against six general-purpose judges reading identical windows. Outcome supervision lifts failure prediction from near chance to a useful signal at the first check, at one to two orders of magnitude lower cost per check than any tested judge. This translates into savings: relative to completing every task, StopLoss lowers cost per retained solve by 4.5\% on Verified…
PDF arXiv Code
arXiv cs.AICodeOct 17▲ 5
HSPR: Recovering Software-Agent Benchmark Scores via Hierarchical Posterior Inference
Wanyi Chen, Wanqin Shen, Yuanchen Shi, Zhensu Sun, Fang Kong
Evaluating software agents on a full benchmark is expensive. A small task subset can preserve agent rankings while yielding inaccurate scores. Historical responses provide useful structure, but tasks with identical source responses can have different outcomes for a new target agent. We introduce Hierarchical Score Posterior Recovery (HSPR), which combines source-fitted response factors with uncertain group success rates. Observed target outcomes update the posteriors over the target latent state and group rates; posterior-guided acquisition selects subsequent tasks. Its score-based criterion equals the expected one-step reduction in full-score posterior variance. Observed successes and posterior-predictive counts for unqueried tasks yield a full-score posterior. The final estimate uses its median with source-selected calibration. At a 10% task budget on four task banks from the SWE-bench family, HSPR achieves the lowest mean absolute error (MAE) on three banks and the lowest maximum absolute error (MaxErr) on all four among the methods compared. Removing group uncertainty or replacing adaptive queries increases macro MAE in the respective controls. Macro MAE decreases across…
PDF arXiv Code
arXiv cs.CLCodeOct 16▲ 122
OR-License: Evaluating LLM Agents as Real-World Operations Research Practitioners
Peisong Wang, Feixue Liu, Yize Meng, Bowen Liu, Ruoyu Chen +14
Can large language model (LLM) agents deliver expert-level operations research (OR) solutions directly from original client materials, without prior expert reformulation? Existing OR benchmarks typically begin with self-contained descriptions or predefined optimization tasks, underrepresenting the translation of original business materials into professionally validated solutions. We introduce OR-License, an end-to-end benchmark of 31 instances across 10 industrial scenarios from paid OR engagements totaling USD 3.68 million. It preserves original requirements, input data, deliverable formats, and information boundaries after de-identification. To study information acquisition, we introduce ORSim, a controlled client simulator that discloses business facts without supplying OR expertise. Hidden evaluation checks feasibility and compares objective quality against original expert solutions accepted by clients before benchmark construction. Deployment-Adjusted Quality (DAQ) averages expert-relative quality across instances, assigning zero to infeasible deliveries. Across 11 frontier LLMs using coding-agent harnesses, several exceed the expert reference in conditional objective…
PDF arXiv Code
arXiv cs.CLCodeOct 16▲ 17
DiscoveryRSI: Recursively Learning to Discover Valuable Hypotheses
Jianyuan Zhong, Chenwei Wu, Zhaochen Yu, Xixian Jiang, Xuewei Li +19
Scientific discovery requires developing worthwhile hypotheses into practical investigations, yet learning from this evolving research process remains challenging. We introduce an operational definition of valuable and testable research hypotheses: evidence-grounded claims about consequential uncertainties, coupled with feasible investigations, explicit decision criteria, and alternative explanations. Our DiscoveryRSI framework develops hypothesis trees through multi-agent evolutionary search, compiles selected investigations into executable Harbor tasks, and recursively learns through three mechanisms: hypothesis credit optimization attributes scientific assessments to documented research contributions; harness replay identifies productive strategies and recurring failures to refine research workflows; and rediscovery applies accepted improvements to new investigations, generating further experience for subsequent learning. We also introduce DFM-Bench, which provides comprehensive assessment of hypothesis quality and cumulative research progress at the levels of hypotheses, revisions, development paths and research runs. Across four life-science domains, automated evaluation…
PDF arXiv Code

Activity · 12 weeks

Coming with more sources: repos

Fields · AI · updated 33m ago

Agents

LLM agents: planning, tool use, multi-agent systems, computer/web agents and agent benchmarks.

500papers this week+0.4% vs last week

All papers · 2.4k With code · 109

Newest Most relevant

2.3k papers this month

Open problems in Agents

Evaluating long-horizon agentic behavior in real-world tasks4 papers raise it · 3 address it
Current benchmarks often simplify or omit the exploration, strategy adjustment, and verification needed for sustained autonomous work.
- Scaling Agentic Data for Long-Horizon Terminal Intelligenceraises · addresses
- FDE-Gym: The Last Mile of AI Deliveryraises · addresses
- AgentSWE: Can Coding Agents Build the Agent You Actually Want?raises · addresses
- Into the Unknown: Benchmarking AI agents on open scientific research problemsraises
Faithful agent development from natural-language requirements1 paper raises it · 1 addresses it
Agents often fail to deliver systems that truly meet the specified requirements, not just appear to.
- AgentSWE: Can Coding Agents Build the Agent You Actually Want?raises · addresses
Learning from experience in unfamiliar environments1 paper raises it · 1 addresses it
Agents struggle to acquire and retain new knowledge through interaction rather than relying on pretrained knowledge.
- Learn2Play Bench: How Well Do LLM Agents Learn from Experience in Unfamiliar Environments?raises · addresses
Detecting when past context matters in long conversations1 paper raises it · 0 address it
AI systems cannot reliably identify when historical information is relevant to the current interaction.
- RealCompanion: Benchmarking Human Understanding from Reasoning over Longitudinal Real-World Conversationsraises
Open and reproducible agent stacks for computer use1 paper raises it · 2 address it
Frontier computer-use agents are often closed, limiting systematic study and community collaboration.
- OpenGCUA: Toward a Fully Open Stack for Generalist Computer Useraises · addresses
- MAI-Gym: An Open All-in-One Platform for GUI Agent Evaluation and RL Training across Mobile and Computer Environmentsaddresses

Named by an AI from these papers' abstracts.