Papers

Field

Top tools here

Coming with more sources: stars history

All tools

Agno4 releases this week
FastMCP6 releases this week
n8n+25% downloads · 148.3k/wk+25.0%
DeepEval+15% downloads · 769.3k/wk+14.7%
Ragas+14% downloads · 319.9k/wk+13.6%

Top tools here

Open problems in Agents

Long-horizon task execution and evidence accumulation3 papers raise it · 2 address it
Agents struggle to sustain multi-step reasoning, adapt strategies, and turn intermediate results into final deliverables in real-world tasks.
- Scaling Agentic Data for Long-Horizon Terminal Intelligenceraises · addresses
- FDE-Gym: The Last Mile of AI Deliveryraises · addresses
- AgentSWE: Can Coding Agents Build the Agent You Actually Want?raises
Realistic evaluation of agentic learning from experience1 paper raises it · 1 addresses it
Existing benchmarks often test reasoning with pre-existing knowledge rather than learning from novel, unfamiliar environments.
- Learn2Play Bench: How Well Do LLM Agents Learn from Experience in Unfamiliar Environments?raises · addresses
Faithful agent development from natural-language requirements1 paper raises it · 1 addresses it
Coding agents often fail to deliver agents that truly meet the specified requirements, not just appear to.
- AgentSWE: Can Coding Agents Build the Agent You Actually Want?raises · addresses
Detecting when past context matters in long conversations1 paper raises it · 0 address it
AI systems cannot reliably identify when historical context is relevant for current interactions in long-term companionship.
- RealCompanion: Benchmarking Human Understanding from Reasoning over Longitudinal Real-World Conversationsraises
Scalable and realistic GUI agent training and evaluation1 paper raises it · 2 address it
High cost and scarcity of realistic interactive environments limit progress in GUI agent training and evaluation.
- MAI-Gym: An Open All-in-One Platform for GUI Agent Evaluation and RL Training across Mobile and Computer Environmentsraises · addresses
- OpenGCUA: Toward a Fully Open Stack for Generalist Computer Useaddresses

Named by an AI from these papers' abstracts.

Open problems

Recently funded

Market

Asymptote LabsSeed · Oct 20$5.5M
GreptileSeries B+ · Oct 19$50M
VesslaPre-seed · Oct 19$3.3M
SupaHumanSeries A · Oct 19$1.5M
BoxLiteSeed · Oct 19undisclosed

Recently funded

Topics in this field

coding agents · code generation · coding agent286 papers85 this week
skills · tasks · ai308 papers62 this week
recursive self improvement · harness evolution · adaptive260 papers50 this week
tool using agents · agent systems · multi agent209 papers48 this week
ai agents · indirect prompt injection · adversarial314 papers47 this week
vision language · safety evaluations · safety alignment115 papers38 this week

Topics in this field

Active contributors · 90 days

Active contributors

Activity · 12 weeks

Coming with more sources: repos

Activity

Recent papers

All in Research →

arXiv cs.CLCodeOct 1▲ 46
Science Utopia? Closed-Loop LLM Simulation of Academic Research Ecosystems
Yiqiao Jin, Yiyang Wang, Lucheng Fu, Bing He, Siheng Xiong +6
Scientific progress emerges from a longitudinal ecosystem in which researchers, institutions, funding agencies, collaboration networks, and the scientific literature co-evolve. As AI becomes increasingly involved throughout the scientific research cycle, understanding these interconnected and evolving processes becomes increasingly important. We introduce SciUtopia, a persistent, closed-loop LLM-agent simulation framework for studying academic research ecosystems. SciUtopia models interconnected scientific processes such as research-direction choice, collaboration, submission, peer review, resubmission, citation, funding, and researcher attrition, while maintaining evolving states across simulated years. Its configurable institutional mechanisms and information channels provide a controlled testbed for matched counterfactual experiments and targeted interventions. Across 61 simulation worlds, SciUtopia simulates over 40,000 researchers from 8,000 institutions, producing around 400,000 publication decisions and 1.2 million LLM-generated peer reviews. Using these longitudinal simulations, we find that rejection-driven resubmission substantially amplifies reviewer burden beyond…
PDF arXiv Code
arXiv cs.AICodeOct 1▲ 63
VeriHarness: Scaling Agentic Verification for Long-Horizon Tasks
Caiqi Zhang, Rujun Han, Zifeng Wang, Zoey CuiZhu, Nigel Collier +2
As LLM agents undertake increasingly complex, long-horizon tasks, verifying their outputs becomes increasingly challenging. We study how verification capability can be strengthened with a fixed base model, without access to reference answers or grading rubrics at test time. Repeated sampling yields multiple rollouts that can contain complementary correct claims, but we need a reliable verification mechanism to determine which claims to trust. We first find that disagreement often exposes correct alternatives, while consensus can conceal errors. These observations motivate VeriHarness, which turns the underlying LLM a generator uses into an agentic verifier by giving it a workspace, evidence tools, and reusable verification skills. A disagreement resolver checks competing claims against environmental evidence, while a consensus challenger tests shared claims and searches for omitted requirements. Their findings guide the selection and revision of the final artifact. Across five long-horizon workspace benchmarks and two frontier models, VeriHarness achieves the highest selection scores among the evaluated baselines. Evidence-backed revision further improves average performance,…
PDF arXiv Code
arXiv cs.LGCodeOct 1▲ 23
GUI-HARVEST: Self-Improving GUI Agents through Evidence-Driven Harness Evolution
Geyi Yang, Zikun Qu, Xiang Li, Zhiyong Wang, Min Zhang +2
The executable harness surrounding a GUI model determines how observations are assembled, actions are executed, and verification, recovery, and termination are controlled. Compared with harness optimization for non-GUI agents, automatically optimizing this harness poses three coupled challenges: reconciling model intent with observed visual effects, diagnosing failures under variable execution outcomes, and identifying recurrent failure patterns across tasks and translating them into reusable runtime changes. We introduce GUI-HARVEST, an automatic harness optimizer that enables self-improving GUI agents with frozen backbone models. First, to ground diagnosis in observed action effects, it aligns model outputs and executed actions with before-and-after screenshots, tying findings to specific interface transitions. Second, to account for execution variability, it treats repeated runs of the same task as a joint evidence unit, using within-task comparisons to locate outcome-relevant behavioral differences. Third, it consolidates verified findings across tasks into recurring failure patterns, maps them to bounded source-code edits with predictions recorded before evaluation, and…
PDF arXiv Code
arXiv cs.AICodeOct 1
Finding the Right Fit: Model-Harness Interactions across Agent Tasks
Yixuan Li, Yiyun Zhou, Yao Long Teng, Fuchao Yang, Yanchen Deng +4
Choosing an agent system means choosing both a language model and the harness through which it acts. We ask whether a strong model, harness, or pairing stays strong when the setting changes. We evaluate 66 configurations: four configurable harnesses (OpenHands, DeepSeek Harness, PI, and openJiuwen) paired with five models on TUA-Bench, ALE-CLI, and Terminal-Bench 4, plus the native Codex-GPT and Claude Code-Claude pairings. Model rankings reverse across harnesses. On Terminal-Bench 4, Claude leads GPT by 7.94 points in OpenHands but trails it by 30.16 points in PI. For four of the five models, the best harness changes from one benchmark to another, yet some pairings hold: openJiuwen gives Kimi its highest score on all three benchmarks, by 5.61 to 11.11 points. A model's own vendor harness is not reliably its best, and higher cost does not reliably buy a higher score. On Terminal-Bench 4, GPT scores higher under PI than under DSH at less than a quarter of the cost per task. Matched trajectories suggest why fit varies. Models start almost all repairs themselves, so much depends on whether the harness hands failures back in a form the model can use. GPT does best with PI's lean…
PDF arXiv Code
arXiv cs.AICodeSep 30
Do Self-Evolving Skills Generalize to Held-Out Tasks?
Xihao Piao, Zifeng Wang, Zhen Chen
AI agents can externalize what they learn from past tasks into reusable \emph{skills}, such as procedures, checklists, code, or other executable artifacts, that can be retrieved and reused when solving new tasks. Self-evolving skill methods keep rewriting these skills after each round of practice on training tasks, and the skill is then used on new tasks of the same kind. We ask a question: does the improvement a skill shows on its training tasks carry over to new test tasks? We test five self-evolving methods and a one-shot skill on six benchmarks, with the same model, the same agent, and the same train/test split for every method. Of the 21 skills that improve on their training tasks, 5 keep all of that improvement on the test tasks, 13 keep part of it, and 3 keep none of it. No existing method is best everywhere. When we read the skills, the ones that carry over badly often fix details that should depend on the task, such as column names and output files, or turn a fix for one failure into a rule for every task. An LLM judge that reads the skill content can often see this: it ranks finished skills the same way the test results do in 86\% of pairs. But it predicts the effect of…
PDF arXiv Code
arXiv cs.AICodeSep 29
Harness Evolution as Learning: Approximation, Generalization, and Optimization Limits of Self-Improving Personal Agents
Zeyu Gan, Zixuan Gong, Yong Liu
As the capabilities of large language models (LLMs) continue to advance, increasing attention is turning to how to translate their abilities into useful behavior. Personal agents bring this question into everyday settings, where models are expected to serve individual users and continually adapt to their preferences. With the underlying model held fixed, such adaptation relies on harness engineering: designing and evolving the surrounding layer that manages context, memory, tools, and execution. Despite rapid progress, the factors governing effective harness evolution remain insufficiently understood. To narrow this gap, we investigate three central questions concerning harness architecture, harness scale, and self-evolution algorithms through complementary empirical and theoretical analyses. Empirically, we introduce a preference-oriented benchmark and systematically characterize the capabilities and limitations of personal agents associated with these three dimensions. Theoretically, we formulate harness evolution as a learning problem and explain these phenomena through approximation, generalization, and optimization errors. Analyses of reachable policies, capacity under…
PDF arXiv Code

Papers

Recently funded

Market

Asymptote LabsSeed · Oct 20$5.5M
GreptileSeries B+ · Oct 19$50M
VesslaPre-seed · Oct 19$3.3M
SupaHumanSeries A · Oct 19$1.5M
BoxLiteSeed · Oct 19undisclosed

Top tools here

Coming with more sources: stars history

All tools

Agno4 releases this week
FastMCP6 releases this week
n8n+25% downloads · 148.3k/wk+25.0%
DeepEval+15% downloads · 769.3k/wk+14.7%
Ragas+14% downloads · 319.9k/wk+13.6%

Topics in this field

coding agents · code generation · coding agent286 papers85 this week
skills · tasks · ai308 papers62 this week
recursive self improvement · harness evolution · adaptive260 papers50 this week
tool using agents · agent systems · multi agent209 papers48 this week
ai agents · indirect prompt injection · adversarial314 papers47 this week
vision language · safety evaluations · safety alignment115 papers38 this week

Active contributors · 90 days

Recent papers

All in Research →

arXiv cs.CLCodeOct 1▲ 46
Science Utopia? Closed-Loop LLM Simulation of Academic Research Ecosystems
Yiqiao Jin, Yiyang Wang, Lucheng Fu, Bing He, Siheng Xiong +6
Scientific progress emerges from a longitudinal ecosystem in which researchers, institutions, funding agencies, collaboration networks, and the scientific literature co-evolve. As AI becomes increasingly involved throughout the scientific research cycle, understanding these interconnected and evolving processes becomes increasingly important. We introduce SciUtopia, a persistent, closed-loop LLM-agent simulation framework for studying academic research ecosystems. SciUtopia models interconnected scientific processes such as research-direction choice, collaboration, submission, peer review, resubmission, citation, funding, and researcher attrition, while maintaining evolving states across simulated years. Its configurable institutional mechanisms and information channels provide a controlled testbed for matched counterfactual experiments and targeted interventions. Across 61 simulation worlds, SciUtopia simulates over 40,000 researchers from 8,000 institutions, producing around 400,000 publication decisions and 1.2 million LLM-generated peer reviews. Using these longitudinal simulations, we find that rejection-driven resubmission substantially amplifies reviewer burden beyond…
PDF arXiv Code
arXiv cs.AICodeOct 1▲ 63
VeriHarness: Scaling Agentic Verification for Long-Horizon Tasks
Caiqi Zhang, Rujun Han, Zifeng Wang, Zoey CuiZhu, Nigel Collier +2
As LLM agents undertake increasingly complex, long-horizon tasks, verifying their outputs becomes increasingly challenging. We study how verification capability can be strengthened with a fixed base model, without access to reference answers or grading rubrics at test time. Repeated sampling yields multiple rollouts that can contain complementary correct claims, but we need a reliable verification mechanism to determine which claims to trust. We first find that disagreement often exposes correct alternatives, while consensus can conceal errors. These observations motivate VeriHarness, which turns the underlying LLM a generator uses into an agentic verifier by giving it a workspace, evidence tools, and reusable verification skills. A disagreement resolver checks competing claims against environmental evidence, while a consensus challenger tests shared claims and searches for omitted requirements. Their findings guide the selection and revision of the final artifact. Across five long-horizon workspace benchmarks and two frontier models, VeriHarness achieves the highest selection scores among the evaluated baselines. Evidence-backed revision further improves average performance,…
PDF arXiv Code
arXiv cs.LGCodeOct 1▲ 23
GUI-HARVEST: Self-Improving GUI Agents through Evidence-Driven Harness Evolution
Geyi Yang, Zikun Qu, Xiang Li, Zhiyong Wang, Min Zhang +2
The executable harness surrounding a GUI model determines how observations are assembled, actions are executed, and verification, recovery, and termination are controlled. Compared with harness optimization for non-GUI agents, automatically optimizing this harness poses three coupled challenges: reconciling model intent with observed visual effects, diagnosing failures under variable execution outcomes, and identifying recurrent failure patterns across tasks and translating them into reusable runtime changes. We introduce GUI-HARVEST, an automatic harness optimizer that enables self-improving GUI agents with frozen backbone models. First, to ground diagnosis in observed action effects, it aligns model outputs and executed actions with before-and-after screenshots, tying findings to specific interface transitions. Second, to account for execution variability, it treats repeated runs of the same task as a joint evidence unit, using within-task comparisons to locate outcome-relevant behavioral differences. Third, it consolidates verified findings across tasks into recurring failure patterns, maps them to bounded source-code edits with predictions recorded before evaluation, and…
PDF arXiv Code
arXiv cs.AICodeOct 1
Finding the Right Fit: Model-Harness Interactions across Agent Tasks
Yixuan Li, Yiyun Zhou, Yao Long Teng, Fuchao Yang, Yanchen Deng +4
Choosing an agent system means choosing both a language model and the harness through which it acts. We ask whether a strong model, harness, or pairing stays strong when the setting changes. We evaluate 66 configurations: four configurable harnesses (OpenHands, DeepSeek Harness, PI, and openJiuwen) paired with five models on TUA-Bench, ALE-CLI, and Terminal-Bench 4, plus the native Codex-GPT and Claude Code-Claude pairings. Model rankings reverse across harnesses. On Terminal-Bench 4, Claude leads GPT by 7.94 points in OpenHands but trails it by 30.16 points in PI. For four of the five models, the best harness changes from one benchmark to another, yet some pairings hold: openJiuwen gives Kimi its highest score on all three benchmarks, by 5.61 to 11.11 points. A model's own vendor harness is not reliably its best, and higher cost does not reliably buy a higher score. On Terminal-Bench 4, GPT scores higher under PI than under DSH at less than a quarter of the cost per task. Matched trajectories suggest why fit varies. Models start almost all repairs themselves, so much depends on whether the harness hands failures back in a form the model can use. GPT does best with PI's lean…
PDF arXiv Code
arXiv cs.AICodeSep 30
Do Self-Evolving Skills Generalize to Held-Out Tasks?
Xihao Piao, Zifeng Wang, Zhen Chen
AI agents can externalize what they learn from past tasks into reusable \emph{skills}, such as procedures, checklists, code, or other executable artifacts, that can be retrieved and reused when solving new tasks. Self-evolving skill methods keep rewriting these skills after each round of practice on training tasks, and the skill is then used on new tasks of the same kind. We ask a question: does the improvement a skill shows on its training tasks carry over to new test tasks? We test five self-evolving methods and a one-shot skill on six benchmarks, with the same model, the same agent, and the same train/test split for every method. Of the 21 skills that improve on their training tasks, 5 keep all of that improvement on the test tasks, 13 keep part of it, and 3 keep none of it. No existing method is best everywhere. When we read the skills, the ones that carry over badly often fix details that should depend on the task, such as column names and output files, or turn a fix for one failure into a rule for every task. An LLM judge that reads the skill content can often see this: it ranks finished skills the same way the test results do in 86\% of pairs. But it predicts the effect of…
PDF arXiv Code
arXiv cs.AICodeSep 29
Harness Evolution as Learning: Approximation, Generalization, and Optimization Limits of Self-Improving Personal Agents
Zeyu Gan, Zixuan Gong, Yong Liu
As the capabilities of large language models (LLMs) continue to advance, increasing attention is turning to how to translate their abilities into useful behavior. Personal agents bring this question into everyday settings, where models are expected to serve individual users and continually adapt to their preferences. With the underlying model held fixed, such adaptation relies on harness engineering: designing and evolving the surrounding layer that manages context, memory, tools, and execution. Despite rapid progress, the factors governing effective harness evolution remain insufficiently understood. To narrow this gap, we investigate three central questions concerning harness architecture, harness scale, and self-evolution algorithms through complementary empirical and theoretical analyses. Empirically, we introduce a preference-oriented benchmark and systematically characterize the capabilities and limitations of personal agents associated with these three dimensions. Theoretically, we formulate harness evolution as a learning problem and explain these phenomena through approximation, generalization, and optimization errors. Analyses of reachable policies, capacity under…
PDF arXiv Code

Activity · 12 weeks

Coming with more sources: repos

Fields · AI · updated 16m ago

Agents

LLM agents: planning, tool use, multi-agent systems, computer/web agents and agent benchmarks.

546papers this week+5.4% vs last week

All papers · 2.5k With code · 113

Newest Most relevant

2.4k papers this month

Open problems in Agents

Long-horizon task execution and evidence accumulation3 papers raise it · 2 address it
Agents struggle to sustain multi-step reasoning, adapt strategies, and turn intermediate results into final deliverables in real-world tasks.
- Scaling Agentic Data for Long-Horizon Terminal Intelligenceraises · addresses
- FDE-Gym: The Last Mile of AI Deliveryraises · addresses
- AgentSWE: Can Coding Agents Build the Agent You Actually Want?raises
Realistic evaluation of agentic learning from experience1 paper raises it · 1 addresses it
Existing benchmarks often test reasoning with pre-existing knowledge rather than learning from novel, unfamiliar environments.
- Learn2Play Bench: How Well Do LLM Agents Learn from Experience in Unfamiliar Environments?raises · addresses
Faithful agent development from natural-language requirements1 paper raises it · 1 addresses it
Coding agents often fail to deliver agents that truly meet the specified requirements, not just appear to.
- AgentSWE: Can Coding Agents Build the Agent You Actually Want?raises · addresses
Detecting when past context matters in long conversations1 paper raises it · 0 address it
AI systems cannot reliably identify when historical context is relevant for current interactions in long-term companionship.
- RealCompanion: Benchmarking Human Understanding from Reasoning over Longitudinal Real-World Conversationsraises
Scalable and realistic GUI agent training and evaluation1 paper raises it · 2 address it
High cost and scarcity of realistic interactive environments limit progress in GUI agent training and evaluation.
- MAI-Gym: An Open All-in-One Platform for GUI Agent Evaluation and RL Training across Mobile and Computer Environmentsraises · addresses
- OpenGCUA: Toward a Fully Open Stack for Generalist Computer Useaddresses

Named by an AI from these papers' abstracts.