Papers

Field

Top tools here

All tools

Agno4 releases this week
FastMCP6 releases this week
n8n+25% downloads · 148.3k/wk+25.0%
DeepEval+15% downloads · 769.3k/wk+14.7%
Ragas+14% downloads · 319.9k/wk+13.6%

Top tools here

Open problems in Agents

Long-horizon task execution and memory3 papers raise it · 3 address it
Agents fail to retain, retrieve, and use past information over long interactions, limiting real-world deployment.
- RealCompanion: Benchmarking Human Understanding from Reasoning over Longitudinal Real-World Conversationsraises
- Scaling Agentic Data for Long-Horizon Terminal Intelligenceraises · addresses
- Learn2Play Bench: How Well Do LLM Agents Learn from Experience in Unfamiliar Environments?raises · addresses
- Agent-Reflex: Teaching Language Agents to Act Reflectivelyaddresses
Real-world task ambiguity and unstated constraints3 papers raise it · 2 address it
Agents struggle with ambiguous, incomplete, or evolving requirements in live systems, unlike controlled benchmarks.
- OR-License: Evaluating LLM Agents as Real-World Operations Research Practitionersraises · addresses
- FDE-Gym: The Last Mile of AI Deliveryraises · addresses
- AgentSWE: Can Coding Agents Build the Agent You Actually Want?raises
Faithful agent development from requirements1 paper raises it · 1 addresses it
Coding agents often produce systems that appear correct but fail to meet the actual requirements.
- AgentSWE: Can Coding Agents Build the Agent You Actually Want?raises · addresses
Sequential distraction in multimodal GUI agents1 paper raises it · 1 addresses it
Omni-modal inputs cause agents to focus on irrelevant signals, leading to compounding errors.
- Focused Omni-Modal GUI Agentsraises · addresses
Open, reproducible agent stacks and evaluation1 paper raises it · 1 addresses it
Closed systems and lack of open benchmarks hinder systematic study and improvement of agent designs.
- OpenGCUA: Toward a Fully Open Stack for Generalist Computer Useraises · addresses

Named by an AI from these papers' abstracts.

Open problems

Recently funded

Market

FirstWorkSeries A · Oct 21$15M
Asymptote LabsSeed · Oct 20$5.5M
LumilabPre-seed · Oct 20undisclosed
GreptileSeries B+ · Oct 19$50M
VesslaPre-seed · Oct 19$3.3M

Recently funded

Topics in this field

coding agents · code generation · coding agent286 papers74 this week
skills · tasks · ai308 papers57 this week
ai agents · indirect prompt injection · adversarial314 papers41 this week
recursive self improvement · harness evolution · adaptive260 papers40 this week
tool using agents · agent systems · multi agent209 papers40 this week
vision language · safety evaluations · safety alignment115 papers33 this week

Topics in this field

Active contributors · 90 days

Active contributors

Activity · 12 weeks

Activity

Recent papers

All papers in Research

arXiv cs.AIOct 15
Testing the Planning Claims of LLM-Based Agents against Formal Semantics
Oriol Miro-Lopez-Feliu, Oscar Miró López-Feliu, Victor Gimenez-Abalos, Javier Vazquez-Salceda, Victor Carballo Araruna +1
LLM-based agents are increasingly credited with the competences of practical reasoning: planning, monitoring, and correcting their own actions. Their success is attributed to the prompting strategies and agent architectures built around the model, such as chain-of-thought, search-based prompting, self-refinement, reflection, critics and tool-using loops. Yet these claims are usually supported by end-to-end success rates, which show only whether a task was completed, or by LLM judges, which are not verifiers. We test LLM planning against formal semantics. Tasks are posed in natural language, but each is grounded in a hidden PDDL model that decides whether every action is applicable, whether the goal is reached, and whether a plan exists at all. Across five domains (two of them new, all released with the paper), four models, six prompting strategies and three agent architectures, the best configuration solves 78% of the solvable tasks offline but only 40% online, where agents mostly act without inspecting the state. Unsolvable tasks are recognised in under 10% of episodes, and heavier prompting strategies use 5 to 22 times as many tokens as direct prompting with no clear gain. In…
PDF arXiv
arXiv cs.LGOct 15
CAT-alyzing Honesty: Continuous Adversarial Training for Transferable LLM Honesty
Yuan-Chen Chang, Niklas Herbster, Gauthier Gidel, Tommaso Tosato
Large language models (LLMs) are becoming prevalent in scenarios where honesty is critical. However, despite alignment training, these models still exhibit dishonest behaviour, such as outputting claims that contradict some knowledge or are not supported by what is accessible in context. We introduce Continuous Adversarial Training for Consistent Honesty (CATCH), inspired by a method originally developed to robustly refuse harmful queries. CATCH trains the model towards an honest answer and away from a dishonest one while maintaining the model's capabilities. We evaluate two architectures (Llama-3.3-70B-Instruct and Qwen3-3.6-27B). We show that trained with only hundreds of targets built from self-reporting scenarios, CATCH learns an honest behaviour that robustly generalizes and transfers across distinct benchmarks. It consistently reduces dishonesty under pressure on MASK, increases confessions in the multi-agent game of Among Us, and improves the discovery rates of the model's hidden behaviours in AuditBench. We show that CATCH outperforms prompting and supervised fine-tuning in improving honesty, and it is on par with state-of-the-art honesty activation steering. However,…
PDF arXiv
arXiv cs.AIOct 15
Improving Environment-Agnostic Agents: The Role of Scaffolding
Sergey Rodionov
Agents built on stronger language models may increasingly construct useful procedures for themselves, reducing the advantage of developer-supplied scaffolding. We investigate why this advantage persists and whether an environment-agnostic agent can be improved without environment-specific engineering. We evaluate a Codex-based agent in two nearly identical prompt variants on ARC-AGI-3, AGI Maze, and Terminal-Bench. Each variant is deployed with unchanged code, prompts, and general-purpose tools across environments, receiving environment-specific information as input. Changing only an instruction to encourage reporting difficulties and returning control for automatic continuation improves mean scores on both ARC-AGI-3 and AGI Maze, while Terminal-Bench shows small, mixed changes. Yet the general agent remains below an ARC-AGI-3-specific reference, including with GPT-5.6-sol. The reference prescribes organization and control rather than supplying new game rules or solutions. To investigate this gap, we transfer reset decisions to a scripted controller that follows a convention already given to the agent. This small change recovers part of the specialized agent's advantage. A…
PDF arXiv
arXiv cs.AIOct 15▲ 8
LEASH: Uncertainty-Aware Weak-to-Strong Escalation and Oversight for Computer-Use Agents
Qingni Wang, Changdae Oh, Yuzhe Yang, Sharon Li, Jiachen Yang +2
Computer-use agents increasingly tackle long-horizon tasks, where running a frontier model at every step is expensive and a small open-weight model alone often fails. Weak-to-strong escalation offers a middle ground where a weak agent acts by default and hands control to a strong one only when needed. The core challenge is deciding when to escalate, which requires an uncertainty signal cheap enough to check at every step. However, even after handoff, the strong agent can still get lost, but a black-box API that the strong agent is usually served exposes no internals to monitor. To this end, we propose LEASH, a training-free framework that unleashes the strong agent only when the weak agent's latent uncertainty signals it is needed, and keeps it on a leash afterward. LEASH has three layers: (1) initial task uncertainty for early routing, (2) step-level uncertainty for runtime escalation, and, after handoff, (3) a shadow observer for continued oversight of the strong black-box agent. Here, we use latent uncertainty as a fuel for all three layers. Specifically, we compare the weak model's hidden states with those from its past successful and failed trajectories, and a state that…
PDF arXiv
arXiv cs.AICodeOct 15▲ 3
Harness-Delta Attribution: Dissecting Gains From Harness Evolution
Wenxuan Ding, Qin Lu, Changlong Yu, Sha Li, Shuowei Jin +3
Harness evolution automatically optimizes the prompts, skills, tools, and orchestration code around a fixed-weight model, offering a way to improve agentic systems without updating model weights. The agent harness is iteratively optimized to achieve higher performance on the search set. But how much of this gain reflects benchmark-specific overfitting or additional inference-time compute, and how much may generalize beyond the search set? In this work, we introduce Harness-Delta Attribution (HDA), an evaluation protocol that dissects an evolution gain into artifact-driven overfitting, gains explained by test-time scaling, and the remainder, which gives an upper bound on residual generalizable improvement. We apply HDA to 11 executor models and three harness-evolution frameworks across math, coding, creativity, and agentic tasks. In 21 of these settings that show statistically significant search-set gains, we find that a large share of the improvement is explained by overfitting or test-time scaling. Specifically, among these 21 settings, overfitting and test-time scaling account for 54% and 14% of the gain on average; their combined share exceeds 50% in 14 settings. The residual…
PDF arXiv Code
arXiv cs.AIOct 15
Author-in-the-loop Agentic Reproducibility Auditing for a Machine Learning Conference
Yukun Dong, Linjun Zhang, Tian Zheng
Reproducibility checks during peer review require a shared understanding of what authors' code is expected to reproduce. We present an author-in-the-loop collaborative audit that makes author declarations explicit inputs to an AI agent's investigation and assessment. A structured checklist records reproduction instructions and limitations; a two-stage workflow on GitHub connects agent checks to author clarification, revision, and agent reassessment. The agent separately assesses whether results have been reproduced and whether authors need to provide clarification or make repairs, while human reviewers approve requests and decide whether concerns are resolved. We deployed the system across 36 conference papers and 133 experimental tables and figures, raising 20 issues after human review. Further checks supported closing all the issues, although full reproduction remained unverified in some cases. We further compared repeated audits with complete and blank checklists to examine how author declarations shaped the agent's investigation and assessment. Our work contributes a collaborative workflow and agent design that use author knowledge to guide reproducibility auditing. The…
PDF arXiv

Papers

Top tools here

All tools

Agno4 releases this week
FastMCP6 releases this week
n8n+25% downloads · 148.3k/wk+25.0%
DeepEval+15% downloads · 769.3k/wk+14.7%
Ragas+14% downloads · 319.9k/wk+13.6%

Recently funded

Market

FirstWorkSeries A · Oct 21$15M
Asymptote LabsSeed · Oct 20$5.5M
LumilabPre-seed · Oct 20undisclosed
GreptileSeries B+ · Oct 19$50M
VesslaPre-seed · Oct 19$3.3M

Topics in this field

coding agents · code generation · coding agent286 papers74 this week
skills · tasks · ai308 papers57 this week
ai agents · indirect prompt injection · adversarial314 papers41 this week
recursive self improvement · harness evolution · adaptive260 papers40 this week
tool using agents · agent systems · multi agent209 papers40 this week
vision language · safety evaluations · safety alignment115 papers33 this week

Active contributors · 90 days

Recent papers

All papers in Research

arXiv cs.AIOct 15
Testing the Planning Claims of LLM-Based Agents against Formal Semantics
Oriol Miro-Lopez-Feliu, Oscar Miró López-Feliu, Victor Gimenez-Abalos, Javier Vazquez-Salceda, Victor Carballo Araruna +1
LLM-based agents are increasingly credited with the competences of practical reasoning: planning, monitoring, and correcting their own actions. Their success is attributed to the prompting strategies and agent architectures built around the model, such as chain-of-thought, search-based prompting, self-refinement, reflection, critics and tool-using loops. Yet these claims are usually supported by end-to-end success rates, which show only whether a task was completed, or by LLM judges, which are not verifiers. We test LLM planning against formal semantics. Tasks are posed in natural language, but each is grounded in a hidden PDDL model that decides whether every action is applicable, whether the goal is reached, and whether a plan exists at all. Across five domains (two of them new, all released with the paper), four models, six prompting strategies and three agent architectures, the best configuration solves 78% of the solvable tasks offline but only 40% online, where agents mostly act without inspecting the state. Unsolvable tasks are recognised in under 10% of episodes, and heavier prompting strategies use 5 to 22 times as many tokens as direct prompting with no clear gain. In…
PDF arXiv
arXiv cs.LGOct 15
CAT-alyzing Honesty: Continuous Adversarial Training for Transferable LLM Honesty
Yuan-Chen Chang, Niklas Herbster, Gauthier Gidel, Tommaso Tosato
Large language models (LLMs) are becoming prevalent in scenarios where honesty is critical. However, despite alignment training, these models still exhibit dishonest behaviour, such as outputting claims that contradict some knowledge or are not supported by what is accessible in context. We introduce Continuous Adversarial Training for Consistent Honesty (CATCH), inspired by a method originally developed to robustly refuse harmful queries. CATCH trains the model towards an honest answer and away from a dishonest one while maintaining the model's capabilities. We evaluate two architectures (Llama-3.3-70B-Instruct and Qwen3-3.6-27B). We show that trained with only hundreds of targets built from self-reporting scenarios, CATCH learns an honest behaviour that robustly generalizes and transfers across distinct benchmarks. It consistently reduces dishonesty under pressure on MASK, increases confessions in the multi-agent game of Among Us, and improves the discovery rates of the model's hidden behaviours in AuditBench. We show that CATCH outperforms prompting and supervised fine-tuning in improving honesty, and it is on par with state-of-the-art honesty activation steering. However,…
PDF arXiv
arXiv cs.AIOct 15
Improving Environment-Agnostic Agents: The Role of Scaffolding
Sergey Rodionov
Agents built on stronger language models may increasingly construct useful procedures for themselves, reducing the advantage of developer-supplied scaffolding. We investigate why this advantage persists and whether an environment-agnostic agent can be improved without environment-specific engineering. We evaluate a Codex-based agent in two nearly identical prompt variants on ARC-AGI-3, AGI Maze, and Terminal-Bench. Each variant is deployed with unchanged code, prompts, and general-purpose tools across environments, receiving environment-specific information as input. Changing only an instruction to encourage reporting difficulties and returning control for automatic continuation improves mean scores on both ARC-AGI-3 and AGI Maze, while Terminal-Bench shows small, mixed changes. Yet the general agent remains below an ARC-AGI-3-specific reference, including with GPT-5.6-sol. The reference prescribes organization and control rather than supplying new game rules or solutions. To investigate this gap, we transfer reset decisions to a scripted controller that follows a convention already given to the agent. This small change recovers part of the specialized agent's advantage. A…
PDF arXiv
arXiv cs.AIOct 15▲ 8
LEASH: Uncertainty-Aware Weak-to-Strong Escalation and Oversight for Computer-Use Agents
Qingni Wang, Changdae Oh, Yuzhe Yang, Sharon Li, Jiachen Yang +2
Computer-use agents increasingly tackle long-horizon tasks, where running a frontier model at every step is expensive and a small open-weight model alone often fails. Weak-to-strong escalation offers a middle ground where a weak agent acts by default and hands control to a strong one only when needed. The core challenge is deciding when to escalate, which requires an uncertainty signal cheap enough to check at every step. However, even after handoff, the strong agent can still get lost, but a black-box API that the strong agent is usually served exposes no internals to monitor. To this end, we propose LEASH, a training-free framework that unleashes the strong agent only when the weak agent's latent uncertainty signals it is needed, and keeps it on a leash afterward. LEASH has three layers: (1) initial task uncertainty for early routing, (2) step-level uncertainty for runtime escalation, and, after handoff, (3) a shadow observer for continued oversight of the strong black-box agent. Here, we use latent uncertainty as a fuel for all three layers. Specifically, we compare the weak model's hidden states with those from its past successful and failed trajectories, and a state that…
PDF arXiv
arXiv cs.AICodeOct 15▲ 3
Harness-Delta Attribution: Dissecting Gains From Harness Evolution
Wenxuan Ding, Qin Lu, Changlong Yu, Sha Li, Shuowei Jin +3
Harness evolution automatically optimizes the prompts, skills, tools, and orchestration code around a fixed-weight model, offering a way to improve agentic systems without updating model weights. The agent harness is iteratively optimized to achieve higher performance on the search set. But how much of this gain reflects benchmark-specific overfitting or additional inference-time compute, and how much may generalize beyond the search set? In this work, we introduce Harness-Delta Attribution (HDA), an evaluation protocol that dissects an evolution gain into artifact-driven overfitting, gains explained by test-time scaling, and the remainder, which gives an upper bound on residual generalizable improvement. We apply HDA to 11 executor models and three harness-evolution frameworks across math, coding, creativity, and agentic tasks. In 21 of these settings that show statistically significant search-set gains, we find that a large share of the improvement is explained by overfitting or test-time scaling. Specifically, among these 21 settings, overfitting and test-time scaling account for 54% and 14% of the gain on average; their combined share exceeds 50% in 14 settings. The residual…
PDF arXiv Code
arXiv cs.AIOct 15
Author-in-the-loop Agentic Reproducibility Auditing for a Machine Learning Conference
Yukun Dong, Linjun Zhang, Tian Zheng
Reproducibility checks during peer review require a shared understanding of what authors' code is expected to reproduce. We present an author-in-the-loop collaborative audit that makes author declarations explicit inputs to an AI agent's investigation and assessment. A structured checklist records reproduction instructions and limitations; a two-stage workflow on GitHub connects agent checks to author clarification, revision, and agent reassessment. The agent separately assesses whether results have been reproduced and whether authors need to provide clarification or make repairs, while human reviewers approve requests and decide whether concerns are resolved. We deployed the system across 36 conference papers and 133 experimental tables and figures, raising 20 issues after human review. Further checks supported closing all the issues, although full reproduction remained unverified in some cases. We further compared repeated audits with complete and blank checklists to examine how author declarations shaped the agent's investigation and assessment. Our work contributes a collaborative workflow and agent design that use author knowledge to guide reproducibility auditing. The…
PDF arXiv

Activity · 12 weeks

Fields · AI · updated 9m ago

Agents

LLM agents: planning, tool use, multi-agent systems, computer/web agents and agent benchmarks.

480papers this week−9.8% vs last week

All papers · 2.5k With code · 116

Newest Most relevant

2.4k papers this month

Open problems in Agents

Long-horizon task execution and memory3 papers raise it · 3 address it
Agents fail to retain, retrieve, and use past information over long interactions, limiting real-world deployment.
- RealCompanion: Benchmarking Human Understanding from Reasoning over Longitudinal Real-World Conversationsraises
- Scaling Agentic Data for Long-Horizon Terminal Intelligenceraises · addresses
- Learn2Play Bench: How Well Do LLM Agents Learn from Experience in Unfamiliar Environments?raises · addresses
- Agent-Reflex: Teaching Language Agents to Act Reflectivelyaddresses
Real-world task ambiguity and unstated constraints3 papers raise it · 2 address it
Agents struggle with ambiguous, incomplete, or evolving requirements in live systems, unlike controlled benchmarks.
- OR-License: Evaluating LLM Agents as Real-World Operations Research Practitionersraises · addresses
- FDE-Gym: The Last Mile of AI Deliveryraises · addresses
- AgentSWE: Can Coding Agents Build the Agent You Actually Want?raises
Faithful agent development from requirements1 paper raises it · 1 addresses it
Coding agents often produce systems that appear correct but fail to meet the actual requirements.
- AgentSWE: Can Coding Agents Build the Agent You Actually Want?raises · addresses
Sequential distraction in multimodal GUI agents1 paper raises it · 1 addresses it
Omni-modal inputs cause agents to focus on irrelevant signals, leading to compounding errors.
- Focused Omni-Modal GUI Agentsraises · addresses
Open, reproducible agent stacks and evaluation1 paper raises it · 1 addresses it
Closed systems and lack of open benchmarks hinder systematic study and improvement of agent designs.
- OpenGCUA: Toward a Fully Open Stack for Generalist Computer Useraises · addresses

Named by an AI from these papers' abstracts.