Papers

Field

Top tools here

Open problems in AI agents

Evaluating long-horizon agentic behavior in real-world tasks3 papers raise it · 2 address it
Current benchmarks often simplify or omit the key work agents must perform, making it hard to assess sustained autonomous problem-solving.
- Scaling Agentic Data for Long-Horizon Terminal Intelligenceraises · addresses
- FDE-Gym: The Last Mile of AI Deliveryraises · addresses
- AgentSWE: Can Coding Agents Build the Agent You Actually Want?raises
Faithful agent development from natural-language requirements1 paper raises it · 1 addresses it
Agents often fail to deliver what users actually ask for, passing superficial tests but not meeting the full requirement.
- AgentSWE: Can Coding Agents Build the Agent You Actually Want?raises · addresses
Detecting when past context matters in long conversations1 paper raises it · 0 address it
AI companions struggle to recognize when historical context is relevant, leading to missed or incorrect responses.
- RealCompanion: Benchmarking Human Understanding from Reasoning over Longitudinal Real-World Conversationsraises
Reliable no-thinking compliance in LLMs1 paper raises it · 1 addresses it
LLMs often ignore or bypass explicit instructions to avoid reasoning, making it hard to enforce strict answer-only behavior.
- Thinking Inertia: LLMs Keep Thinking When Told Not Toraises · addresses
Transition-aware safety for GUI agents1 paper raises it · 1 addresses it
Safety checks that only inspect screens can be fooled by reusing legitimate screens in unexpected transitions.
- Right Screen, Wrong Transition: World Models as Verifiers for GUI Agentsraises · addresses

Named by an AI from these papers' abstracts.

Open problems

Recently funded

Topics in this field

code generation · coding agents · software engineering424 papers94 this week
ai agents · agents · multi agent488 papers67 this week
reasoning tasks · reasoning paths · reasoning models381 papers44 this week
agent skills · language model agents · skill evolution268 papers36 this week
harness optimization · agent harnesses · harness evolution279 papers32 this week
research agents · ai agents · autonomous research184 papers29 this week

Topics in this field

Active contributors · 90 days

Active contributors

Activity · 12 weeks

Activity

Recent papers

All papers in Research

arXiv cs.AIOct 20
CUA-Recovery: Benchmarking Error Recovery in Long-Horizon Computer-Use Agents
Xu Hu, Haoming Li, Yuxiang Wang, Jinxin Yang, Kangshuo Li +9
As computer-use agents (CUAs) move to long-horizon workflows spanning multiple applications, undetected errors can silently corrupt persistent state and cause cascading failures in subsequent actions. Reliable error detection and recovery have become critical bottlenecks for building robust CUAs. However, existing benchmarks focus primarily on end-to-end task performance, providing limited insight into whether agents can detect and recover from errors during execution. The few benchmarks that evaluate recovery rely largely on manually injected errors, which may not reflect the types or horizons of failures induced by real agent policies. We introduce CUA-Recovery, a benchmark comprising test and training suites across multiple desktop applications. We construct it using ReRail, a framework for synthesizing long-horizon tasks and trajectories in a persistent desktop environment. ReRail composes MyPCBench tasks into new long-horizon workflows with executable ground truth, collects verifier-confirmed failures from multiple policies, and turns them into verified recovery trajectories. CUA-Recovery comprises a human-verified test suite of 2,250 erroneous states over 100 tasks and a…
PDF arXiv
arXiv cs.SEOct 20
Standards-Guided and Dependency-Aware Code-Level Documentation Generation
Tingxu Han, Wei Song, Ziming Li, Yuchen Chen, Weisong Sun +4
Automatically generating high-quality documentation for software repositories remains challenging: existing approaches largely treat documentation as unconstrained text generation, lacking explicit quality specifications, depending heavily on long-context prompts over raw code, and offering limited support for reproducible evaluation. We argue that code-level documentation generation should instead be a standards-guided software engineering task, in which documentation is generated and assessed against explicit, reusable quality requirements derived from authoritative standards. We define code documentation as natural-language descriptions of repository functions and classes, organized by source-code path. We present CODEDOC, a standards-guided and dependency-aware framework that identifies dependencies between code entities via static call-graph and AST analysis and constructs dependency-aware structured prompts combining local code context, repository paths, and documentation criteria drawn from authoritative standards and style guides. We further introduce an interactive LLM-as-a-judge protocol that supplies criterion-aligned feedback to refine generated documentation along…
PDF arXiv
arXiv cs.CROct 20
Local Allow, Global Deny: Why Per-Action Authorization Does Not Compose in LLM Agents
Anmol Pandey, Aditya Jain, Liang Chen, Carsten Maple
Autonomous LLM agents act through tools under per-action authorization: a local guard screens each tool call in isolation. We show that local authorization does not compose into global authorization. A compositional authorization failure (CAF) is a locally authorized execution, or a pair of locally authorized executions, that violates a global policy spanning intent, purpose, provenance, consent, and delegation. We characterize CAFs along two axes, expressiveness (single-trace safety versus relational k-safety) and observability (a ladder of monitor profiles). Our central result is an impossibility: relational purpose violations form a 2-safety hyperproperty with no per-trace locus of blame, so a per-action monitor that never raises a false alarm detects none of them, independent of the underlying model or decision procedure; they become visible only under cross-execution monitoring, which none of the six guardrails we survey implements. We make this constructive with AL-SYNTH, a SAT/SMT synthesizer (using no language model) that emits minimal CAF witnesses, each checked against an exhaustive oracle. The empirical signature is sharp: across nine LLM judges the per-action view…
PDF arXiv
arXiv cs.LGOct 20
Auditing the Product, Not the Recursion: The Evaluation Evidence Behind Recursive Self-Improvement Claims
Zihan Chen, Di Zhu, Lei Nico Zheng
Recursive self-improvement (RSI) promises AI systems, such as self-evolving LLM agents, that become better at producing further improvements. Evaluating this promise requires separating product claims about the final system from process claims about the improvement process itself. Existing surveys specify the evidence these claims require, but how often studies report that evidence remains unmeasured. We examine this question through full-text coding of 114 eligible studies in a reproducible arXiv search frame frozen in October 2026, including 90 that describe their own systems as RSI. The codebook links claims to evaluation designs. Independent human coding, third-reader adjudication, and a separate blind coding of improver comparisons assess coding reliability. Among the 90 RSI studies, 90% report product evaluation beyond development tasks under a broad criterion that includes partial or unstated separation, while 60% report evaluation on tasks unused for selection. Only 22% audit two or more reruns of the full loop. In a human-coded random sample, the RSI studies show the same contrast: 45% evaluate on tasks unused for selection and 14% audit two or more reruns. Of the 26…
PDF arXiv
arXiv cs.AIOct 20
The Show-Up Attack: When Subagents Win Over Their Supervisor
Maosen Zhang, Jianshuo Dong, Yutong Zhang, Jie Zhang, Tianwei Zhang +1
Multi-agent systems are increasingly deployed to coordinate multiple large language model (LLM) agents. However, this also creates new attack surfaces: malicious agents may have chances to exploit the coordination. In this paper, we propose and study the \textit{show-up attack}, in which a malicious subagent manipulates the supervisor into involving it in tasks beyond its assigned scope. To evaluate this risk, we propose SUA-Arena, which implements 6 attack strategies and contains 188 test cases. Then, we evaluate 6 frontier LLMs and find that all of them are vulnerable to show-up attacks. For example, GPT-5.6-Sol as a supervisor suffers an ASR of up to 0.93. Mitigation strategies such as prompting and restricted communication fail to reduce the risk consistently. These results show that multi-agent systems need explicit mechanisms for enforcing authority boundaries.
PDF arXiv
arXiv cs.AIOct 20
When LLMs Manipulate: Measuring Behavioral, Social, and Temporal Effects
Sadia Asif, Israel Antonio Rosales Laguan, Shumaila Asif, Haris Khan, Muneeb Asif +1
Large language models (LLMs) can influence user trust and decision-making through subtle linguistic cues, yet the behavioral effects of manipulative outputs remain poorly quantified. We introduce a behavioral computing framework for measuring \textit{behavioral vulnerability}, defined as users' susceptibility to manipulative AI responses. Building on DarkPatterns-LLM benchmark, we generate controlled manipulative stimuli using Manipulation Gradient Descent (MGD), targeting biases associated with authority, urgency, and social proof. We analyze participant responses using the Multi-Stakeholder Impact Assessment Network (MSIAN) to model cross-user influence and the Temporal Harm Propagation Model (THPM) to characterize the persistence of behavioral effects over time. In a study involving 120 participants, manipulative framing produced significant changes in trust calibration and compliance ($p < 0.01$), with effects also propagating across social contexts. These findings highlight the need to extend LLM safety evaluation beyond model outputs to measurable behavioral consequences and provide a computational framework for quantifying individual, social, and temporal risks arising…
PDF arXiv

Papers

Topics in this field

code generation · coding agents · software engineering424 papers94 this week
ai agents · agents · multi agent488 papers67 this week
reasoning tasks · reasoning paths · reasoning models381 papers44 this week
agent skills · language model agents · skill evolution268 papers36 this week
harness optimization · agent harnesses · harness evolution279 papers32 this week
research agents · ai agents · autonomous research184 papers29 this week

Active contributors · 90 days

Recent papers

All papers in Research

arXiv cs.AIOct 20
CUA-Recovery: Benchmarking Error Recovery in Long-Horizon Computer-Use Agents
Xu Hu, Haoming Li, Yuxiang Wang, Jinxin Yang, Kangshuo Li +9
As computer-use agents (CUAs) move to long-horizon workflows spanning multiple applications, undetected errors can silently corrupt persistent state and cause cascading failures in subsequent actions. Reliable error detection and recovery have become critical bottlenecks for building robust CUAs. However, existing benchmarks focus primarily on end-to-end task performance, providing limited insight into whether agents can detect and recover from errors during execution. The few benchmarks that evaluate recovery rely largely on manually injected errors, which may not reflect the types or horizons of failures induced by real agent policies. We introduce CUA-Recovery, a benchmark comprising test and training suites across multiple desktop applications. We construct it using ReRail, a framework for synthesizing long-horizon tasks and trajectories in a persistent desktop environment. ReRail composes MyPCBench tasks into new long-horizon workflows with executable ground truth, collects verifier-confirmed failures from multiple policies, and turns them into verified recovery trajectories. CUA-Recovery comprises a human-verified test suite of 2,250 erroneous states over 100 tasks and a…
PDF arXiv
arXiv cs.SEOct 20
Standards-Guided and Dependency-Aware Code-Level Documentation Generation
Tingxu Han, Wei Song, Ziming Li, Yuchen Chen, Weisong Sun +4
Automatically generating high-quality documentation for software repositories remains challenging: existing approaches largely treat documentation as unconstrained text generation, lacking explicit quality specifications, depending heavily on long-context prompts over raw code, and offering limited support for reproducible evaluation. We argue that code-level documentation generation should instead be a standards-guided software engineering task, in which documentation is generated and assessed against explicit, reusable quality requirements derived from authoritative standards. We define code documentation as natural-language descriptions of repository functions and classes, organized by source-code path. We present CODEDOC, a standards-guided and dependency-aware framework that identifies dependencies between code entities via static call-graph and AST analysis and constructs dependency-aware structured prompts combining local code context, repository paths, and documentation criteria drawn from authoritative standards and style guides. We further introduce an interactive LLM-as-a-judge protocol that supplies criterion-aligned feedback to refine generated documentation along…
PDF arXiv
arXiv cs.CROct 20
Local Allow, Global Deny: Why Per-Action Authorization Does Not Compose in LLM Agents
Anmol Pandey, Aditya Jain, Liang Chen, Carsten Maple
Autonomous LLM agents act through tools under per-action authorization: a local guard screens each tool call in isolation. We show that local authorization does not compose into global authorization. A compositional authorization failure (CAF) is a locally authorized execution, or a pair of locally authorized executions, that violates a global policy spanning intent, purpose, provenance, consent, and delegation. We characterize CAFs along two axes, expressiveness (single-trace safety versus relational k-safety) and observability (a ladder of monitor profiles). Our central result is an impossibility: relational purpose violations form a 2-safety hyperproperty with no per-trace locus of blame, so a per-action monitor that never raises a false alarm detects none of them, independent of the underlying model or decision procedure; they become visible only under cross-execution monitoring, which none of the six guardrails we survey implements. We make this constructive with AL-SYNTH, a SAT/SMT synthesizer (using no language model) that emits minimal CAF witnesses, each checked against an exhaustive oracle. The empirical signature is sharp: across nine LLM judges the per-action view…
PDF arXiv
arXiv cs.LGOct 20
Auditing the Product, Not the Recursion: The Evaluation Evidence Behind Recursive Self-Improvement Claims
Zihan Chen, Di Zhu, Lei Nico Zheng
Recursive self-improvement (RSI) promises AI systems, such as self-evolving LLM agents, that become better at producing further improvements. Evaluating this promise requires separating product claims about the final system from process claims about the improvement process itself. Existing surveys specify the evidence these claims require, but how often studies report that evidence remains unmeasured. We examine this question through full-text coding of 114 eligible studies in a reproducible arXiv search frame frozen in October 2026, including 90 that describe their own systems as RSI. The codebook links claims to evaluation designs. Independent human coding, third-reader adjudication, and a separate blind coding of improver comparisons assess coding reliability. Among the 90 RSI studies, 90% report product evaluation beyond development tasks under a broad criterion that includes partial or unstated separation, while 60% report evaluation on tasks unused for selection. Only 22% audit two or more reruns of the full loop. In a human-coded random sample, the RSI studies show the same contrast: 45% evaluate on tasks unused for selection and 14% audit two or more reruns. Of the 26…
PDF arXiv
arXiv cs.AIOct 20
The Show-Up Attack: When Subagents Win Over Their Supervisor
Maosen Zhang, Jianshuo Dong, Yutong Zhang, Jie Zhang, Tianwei Zhang +1
Multi-agent systems are increasingly deployed to coordinate multiple large language model (LLM) agents. However, this also creates new attack surfaces: malicious agents may have chances to exploit the coordination. In this paper, we propose and study the \textit{show-up attack}, in which a malicious subagent manipulates the supervisor into involving it in tasks beyond its assigned scope. To evaluate this risk, we propose SUA-Arena, which implements 6 attack strategies and contains 188 test cases. Then, we evaluate 6 frontier LLMs and find that all of them are vulnerable to show-up attacks. For example, GPT-5.6-Sol as a supervisor suffers an ASR of up to 0.93. Mitigation strategies such as prompting and restricted communication fail to reduce the risk consistently. These results show that multi-agent systems need explicit mechanisms for enforcing authority boundaries.
PDF arXiv
arXiv cs.AIOct 20
When LLMs Manipulate: Measuring Behavioral, Social, and Temporal Effects
Sadia Asif, Israel Antonio Rosales Laguan, Shumaila Asif, Haris Khan, Muneeb Asif +1
Large language models (LLMs) can influence user trust and decision-making through subtle linguistic cues, yet the behavioral effects of manipulative outputs remain poorly quantified. We introduce a behavioral computing framework for measuring \textit{behavioral vulnerability}, defined as users' susceptibility to manipulative AI responses. Building on DarkPatterns-LLM benchmark, we generate controlled manipulative stimuli using Manipulation Gradient Descent (MGD), targeting biases associated with authority, urgency, and social proof. We analyze participant responses using the Multi-Stakeholder Impact Assessment Network (MSIAN) to model cross-user influence and the Temporal Harm Propagation Model (THPM) to characterize the persistence of behavioral effects over time. In a study involving 120 participants, manipulative framing produced significant changes in trust calibration and compliance ($p < 0.01$), with effects also propagating across social contexts. These findings highlight the need to extend LLM safety evaluation beyond model outputs to measurable behavioral consequences and provide a computational framework for quantifying individual, social, and temporal risks arising…
PDF arXiv

Activity · 12 weeks

Fields · AI · updated 40m ago

AI agents

Found by the topic model: ai agents, code generation, reasoning tasks.

480papers this week−28.8% vs last week

All papers · 3.3k With code · 175

Newest Most relevant

2.7k papers this month

Open problems in AI agents

Evaluating long-horizon agentic behavior in real-world tasks3 papers raise it · 2 address it
Current benchmarks often simplify or omit the key work agents must perform, making it hard to assess sustained autonomous problem-solving.
- Scaling Agentic Data for Long-Horizon Terminal Intelligenceraises · addresses
- FDE-Gym: The Last Mile of AI Deliveryraises · addresses
- AgentSWE: Can Coding Agents Build the Agent You Actually Want?raises
Faithful agent development from natural-language requirements1 paper raises it · 1 addresses it
Agents often fail to deliver what users actually ask for, passing superficial tests but not meeting the full requirement.
- AgentSWE: Can Coding Agents Build the Agent You Actually Want?raises · addresses
Detecting when past context matters in long conversations1 paper raises it · 0 address it
AI companions struggle to recognize when historical context is relevant, leading to missed or incorrect responses.
- RealCompanion: Benchmarking Human Understanding from Reasoning over Longitudinal Real-World Conversationsraises
Reliable no-thinking compliance in LLMs1 paper raises it · 1 addresses it
LLMs often ignore or bypass explicit instructions to avoid reasoning, making it hard to enforce strict answer-only behavior.
- Thinking Inertia: LLMs Keep Thinking When Told Not Toraises · addresses
Transition-aware safety for GUI agents1 paper raises it · 1 addresses it
Safety checks that only inspect screens can be fooled by reusing legitimate screens in unexpected transitions.
- Right Screen, Wrong Transition: World Models as Verifiers for GUI Agentsraises · addresses

Named by an AI from these papers' abstracts.