Papers

Field

Top tools here

Open problems in AI agents

Long-horizon reasoning and memory in real-world agent tasks3 papers raise it · 3 address it
Agents fail to retain, retrieve, and use past context over long interactions, limiting their ability to understand users or solve complex tasks.
- RealCompanion: Benchmarking Human Understanding from Reasoning over Longitudinal Real-World Conversationsraises
- Scaling Agentic Data for Long-Horizon Terminal Intelligenceraises · addresses
- Learn2Play Bench: How Well Do LLM Agents Learn from Experience in Unfamiliar Environments?raises · addresses
- Agent-Reflex: Teaching Language Agents to Act Reflectivelyaddresses
Ambiguous and dynamic real-world task requirements2 papers raise it · 2 address it
Agents struggle with unclear, evolving, or unstated constraints in real deployments, unlike well-specified benchmarks.
- FDE-Gym: The Last Mile of AI Deliveryraises · addresses
- AgentSWE: Can Coding Agents Build the Agent You Actually Want?raises · addresses
Reliable no-thinking compliance in LLMs1 paper raises it · 0 address it
Models often ignore or bypass explicit instructions to avoid reasoning, making them unreliable for constrained or safety-critical use.
- Thinking Inertia: LLMs Keep Thinking When Told Not Toraises
Verification of agent actions and transitions1 paper raises it · 1 addresses it
Agents need to detect harmful or unexpected transitions, not just static states, to ensure safety and correctness.
- Right Screen, Wrong Transition: World Models as Verifiers for GUI Agentsraises · addresses
Faithful agent development from natural-language requirements1 paper raises it · 1 addresses it
Coding agents often produce systems that appear to meet requirements but fail to deliver the intended behavior.
- AgentSWE: Can Coding Agents Build the Agent You Actually Want?raises · addresses

Named by an AI from these papers' abstracts.

Open problems

Recently funded

Topics in this field

code generation · coding agents · software engineering384 papers79 this week
ai agents · agents · multi agent462 papers51 this week
reasoning tasks · reasoning paths · reasoning models364 papers32 this week
harness optimization · agent harnesses · harness evolution270 papers32 this week
agent skills · language model agents · skill evolution247 papers22 this week
ai agents · multi agent llm · multi agent195 papers22 this week

Topics in this field

Active contributors · 90 days

Active contributors

Activity · 12 weeks

Activity

Recent papers

All papers in Research

arXiv cs.SEOct 21
Detection Is Not Enough: Reasoning Traces as Testable Program Hypotheses
Vaidehi Hegde
When a language model is deployed, no reference solution exists, so the reasoning trace it writes is the only evidence for judging its answer. The reference-free substitutes each fail differently: self-assessment degrades the answer it was meant to protect, sampling agreement measures stability rather than correctness, and perturbation studies read the expected answer off a human-written template, grading a model rather than a trace. We ask how far a single trace can be verified with no reference solution, no judge model and no weight update, and find, for a frozen LLaMA-2-7B-chat on GSM8K, that detection is not what limits such a pipeline. Checking a trace against itself reaches an AUROC of 0.741 and stops there for a structural reason: two thirds of wrong traces state no false equation, having executed a mistaken plan correctly, and on wrong-plan failures the detector flags a wrong trace less often than a right one. A candidate-derived metamorphic probe exposes many of these failures: we abstract the trace into a program over the question's quantities, perturb one quantity, execute the program, and ask the model the rewritten question once, so every trace supplies its own test.…
PDF arXiv
arXiv cs.LGOct 21
Parallelizable Probabilistic Latent Reasoning for Diverse Reasoning Trajectories
Seokhyun Lee, Seongjun Lee, Changhee Lee
Language models relying on verbalized Chain-of-Thought (CoT) are constrained to token-by-token generation in the language space, where many tokens serve textual coherence rather than core reasoning. This entangles genuine differences in logical trajectories with superficial variations in verbalization. Moving reasoning into a compact, continuous latent space offers a promising alternative for efficiently representing and exploring diverse reasoning trajectories. However, current latent models remain largely deterministic, capturing only a single reasoning trajectory rather than modeling a distribution over multiple valid alternatives. While stochastic approaches parameterize distributions over latent reasoning steps, they frequently suffer from unstable optimization and limited empirical performance. Furthermore, their reliance on autoregressive latent prediction during training precludes effective parallelization, leading to prohibitively long training times. To overcome these bottlenecks, we propose Probabilistic Latent Reasoning Models (PLRM), a parallelizable variational framework for probabilistic latent reasoning. PLRM adopts a JEPA-style objective in which the posterior…
PDF arXiv
arXiv cs.CLOct 21
Right On Q: Recovering Reasoning under Quantization via Self-Generated Trajectories
Boyuan Zheng, Jing Xiong, Chenwei Wu, Hui Shen, Zesen Zhao +6
Quantization-aware training (QAT) aims to recover reasoning capabilities in low-bit large language models (LLMs). Existing recipes typically supervise the quantized student with the original full-precision model on fixed response sequences. However, the resulting low-bit quantized models still struggle on long-horizon reasoning tasks and often fail to terminate within a token budget that suffices for the full-precision model. We identify two challenges in closing this gap. First, fixed-response supervision leaves a mismatch between training prefixes and those encountered during student generation, and this trajectory divergence grows as responses lengthen. Second, on its own prefixes, the student underpredicts some next-token candidates and overpredicts others, and this distributional misalignment becomes more frequent at high-entropy prefixes. We present OnQ, a low-bit recovery pipeline that combines on-policy distillation and candidate-wise Dynamic KL. Starting from block-wise calibration, the quantized student generates responses and receives full-precision teacher supervision on its own prefixes. Dynamic KL weights forward and reverse KL by each candidate's teacher-student…
PDF arXiv
arXiv cs.LGOct 21
Benchmarking and Error Analysis of Deep Search and Deep Research LLM Agents
Zixuan Yu, Zhenheng Tang, Xiaowen Chu, Bo Han
As Large Language Models (LLMs) evolve into autonomous Deep Search and Deep Research Agents capable of executing long-horizon, multi-step tasks, evaluating their reliability becomes increasingly critical. However, existing evaluation paradigms predominantly focus on final task outcomes, leaving a significant gap in understanding the intermediate execution traces, where agents are highly susceptible to various errors. To address this critical gap, we introduce a comprehensive methodology and analytical framework designed specifically to diagnose errors in intermediate workflows. Building upon this framework, we establish a structured taxonomy that categorizes these vulnerabilities into two primary dimensions: System-Level Errors and Task-Level Errors. Through a systematic examination of step-by-step execution traces and agent-environment interactions, we reveal how these underlying errors affect overall performance. Finally, through empirical experiments and systematic analysis of the identified error types, we provide actionable design guidelines, aiming to enhance the robustness, reasoning capabilities, and reliability of Deep Search and Deep Research agents.
PDF arXiv
arXiv cs.CLOct 21
Policy Central: Detecting Conflicts, Redundancy, and Similarity in Deployed Police Policy
Holly Baker, Michael Kuo, Kyana Pollas
Policy Central is a deployed system that helps police departments find relationships between sections of their written policy: whether two sections conflict, are redundant, are merely similar, or are different. We describe the system and compare two architectures. Retrieve-Judge, a retrieve-then-classify pipeline, runs as a cheap library-wide health signal; Screen-Confirm, a screen-then-confirm cascade, is a selective audit of the documents a reviewer chooses, checking them exhaustively against one another. On a 302-pair benchmark drawn from one department's ~84 real policy documents, given each pair Screen-Confirm reaches 0.795 aggregate recall and 0.885 on Conflict, so current models judge well. Finding the pair is the hard part: Retrieve-Judge's composed recall falls to 0.457, and Screen-Confirm recovers 0.472 (0.615 on Conflict). Today these pairs are found by reading the whole library, nearly 20,000 sections, checked against one another and re-checked after every revision. A scan that returns half of them in minutes, at low cost, and can be re-run on demand changes the economics of policy review even before recall improves.
PDF arXiv
arXiv cs.SEOct 21
Do Coding Agents Understand Why Systems Crash---and Why They Don't?
Chenxi Huang, Haonan Wang, Jinghe Zhang, Aleksandr Nogikh, Franjo Ivančić +5
Coding agents are increasingly capable of fixing real-world bugs, yet they still struggle with large systems codebases such as the Linux kernel. To understand why, we test how agents reason about execution in these codebases through crash reasoning: predicting from source code whether a given input crashes the system. A correct prediction alone, however, does not establish this ability, since it may come from merely spotting suspicious code. In this paper, we propose source interventions: we pair source trees that differ by a single commit, so that the same input crashes only one of them, and test whether agents predict crashes more often on the crashing tree. Experiments with four coding agents on SysRB, our dataset of 934 such pairs from Linux and OSS-Fuzz projects, yield three main findings. First, agents are biased toward predicting no crash: they rarely raise false alarms but miss many real crashes, and even the best agent misses over a quarter of them. Second, locating the code that determines the crash outcome is not enough: agents often misjudge its effect, especially when the crash occurs far from that code. Third, testing only on crashing code is misleading: agents that…
PDF arXiv

Papers

Topics in this field

code generation · coding agents · software engineering384 papers79 this week
ai agents · agents · multi agent462 papers51 this week
reasoning tasks · reasoning paths · reasoning models364 papers32 this week
harness optimization · agent harnesses · harness evolution270 papers32 this week
agent skills · language model agents · skill evolution247 papers22 this week
ai agents · multi agent llm · multi agent195 papers22 this week

Active contributors · 90 days

Recent papers

All papers in Research

arXiv cs.SEOct 21
Detection Is Not Enough: Reasoning Traces as Testable Program Hypotheses
Vaidehi Hegde
When a language model is deployed, no reference solution exists, so the reasoning trace it writes is the only evidence for judging its answer. The reference-free substitutes each fail differently: self-assessment degrades the answer it was meant to protect, sampling agreement measures stability rather than correctness, and perturbation studies read the expected answer off a human-written template, grading a model rather than a trace. We ask how far a single trace can be verified with no reference solution, no judge model and no weight update, and find, for a frozen LLaMA-2-7B-chat on GSM8K, that detection is not what limits such a pipeline. Checking a trace against itself reaches an AUROC of 0.741 and stops there for a structural reason: two thirds of wrong traces state no false equation, having executed a mistaken plan correctly, and on wrong-plan failures the detector flags a wrong trace less often than a right one. A candidate-derived metamorphic probe exposes many of these failures: we abstract the trace into a program over the question's quantities, perturb one quantity, execute the program, and ask the model the rewritten question once, so every trace supplies its own test.…
PDF arXiv
arXiv cs.LGOct 21
Parallelizable Probabilistic Latent Reasoning for Diverse Reasoning Trajectories
Seokhyun Lee, Seongjun Lee, Changhee Lee
Language models relying on verbalized Chain-of-Thought (CoT) are constrained to token-by-token generation in the language space, where many tokens serve textual coherence rather than core reasoning. This entangles genuine differences in logical trajectories with superficial variations in verbalization. Moving reasoning into a compact, continuous latent space offers a promising alternative for efficiently representing and exploring diverse reasoning trajectories. However, current latent models remain largely deterministic, capturing only a single reasoning trajectory rather than modeling a distribution over multiple valid alternatives. While stochastic approaches parameterize distributions over latent reasoning steps, they frequently suffer from unstable optimization and limited empirical performance. Furthermore, their reliance on autoregressive latent prediction during training precludes effective parallelization, leading to prohibitively long training times. To overcome these bottlenecks, we propose Probabilistic Latent Reasoning Models (PLRM), a parallelizable variational framework for probabilistic latent reasoning. PLRM adopts a JEPA-style objective in which the posterior…
PDF arXiv
arXiv cs.CLOct 21
Right On Q: Recovering Reasoning under Quantization via Self-Generated Trajectories
Boyuan Zheng, Jing Xiong, Chenwei Wu, Hui Shen, Zesen Zhao +6
Quantization-aware training (QAT) aims to recover reasoning capabilities in low-bit large language models (LLMs). Existing recipes typically supervise the quantized student with the original full-precision model on fixed response sequences. However, the resulting low-bit quantized models still struggle on long-horizon reasoning tasks and often fail to terminate within a token budget that suffices for the full-precision model. We identify two challenges in closing this gap. First, fixed-response supervision leaves a mismatch between training prefixes and those encountered during student generation, and this trajectory divergence grows as responses lengthen. Second, on its own prefixes, the student underpredicts some next-token candidates and overpredicts others, and this distributional misalignment becomes more frequent at high-entropy prefixes. We present OnQ, a low-bit recovery pipeline that combines on-policy distillation and candidate-wise Dynamic KL. Starting from block-wise calibration, the quantized student generates responses and receives full-precision teacher supervision on its own prefixes. Dynamic KL weights forward and reverse KL by each candidate's teacher-student…
PDF arXiv
arXiv cs.LGOct 21
Benchmarking and Error Analysis of Deep Search and Deep Research LLM Agents
Zixuan Yu, Zhenheng Tang, Xiaowen Chu, Bo Han
As Large Language Models (LLMs) evolve into autonomous Deep Search and Deep Research Agents capable of executing long-horizon, multi-step tasks, evaluating their reliability becomes increasingly critical. However, existing evaluation paradigms predominantly focus on final task outcomes, leaving a significant gap in understanding the intermediate execution traces, where agents are highly susceptible to various errors. To address this critical gap, we introduce a comprehensive methodology and analytical framework designed specifically to diagnose errors in intermediate workflows. Building upon this framework, we establish a structured taxonomy that categorizes these vulnerabilities into two primary dimensions: System-Level Errors and Task-Level Errors. Through a systematic examination of step-by-step execution traces and agent-environment interactions, we reveal how these underlying errors affect overall performance. Finally, through empirical experiments and systematic analysis of the identified error types, we provide actionable design guidelines, aiming to enhance the robustness, reasoning capabilities, and reliability of Deep Search and Deep Research agents.
PDF arXiv
arXiv cs.CLOct 21
Policy Central: Detecting Conflicts, Redundancy, and Similarity in Deployed Police Policy
Holly Baker, Michael Kuo, Kyana Pollas
Policy Central is a deployed system that helps police departments find relationships between sections of their written policy: whether two sections conflict, are redundant, are merely similar, or are different. We describe the system and compare two architectures. Retrieve-Judge, a retrieve-then-classify pipeline, runs as a cheap library-wide health signal; Screen-Confirm, a screen-then-confirm cascade, is a selective audit of the documents a reviewer chooses, checking them exhaustively against one another. On a 302-pair benchmark drawn from one department's ~84 real policy documents, given each pair Screen-Confirm reaches 0.795 aggregate recall and 0.885 on Conflict, so current models judge well. Finding the pair is the hard part: Retrieve-Judge's composed recall falls to 0.457, and Screen-Confirm recovers 0.472 (0.615 on Conflict). Today these pairs are found by reading the whole library, nearly 20,000 sections, checked against one another and re-checked after every revision. A scan that returns half of them in minutes, at low cost, and can be re-run on demand changes the economics of policy review even before recall improves.
PDF arXiv
arXiv cs.SEOct 21
Do Coding Agents Understand Why Systems Crash---and Why They Don't?
Chenxi Huang, Haonan Wang, Jinghe Zhang, Aleksandr Nogikh, Franjo Ivančić +5
Coding agents are increasingly capable of fixing real-world bugs, yet they still struggle with large systems codebases such as the Linux kernel. To understand why, we test how agents reason about execution in these codebases through crash reasoning: predicting from source code whether a given input crashes the system. A correct prediction alone, however, does not establish this ability, since it may come from merely spotting suspicious code. In this paper, we propose source interventions: we pair source trees that differ by a single commit, so that the same input crashes only one of them, and test whether agents predict crashes more often on the crashing tree. Experiments with four coding agents on SysRB, our dataset of 934 such pairs from Linux and OSS-Fuzz projects, yield three main findings. First, agents are biased toward predicting no crash: they rarely raise false alarms but miss many real crashes, and even the best agent misses over a quarter of them. Second, locating the code that determines the crash outcome is not enough: agents often misjudge its effect, especially when the crash occurs far from that code. Third, testing only on crashing code is misleading: agents that…
PDF arXiv

Activity · 12 weeks

Fields · AI · updated 55m ago

AI agents

Found by the topic model: ai agents, code generation, reasoning tasks.

370papers this week−45.1% vs last week

All papers · 3.1k With code · 149

Newest Most relevant

2.6k papers this month

Open problems in AI agents

Long-horizon reasoning and memory in real-world agent tasks3 papers raise it · 3 address it
Agents fail to retain, retrieve, and use past context over long interactions, limiting their ability to understand users or solve complex tasks.
- RealCompanion: Benchmarking Human Understanding from Reasoning over Longitudinal Real-World Conversationsraises
- Scaling Agentic Data for Long-Horizon Terminal Intelligenceraises · addresses
- Learn2Play Bench: How Well Do LLM Agents Learn from Experience in Unfamiliar Environments?raises · addresses
- Agent-Reflex: Teaching Language Agents to Act Reflectivelyaddresses
Ambiguous and dynamic real-world task requirements2 papers raise it · 2 address it
Agents struggle with unclear, evolving, or unstated constraints in real deployments, unlike well-specified benchmarks.
- FDE-Gym: The Last Mile of AI Deliveryraises · addresses
- AgentSWE: Can Coding Agents Build the Agent You Actually Want?raises · addresses
Reliable no-thinking compliance in LLMs1 paper raises it · 0 address it
Models often ignore or bypass explicit instructions to avoid reasoning, making them unreliable for constrained or safety-critical use.
- Thinking Inertia: LLMs Keep Thinking When Told Not Toraises
Verification of agent actions and transitions1 paper raises it · 1 addresses it
Agents need to detect harmful or unexpected transitions, not just static states, to ensure safety and correctness.
- Right Screen, Wrong Transition: World Models as Verifiers for GUI Agentsraises · addresses
Faithful agent development from natural-language requirements1 paper raises it · 1 addresses it
Coding agents often produce systems that appear to meet requirements but fail to deliver the intended behavior.
- AgentSWE: Can Coding Agents Build the Agent You Actually Want?raises · addresses

Named by an AI from these papers' abstracts.