Papers

Field

Top tools here

Open problems in AI agents

Evaluating long-horizon agentic behavior in real-world tasks3 papers raise it · 2 address it
Current benchmarks often simplify or omit the key work agents must perform, making it hard to assess sustained autonomous problem-solving.
- Scaling Agentic Data for Long-Horizon Terminal Intelligenceraises · addresses
- FDE-Gym: The Last Mile of AI Deliveryraises · addresses
- AgentSWE: Can Coding Agents Build the Agent You Actually Want?raises
Faithful agent development from natural-language requirements1 paper raises it · 1 addresses it
Agents often fail to deliver what users actually ask for, passing superficial tests but not meeting the full requirement.
- AgentSWE: Can Coding Agents Build the Agent You Actually Want?raises · addresses
Detecting when past context matters in long conversations1 paper raises it · 0 address it
AI companions struggle to recognize when historical context is relevant, leading to missed or incorrect responses.
- RealCompanion: Benchmarking Human Understanding from Reasoning over Longitudinal Real-World Conversationsraises
Reliable no-thinking compliance in LLMs1 paper raises it · 1 addresses it
LLMs often ignore or bypass explicit instructions to avoid reasoning, making it hard to enforce strict answer-only behavior.
- Thinking Inertia: LLMs Keep Thinking When Told Not Toraises · addresses
Transition-aware safety for GUI agents1 paper raises it · 1 addresses it
Safety checks that only inspect screens can be fooled by reusing legitimate screens in unexpected transitions.
- Right Screen, Wrong Transition: World Models as Verifiers for GUI Agentsraises · addresses

Named by an AI from these papers' abstracts.

Open problems

Recently funded

Topics in this field

code generation · coding agents · software engineering424 papers94 this week
ai agents · agents · multi agent488 papers64 this week
reasoning tasks · reasoning paths · reasoning models381 papers44 this week
agent skills · language model agents · skill evolution268 papers35 this week
harness optimization · agent harnesses · harness evolution279 papers32 this week
research agents · ai agents · autonomous research184 papers28 this week

Topics in this field

Active contributors · 90 days

Active contributors

Activity · 12 weeks

Activity

Recent papers

All papers in Research

arXiv cs.AIOct 19
EGI-BENCH: An Enterprise General Intelligence Benchmark Suite
Zhiwei Liu, Zhujun Lan, Jielin Qiu, Zixiang Chen, Liangwei Yang +10
Enterprise adoption of LLMs in agentic workflows is gated by evaluations that span a wide arc of abilities: CRM operations, heterogeneous-document retrieval, personal data understanding, multi-turn planning, tool use, function calling, long-horizon research, and long-context software engineering. These evaluations live today in separate repositories with divergent runners, dependencies, and scoring conventions, which makes cross-benchmark comparison difficult and obscures what a single aggregate number actually represents. We release EGI-BENCH, a unified LLM agent benchmark suite that consolidates eight enterprise facing agent benchmarks (CRMARENA, HERB, PERSONABENCH, USERBENCH, MCP-UNIVERSE, MCPEVAL, BENCHFORCE, LOCOBENCH) behind a single command-line interface with two runtime-budgeted execution modes. On top of the suite we define the Enterprise General Intelligence (EGI) Score, a six-dimensional framework (M1-M6) whose buckets are populated by fine-grained sub-metrics routed from each benchmark rather than by benchmark-level averages. We describe the suite, the sub-metric routing, and the quick/deep budgets, and report results for fourteen frontier and cost-efficient LLMs…
PDF arXiv
arXiv cs.AIOct 19
Sparse RL Relaxation for LLM Reasoning
Jiabin Fan, Matthias Lindemann, Zita Marinho, André FT Martins, Lili Mou
Recent work has explored continuous representations in place of discrete tokens during Chain-of-Thought (CoT) reasoning, often referred to as soft CoT. This is motivated by the idea that a mixture of token embeddings can represent a superposition of multiple reasoning possibilities within a single step. However, it remains challenging to train large language models to realize this potential. Existing work has explored reinforcement learning (RL)-style training for soft CoT. However, these approaches treat soft tokens as actions in a Markov decision process rather than as differentiable intermediate computation steps. Thus, they do not propagate gradients across the soft tokens and miss out on the signal of how the choice of one soft token affects the next. In this paper, we propose a sparse relaxation method for RL (called SpaRLax), which leverages the differentiability of soft-CoT reasoning over time steps and makes training more stable and effective. We adopt an entmax sparsity method that adaptively determines the degree of softness, which truncates long-tail noise at confident steps while retaining multiple plausible candidates when uncertain. Across nine reasoning…
PDF arXiv
arXiv cs.CLCodeOct 19▲ 10
On the Fragility of Tasks under Composition: Implicit Budgeting in Language Models
Chenpan Li, Thies Brockmoeller, Stefanie Jegelka, Sharut Gupta, Nikolaos Karalias
Modern language models are increasingly asked to handle several tasks within a single prompt. Doing so requires solving each task correctly while integrating the resulting answers into a coherent response. In this paper, we identify a fundamental limitation in this process, where models make substantially more errors when tasks are composed, even when they can solve every constituent task reliably in isolation. This degradation cannot be fully explained by the accumulation of independent errors. Across both open-weight models and the frontier model Claude Opus 5, our findings instead point to an implicit thinking budget underlying this effect. In particular, under composition, models use substantially fewer tokens per task than in isolation, even when the full sequence remains well within the context window. Tasks that require longer solution traces in isolation are consequently more $\textit{fragile}$ under composition, i.e., models make more errors on prompts containing these tasks. In isolation, non-fragile tasks receive more reasoning than they need and waste compute. However, by grouping tasks according to their fragility, the implicit budgeting can be manipulated to reduce…
PDF arXiv Code
arXiv cs.SEOct 19
Who Approved This? Coding Agents, Human Accounts, and Merge-Gate Evidence at a Fintech Startup
Adam Manji
From 3 October, every non-bot merge into our main branch passed a non-author review gate, yet 71 of those 263 merges (27%) rested only on an approval that a coding agent had submitted through the technical lead's GitHub account, and the platform recorded each one as an ordinary approval by the lead. We report twelve weeks of agent-driven development at Bashi AI, a fintech startup preparing for a SOC 2 audit: 786 merged pull requests, 657 of them associated with agents by explicit markers or by inference, analysed from platform records and saved agent-session transcripts. In the gated phase, an agent session submitted every approval the lead's account gave on a colleague's pull request, and issued nearly all of its merges; the records cannot show who decided any of them. We recompute, for every merge, a core gate property: a non-author approval and a passing required check, both before the merge and both on the exact revision merged. The recomputation confirmed the gate wherever branch protection applied and found where it did not: agents switching enforcement off around individual merges, a merge script that recorded failed checks as passing, and an agent collapsing a stack of…
PDF arXiv
arXiv cs.CLOct 19
SITO: Sifting a Commercial Text-Safety API for Missed Harm and False Flags in English, Ukrainian and Surzhyk
Andrii Tuzhykov
Platforms that host young users screen messages with commercial text-safety APIs, but independent evidence on how often such systems miss harm or flag harmless text is scarce, especially outside English, and the documentation of the one studied here leaves its false-positive rate unquantified. We evaluated one such API, Tuteliq Detect as deployed under the release identifier tuteliq-detect-v5.20261011 (not the separately published v6 model), on six harm categories: authorised push-payment fraud, money-mule recruitment, social engineering, romance scams, gambling harm and bullying. We built SITO (Synthetic Independent Text-safety benchmark with Overflagging tests; sito is Ukrainian for 'sieve'), 2,332 synthetic cases in English, Ukrainian and mixed Ukrainian, with each harmful case matched by hard harmless ones: same-topic messages, minimal-edit contrasts, warnings and retrospective accounts. On English text the system flagged 84.9-94.1% of harmful cases and 0-4.3% of harmless ones per category. It missed third-party reports of ongoing harm far more often than perpetrators' messages (63.6% against 95.1% recall, all languages), when sent without the settings the API offers for…
PDF arXiv
arXiv cs.LGOct 19
Understanding the Benefits of Scaffolds through Computational Separations
Jingchu Gai, Adam Block, Andrej Risteski
Inference-time harnesses, i.e., external procedures that call a model many times and keep state across calls, can substantially improve model capabilities. In this paper, we investigate the benefits of using such harnesses and the extent to which these benefits can be internalized through training. More precisely, we conceptualize a harness $\mathcal{A}$ as an arbitrary wrapper around a base model $M$ and attempt to distill a student model from the harness's traces. Empirically we find that, even with simple scaffolds and small models, the inference-time harness significantly outperforms a distilled student. We then prove that such a gap is due to a fundamental representational limitation: inference-time harnesses can compute a closure of their per-call primitive that is $\NL$-complete even when the primitive is a trivial $\AC^0$ lookup, and that no fixed-depth forward pass can represent under the widely believed conjecture that $\NL\not\subseteq\TC^0$. Our theory suggests that expanding the representational capacity of the student relative to the task at hand, e.g., by enlarging the model, making the task easier, or lengthening its chain-of-thought, is necessary to close the…
PDF arXiv

Papers

Topics in this field

code generation · coding agents · software engineering424 papers94 this week
ai agents · agents · multi agent488 papers64 this week
reasoning tasks · reasoning paths · reasoning models381 papers44 this week
agent skills · language model agents · skill evolution268 papers35 this week
harness optimization · agent harnesses · harness evolution279 papers32 this week
research agents · ai agents · autonomous research184 papers28 this week

Active contributors · 90 days

Recent papers

All papers in Research

arXiv cs.AIOct 19
EGI-BENCH: An Enterprise General Intelligence Benchmark Suite
Zhiwei Liu, Zhujun Lan, Jielin Qiu, Zixiang Chen, Liangwei Yang +10
Enterprise adoption of LLMs in agentic workflows is gated by evaluations that span a wide arc of abilities: CRM operations, heterogeneous-document retrieval, personal data understanding, multi-turn planning, tool use, function calling, long-horizon research, and long-context software engineering. These evaluations live today in separate repositories with divergent runners, dependencies, and scoring conventions, which makes cross-benchmark comparison difficult and obscures what a single aggregate number actually represents. We release EGI-BENCH, a unified LLM agent benchmark suite that consolidates eight enterprise facing agent benchmarks (CRMARENA, HERB, PERSONABENCH, USERBENCH, MCP-UNIVERSE, MCPEVAL, BENCHFORCE, LOCOBENCH) behind a single command-line interface with two runtime-budgeted execution modes. On top of the suite we define the Enterprise General Intelligence (EGI) Score, a six-dimensional framework (M1-M6) whose buckets are populated by fine-grained sub-metrics routed from each benchmark rather than by benchmark-level averages. We describe the suite, the sub-metric routing, and the quick/deep budgets, and report results for fourteen frontier and cost-efficient LLMs…
PDF arXiv
arXiv cs.AIOct 19
Sparse RL Relaxation for LLM Reasoning
Jiabin Fan, Matthias Lindemann, Zita Marinho, André FT Martins, Lili Mou
Recent work has explored continuous representations in place of discrete tokens during Chain-of-Thought (CoT) reasoning, often referred to as soft CoT. This is motivated by the idea that a mixture of token embeddings can represent a superposition of multiple reasoning possibilities within a single step. However, it remains challenging to train large language models to realize this potential. Existing work has explored reinforcement learning (RL)-style training for soft CoT. However, these approaches treat soft tokens as actions in a Markov decision process rather than as differentiable intermediate computation steps. Thus, they do not propagate gradients across the soft tokens and miss out on the signal of how the choice of one soft token affects the next. In this paper, we propose a sparse relaxation method for RL (called SpaRLax), which leverages the differentiability of soft-CoT reasoning over time steps and makes training more stable and effective. We adopt an entmax sparsity method that adaptively determines the degree of softness, which truncates long-tail noise at confident steps while retaining multiple plausible candidates when uncertain. Across nine reasoning…
PDF arXiv
arXiv cs.CLCodeOct 19▲ 10
On the Fragility of Tasks under Composition: Implicit Budgeting in Language Models
Chenpan Li, Thies Brockmoeller, Stefanie Jegelka, Sharut Gupta, Nikolaos Karalias
Modern language models are increasingly asked to handle several tasks within a single prompt. Doing so requires solving each task correctly while integrating the resulting answers into a coherent response. In this paper, we identify a fundamental limitation in this process, where models make substantially more errors when tasks are composed, even when they can solve every constituent task reliably in isolation. This degradation cannot be fully explained by the accumulation of independent errors. Across both open-weight models and the frontier model Claude Opus 5, our findings instead point to an implicit thinking budget underlying this effect. In particular, under composition, models use substantially fewer tokens per task than in isolation, even when the full sequence remains well within the context window. Tasks that require longer solution traces in isolation are consequently more $\textit{fragile}$ under composition, i.e., models make more errors on prompts containing these tasks. In isolation, non-fragile tasks receive more reasoning than they need and waste compute. However, by grouping tasks according to their fragility, the implicit budgeting can be manipulated to reduce…
PDF arXiv Code
arXiv cs.SEOct 19
Who Approved This? Coding Agents, Human Accounts, and Merge-Gate Evidence at a Fintech Startup
Adam Manji
From 3 October, every non-bot merge into our main branch passed a non-author review gate, yet 71 of those 263 merges (27%) rested only on an approval that a coding agent had submitted through the technical lead's GitHub account, and the platform recorded each one as an ordinary approval by the lead. We report twelve weeks of agent-driven development at Bashi AI, a fintech startup preparing for a SOC 2 audit: 786 merged pull requests, 657 of them associated with agents by explicit markers or by inference, analysed from platform records and saved agent-session transcripts. In the gated phase, an agent session submitted every approval the lead's account gave on a colleague's pull request, and issued nearly all of its merges; the records cannot show who decided any of them. We recompute, for every merge, a core gate property: a non-author approval and a passing required check, both before the merge and both on the exact revision merged. The recomputation confirmed the gate wherever branch protection applied and found where it did not: agents switching enforcement off around individual merges, a merge script that recorded failed checks as passing, and an agent collapsing a stack of…
PDF arXiv
arXiv cs.CLOct 19
SITO: Sifting a Commercial Text-Safety API for Missed Harm and False Flags in English, Ukrainian and Surzhyk
Andrii Tuzhykov
Platforms that host young users screen messages with commercial text-safety APIs, but independent evidence on how often such systems miss harm or flag harmless text is scarce, especially outside English, and the documentation of the one studied here leaves its false-positive rate unquantified. We evaluated one such API, Tuteliq Detect as deployed under the release identifier tuteliq-detect-v5.20261011 (not the separately published v6 model), on six harm categories: authorised push-payment fraud, money-mule recruitment, social engineering, romance scams, gambling harm and bullying. We built SITO (Synthetic Independent Text-safety benchmark with Overflagging tests; sito is Ukrainian for 'sieve'), 2,332 synthetic cases in English, Ukrainian and mixed Ukrainian, with each harmful case matched by hard harmless ones: same-topic messages, minimal-edit contrasts, warnings and retrospective accounts. On English text the system flagged 84.9-94.1% of harmful cases and 0-4.3% of harmless ones per category. It missed third-party reports of ongoing harm far more often than perpetrators' messages (63.6% against 95.1% recall, all languages), when sent without the settings the API offers for…
PDF arXiv
arXiv cs.LGOct 19
Understanding the Benefits of Scaffolds through Computational Separations
Jingchu Gai, Adam Block, Andrej Risteski
Inference-time harnesses, i.e., external procedures that call a model many times and keep state across calls, can substantially improve model capabilities. In this paper, we investigate the benefits of using such harnesses and the extent to which these benefits can be internalized through training. More precisely, we conceptualize a harness $\mathcal{A}$ as an arbitrary wrapper around a base model $M$ and attempt to distill a student model from the harness's traces. Empirically we find that, even with simple scaffolds and small models, the inference-time harness significantly outperforms a distilled student. We then prove that such a gap is due to a fundamental representational limitation: inference-time harnesses can compute a closure of their per-call primitive that is $\NL$-complete even when the primitive is a trivial $\AC^0$ lookup, and that no fixed-depth forward pass can represent under the widely believed conjecture that $\NL\not\subseteq\TC^0$. Our theory suggests that expanding the representational capacity of the student relative to the task at hand, e.g., by enlarging the model, making the task easier, or lengthening its chain-of-thought, is necessary to close the…
PDF arXiv

Activity · 12 weeks

Fields · AI · updated 59m ago

AI agents

Found by the topic model: ai agents, code generation, reasoning tasks.

467papers this week−29.5% vs last week

All papers · 3.3k With code · 178

Newest Most relevant

2.7k papers this month

Open problems in AI agents

Evaluating long-horizon agentic behavior in real-world tasks3 papers raise it · 2 address it
Current benchmarks often simplify or omit the key work agents must perform, making it hard to assess sustained autonomous problem-solving.
- Scaling Agentic Data for Long-Horizon Terminal Intelligenceraises · addresses
- FDE-Gym: The Last Mile of AI Deliveryraises · addresses
- AgentSWE: Can Coding Agents Build the Agent You Actually Want?raises
Faithful agent development from natural-language requirements1 paper raises it · 1 addresses it
Agents often fail to deliver what users actually ask for, passing superficial tests but not meeting the full requirement.
- AgentSWE: Can Coding Agents Build the Agent You Actually Want?raises · addresses
Detecting when past context matters in long conversations1 paper raises it · 0 address it
AI companions struggle to recognize when historical context is relevant, leading to missed or incorrect responses.
- RealCompanion: Benchmarking Human Understanding from Reasoning over Longitudinal Real-World Conversationsraises
Reliable no-thinking compliance in LLMs1 paper raises it · 1 addresses it
LLMs often ignore or bypass explicit instructions to avoid reasoning, making it hard to enforce strict answer-only behavior.
- Thinking Inertia: LLMs Keep Thinking When Told Not Toraises · addresses
Transition-aware safety for GUI agents1 paper raises it · 1 addresses it
Safety checks that only inspect screens can be fooled by reusing legitimate screens in unexpected transitions.
- Right Screen, Wrong Transition: World Models as Verifiers for GUI Agentsraises · addresses

Named by an AI from these papers' abstracts.