Papers

Field

Top tools here

Open problems in AI agents

Evaluating agent harness reliability and failure attribution1 paper raises it · 1 addresses it
Current benchmarks focus on model capabilities rather than the reliability of the agent harness, which is critical for real-world deployment.
- Trial by Trace: Benchmarking AI Agent Harnesses for Reliability and Failure Attributionraises · addresses
Long-horizon reasoning and memory in agents2 papers raise it · 2 address it
Agents struggle to maintain coherence, adapt strategies, and use past evidence over long interactions, limiting their ability to solve complex, real-world tasks.
Faithful agent development from natural-language requirements1 paper raises it · 1 addresses it
Coding agents often fail to deliver agents that truly meet the specified requirements, not just appear to.
- AgentSWE: Can Coding Agents Build the Agent You Actually Want?raises · addresses
Realistic evaluation of agent delivery in ambiguous environments1 paper raises it · 1 addresses it
Existing benchmarks do not capture the ambiguity, live systems, and unstated constraints of real-world delivery tasks.
- FDE-Gym: The Last Mile of AI Deliveryraises · addresses
Controlling and verifying no-thinking behavior in LLMs1 paper raises it · 1 addresses it
LLMs often ignore explicit instructions to avoid reasoning, making it hard to enforce strict answer-only outputs.
- Thinking Inertia: LLMs Keep Thinking When Told Not Toraises · addresses

Named by an AI from these papers' abstracts.

Open problems

Recently funded

Topics in this field

code generation · coding agents · software engineering447 papers104 this week
ai agents · agents · multi agent500 papers65 this week
agent skills · language model agents · skill evolution275 papers41 this week
reasoning tasks · reasoning paths · reasoning models385 papers36 this week
harness optimization · agent harnesses · harness evolution288 papers33 this week
ai agents · multi agent llm · multi agent215 papers33 this week

Topics in this field

Active contributors · 90 days

Active contributors

Activity · 12 weeks

Activity

Recent papers

All papers in Research

arXiv cs.AIOct 18
When Empathy Misses the Goal: A Benchmark for Goal Displacement in LLM Advice
Dean Ariel, Guy Laban
Conversational large language models can respond with warmth and reassurance while failing to advance the user's stated goal. We conceptualize this failure as $\textbf{goal displacement}$, the substitution of persona-maintenance content for goal-relevant substance. To measure it, we introduce $\textbf{GDB}$ (Goal Displacement Benchmark), comprising 230 scenarios across nine domains and $4{,}134$ human-scored responses from 18 models. A $2 \times 2$ taxonomy crosses hard constraints with emotional distress, distinguishing failures to confront infeasible goals or combine empathy with practical guidance from reverse displacement, where unsolicited problem-solving replaces requested emotional support. A deletion-test rubric assesses whether substantive help remains after purely supportive sentences are removed. Independent raters achieve weighted $κ= 0.82$ against author scores for displacement severity. Trap displacement ranges from $0.0\%$ to $37.7\%$ and is strongly negatively correlated with Chatbot Arena Elo ($ρ= -0.706$, $n = 15$). Reverse displacement reaches $41.9\%$, including substantial rates in models with negligible trap displacement. A factorial probe supports a…
PDF arXiv
arXiv cs.AIOct 18
Assessing the AI Readiness of Construction Datasets: A Multidimensional Operational Framework
Xiaoyu Hou, Bo Xiao, Abdolmajid Erfani, Ronghua Xu, Yu Cai
The rapid growth of artificial intelligence (AI) applications in construction has increased the demand for datasets suitable for reliable model development, evaluation, and reuse. However, construction datasets vary widely in documentation, quality, governance, and evaluation support, and consistent criteria for assessing AI readiness remain limited. This study develops a Construction Dataset AI Readiness Framework with 30 attributes across four dimensions: Metadata Completeness, Data Quality and Representativeness, Data Governance and Accessibility, and AI Utility. The framework uses attribute-specific scoring, applicability screening, and automated and manual verification. It was evaluated using 21 publicly available datasets across seven data-type groups. Overall readiness scores ranged from 0.720 to 0.964, with a mean of 0.879. Hybrid verification achieved 90.5% agreement between automated suggestions and manually verified scores. The framework supports consistent assessment of heterogeneous construction datasets and identification of readiness gaps.
PDF arXiv
arXiv cs.LGOct 18
Consistent LLM Call Handling Across Java Frameworks: A Reference Implementation and Benchmark
Oleksandr Klymenko
Java applications reach large language models (LLMs) through an integration framework, and rely on it to handle a failing provider: to retry what can be retried and nothing else, to wait as long as the provider asks, never to present a truncated answer as complete, and to record each failure in the application's metrics. We state these expectations as a six-clause reliability contract and measure Spring AI 2.0.1 and LangChain4j 1.21.0 against it with a scripted OpenAI-compatible provider: fifteen failure scenarios, synchronous and streaming, three repetitions. LangChain4j meets it in 47 of 77 applicable cases with its defaults and in 51 of 83 when configured explicitly, Spring AI in 72 of 77 and 76 of 83; both deliver a stream that ended without its terminator as a complete answer. We then place one implementation of the contract beneath both frameworks, at the HTTP boundary - an OkHttp interceptor for Spring AI, an HttpClient for LangChain4j, and a java.net.http transport for code without a framework - sharing a 725-line core that depends only on the JDK and the Micrometer Observation API. With it, all three clients behave identically and meet the contract in 90 of 91 cases; the…
PDF arXiv
arXiv cs.AIOct 18
On Reasoning Correctness Directions
Munachiso Samuel Nwadike, Zayd M. K. Zuhri, Rifo Genadi, Abzal Nurgazy, Ali Mekky +3
We present experimental evidence suggesting that information about whether a model's reasoning is on the right track can be linearly decodable without defining a causally useful direction for steering. To study this, we introduce ChainSearch, a controlled arithmetic reasoning task that extends Countdown with explicit search and backtracking. We train a 3-million-parameter model to solve this task. Its compact size allows us to study internal model correctness directions in detail using linear probes, mean activation differences, and direct optimisation, as part of a rigorous search for internal reasoning-correctness directions. We find that while the model contains statistically significant internal directions of reasoning correctness, none of the resulting directions consistently improves reasoning under causal intervention. This calls into question whether the linear representation hypothesis is sufficient for identifying causally useful directions in the context of reasoning. Our code is made available together with this work.
PDF arXiv
arXiv cs.AIOct 18
Stepwise Reasoning Verification: Don't Generate, Classify!
Yifei Shi, Michalis Korakakis, Rami Aly, Eric Chamoun, Lukasz Szpruch +1
Verifying the reasoning of large language models is increasingly important as they are applied to complex mathematical and scientific problems. Existing work focuses on large decoder models and reports mixed results for generative and discriminative verifiers, leaving unclear which paradigm is more effective and whether large models are necessary. In this work, we cast stepwise verification as a natural language inference task: the problem and partial solution prefix form the premise, while the next reasoning step serves as the hypothesis to be classified as entailed or non-entailed. This formulation enables error localisation at individual reasoning steps and direct comparison of encoder classification, decoder classification, and decoder verdict generation across model scales and annotation budgets. Across five reasoning datasets, a 0.15B encoder classifier outperforms most generative verifiers with up to 4B parameters. On identical decoder backbones, classification outperforms generation from 0.5B to 4B parameters and matches it on the strongest 7B backbone---reaching comparable performance with roughly one tenth as many annotated problems. Our findings establish…
PDF arXiv
arXiv cs.AIOct 18
When Form Changes but Logic Doesn't: Building Logic-invariant LLMs through Logical Structure
Xuyuan Liu, Xinshuai Dong, Elynn Chen, Yujun Yan
Large language models (LLMs) have achieved strong performance across diverse reasoning tasks, yet it remains unclear whether this performance reflects an understanding of underlying logical structure or a reliance on shallow semantic cues. A key indicator of such understanding is whether models treat logically equivalent inputs consistently, producing the same predictions despite differences in expression. Through systematic evaluation on curated data, we show that even strong LLMs often fail this test: they do not consistently preserve predictions across logically equivalent inputs and generalize poorly to unseen logical forms. To address this limitation, we propose LoGIcal STructure-guided reasoning (LoGIST), a framework that guides LLM reasoning with explicit logical structure. LoGIST maps each input instance to a decision diagram, a directed acyclic graph that captures the underlying logical structure, and encodes this graph into a logical embedding that conditions the LLM's reasoning process. Across multiple training settings, LoGIST reduces inconsistency under logical equivalence and improves generalization to unseen logical forms, suggesting that logical invariance is a…
PDF arXiv

Papers

Topics in this field

code generation · coding agents · software engineering447 papers104 this week
ai agents · agents · multi agent500 papers65 this week
agent skills · language model agents · skill evolution275 papers41 this week
reasoning tasks · reasoning paths · reasoning models385 papers36 this week
harness optimization · agent harnesses · harness evolution288 papers33 this week
ai agents · multi agent llm · multi agent215 papers33 this week

Active contributors · 90 days

Recent papers

All papers in Research

arXiv cs.AIOct 18
When Empathy Misses the Goal: A Benchmark for Goal Displacement in LLM Advice
Dean Ariel, Guy Laban
Conversational large language models can respond with warmth and reassurance while failing to advance the user's stated goal. We conceptualize this failure as $\textbf{goal displacement}$, the substitution of persona-maintenance content for goal-relevant substance. To measure it, we introduce $\textbf{GDB}$ (Goal Displacement Benchmark), comprising 230 scenarios across nine domains and $4{,}134$ human-scored responses from 18 models. A $2 \times 2$ taxonomy crosses hard constraints with emotional distress, distinguishing failures to confront infeasible goals or combine empathy with practical guidance from reverse displacement, where unsolicited problem-solving replaces requested emotional support. A deletion-test rubric assesses whether substantive help remains after purely supportive sentences are removed. Independent raters achieve weighted $κ= 0.82$ against author scores for displacement severity. Trap displacement ranges from $0.0\%$ to $37.7\%$ and is strongly negatively correlated with Chatbot Arena Elo ($ρ= -0.706$, $n = 15$). Reverse displacement reaches $41.9\%$, including substantial rates in models with negligible trap displacement. A factorial probe supports a…
PDF arXiv
arXiv cs.AIOct 18
Assessing the AI Readiness of Construction Datasets: A Multidimensional Operational Framework
Xiaoyu Hou, Bo Xiao, Abdolmajid Erfani, Ronghua Xu, Yu Cai
The rapid growth of artificial intelligence (AI) applications in construction has increased the demand for datasets suitable for reliable model development, evaluation, and reuse. However, construction datasets vary widely in documentation, quality, governance, and evaluation support, and consistent criteria for assessing AI readiness remain limited. This study develops a Construction Dataset AI Readiness Framework with 30 attributes across four dimensions: Metadata Completeness, Data Quality and Representativeness, Data Governance and Accessibility, and AI Utility. The framework uses attribute-specific scoring, applicability screening, and automated and manual verification. It was evaluated using 21 publicly available datasets across seven data-type groups. Overall readiness scores ranged from 0.720 to 0.964, with a mean of 0.879. Hybrid verification achieved 90.5% agreement between automated suggestions and manually verified scores. The framework supports consistent assessment of heterogeneous construction datasets and identification of readiness gaps.
PDF arXiv
arXiv cs.LGOct 18
Consistent LLM Call Handling Across Java Frameworks: A Reference Implementation and Benchmark
Oleksandr Klymenko
Java applications reach large language models (LLMs) through an integration framework, and rely on it to handle a failing provider: to retry what can be retried and nothing else, to wait as long as the provider asks, never to present a truncated answer as complete, and to record each failure in the application's metrics. We state these expectations as a six-clause reliability contract and measure Spring AI 2.0.1 and LangChain4j 1.21.0 against it with a scripted OpenAI-compatible provider: fifteen failure scenarios, synchronous and streaming, three repetitions. LangChain4j meets it in 47 of 77 applicable cases with its defaults and in 51 of 83 when configured explicitly, Spring AI in 72 of 77 and 76 of 83; both deliver a stream that ended without its terminator as a complete answer. We then place one implementation of the contract beneath both frameworks, at the HTTP boundary - an OkHttp interceptor for Spring AI, an HttpClient for LangChain4j, and a java.net.http transport for code without a framework - sharing a 725-line core that depends only on the JDK and the Micrometer Observation API. With it, all three clients behave identically and meet the contract in 90 of 91 cases; the…
PDF arXiv
arXiv cs.AIOct 18
On Reasoning Correctness Directions
Munachiso Samuel Nwadike, Zayd M. K. Zuhri, Rifo Genadi, Abzal Nurgazy, Ali Mekky +3
We present experimental evidence suggesting that information about whether a model's reasoning is on the right track can be linearly decodable without defining a causally useful direction for steering. To study this, we introduce ChainSearch, a controlled arithmetic reasoning task that extends Countdown with explicit search and backtracking. We train a 3-million-parameter model to solve this task. Its compact size allows us to study internal model correctness directions in detail using linear probes, mean activation differences, and direct optimisation, as part of a rigorous search for internal reasoning-correctness directions. We find that while the model contains statistically significant internal directions of reasoning correctness, none of the resulting directions consistently improves reasoning under causal intervention. This calls into question whether the linear representation hypothesis is sufficient for identifying causally useful directions in the context of reasoning. Our code is made available together with this work.
PDF arXiv
arXiv cs.AIOct 18
Stepwise Reasoning Verification: Don't Generate, Classify!
Yifei Shi, Michalis Korakakis, Rami Aly, Eric Chamoun, Lukasz Szpruch +1
Verifying the reasoning of large language models is increasingly important as they are applied to complex mathematical and scientific problems. Existing work focuses on large decoder models and reports mixed results for generative and discriminative verifiers, leaving unclear which paradigm is more effective and whether large models are necessary. In this work, we cast stepwise verification as a natural language inference task: the problem and partial solution prefix form the premise, while the next reasoning step serves as the hypothesis to be classified as entailed or non-entailed. This formulation enables error localisation at individual reasoning steps and direct comparison of encoder classification, decoder classification, and decoder verdict generation across model scales and annotation budgets. Across five reasoning datasets, a 0.15B encoder classifier outperforms most generative verifiers with up to 4B parameters. On identical decoder backbones, classification outperforms generation from 0.5B to 4B parameters and matches it on the strongest 7B backbone---reaching comparable performance with roughly one tenth as many annotated problems. Our findings establish…
PDF arXiv
arXiv cs.AIOct 18
When Form Changes but Logic Doesn't: Building Logic-invariant LLMs through Logical Structure
Xuyuan Liu, Xinshuai Dong, Elynn Chen, Yujun Yan
Large language models (LLMs) have achieved strong performance across diverse reasoning tasks, yet it remains unclear whether this performance reflects an understanding of underlying logical structure or a reliance on shallow semantic cues. A key indicator of such understanding is whether models treat logically equivalent inputs consistently, producing the same predictions despite differences in expression. Through systematic evaluation on curated data, we show that even strong LLMs often fail this test: they do not consistently preserve predictions across logically equivalent inputs and generalize poorly to unseen logical forms. To address this limitation, we propose LoGIcal STructure-guided reasoning (LoGIST), a framework that guides LLM reasoning with explicit logical structure. LoGIST maps each input instance to a decision diagram, a directed acyclic graph that captures the underlying logical structure, and encodes this graph into a logical embedding that conditions the LLM's reasoning process. Across multiple training settings, LoGIST reduces inconsistency under logical equivalence and improves generalization to unseen logical forms, suggesting that logical invariance is a…
PDF arXiv

Activity · 12 weeks

Fields · AI · updated 7m ago

AI agents

Found by the topic model: ai agents, code generation, reasoning tasks.

505papers this week−20.2% vs last week

All papers · 3.4k With code · 182

Newest Most relevant

2.7k papers this month

Open problems in AI agents

Evaluating agent harness reliability and failure attribution1 paper raises it · 1 addresses it
Current benchmarks focus on model capabilities rather than the reliability of the agent harness, which is critical for real-world deployment.
- Trial by Trace: Benchmarking AI Agent Harnesses for Reliability and Failure Attributionraises · addresses
Long-horizon reasoning and memory in agents2 papers raise it · 2 address it
Agents struggle to maintain coherence, adapt strategies, and use past evidence over long interactions, limiting their ability to solve complex, real-world tasks.
Faithful agent development from natural-language requirements1 paper raises it · 1 addresses it
Coding agents often fail to deliver agents that truly meet the specified requirements, not just appear to.
- AgentSWE: Can Coding Agents Build the Agent You Actually Want?raises · addresses
Realistic evaluation of agent delivery in ambiguous environments1 paper raises it · 1 addresses it
Existing benchmarks do not capture the ambiguity, live systems, and unstated constraints of real-world delivery tasks.
- FDE-Gym: The Last Mile of AI Deliveryraises · addresses
Controlling and verifying no-thinking behavior in LLMs1 paper raises it · 1 addresses it
LLMs often ignore explicit instructions to avoid reasoning, making it hard to enforce strict answer-only outputs.
- Thinking Inertia: LLMs Keep Thinking When Told Not Toraises · addresses

Named by an AI from these papers' abstracts.