Papers

Field

Top tools here

All tools

jcode+233% downloads · 25.5k/wk+233.1%
billion-context+259% downloads · 1.1M/wk+258.8%
oh-my-claudecode+47% downloads · 7.7k/wk+47.4%
TanStack AI+68% downloads · 1M/wk+67.7%
Vibe-Trading+100% downloads · 3.1k/wk+100.3%

Top tools here

Open problems in Agents

Long-horizon task completion with exploration and verification3 papers raise it · 2 address it
Agents struggle to accumulate evidence, adjust strategies, and verify results over long, open-ended tasks in real-world settings.
- Scaling Agentic Data for Long-Horizon Terminal Intelligenceraises · addresses
- FDE-Gym: The Last Mile of AI Deliveryraises · addresses
- Into the Unknown: Benchmarking AI agents on open scientific research problemsraises
Faithful agent development from natural-language requirements1 paper raises it · 1 addresses it
Coding agents often produce systems that appear to meet requirements but fail to deliver the intended behavior on held-out cases.
- AgentSWE: Can Coding Agents Build the Agent You Actually Want?raises · addresses
Detecting when past context matters in long conversations1 paper raises it · 0 address it
Agents cannot reliably identify when historical information is relevant, leading to missed or misapplied context.
- RealCompanion: Benchmarking Human Understanding from Reasoning over Longitudinal Real-World Conversationsraises
Transition-aware safety for GUI agents1 paper raises it · 1 addresses it
Safety checks that only inspect screens can be bypassed by reusing legitimate screens in unexpected transitions.
- Right Screen, Wrong Transition: World Models as Verifiers for GUI Agentsraises · addresses
Learning from experience in unfamiliar environments1 paper raises it · 1 addresses it
Agents often rely on pretrained knowledge rather than acquiring new knowledge through interaction in novel settings.
- Learn2Play Bench: How Well Do LLM Agents Learn from Experience in Unfamiliar Environments?raises · addresses

Named by an AI from these papers' abstracts.

Open problems

Recently funded

Market

Hongxiong AISeries A · Oct 25undisclosed
Pathos AISeries B+ · Oct 23$237M
Tolan AI Companion AppStage not given · Oct 23$30M
hexafarmsStage not given · Oct 23$5.4M
CygridPre-seed · Oct 23$3M

Recently funded

Topics in this field

coding agents · code generation · coding agent318 papers62 this week
recursive self improvement · harness evolution · adaptive278 papers37 this week
tool using agents · agent systems · multi agent223 papers35 this week
ai agents · indirect prompt injection · adversarial328 papers32 this week
skills · tasks · ai321 papers32 this week
agent · agentic · llm agents179 papers32 this week

Topics in this field

Active contributors · 90 days

Active contributors

Activity · 12 weeks

Activity

Recent papers

All papers in Research

arXiv cs.AIOct 7
Cost-Efficient Theorem Proving via Agent Orchestration in Program Verification
Shuangjie Yao, Nikolaus Holzer, Mark Paul Santolucito, Baishakhi Ray, Suman Jana +1
Program verification establishes software correctness through machine-checkable proofs constructed in theorem provers. It's a guarantee especially valuable for code generated by large language models (LLMs), which is fluent but carries no assurance of correctness. Almost all existing provers, however, pursue pass rates alone at whatever sampling or search budget it takes, and overlook the success-vs-cost frontier; yet real software often carries hundreds of interdependent proof obligations, so what matters at scale is not whether one theorem can be proved, but how many can be proved economically. We introduce CoCo-Prover, which formalizes cost-efficient program proving as metalevel decision-making under cost, grounded on two-level proof graphs: an AND/OR proof hypergraph within each declaration is joined to a lemma-dependency graph across declarations; and at each step, it answers two questions: which open goals to select, and which actions to purchase on these goals. Selection stays symbolic as a topological pass over the proof graphs. Action choice is agent orchestration via metalevel decision-making: an agentic router treats every bounded specialist invocation as a separately…
PDF arXiv
arXiv cs.AIOct 7
Shared and structured inputs undermine collective random choice by reasoning AI agents
Takahiro Ezaki, Naoto Imura, Katsuhiro Nishinari
Random selection is widely used in resource allocation and auditing, making reliable implementation essential for AI-agent systems. Behavioural tests across six reasoning models uncovered threshold and divisibility rules used in identifier-based choices. For threshold-following GPT-6 Sol and Gemini 3.8 Flash, single-agent measurements prospectively predicted correlated participation under shared identifiers and biased participation under distinct identifiers with common timestamp bits. Changing dates, formats and identifier labels revealed when these predictions held. Explicit instructions to randomize independently reduced but did not eliminate shared-input correlation. To test implications for oversight, we asked four models to select customer requests randomly for human review. GPT-6 Sol approached the target rate while selecting predictably from identifiers; the others rarely selected requests. All four closely followed supplied random draws. These findings expose collective and audit vulnerabilities that selection rates alone miss, making input-dependent bias, correlation and predictability central targets for agent evaluation.
PDF arXiv
arXiv cs.CROct 7
When AI Finds Hidden Messages, Does It Report?
William Guey, Rashik Jahangir, Pierrick Bougault, Vitor D. de Moura, Wei Zhang +1
When an assistant encounters a message for another AI, does it tell its user? Four fixed model-provider deployments perform simulated source tasks in 1,280 ordinary-note and 128 enhanced-note sessions. Harmless and harmful messages have matched plaintext and ROT13 versions, with no-message controls. Observers receive no decoder or decoded meaning; a requested reference code incentivizes inspection. Asking for reports increases rule-detected notifications identifying another AI as recipient by 53.1 percentage points for harmless ROT13 messages and 54.7 for harmful ones. This is a joint inspection, recognition, and notification effect; missing-response bounds are 38.3--77.3 and 36.7--78.1 points. Model-based trace checks identify eleven ordinary plaintext cases where agents interpret the message but do not notify their user. Seven encoded omissions are verified with enhanced notes; ordinary encoded omissions remain unverified. Seven simulated filename disclosures coexist with accurate review-status answers, and two answers use a planted false count. Interpretation, notification, and authorized task performance are distinct outcomes.
PDF arXiv
arXiv cs.LGCodeOct 7▲ 14
Coding-Agent Benchmarks Should Match Their Users' Task Flows
Igor Slinko, Yaroslav Golubev, Sergey Titov
The evaluation of coding agents generally strives to be as realistic as possible. In our study, we collect 4,782 agent sessions of real software engineers in JetBrains IDEs, which we call Production Sessions. Since our subject is interactive agents, we study the sessions with at least three user messages (33% of the sample). These long sessions differ from issue-derived benchmark tasks in two ways: (i) user requests span a far wider mix of task types - questions about the project's code, planning, review, refactoring, execution - and (ii) users switch between types throughout a session. Long-session samples from three public interaction corpora exhibit markedly different Task Flows (the distributions of session lengths, task types, and type-to-type transitions), so no single interaction distribution is universally realistic: benchmarks should name a target use case and calibrate to measurements from it. We present SWE-TaskFlow, an approach for transforming any issue-derived benchmark: it preserves the verified tasks and tests while steering the interaction toward a target Task Flow through prompt splitting and verifiable repository QA, with a TaskFlow Alignment Score (TFAS) for…
PDF arXiv Code
arXiv cs.AIOct 7
How Do Agentic LLMs Decide to Call Tools? A Tool-Call Vector Shaped by Suppression
Xijie Gong, Tingxu Han, Jiahao Zhang, Wei Song, Ziqi Ding +3
Tool calling, invoking external tools on demand, is central to agentic LLMs, yet the mechanism that decides whether a model calls a tool or responds directly remains poorly understood. Agentic prompts are long and heavily scaffolded, combining role instructions, tool schemas, format templates, and the user's request across hundreds of tokens, creating a noisy, highly entangled context in which no single controllable variable for mechanistic analysis is obvious. To obtain such a variable, we propose a method that converts complex agentic prompts into minimal contrastive pairs in which a single request verb determines the tool-call decision: replacing an execution-verb (e.g., \textit{write}) with an analysis-verb (e.g., \textit{discuss}) reliably flips the decision, suggesting it is mediated by a compact internal state. We construct 500 such paired prompts across Python, Java, and C++ (300 for mechanistic analysis, 200 held out for evaluation). We trace the decision to a vector, $μ_Δ$, that is both causally necessary and sufficient and generalizes beyond the discovery prompts to native multi-turn $τ^2$-Bench trajectories and verb-free requests. Behavioral ablations show that the…
PDF arXiv
arXiv cs.SEOct 7
TestJack: Should you trust the results in coding benchmarks? Agentic Coding Benchmarks Auditing via Evaluator Evolution
Shuangjie Yao, Hao Wang, Koushik Sen, Simin Chen, Baishakhi Ray +1
Large language model (LLM) agents are rapidly reshaping software engineering, accompanied by an explosion of new code benchmarks. Yet nearly all existing benchmarks still rely on the same decades-old criterion: a solution is correct if it passes a fixed set of unit tests. Such tests are often insufficient: they check only part of what the task requires, so agents can reward hack them or silently miss required behavior while still passing every test. As a result, higher benchmark scores may partly reflect better adaptation to the evaluator rather than better problem solving. Existing works focus on static test augmentation: they strengthen each task's tests once, before any trial is seen, and thus overlook how real trials actually fail. We introduce TestJack, a scalable framework for evaluating patches beyond fixed tests. For each trial, TestJack generates tests targeting prompt requirements the patch may violate, retains only tests passed by the ground-truth patch, and re-examines any trial failures. Each confirmed failure is thus supported by a replayable test. To reduce evaluation cost, we also introduce a lightweight variant which audits a random sample of trials in depth and…
PDF arXiv

Papers

Top tools here

All tools

jcode+233% downloads · 25.5k/wk+233.1%
billion-context+259% downloads · 1.1M/wk+258.8%
oh-my-claudecode+47% downloads · 7.7k/wk+47.4%
TanStack AI+68% downloads · 1M/wk+67.7%
Vibe-Trading+100% downloads · 3.1k/wk+100.3%

Recently funded

Market

Hongxiong AISeries A · Oct 25undisclosed
Pathos AISeries B+ · Oct 23$237M
Tolan AI Companion AppStage not given · Oct 23$30M
hexafarmsStage not given · Oct 23$5.4M
CygridPre-seed · Oct 23$3M

Topics in this field

coding agents · code generation · coding agent318 papers62 this week
recursive self improvement · harness evolution · adaptive278 papers37 this week
tool using agents · agent systems · multi agent223 papers35 this week
ai agents · indirect prompt injection · adversarial328 papers32 this week
skills · tasks · ai321 papers32 this week
agent · agentic · llm agents179 papers32 this week

Active contributors · 90 days

Activity · 12 weeks

Recent papers

All papers in Research

arXiv cs.AIOct 7
Cost-Efficient Theorem Proving via Agent Orchestration in Program Verification
Shuangjie Yao, Nikolaus Holzer, Mark Paul Santolucito, Baishakhi Ray, Suman Jana +1
Program verification establishes software correctness through machine-checkable proofs constructed in theorem provers. It's a guarantee especially valuable for code generated by large language models (LLMs), which is fluent but carries no assurance of correctness. Almost all existing provers, however, pursue pass rates alone at whatever sampling or search budget it takes, and overlook the success-vs-cost frontier; yet real software often carries hundreds of interdependent proof obligations, so what matters at scale is not whether one theorem can be proved, but how many can be proved economically. We introduce CoCo-Prover, which formalizes cost-efficient program proving as metalevel decision-making under cost, grounded on two-level proof graphs: an AND/OR proof hypergraph within each declaration is joined to a lemma-dependency graph across declarations; and at each step, it answers two questions: which open goals to select, and which actions to purchase on these goals. Selection stays symbolic as a topological pass over the proof graphs. Action choice is agent orchestration via metalevel decision-making: an agentic router treats every bounded specialist invocation as a separately…
PDF arXiv
arXiv cs.AIOct 7
Shared and structured inputs undermine collective random choice by reasoning AI agents
Takahiro Ezaki, Naoto Imura, Katsuhiro Nishinari
Random selection is widely used in resource allocation and auditing, making reliable implementation essential for AI-agent systems. Behavioural tests across six reasoning models uncovered threshold and divisibility rules used in identifier-based choices. For threshold-following GPT-6 Sol and Gemini 3.8 Flash, single-agent measurements prospectively predicted correlated participation under shared identifiers and biased participation under distinct identifiers with common timestamp bits. Changing dates, formats and identifier labels revealed when these predictions held. Explicit instructions to randomize independently reduced but did not eliminate shared-input correlation. To test implications for oversight, we asked four models to select customer requests randomly for human review. GPT-6 Sol approached the target rate while selecting predictably from identifiers; the others rarely selected requests. All four closely followed supplied random draws. These findings expose collective and audit vulnerabilities that selection rates alone miss, making input-dependent bias, correlation and predictability central targets for agent evaluation.
PDF arXiv
arXiv cs.CROct 7
When AI Finds Hidden Messages, Does It Report?
William Guey, Rashik Jahangir, Pierrick Bougault, Vitor D. de Moura, Wei Zhang +1
When an assistant encounters a message for another AI, does it tell its user? Four fixed model-provider deployments perform simulated source tasks in 1,280 ordinary-note and 128 enhanced-note sessions. Harmless and harmful messages have matched plaintext and ROT13 versions, with no-message controls. Observers receive no decoder or decoded meaning; a requested reference code incentivizes inspection. Asking for reports increases rule-detected notifications identifying another AI as recipient by 53.1 percentage points for harmless ROT13 messages and 54.7 for harmful ones. This is a joint inspection, recognition, and notification effect; missing-response bounds are 38.3--77.3 and 36.7--78.1 points. Model-based trace checks identify eleven ordinary plaintext cases where agents interpret the message but do not notify their user. Seven encoded omissions are verified with enhanced notes; ordinary encoded omissions remain unverified. Seven simulated filename disclosures coexist with accurate review-status answers, and two answers use a planted false count. Interpretation, notification, and authorized task performance are distinct outcomes.
PDF arXiv
arXiv cs.LGCodeOct 7▲ 14
Coding-Agent Benchmarks Should Match Their Users' Task Flows
Igor Slinko, Yaroslav Golubev, Sergey Titov
The evaluation of coding agents generally strives to be as realistic as possible. In our study, we collect 4,782 agent sessions of real software engineers in JetBrains IDEs, which we call Production Sessions. Since our subject is interactive agents, we study the sessions with at least three user messages (33% of the sample). These long sessions differ from issue-derived benchmark tasks in two ways: (i) user requests span a far wider mix of task types - questions about the project's code, planning, review, refactoring, execution - and (ii) users switch between types throughout a session. Long-session samples from three public interaction corpora exhibit markedly different Task Flows (the distributions of session lengths, task types, and type-to-type transitions), so no single interaction distribution is universally realistic: benchmarks should name a target use case and calibrate to measurements from it. We present SWE-TaskFlow, an approach for transforming any issue-derived benchmark: it preserves the verified tasks and tests while steering the interaction toward a target Task Flow through prompt splitting and verifiable repository QA, with a TaskFlow Alignment Score (TFAS) for…
PDF arXiv Code
arXiv cs.AIOct 7
How Do Agentic LLMs Decide to Call Tools? A Tool-Call Vector Shaped by Suppression
Xijie Gong, Tingxu Han, Jiahao Zhang, Wei Song, Ziqi Ding +3
Tool calling, invoking external tools on demand, is central to agentic LLMs, yet the mechanism that decides whether a model calls a tool or responds directly remains poorly understood. Agentic prompts are long and heavily scaffolded, combining role instructions, tool schemas, format templates, and the user's request across hundreds of tokens, creating a noisy, highly entangled context in which no single controllable variable for mechanistic analysis is obvious. To obtain such a variable, we propose a method that converts complex agentic prompts into minimal contrastive pairs in which a single request verb determines the tool-call decision: replacing an execution-verb (e.g., \textit{write}) with an analysis-verb (e.g., \textit{discuss}) reliably flips the decision, suggesting it is mediated by a compact internal state. We construct 500 such paired prompts across Python, Java, and C++ (300 for mechanistic analysis, 200 held out for evaluation). We trace the decision to a vector, $μ_Δ$, that is both causally necessary and sufficient and generalizes beyond the discovery prompts to native multi-turn $τ^2$-Bench trajectories and verb-free requests. Behavioral ablations show that the…
PDF arXiv
arXiv cs.SEOct 7
TestJack: Should you trust the results in coding benchmarks? Agentic Coding Benchmarks Auditing via Evaluator Evolution
Shuangjie Yao, Hao Wang, Koushik Sen, Simin Chen, Baishakhi Ray +1
Large language model (LLM) agents are rapidly reshaping software engineering, accompanied by an explosion of new code benchmarks. Yet nearly all existing benchmarks still rely on the same decades-old criterion: a solution is correct if it passes a fixed set of unit tests. Such tests are often insufficient: they check only part of what the task requires, so agents can reward hack them or silently miss required behavior while still passing every test. As a result, higher benchmark scores may partly reflect better adaptation to the evaluator rather than better problem solving. Existing works focus on static test augmentation: they strengthen each task's tests once, before any trial is seen, and thus overlook how real trials actually fail. We introduce TestJack, a scalable framework for evaluating patches beyond fixed tests. For each trial, TestJack generates tests targeting prompt requirements the patch may violate, retains only tests passed by the ground-truth patch, and re-examines any trial failures. Each confirmed failure is thus supported by a replayable test. To reduce evaluation cost, we also introduce a lightweight variant which audits a random sample of trials in depth and…
PDF arXiv

Open problems in Agents

Long-horizon task completion with exploration and verification3 papers raise it · 2 address it
Agents struggle to accumulate evidence, adjust strategies, and verify results over long, open-ended tasks in real-world settings.
- Scaling Agentic Data for Long-Horizon Terminal Intelligenceraises · addresses
- FDE-Gym: The Last Mile of AI Deliveryraises · addresses
- Into the Unknown: Benchmarking AI agents on open scientific research problemsraises
Faithful agent development from natural-language requirements1 paper raises it · 1 addresses it
Coding agents often produce systems that appear to meet requirements but fail to deliver the intended behavior on held-out cases.
- AgentSWE: Can Coding Agents Build the Agent You Actually Want?raises · addresses
Detecting when past context matters in long conversations1 paper raises it · 0 address it
Agents cannot reliably identify when historical information is relevant, leading to missed or misapplied context.
- RealCompanion: Benchmarking Human Understanding from Reasoning over Longitudinal Real-World Conversationsraises
Transition-aware safety for GUI agents1 paper raises it · 1 addresses it
Safety checks that only inspect screens can be bypassed by reusing legitimate screens in unexpected transitions.
- Right Screen, Wrong Transition: World Models as Verifiers for GUI Agentsraises · addresses
Learning from experience in unfamiliar environments1 paper raises it · 1 addresses it
Agents often rely on pretrained knowledge rather than acquiring new knowledge through interaction in novel settings.
- Learn2Play Bench: How Well Do LLM Agents Learn from Experience in Unfamiliar Environments?raises · addresses

Named by an AI from these papers' abstracts.

Fields · AI · updated 44m ago

Agents

LLM agents: planning, tool use, multi-agent systems, computer/web agents and agent benchmarks.

392papers this week−35.3% vs last week

All papers · 2.7k With code · 143

Newest Most relevant

2.4k papers this month