Papers

Field

Top tools here

All tools

jcode+233% downloads · 25.5k/wk+233.1%
billion-context+259% downloads · 1.1M/wk+258.8%
oh-my-claudecode+47% downloads · 7.7k/wk+47.4%
TanStack AI+68% downloads · 1M/wk+67.7%
Vibe-Trading+100% downloads · 3.1k/wk+100.3%

Top tools here

Open problems in Agents

Long-horizon task completion with exploration and verification3 papers raise it · 2 address it
Agents struggle to accumulate evidence, adjust strategies, and verify results over long, open-ended tasks in real-world settings.
- Scaling Agentic Data for Long-Horizon Terminal Intelligenceraises · addresses
- FDE-Gym: The Last Mile of AI Deliveryraises · addresses
- Into the Unknown: Benchmarking AI agents on open scientific research problemsraises
Faithful agent development from natural-language requirements1 paper raises it · 1 addresses it
Coding agents often produce systems that appear to meet requirements but fail to deliver the intended behavior on held-out cases.
- AgentSWE: Can Coding Agents Build the Agent You Actually Want?raises · addresses
Detecting when past context matters in long conversations1 paper raises it · 0 address it
Agents cannot reliably identify when historical information is relevant, leading to missed or misapplied context.
- RealCompanion: Benchmarking Human Understanding from Reasoning over Longitudinal Real-World Conversationsraises
Transition-aware safety for GUI agents1 paper raises it · 1 addresses it
Safety checks that only inspect screens can be bypassed by reusing legitimate screens in unexpected transitions.
- Right Screen, Wrong Transition: World Models as Verifiers for GUI Agentsraises · addresses
Learning from experience in unfamiliar environments1 paper raises it · 1 addresses it
Agents often rely on pretrained knowledge rather than acquiring new knowledge through interaction in novel settings.
- Learn2Play Bench: How Well Do LLM Agents Learn from Experience in Unfamiliar Environments?raises · addresses

Named by an AI from these papers' abstracts.

Open problems

Recently funded

Market

Hongxiong AISeries A · Oct 25undisclosed
Pathos AISeries B+ · Oct 23$237M
Tolan AI Companion AppStage not given · Oct 23$30M
hexafarmsStage not given · Oct 23$5.4M
CygridPre-seed · Oct 23$3M

Recently funded

Topics in this field

coding agents · code generation · coding agent318 papers62 this week
recursive self improvement · harness evolution · adaptive278 papers37 this week
tool using agents · agent systems · multi agent223 papers35 this week
ai agents · indirect prompt injection · adversarial328 papers32 this week
agent · agentic · llm agents179 papers32 this week
skills · tasks · ai321 papers31 this week

Topics in this field

Active contributors · 90 days

Active contributors

Activity · 12 weeks

Activity

Recent papers

All papers in Research

arXiv cs.CROct 7
From Investigation Failures to Reliable SOC Agents: Understanding and Improving LLM-Based Alert Triage
Saimon Amanuel Tsegai, Alex Kantchelian, Danfeng, Yao, Peng Gao
Security operations centers (SOCs) must triage large volumes of alerts, most of which are benign, while missed attacks can remain uninvestigated. Tool-using large language model (LLM) agents can retrieve evidence during triage, but it remains unclear how reasoning strategies determine what to gather and when an investigation is sufficient to close an alert. We study five representative approaches spanning single-pass tool use, iterative retrieval, sampled investigations, self-review, and explicit verification. To support this study, we build ALERT-BENCH, an interactive benchmark that replays enterprise telemetry through a live SIEM and requires each system to retrieve evidence. Across 1,247 alerts from a multi-stage attack scenario, every approach missed at least 40.4% of attack-related alerts. Trace analysis shows that attack alerts are more likely to be dismissed when searches return no records, same-context review has negative net correction, and dismissal receives no consistently stronger investigation than escalation. Based on these findings, we further design AIDA (Adversarial Investigation and Dialectical Analysis), a multi-agent framework that requires an explicit…
PDF arXiv
arXiv cs.SEOct 7
Beyond Type-checking: Towards Holistic Evaluation of Formal Specification Generation
Srijith Nair, Aditya Vempaty, Jia Liu, Ashish Jagmohan
When generating verifiable code, natural language requirements are mapped to machine checked code using LLMs and agentic workflows. A crucial component of this pipeline is specification generation (SpecGen), which produces a formal contract against which an agent can prove implementation correctness. Proof generation can obtain deterministic feedback from a theorem prover, but SpecGen lacks a definitive check that a generated specification captures the user's intent. A checked proof can therefore establish correctness against a specification that misrepresents the intended behaviour. We take a step towards holistic SpecGen evaluation with a unified dataset assembled from $350$ existing Lean tasks, including $189$ from VERINA and $161$ from CLEVER, and a framework covering formal validity, reference similarity and equivalence, and behavioural adequacy. We distinguish acceptance of required inputs from acceptance of valid outputs and rejection of invalid outputs, while making each metric's evidence scope explicit. Across four SpecGen configurations, restricting the generalized tree edit distance (GTED) comparison, a reference similarity measure, to $32$ jointly measurable VERINA…
PDF arXiv
arXiv cs.AIOct 7
The AI Evaluation Ecosystem
Yash Dave, Sang T. Truong, Serena Wang, Sanmi Koyejo
AI evaluation shapes the decisions of model providers, users, funders, and regulators. We argue that designing valid benchmarks requires contextualizing design choices in the dynamics of this ecosystem of actors. We develop a simulation architecture that combines rule-based market dynamics with LLM-driven strategic actors, building on advances in Generative Agent-Based Modeling (GABM). We model benchmarks, consumer needs, and provider capabilities as vectors over a six-dimensional capability space (reasoning, coding, knowledge, safety, communication, agentic), with structural information partitions across actors. As a case study, we apply this stylized simulation to explore benchmark holdout design. We find that moving from public benchmarks to private holdout benchmarks shrinks the gap between benchmark scores and user satisfaction on most benchmarks but widens it on a few, depending on where holdout weights shift scoring credit. We stress-test our findings at both the instrument and case-study level, drawing on the V&V framework of Sargent (2013) and GABM-specific evidence criteria. Beyond holdout design, our simulation is a hypothesis-generating sandbox for studying how…
PDF arXiv
arXiv cs.CROct 7
Package Hallucination Attacks on Coding Agents through Prompt Injection in Rule Files
Yupu Wang, Zhengyuan Jiang, Reachal Wang, Neil Zhenqiang Gong
Modern agentic coding frameworks increasingly rely on community-shared rule files (e.g., AGENTS.md or .cursorrules) to guide autonomous code generation, yet the security risks of this pipeline remain underexplored. To bridge this gap, we introduce the package hallucination attack, where an attacker injects malicious prompts into benign rule files to induce coding agents to replace legitimate dependencies with attacker-controlled packages. To obtain effective malicious prompts injected into rule files, we propose PackHallu, an evolutionary optimization framework that iteratively rewrites these injected prompts using trajectory-level feedback and LLM-guided mutations. Evaluations across multiple benchmarks, LLMs, and agent frameworks show that PackHallu achieves high attack success rates and strong transferability across diverse models and agent combinations. Our findings demonstrate that coding agents are vulnerable to package hallucination attacks, highlighting the urgent need for stronger security safeguards in autonomous coding systems.
PDF arXiv
arXiv cs.AIOct 7▲ 14
We Query, Therefore We Compute: On Oracle Computation beyond the Machine, with an Application to Agents
Kefan Liu, Fengning Ou, Yelin Luo, Jingdi Lei
Agentic systems use large language models (LLMs) to carry out concrete tasks. Prior work often borrows abstractions such as scheduling, caching or isolation piecemeal from operating systems, so the mechanisms it builds share little common ground, and the shared view of the two forms of agentic system, Workflows and Agents, is limited. We construct an abstract machine that provides both. We treat the LLM as an Oracle and extend a two-stack pushdown automaton with one instruction, which hands the Oracle a whole stack as its query and appends the answer to that same stack. The machine thus performs two computations, the Oracle's and a Turing-complete one that we call the Priestess. A stack that the program only appends to grows autoregressively, as an agent's context does. Two symmetry breakings, S in storage and T in transitions, make a Priestess program the operating system of the programs the Oracle runs, and produce the Agent and the Workflow as the two placements of a task's program. For internally autoregressive Oracles, the two computations synchronize at the end of every answer under certain conditions, and through that synchronization we model caching and analyse…
PDF arXiv
arXiv cs.AIOct 6
Trajectory Abstraction for the Science of Language Agent Behavior
Tianqiang Yan
Scientific studies of language agents need behavioral variables that support hypotheses across tasks and models. We formulate this research problem as learning and testing a hierarchy of trajectory abstractions. A concrete recursive procedure first measures role- and phase-indexed events, proposes temporally constrained relations, and tests their stability across conditions. It then constructs episode-level motif variables from selected relations and repeats the analysis on those variables. Explicit measurement functions connect every abstraction level to the original trajectories. Observations and randomized protocol experiments assess the resulting hypotheses, while comparisons between intervention realizations determine whether an abstraction should be retained, refined, or restricted. We derive a finite-depth bound for accepted reductions, identify protocol effects on fixed abstractions, and characterize realization disagreement and composition of abstraction error. A finite-sample test makes projected intervention consistency operational, and constructed examples illustrate motif construction and abstraction refinement. The formulation distinguishes this experimental…
PDF arXiv

Papers

Top tools here

All tools

jcode+233% downloads · 25.5k/wk+233.1%
billion-context+259% downloads · 1.1M/wk+258.8%
oh-my-claudecode+47% downloads · 7.7k/wk+47.4%
TanStack AI+68% downloads · 1M/wk+67.7%
Vibe-Trading+100% downloads · 3.1k/wk+100.3%

Recently funded

Market

Hongxiong AISeries A · Oct 25undisclosed
Pathos AISeries B+ · Oct 23$237M
Tolan AI Companion AppStage not given · Oct 23$30M
hexafarmsStage not given · Oct 23$5.4M
CygridPre-seed · Oct 23$3M

Topics in this field

coding agents · code generation · coding agent318 papers62 this week
recursive self improvement · harness evolution · adaptive278 papers37 this week
tool using agents · agent systems · multi agent223 papers35 this week
ai agents · indirect prompt injection · adversarial328 papers32 this week
agent · agentic · llm agents179 papers32 this week
skills · tasks · ai321 papers31 this week

Active contributors · 90 days

Recent papers

All papers in Research

arXiv cs.CROct 7
From Investigation Failures to Reliable SOC Agents: Understanding and Improving LLM-Based Alert Triage
Saimon Amanuel Tsegai, Alex Kantchelian, Danfeng, Yao, Peng Gao
Security operations centers (SOCs) must triage large volumes of alerts, most of which are benign, while missed attacks can remain uninvestigated. Tool-using large language model (LLM) agents can retrieve evidence during triage, but it remains unclear how reasoning strategies determine what to gather and when an investigation is sufficient to close an alert. We study five representative approaches spanning single-pass tool use, iterative retrieval, sampled investigations, self-review, and explicit verification. To support this study, we build ALERT-BENCH, an interactive benchmark that replays enterprise telemetry through a live SIEM and requires each system to retrieve evidence. Across 1,247 alerts from a multi-stage attack scenario, every approach missed at least 40.4% of attack-related alerts. Trace analysis shows that attack alerts are more likely to be dismissed when searches return no records, same-context review has negative net correction, and dismissal receives no consistently stronger investigation than escalation. Based on these findings, we further design AIDA (Adversarial Investigation and Dialectical Analysis), a multi-agent framework that requires an explicit…
PDF arXiv
arXiv cs.SEOct 7
Beyond Type-checking: Towards Holistic Evaluation of Formal Specification Generation
Srijith Nair, Aditya Vempaty, Jia Liu, Ashish Jagmohan
When generating verifiable code, natural language requirements are mapped to machine checked code using LLMs and agentic workflows. A crucial component of this pipeline is specification generation (SpecGen), which produces a formal contract against which an agent can prove implementation correctness. Proof generation can obtain deterministic feedback from a theorem prover, but SpecGen lacks a definitive check that a generated specification captures the user's intent. A checked proof can therefore establish correctness against a specification that misrepresents the intended behaviour. We take a step towards holistic SpecGen evaluation with a unified dataset assembled from $350$ existing Lean tasks, including $189$ from VERINA and $161$ from CLEVER, and a framework covering formal validity, reference similarity and equivalence, and behavioural adequacy. We distinguish acceptance of required inputs from acceptance of valid outputs and rejection of invalid outputs, while making each metric's evidence scope explicit. Across four SpecGen configurations, restricting the generalized tree edit distance (GTED) comparison, a reference similarity measure, to $32$ jointly measurable VERINA…
PDF arXiv
arXiv cs.AIOct 7
The AI Evaluation Ecosystem
Yash Dave, Sang T. Truong, Serena Wang, Sanmi Koyejo
AI evaluation shapes the decisions of model providers, users, funders, and regulators. We argue that designing valid benchmarks requires contextualizing design choices in the dynamics of this ecosystem of actors. We develop a simulation architecture that combines rule-based market dynamics with LLM-driven strategic actors, building on advances in Generative Agent-Based Modeling (GABM). We model benchmarks, consumer needs, and provider capabilities as vectors over a six-dimensional capability space (reasoning, coding, knowledge, safety, communication, agentic), with structural information partitions across actors. As a case study, we apply this stylized simulation to explore benchmark holdout design. We find that moving from public benchmarks to private holdout benchmarks shrinks the gap between benchmark scores and user satisfaction on most benchmarks but widens it on a few, depending on where holdout weights shift scoring credit. We stress-test our findings at both the instrument and case-study level, drawing on the V&V framework of Sargent (2013) and GABM-specific evidence criteria. Beyond holdout design, our simulation is a hypothesis-generating sandbox for studying how…
PDF arXiv
arXiv cs.CROct 7
Package Hallucination Attacks on Coding Agents through Prompt Injection in Rule Files
Yupu Wang, Zhengyuan Jiang, Reachal Wang, Neil Zhenqiang Gong
Modern agentic coding frameworks increasingly rely on community-shared rule files (e.g., AGENTS.md or .cursorrules) to guide autonomous code generation, yet the security risks of this pipeline remain underexplored. To bridge this gap, we introduce the package hallucination attack, where an attacker injects malicious prompts into benign rule files to induce coding agents to replace legitimate dependencies with attacker-controlled packages. To obtain effective malicious prompts injected into rule files, we propose PackHallu, an evolutionary optimization framework that iteratively rewrites these injected prompts using trajectory-level feedback and LLM-guided mutations. Evaluations across multiple benchmarks, LLMs, and agent frameworks show that PackHallu achieves high attack success rates and strong transferability across diverse models and agent combinations. Our findings demonstrate that coding agents are vulnerable to package hallucination attacks, highlighting the urgent need for stronger security safeguards in autonomous coding systems.
PDF arXiv
arXiv cs.AIOct 7▲ 14
We Query, Therefore We Compute: On Oracle Computation beyond the Machine, with an Application to Agents
Kefan Liu, Fengning Ou, Yelin Luo, Jingdi Lei
Agentic systems use large language models (LLMs) to carry out concrete tasks. Prior work often borrows abstractions such as scheduling, caching or isolation piecemeal from operating systems, so the mechanisms it builds share little common ground, and the shared view of the two forms of agentic system, Workflows and Agents, is limited. We construct an abstract machine that provides both. We treat the LLM as an Oracle and extend a two-stack pushdown automaton with one instruction, which hands the Oracle a whole stack as its query and appends the answer to that same stack. The machine thus performs two computations, the Oracle's and a Turing-complete one that we call the Priestess. A stack that the program only appends to grows autoregressively, as an agent's context does. Two symmetry breakings, S in storage and T in transitions, make a Priestess program the operating system of the programs the Oracle runs, and produce the Agent and the Workflow as the two placements of a task's program. For internally autoregressive Oracles, the two computations synchronize at the end of every answer under certain conditions, and through that synchronization we model caching and analyse…
PDF arXiv
arXiv cs.AIOct 6
Trajectory Abstraction for the Science of Language Agent Behavior
Tianqiang Yan
Scientific studies of language agents need behavioral variables that support hypotheses across tasks and models. We formulate this research problem as learning and testing a hierarchy of trajectory abstractions. A concrete recursive procedure first measures role- and phase-indexed events, proposes temporally constrained relations, and tests their stability across conditions. It then constructs episode-level motif variables from selected relations and repeats the analysis on those variables. Explicit measurement functions connect every abstraction level to the original trajectories. Observations and randomized protocol experiments assess the resulting hypotheses, while comparisons between intervention realizations determine whether an abstraction should be retained, refined, or restricted. We derive a finite-depth bound for accepted reductions, identify protocol effects on fixed abstractions, and characterize realization disagreement and composition of abstraction error. A finite-sample test makes projected intervention consistency operational, and constructed examples illustrate motif construction and abstraction refinement. The formulation distinguishes this experimental…
PDF arXiv

Activity · 12 weeks

Fields · AI · updated 26m ago

Agents

LLM agents: planning, tool use, multi-agent systems, computer/web agents and agent benchmarks.

388papers this week−35.8% vs last week

All papers · 2.7k With code · 143

Newest Most relevant

2.3k papers this month

Open problems in Agents

Long-horizon task completion with exploration and verification3 papers raise it · 2 address it
Agents struggle to accumulate evidence, adjust strategies, and verify results over long, open-ended tasks in real-world settings.
- Scaling Agentic Data for Long-Horizon Terminal Intelligenceraises · addresses
- FDE-Gym: The Last Mile of AI Deliveryraises · addresses
- Into the Unknown: Benchmarking AI agents on open scientific research problemsraises
Faithful agent development from natural-language requirements1 paper raises it · 1 addresses it
Coding agents often produce systems that appear to meet requirements but fail to deliver the intended behavior on held-out cases.
- AgentSWE: Can Coding Agents Build the Agent You Actually Want?raises · addresses
Detecting when past context matters in long conversations1 paper raises it · 0 address it
Agents cannot reliably identify when historical information is relevant, leading to missed or misapplied context.
- RealCompanion: Benchmarking Human Understanding from Reasoning over Longitudinal Real-World Conversationsraises
Transition-aware safety for GUI agents1 paper raises it · 1 addresses it
Safety checks that only inspect screens can be bypassed by reusing legitimate screens in unexpected transitions.
- Right Screen, Wrong Transition: World Models as Verifiers for GUI Agentsraises · addresses
Learning from experience in unfamiliar environments1 paper raises it · 1 addresses it
Agents often rely on pretrained knowledge rather than acquiring new knowledge through interaction in novel settings.
- Learn2Play Bench: How Well Do LLM Agents Learn from Experience in Unfamiliar Environments?raises · addresses

Named by an AI from these papers' abstracts.