Papers

Field

Top tools here

All tools

Agno5 releases this week
FastMCP6 releases this week
Pydantic AI+17% downloads · 1.5M/wk+17.0%
n8n+25% downloads · 148.3k/wk+25.0%
smolagents+19% downloads · 125.9k/wk+18.7%

Top tools here

Open problems in Agents

Long-horizon task execution and memory3 papers raise it · 3 address it
Agents fail to retain, retrieve, and use past information over long interactions, limiting real-world deployment.
- RealCompanion: Benchmarking Human Understanding from Reasoning over Longitudinal Real-World Conversationsraises
- Scaling Agentic Data for Long-Horizon Terminal Intelligenceraises · addresses
- Learn2Play Bench: How Well Do LLM Agents Learn from Experience in Unfamiliar Environments?raises · addresses
- Agent-Reflex: Teaching Language Agents to Act Reflectivelyaddresses
Real-world task ambiguity and unstated constraints3 papers raise it · 2 address it
Agents struggle with ambiguous, incomplete, or evolving requirements in live systems, unlike controlled benchmarks.
- OR-License: Evaluating LLM Agents as Real-World Operations Research Practitionersraises · addresses
- FDE-Gym: The Last Mile of AI Deliveryraises · addresses
- AgentSWE: Can Coding Agents Build the Agent You Actually Want?raises
Faithful agent development from requirements1 paper raises it · 1 addresses it
Coding agents often produce systems that appear correct but fail to meet the actual requirements.
- AgentSWE: Can Coding Agents Build the Agent You Actually Want?raises · addresses
Sequential distraction in multimodal GUI agents1 paper raises it · 1 addresses it
Omni-modal inputs cause agents to focus on irrelevant signals, leading to compounding errors.
- Focused Omni-Modal GUI Agentsraises · addresses
Open, reproducible agent stacks and evaluation1 paper raises it · 1 addresses it
Closed systems and lack of open benchmarks hinder systematic study and improvement of agent designs.
- OpenGCUA: Toward a Fully Open Stack for Generalist Computer Useraises · addresses

Named by an AI from these papers' abstracts.

Open problems

Recently funded

Market

FirstWorkSeries A · Oct 21$15M
Asymptote LabsSeed · Oct 20$5.5M
LumilabPre-seed · Oct 20undisclosed
GreptileSeries B+ · Oct 19$50M
VesslaPre-seed · Oct 19$3.3M

Recently funded

Topics in this field

coding agents · code generation · coding agent303 papers82 this week
skills · tasks · ai315 papers55 this week
recursive self improvement · harness evolution · adaptive271 papers50 this week
ai agents · indirect prompt injection · adversarial319 papers42 this week
tool using agents · agent systems · multi agent216 papers39 this week
agent · agentic · llm agents172 papers39 this week

Topics in this field

Active contributors · 90 days

Active contributors

Activity · 12 weeks

Activity

Recent papers

All papers in Research

arXiv cs.AIOct 14
Are You Reading the Right Thing? Rethinking Internal Signals for Prompt-Injection Detection
Sarthak Gupta, Priyanka Tembey
Several recent studies propose detecting prompt injections from a language model's internal state. However, these methods do not measure a single security property. Residual-stream probes, attention scores, role projections, exposure probes, and concept readouts may capture injection presence, recognition, information routing, or behavioral influence. It is therefore unclear what differences in prompt-injection detection performance reveal about the internal signals these methods read. We study how five representation-reading methods vary under a common structured indirect prompt-injection setup on Qwen3.6-35B-A3B and Gemma-4-26B-A4B-it. We evaluate each method by how well it distinguishes injected prompts from benign controls. We then further analyze each detection method's ability to detect attacks in cases where they are successful and in cases where they are not. This helps us better assess what properties each detection method might rely on. Methods trained to detect injection presence or attack-related concepts generally separate unsuccessful injections more strongly from benign inputs, whereas attention diversion, exposure, and role-confusion-based methods separate…
PDF arXiv
arXiv cs.SEOct 14
BenchChecker: Assessing the Credibility of Bug-Fixing Benchmarks for LLMs
Di Wu, Xu He, Shu Wang, Kun Sun
While Large Language Models (LLMs) show promise for Automatic Program Repair (APR), existing benchmarks suffer from data contamination, in which test samples appear in training sets, artificially inflating performance. To address this, we propose BenchChecker, a framework compatible with both open-source and closed-source models that supports more credible evaluation by detecting such contamination. Grounded in the observation that persistent patches likely remain in recent repository snapshots used for training, BenchChecker employs a forensics-based two-stage detection process using only textual model outputs together with public repository history. The first stage conducts a repository presence test by probing the model with code prefixes to infer training inclusion; the second stage performs a patch presence test to verify patch persistence via similarity alignment and test-suite verification. Evaluations using StarCoder-15B, StarCoder2-15B, and CodeGen2-16B trained on The Stack v2 demonstrate that BenchChecker achieves robust detection with AUC scores around 0.9 across six benchmarks. Furthermore, applying BenchChecker to SWE-bench Verified and Multi-SWE-bench reveals that…
PDF arXiv
arXiv cs.AIOct 14
EvoProbe: Evaluation-Driven Benchmark Evolution for LLM Agents
Qianhong Guo, Wei Xie, Xiaofang Cai, Shuoyoucheng Ma, Chengxin Hu +1
Static benchmarks for large language model (LLM) agents face saturation and data contamination risks, motivating continual updates to evaluation tasks. However, aggregate scores and success rates offer limited insight into specific problem-solving bottlenecks and provide limited guidance for subsequent task design. We propose EvoProbe, an evaluation-driven benchmark evolution framework that integrates task generation, diagnostic evaluation, and feedback-driven regeneration into a closed loop. It combines verifiable subtask outcome scoring with evidence-supported multi-judge trajectory evaluation and distills phase-level bottlenecks, success patterns, and failure patterns into diagnostic feedback for generating new executable tasks. Across two rounds of cybersecurity experiments, evaluations of eight models on 66 tasks reveal phase-level performance differences and common failure patterns beyond aggregate scores. In a cross-round comparison with the Examiner model set and solver configuration held fixed, all eight models achieve lower Outcome scores on the updated tasks, with relative decreases of 11.8\% in mean Outcome and 40.0\% in mean Pass\textasciicircum{}3. These results…
PDF arXiv
arXiv cs.AIOct 14▲ 9
Evaluating Agentic Capabilities Before Post-Training: An End-to-End Approach
Jiarui Qin, Junjie Huang, Yunjia Xi, Di Yin, Yong Yu +3
Agentic capabilities are increasingly important for modern LLMs, yet remain difficult to evaluate during pretraining. Existing approaches typically rely on static trajectory-derived tasks or likelihood-based proxies, which are lightweight but provide limited visibility into how base models behave in interactive environments. We introduce ASAP (Agent Scaffold Adaptation for Pretraining Evaluation), a lightweight framework for eliciting and evaluating agentic behavior from pretrained base models on end-to-end tasks. Rather than converting base models into chat-style agents, ASAP preserves the completion interface and adapts existing agent scaffolds through structured trajectory continuation, short reference prefixes, and minimal recovery mechanisms. Across software-engineering and deep-research benchmarks, ASAP yields discriminative end-to-end performance across base models and pretraining checkpoints. Beyond aggregate scores, ASAP exposes model-generated execution trajectories, enabling analysis of exploration, editing, testing, recovery, and task completion across training stages. These trajectories provide a behavioral view of how agent-relevant capabilities change during…
PDF arXiv
arXiv cs.AIOct 14▲ 6
AgentSafeEvo: Adversarial Co-Evolution for LLM Agent Safety
Chenxu Wang, Xiaoyu Wen, Han Qi, Ziyuan Zhou, Cheng Jin +2
Large language model (LLM) agents accomplish complex tasks through multi-turn interactions with external tools. However, malicious content embedded in user queries, tool descriptions, or environment observations can manipulate their decisions and induce unsafe actions. Adversarial co-evolution has emerged as an effective approach to strengthening LLM safety, but existing work primarily focuses on language-level safety or indirect prompt injection in web tasks. In this paper, we introduce AgentSafeEvo, a co-evolutionary reinforcement learning framework for agent safety built on an attacker-defender game spanning diverse tasks and attack types. The attacker learns to construct adversarial agent tasks that expose weaknesses in the defender's tool use, while the defender learns to complete tasks safely under these attacks. We further introduce learnability-guided task weighting, which uses within-task defender-reward variability to adjust each task's contribution to attacker and defender updates. Extensive experiments on agent safety and general capability benchmarks demonstrate that our method improves adversarial robustness while maintaining general agent capabilities.
PDF arXiv
arXiv cs.ROCodeOct 14▲ 9
From Failures to Learnable Worlds: Policy-Conditioned Environment Generation for Navigation Agents
Yue Zhang, Han Lin, Zun Wang, Yidong Huang, Zaid Khan +1
Simulated environments enable scalable training data generation for embodied agents, but more data does not necessarily provide what a particular agent needs to learn. Existing environment-generation pipelines largely optimize generic properties such as realism, diversity, or task coverage, while different pretrained policies can fail for different environmental reasons. We therefore introduce WEDGE, a framework for weakness-guided navigation learning in which the failure modes of a pretrained agent guide the generation of training environments at each adaptation round. Given a pretrained policy, WEDGE uses an LLM to propose executable weakness features from contrasting successful and failed rollouts, automatically validates them on the diagnostic set, and selects the strongest validated weakness based on its success-rate gap. The validated weaknesses are then mapped to the environmental factors that control them. WEDGE varies these factors to construct environments in the difficulty regimes where the policy struggles, producing targeted training experience for adaptation. Across three navigation agents with different adaptation mechanisms, we find that policies exhibit distinct…
PDF arXiv Code

Papers

Top tools here

All tools

Agno5 releases this week
FastMCP6 releases this week
Pydantic AI+17% downloads · 1.5M/wk+17.0%
n8n+25% downloads · 148.3k/wk+25.0%
smolagents+19% downloads · 125.9k/wk+18.7%

Recently funded

Market

FirstWorkSeries A · Oct 21$15M
Asymptote LabsSeed · Oct 20$5.5M
LumilabPre-seed · Oct 20undisclosed
GreptileSeries B+ · Oct 19$50M
VesslaPre-seed · Oct 19$3.3M

Topics in this field

coding agents · code generation · coding agent303 papers82 this week
skills · tasks · ai315 papers55 this week
recursive self improvement · harness evolution · adaptive271 papers50 this week
ai agents · indirect prompt injection · adversarial319 papers42 this week
tool using agents · agent systems · multi agent216 papers39 this week
agent · agentic · llm agents172 papers39 this week

Active contributors · 90 days

Recent papers

All papers in Research

arXiv cs.AIOct 14
Are You Reading the Right Thing? Rethinking Internal Signals for Prompt-Injection Detection
Sarthak Gupta, Priyanka Tembey
Several recent studies propose detecting prompt injections from a language model's internal state. However, these methods do not measure a single security property. Residual-stream probes, attention scores, role projections, exposure probes, and concept readouts may capture injection presence, recognition, information routing, or behavioral influence. It is therefore unclear what differences in prompt-injection detection performance reveal about the internal signals these methods read. We study how five representation-reading methods vary under a common structured indirect prompt-injection setup on Qwen3.6-35B-A3B and Gemma-4-26B-A4B-it. We evaluate each method by how well it distinguishes injected prompts from benign controls. We then further analyze each detection method's ability to detect attacks in cases where they are successful and in cases where they are not. This helps us better assess what properties each detection method might rely on. Methods trained to detect injection presence or attack-related concepts generally separate unsuccessful injections more strongly from benign inputs, whereas attention diversion, exposure, and role-confusion-based methods separate…
PDF arXiv
arXiv cs.SEOct 14
BenchChecker: Assessing the Credibility of Bug-Fixing Benchmarks for LLMs
Di Wu, Xu He, Shu Wang, Kun Sun
While Large Language Models (LLMs) show promise for Automatic Program Repair (APR), existing benchmarks suffer from data contamination, in which test samples appear in training sets, artificially inflating performance. To address this, we propose BenchChecker, a framework compatible with both open-source and closed-source models that supports more credible evaluation by detecting such contamination. Grounded in the observation that persistent patches likely remain in recent repository snapshots used for training, BenchChecker employs a forensics-based two-stage detection process using only textual model outputs together with public repository history. The first stage conducts a repository presence test by probing the model with code prefixes to infer training inclusion; the second stage performs a patch presence test to verify patch persistence via similarity alignment and test-suite verification. Evaluations using StarCoder-15B, StarCoder2-15B, and CodeGen2-16B trained on The Stack v2 demonstrate that BenchChecker achieves robust detection with AUC scores around 0.9 across six benchmarks. Furthermore, applying BenchChecker to SWE-bench Verified and Multi-SWE-bench reveals that…
PDF arXiv
arXiv cs.AIOct 14
EvoProbe: Evaluation-Driven Benchmark Evolution for LLM Agents
Qianhong Guo, Wei Xie, Xiaofang Cai, Shuoyoucheng Ma, Chengxin Hu +1
Static benchmarks for large language model (LLM) agents face saturation and data contamination risks, motivating continual updates to evaluation tasks. However, aggregate scores and success rates offer limited insight into specific problem-solving bottlenecks and provide limited guidance for subsequent task design. We propose EvoProbe, an evaluation-driven benchmark evolution framework that integrates task generation, diagnostic evaluation, and feedback-driven regeneration into a closed loop. It combines verifiable subtask outcome scoring with evidence-supported multi-judge trajectory evaluation and distills phase-level bottlenecks, success patterns, and failure patterns into diagnostic feedback for generating new executable tasks. Across two rounds of cybersecurity experiments, evaluations of eight models on 66 tasks reveal phase-level performance differences and common failure patterns beyond aggregate scores. In a cross-round comparison with the Examiner model set and solver configuration held fixed, all eight models achieve lower Outcome scores on the updated tasks, with relative decreases of 11.8\% in mean Outcome and 40.0\% in mean Pass\textasciicircum{}3. These results…
PDF arXiv
arXiv cs.AIOct 14▲ 9
Evaluating Agentic Capabilities Before Post-Training: An End-to-End Approach
Jiarui Qin, Junjie Huang, Yunjia Xi, Di Yin, Yong Yu +3
Agentic capabilities are increasingly important for modern LLMs, yet remain difficult to evaluate during pretraining. Existing approaches typically rely on static trajectory-derived tasks or likelihood-based proxies, which are lightweight but provide limited visibility into how base models behave in interactive environments. We introduce ASAP (Agent Scaffold Adaptation for Pretraining Evaluation), a lightweight framework for eliciting and evaluating agentic behavior from pretrained base models on end-to-end tasks. Rather than converting base models into chat-style agents, ASAP preserves the completion interface and adapts existing agent scaffolds through structured trajectory continuation, short reference prefixes, and minimal recovery mechanisms. Across software-engineering and deep-research benchmarks, ASAP yields discriminative end-to-end performance across base models and pretraining checkpoints. Beyond aggregate scores, ASAP exposes model-generated execution trajectories, enabling analysis of exploration, editing, testing, recovery, and task completion across training stages. These trajectories provide a behavioral view of how agent-relevant capabilities change during…
PDF arXiv
arXiv cs.AIOct 14▲ 6
AgentSafeEvo: Adversarial Co-Evolution for LLM Agent Safety
Chenxu Wang, Xiaoyu Wen, Han Qi, Ziyuan Zhou, Cheng Jin +2
Large language model (LLM) agents accomplish complex tasks through multi-turn interactions with external tools. However, malicious content embedded in user queries, tool descriptions, or environment observations can manipulate their decisions and induce unsafe actions. Adversarial co-evolution has emerged as an effective approach to strengthening LLM safety, but existing work primarily focuses on language-level safety or indirect prompt injection in web tasks. In this paper, we introduce AgentSafeEvo, a co-evolutionary reinforcement learning framework for agent safety built on an attacker-defender game spanning diverse tasks and attack types. The attacker learns to construct adversarial agent tasks that expose weaknesses in the defender's tool use, while the defender learns to complete tasks safely under these attacks. We further introduce learnability-guided task weighting, which uses within-task defender-reward variability to adjust each task's contribution to attacker and defender updates. Extensive experiments on agent safety and general capability benchmarks demonstrate that our method improves adversarial robustness while maintaining general agent capabilities.
PDF arXiv
arXiv cs.ROCodeOct 14▲ 9
From Failures to Learnable Worlds: Policy-Conditioned Environment Generation for Navigation Agents
Yue Zhang, Han Lin, Zun Wang, Yidong Huang, Zaid Khan +1
Simulated environments enable scalable training data generation for embodied agents, but more data does not necessarily provide what a particular agent needs to learn. Existing environment-generation pipelines largely optimize generic properties such as realism, diversity, or task coverage, while different pretrained policies can fail for different environmental reasons. We therefore introduce WEDGE, a framework for weakness-guided navigation learning in which the failure modes of a pretrained agent guide the generation of training environments at each adaptation round. Given a pretrained policy, WEDGE uses an LLM to propose executable weakness features from contrasting successful and failed rollouts, automatically validates them on the diagnostic set, and selects the strongest validated weakness based on its success-rate gap. The validated weaknesses are then mapped to the environmental factors that control them. WEDGE varies these factors to construct environments in the difficulty regimes where the policy struggles, producing targeted training experience for adaptation. Across three navigation agents with different adaptation mechanisms, we find that policies exhibit distinct…
PDF arXiv Code

Activity · 12 weeks

Fields · AI · updated 26m ago

Agents

LLM agents: planning, tool use, multi-agent systems, computer/web agents and agent benchmarks.

522papers this week−6.5% vs last week

All papers · 2.6k With code · 123

Newest Most relevant

2.4k papers this month

Open problems in Agents

Long-horizon task execution and memory3 papers raise it · 3 address it
Agents fail to retain, retrieve, and use past information over long interactions, limiting real-world deployment.
- RealCompanion: Benchmarking Human Understanding from Reasoning over Longitudinal Real-World Conversationsraises
- Scaling Agentic Data for Long-Horizon Terminal Intelligenceraises · addresses
- Learn2Play Bench: How Well Do LLM Agents Learn from Experience in Unfamiliar Environments?raises · addresses
- Agent-Reflex: Teaching Language Agents to Act Reflectivelyaddresses
Real-world task ambiguity and unstated constraints3 papers raise it · 2 address it
Agents struggle with ambiguous, incomplete, or evolving requirements in live systems, unlike controlled benchmarks.
- OR-License: Evaluating LLM Agents as Real-World Operations Research Practitionersraises · addresses
- FDE-Gym: The Last Mile of AI Deliveryraises · addresses
- AgentSWE: Can Coding Agents Build the Agent You Actually Want?raises
Faithful agent development from requirements1 paper raises it · 1 addresses it
Coding agents often produce systems that appear correct but fail to meet the actual requirements.
- AgentSWE: Can Coding Agents Build the Agent You Actually Want?raises · addresses
Sequential distraction in multimodal GUI agents1 paper raises it · 1 addresses it
Omni-modal inputs cause agents to focus on irrelevant signals, leading to compounding errors.
- Focused Omni-Modal GUI Agentsraises · addresses
Open, reproducible agent stacks and evaluation1 paper raises it · 1 addresses it
Closed systems and lack of open benchmarks hinder systematic study and improvement of agent designs.
- OpenGCUA: Toward a Fully Open Stack for Generalist Computer Useraises · addresses

Named by an AI from these papers' abstracts.