Papers

Field

Top tools here

Open problems in Agents & indirect prompt injection

Lack of benchmarks for agent harness reliability2 papers raise it · 2 address it
Existing benchmarks focus on model capabilities rather than the reliability of the harnesses that support agent operation, leaving a gap in evaluating critical infrastructure.
- Trial by Trace: Benchmarking AI Agent Harnesses for Reliability and Failure Attributionraises · addresses
- Doc2Harness: Benchmarking LLMs on Long-Horizon Harness Engineering of Coding Agentsraises · addresses
Agents fail to meet real-world requirements3 papers raise it · 3 address it
Coding agents often produce outputs that appear correct but do not fully satisfy the intended requirements, especially in long-horizon or cross-repository tasks.
- AgentSWE: Can Coding Agents Build the Agent You Actually Want?raises · addresses
- CrossSWE: Benchmarking Cross-Repository Joint Development for Coding Agentsraises · addresses
- Doc2Harness: Benchmarking LLMs on Long-Horizon Harness Engineering of Coding Agentsraises · addresses
Security of agent-generated code1 paper raises it · 1 addresses it
Agents frequently produce functional but insecure code, with high rates of vulnerabilities even when functional requirements are met.
- Last Secure Code Benchmark: Is the Code You Vibe Secure?raises · addresses
Mosaic-style confidentiality leaks in agents1 paper raises it · 1 addresses it
Agents can inadvertently disclose sensitive information by combining non-sensitive data from multiple sources, bypassing safeguards.
- Agent Mosaic: Recognizing Confidentiality Leaks in Agentic LLM Systemsraises · addresses
Trustworthy recursive self-improvement1 paper raises it · 1 addresses it
Self-improving agents risk flawed evaluations, capability degradation, or constraint violations, making it hard to ensure trustworthy and sustained improvement.
- Trustworthy Recursive Self-Improvement in the Era of Large Models: Foundation, Risks, Evaluation and Open Challengesraises · addresses

Named by an AI from these papers' abstracts.

Open problems

Recently funded

Topics in this field

indirect prompt injection · prompt injection · ai agents472 papers40 this week
agents · tool · multi agent479 papers39 this week
coding agents · coding agent · code generation283 papers39 this week
code generation · language models llms · compiler199 papers36 this week
specification generation · formal specifications · formalization116 papers26 this week
proof assistant · proof search · theorem provers122 papers16 this week

Topics in this field

Active contributors · 90 days

Active contributors

Activity · 12 weeks

Activity

Recent papers

All papers in Research

arXiv cs.AISep 27
SpecRead: A Benchmark for Measuring Whether Language Models Understand Hardware Specifications
Feilian Huang
Existing benchmarks for large language models (LLMs) in hardware design evaluate downstream artifacts such as generated RTL, assertions, or testbenches. When a model fails such a benchmark, the failure is ambiguous: it may have misread the specification, or it may have understood the specification and failed to write the code. We present SpecRead, a benchmark that isolates specification comprehension from generation ability. SpecRead v2.1 contains 385 questions over 10 open-source OpenTitan IP blocks: exact retrieval, cross-section reasoning, contradiction detection in mutated specifications, and spec-RTL consistency checking, plus 82 controls (41 distractor, 41 consistent-RTL). Type-4 items are built from real RTL mutations; we retain only mutations that Icarus Verilog simulation shows to change observable behavior. A with-spec vs. without-spec ablation suggests the questions require the excerpt, not training recall alone (without-spec accuracy 3/20 on the t1/t2 subset), though memorization of the source text may still help spot mutations. As an initial characterization with a small model, Ministral-3B scores 33.2% overall (128/385; macro average 39.0%): 55.2% on retrieval,…
PDF arXiv
arXiv cs.AISep 27
Auditing Agent Actions through Query-Conditioned Attribution
Yifan Liu, Praveen Venkateswaran, Abdulhamid Adebayo, Dong Wang
LLM agents increasingly take consequential actions through interactions with users, policies, and external tools. Auditing these agents requires automated attribution of realized actions to their historical basis. However, existing attribution formulations do not provide question-specific traces for diverse auditing objectives. Additionally, when access to the acting model is limited (e.g., in API-only deployments), applicable methods commonly rely on costly input perturbations or external LLM analysis of complete trajectories. We therefore formulate query-conditioned agent action attribution, a new task that takes a natural-language auditing query as input and recovers the source and ordered intermediate evidence for the query-specified aspect of an action. We instantiate this task with $A^3Bench$, a benchmark comprising 1,396 auditing queries across policy basis, parameter provenance, failure propagation, and unsafe-behavior tracing. To enable efficient, query-specific attribution, we use small open-weight models as attribution proposers that combine query-conditioned gradient saliency with query-semantic relevance to rank history units. Our proposer consistently achieves…
PDF arXiv
arXiv cs.AISep 27
Audit-First VAPO: Risk-Certified Selective Updates under Imperfect Verification
Miaobo Hu, Shuhao Hu, Xiaobo Guo, Xin Wang, Bokun Wang +3
Imperfect verifiers can assign a harmful update direction even when clipping and regularization bound its magnitude. We introduce Audit-First VAPO, which separates discrete directional admission from continuous magnitude control. An observation-only accept-appeal-abstain policy uses a finite secondary-verification budget; its action trace is frozen before clean labels are joined. Simultaneous finite-sample bounds then certify selected harmful risk, coverage, and verifier-call rate over a predeclared policy family. Conditional Hoeffding-Azuma bounds account for the dependence induced by shared budgets, and rollout or verifier changes initiate a new certification stage. After admission, a bounded trust-clip-KL actuator controls magnitude. We evaluate two models on two reasoning benchmarks against static RLVR, matched-random selection, confidence thresholding, noise correction, and verifier augmentation. On Qwen3.5-0.8B and GSM8K at target risk $ρ=0.08$, RC-VAPO achieves 74.1% accuracy, selected harmful risk 0.0697, coverage 0.4125, and relative verifier cost $1.16\times$. At matched coverage and update magnitude, its selected-risk difference from matched random is -0.0260 with…
PDF arXiv
arXiv cs.AISep 27
AgentBoundary: Counterfactual Evaluation of Safety in Tool-Using LLM Agents
Tianzhuo Yang, Zirui Mi, Yantao Huang, Guoxi Zhang, Jiawei Chen +2
Safety alignment for large language models (LLMs) in conversational settings is largely framed around whether to answer or refuse a request. In agentic settings, however, the same models must decide whether to act as permission-critical evidence emerges during execution. This creates a distinct challenge: apparent risk, action permissibility, and task competence are easily confounded, making agentic over-refusal difficult to distinguish from ordinary task failure. To address this, we introduce AgentBound, the first four-way counterfactual generation-and-evaluation framework for tool-using agent safety. AgentBound transforms the same executable workflow by independently varying apparent risk and action permissibility, enabling controlled comparisons of risky-looking but authorized tasks and routine-looking but unauthorized tasks. These comparisons jointly diagnose over-refusal and unsafe compliance while controlling for task competence. We instantiate AgentBound as a human-validated 4,000-task evaluation suite with trajectory-based and post-state-based judgments. Across 17 model and harness configurations, high safety frequently coexists with poor authorized-task completion:…
PDF arXiv
arXiv cs.LGSep 27
LLM4Trust: Exploring the Capabilities of Large Language Models for Trust Evaluation
Jie Wang, Yanbo Sun, Zheng Yan, Jiahe Lan, Elisa Bertino
Trust evaluation plays a critical role in cybersecurity by supporting risk mitigation and decision-making. A variety of trust evaluation methods have been proposed, with learning-based approaches offering high accuracy and automation. However, they often require substantial ground truth, suffer from low training efficiency, lack support for basic trust properties, and provide limited explainability. Large Language Models (LLMs) offer a compelling alternative due to their strong zero-/few-shot reasoning abilities and broad knowledge. To this end, we propose LLM4Trust, the first benchmark framework that systematically explores the capabilities of LLMs for trust evaluation. We first construct diverse trust graphs to model five basic trust properties and design corresponding property understanding tasks. We then assess the ability of eight representative LLMs to understand these properties under nine prompt methods. Based on this exploration, we identify the most effective LLM-prompt combinations and apply them to five real-world datasets for validating LLMs' trust evaluation capability. During this process, we propose two strategies to extract key information from large-scale trust…
PDF arXiv
arXiv cs.LOSep 27
Protected Cores Are Not Enough: Certifying AI-Proposed Revisions of Temporal Specifications
Ruggero Lanotte
Runtime monitoring traditionally evaluates a specification that is fixed before execution or externally modified when requirements change. In learning-enabled and data-intensive systems, however, the temporal relationships represented by a specification may themselves evolve. Allowing an AI component to directly replace a formal specification is unsafe: it may overfit transient behavior, weaken protected requirements, or activate statistically unsupported revisions. We introduce an intersymbolic architecture in which an untrusted AI proposer suggests temporal specification revisions and a symbolic governor controls their activation. Two results organize the framework. First, origin-version semantics makes the outcome of each obligation invariant to later revisions. Second, aggregate certification can conceal systematic failures on protected triggers; simultaneous aggregate and core-conditional post-selection certification controls both targets. A structural invariant preserves designer-protected components, and a proposer-independent lifetime error bound supports repeated activation decisions. The statistical bound concerns the predictable means of completed certification…
PDF arXiv

Papers

Topics in this field

indirect prompt injection · prompt injection · ai agents472 papers40 this week
agents · tool · multi agent479 papers39 this week
coding agents · coding agent · code generation283 papers39 this week
code generation · language models llms · compiler199 papers36 this week
specification generation · formal specifications · formalization116 papers26 this week
proof assistant · proof search · theorem provers122 papers16 this week

Recent papers

All papers in Research

arXiv cs.AISep 27
SpecRead: A Benchmark for Measuring Whether Language Models Understand Hardware Specifications
Feilian Huang
Existing benchmarks for large language models (LLMs) in hardware design evaluate downstream artifacts such as generated RTL, assertions, or testbenches. When a model fails such a benchmark, the failure is ambiguous: it may have misread the specification, or it may have understood the specification and failed to write the code. We present SpecRead, a benchmark that isolates specification comprehension from generation ability. SpecRead v2.1 contains 385 questions over 10 open-source OpenTitan IP blocks: exact retrieval, cross-section reasoning, contradiction detection in mutated specifications, and spec-RTL consistency checking, plus 82 controls (41 distractor, 41 consistent-RTL). Type-4 items are built from real RTL mutations; we retain only mutations that Icarus Verilog simulation shows to change observable behavior. A with-spec vs. without-spec ablation suggests the questions require the excerpt, not training recall alone (without-spec accuracy 3/20 on the t1/t2 subset), though memorization of the source text may still help spot mutations. As an initial characterization with a small model, Ministral-3B scores 33.2% overall (128/385; macro average 39.0%): 55.2% on retrieval,…
PDF arXiv
arXiv cs.AISep 27
Auditing Agent Actions through Query-Conditioned Attribution
Yifan Liu, Praveen Venkateswaran, Abdulhamid Adebayo, Dong Wang
LLM agents increasingly take consequential actions through interactions with users, policies, and external tools. Auditing these agents requires automated attribution of realized actions to their historical basis. However, existing attribution formulations do not provide question-specific traces for diverse auditing objectives. Additionally, when access to the acting model is limited (e.g., in API-only deployments), applicable methods commonly rely on costly input perturbations or external LLM analysis of complete trajectories. We therefore formulate query-conditioned agent action attribution, a new task that takes a natural-language auditing query as input and recovers the source and ordered intermediate evidence for the query-specified aspect of an action. We instantiate this task with $A^3Bench$, a benchmark comprising 1,396 auditing queries across policy basis, parameter provenance, failure propagation, and unsafe-behavior tracing. To enable efficient, query-specific attribution, we use small open-weight models as attribution proposers that combine query-conditioned gradient saliency with query-semantic relevance to rank history units. Our proposer consistently achieves…
PDF arXiv
arXiv cs.AISep 27
Audit-First VAPO: Risk-Certified Selective Updates under Imperfect Verification
Miaobo Hu, Shuhao Hu, Xiaobo Guo, Xin Wang, Bokun Wang +3
Imperfect verifiers can assign a harmful update direction even when clipping and regularization bound its magnitude. We introduce Audit-First VAPO, which separates discrete directional admission from continuous magnitude control. An observation-only accept-appeal-abstain policy uses a finite secondary-verification budget; its action trace is frozen before clean labels are joined. Simultaneous finite-sample bounds then certify selected harmful risk, coverage, and verifier-call rate over a predeclared policy family. Conditional Hoeffding-Azuma bounds account for the dependence induced by shared budgets, and rollout or verifier changes initiate a new certification stage. After admission, a bounded trust-clip-KL actuator controls magnitude. We evaluate two models on two reasoning benchmarks against static RLVR, matched-random selection, confidence thresholding, noise correction, and verifier augmentation. On Qwen3.5-0.8B and GSM8K at target risk $ρ=0.08$, RC-VAPO achieves 74.1% accuracy, selected harmful risk 0.0697, coverage 0.4125, and relative verifier cost $1.16\times$. At matched coverage and update magnitude, its selected-risk difference from matched random is -0.0260 with…
PDF arXiv
arXiv cs.AISep 27
AgentBoundary: Counterfactual Evaluation of Safety in Tool-Using LLM Agents
Tianzhuo Yang, Zirui Mi, Yantao Huang, Guoxi Zhang, Jiawei Chen +2
Safety alignment for large language models (LLMs) in conversational settings is largely framed around whether to answer or refuse a request. In agentic settings, however, the same models must decide whether to act as permission-critical evidence emerges during execution. This creates a distinct challenge: apparent risk, action permissibility, and task competence are easily confounded, making agentic over-refusal difficult to distinguish from ordinary task failure. To address this, we introduce AgentBound, the first four-way counterfactual generation-and-evaluation framework for tool-using agent safety. AgentBound transforms the same executable workflow by independently varying apparent risk and action permissibility, enabling controlled comparisons of risky-looking but authorized tasks and routine-looking but unauthorized tasks. These comparisons jointly diagnose over-refusal and unsafe compliance while controlling for task competence. We instantiate AgentBound as a human-validated 4,000-task evaluation suite with trajectory-based and post-state-based judgments. Across 17 model and harness configurations, high safety frequently coexists with poor authorized-task completion:…
PDF arXiv
arXiv cs.LGSep 27
LLM4Trust: Exploring the Capabilities of Large Language Models for Trust Evaluation
Jie Wang, Yanbo Sun, Zheng Yan, Jiahe Lan, Elisa Bertino
Trust evaluation plays a critical role in cybersecurity by supporting risk mitigation and decision-making. A variety of trust evaluation methods have been proposed, with learning-based approaches offering high accuracy and automation. However, they often require substantial ground truth, suffer from low training efficiency, lack support for basic trust properties, and provide limited explainability. Large Language Models (LLMs) offer a compelling alternative due to their strong zero-/few-shot reasoning abilities and broad knowledge. To this end, we propose LLM4Trust, the first benchmark framework that systematically explores the capabilities of LLMs for trust evaluation. We first construct diverse trust graphs to model five basic trust properties and design corresponding property understanding tasks. We then assess the ability of eight representative LLMs to understand these properties under nine prompt methods. Based on this exploration, we identify the most effective LLM-prompt combinations and apply them to five real-world datasets for validating LLMs' trust evaluation capability. During this process, we propose two strategies to extract key information from large-scale trust…
PDF arXiv
arXiv cs.LOSep 27
Protected Cores Are Not Enough: Certifying AI-Proposed Revisions of Temporal Specifications
Ruggero Lanotte
Runtime monitoring traditionally evaluates a specification that is fixed before execution or externally modified when requirements change. In learning-enabled and data-intensive systems, however, the temporal relationships represented by a specification may themselves evolve. Allowing an AI component to directly replace a formal specification is unsafe: it may overfit transient behavior, weaken protected requirements, or activate statistically unsupported revisions. We introduce an intersymbolic architecture in which an untrusted AI proposer suggests temporal specification revisions and a symbolic governor controls their activation. Two results organize the framework. First, origin-version semantics makes the outcome of each obligation invariant to later revisions. Second, aggregate certification can conceal systematic failures on protected triggers; simultaneous aggregate and core-conditional post-selection certification controls both targets. A structural invariant preserves designer-protected components, and a proposer-independent lifetime error bound supports repeated activation decisions. The statistical bound concerns the predictable means of completed certification…
PDF arXiv

Active contributors · 90 days

Activity · 12 weeks

Fields · AI · updated 5m ago

Agents & indirect prompt injection

Found by the topic model: agents, indirect prompt injection, code generation.

209papers this week−19.3% vs last week

All papers · 1.8k With code · 44

Newest Most relevant

1.1k papers this month

Open problems in Agents & indirect prompt injection

Lack of benchmarks for agent harness reliability2 papers raise it · 2 address it
Existing benchmarks focus on model capabilities rather than the reliability of the harnesses that support agent operation, leaving a gap in evaluating critical infrastructure.
- Trial by Trace: Benchmarking AI Agent Harnesses for Reliability and Failure Attributionraises · addresses
- Doc2Harness: Benchmarking LLMs on Long-Horizon Harness Engineering of Coding Agentsraises · addresses
Agents fail to meet real-world requirements3 papers raise it · 3 address it
Coding agents often produce outputs that appear correct but do not fully satisfy the intended requirements, especially in long-horizon or cross-repository tasks.
- AgentSWE: Can Coding Agents Build the Agent You Actually Want?raises · addresses
- CrossSWE: Benchmarking Cross-Repository Joint Development for Coding Agentsraises · addresses
- Doc2Harness: Benchmarking LLMs on Long-Horizon Harness Engineering of Coding Agentsraises · addresses
Security of agent-generated code1 paper raises it · 1 addresses it
Agents frequently produce functional but insecure code, with high rates of vulnerabilities even when functional requirements are met.
- Last Secure Code Benchmark: Is the Code You Vibe Secure?raises · addresses
Mosaic-style confidentiality leaks in agents1 paper raises it · 1 addresses it
Agents can inadvertently disclose sensitive information by combining non-sensitive data from multiple sources, bypassing safeguards.
- Agent Mosaic: Recognizing Confidentiality Leaks in Agentic LLM Systemsraises · addresses
Trustworthy recursive self-improvement1 paper raises it · 1 addresses it
Self-improving agents risk flawed evaluations, capability degradation, or constraint violations, making it hard to ensure trustworthy and sustained improvement.
- Trustworthy Recursive Self-Improvement in the Era of Large Models: Foundation, Risks, Evaluation and Open Challengesraises · addresses

Named by an AI from these papers' abstracts.