Papers

Field

Top tools here

Open problems in Agents & indirect prompt injection

Lack of benchmarks for agent harness reliability2 papers raise it · 2 address it
Existing benchmarks focus on model capabilities rather than the reliability of the harnesses that support agent operation, leaving a gap in evaluating critical infrastructure.
- Trial by Trace: Benchmarking AI Agent Harnesses for Reliability and Failure Attributionraises · addresses
- Doc2Harness: Benchmarking LLMs on Long-Horizon Harness Engineering of Coding Agentsraises · addresses
Agents fail to meet real-world requirements3 papers raise it · 3 address it
Coding agents often produce outputs that appear correct but do not fully satisfy the intended requirements, especially in long-horizon or cross-repository tasks.
- AgentSWE: Can Coding Agents Build the Agent You Actually Want?raises · addresses
- CrossSWE: Benchmarking Cross-Repository Joint Development for Coding Agentsraises · addresses
- Doc2Harness: Benchmarking LLMs on Long-Horizon Harness Engineering of Coding Agentsraises · addresses
Security of agent-generated code1 paper raises it · 1 addresses it
Agents frequently produce functional but insecure code, with high rates of vulnerabilities even when functional requirements are met.
- Last Secure Code Benchmark: Is the Code You Vibe Secure?raises · addresses
Mosaic-style confidentiality leaks in agents1 paper raises it · 1 addresses it
Agents can inadvertently disclose sensitive information by combining non-sensitive data from multiple sources, bypassing safeguards.
- Agent Mosaic: Recognizing Confidentiality Leaks in Agentic LLM Systemsraises · addresses
Trustworthy recursive self-improvement1 paper raises it · 1 addresses it
Self-improving agents risk flawed evaluations, capability degradation, or constraint violations, making it hard to ensure trustworthy and sustained improvement.
- Trustworthy Recursive Self-Improvement in the Era of Large Models: Foundation, Risks, Evaluation and Open Challengesraises · addresses

Named by an AI from these papers' abstracts.

Open problems

Recently funded

Topics in this field

indirect prompt injection · prompt injection · ai agents480 papers43 this week
code generation · language models llms · compiler207 papers39 this week
agents · tool · multi agent486 papers37 this week
coding agents · coding agent · code generation290 papers35 this week
specification generation · formal specifications · formalization121 papers25 this week
proof assistant · proof search · theorem provers127 papers19 this week

Topics in this field

Active contributors · 90 days

Active contributors

Activity · 12 weeks

Activity

Recent papers

All papers in Research

arXiv cs.AISep 21
Lean Pool: An AI-Maintained Archive of Formalized Mathematics
Vasily Ilin
Lean Pool is a repository of formalized mathematics. It is grown, maintained and optimized by AI agents.
PDF arXiv
arXiv cs.CLSep 21
Human-LLM Deliberation as Interactive Proof: Conditions for Verifiability Without Transparency
Baotong Zhang, Dean Foster, João Sedoc
When an LLM supplies an argument a user could not produce, how can the user decide whether to accept its claim? Inspired by interactive proofs, we pair a computationally unbounded LLM prover with a resource-bounded human verifier. The verifier checks requested details without seeing the LLM's internals; passed checks accumulate evidence toward an acceptance threshold. Given task-supplied bounds on false passes and human checking errors, we prove anytime-valid soundness against adaptive provers: the probability of ever accepting a false claim is at most a chosen error level. For a true claim, acceptance within the budget has probability at least the product of per-round probabilities that the LLM responds adequately and the human checks correctly, provided the threshold is reachable in time. Each additional check guards against false claims but gives a true claim another chance to fail. Which review lengths meet both error targets depends on the verifier's effort, cognitive load, expertise, and fatigue. With equal resources, we identify conditions under which the bounds certify a review by local checks but not a one-shot check. In LLM-only simulations on GPQA Diamond and…
PDF arXiv
arXiv cs.AISep 21
Silent Failures in Agent-Tool Interaction: An Audit of ToolUniverse
Devansh Singh, Shreya Gopalan, Sundaraparipurnan Narayanan
Agentic AI systems are increasingly adopting automated pipelines that integrate multiple tools. While prior research have studied task success and task completion of such agentic systems, research on agent to tool interaction chain via a wrapper and API, specifically in biological agentic workflows, remains limited. This study investigates specific failures in agent to tool interaction where a tool invocation appears successful but some or all of the information or functionality from the tool via its API / wrapper is incomplete or missing and there are no communications to the user or the agent regarding the missing information. We call these 'silent failures' because the user or the agents are not aware that such a failure has occurred. For the purpose of this study, we developed an audit mechanism to identify such silent failures in agent to tool interaction chains, by examining 15 scientific tools (and their associated API documentation and tool documentations) integrated within the ToolUniverse environment (ToolUniverse serves as our experimental environment rather than the object of the study itself). We structure our study around 7 failure locus, characterising where the…
PDF arXiv
arXiv cs.AISep 21
Convex AI Compositionality and the Governance of AI System Populations
Andrea Ferrario
AI governance increasingly requires providers and public authorities to reason about multiple AI instantiations, alternative versions, and deployment configurations of multiple AI systems. Yet current regulation remains predominantly single-system-centric, acknowledging such multiplicity only sparsely without treating collections of related AI systems as governance objects. This creates an AI population governance problem: determining which instantiations can be meaningfully considered together and how their changing configurations can be represented and monitored. The first requirement has recently been addressed through trustworthiness-based accounts of AI identity. We address the second by introducing convex AI compositionality: a formal representation of the configurations generated by finite AI system populations that uses convex spaces. The core idea is that convex compositions of the operational states that a population of AI system instantiations may occupy over time are compatible with lifecycle reachability across the population and can preserve the formal identity relations between these systems. Well-known statistical and geometric constructions, such as weighted…
PDF arXiv
arXiv cs.SESep 21
Spec2COBOLRot: An Agentic-AI Degradation Loop for Realistic COBOL Corpus Generation
Jean-Baptiste Espinasse, Djamel Eddine Khelladi, Mathieu Acher
COBOL remains widely deployed, yet representative corpora reflecting real production code are rarely available, limiting rigorous benchmarking of modernization approaches. We propose a systematic agentic AI pipeline for generating realistic COBOL programs, combining specification-driven generation with iterative degradation guided by patterns and complexity targets extracted from real production code. Here, realism is understood as structural fidelity to production code as captured by our metrics. We evaluate whether degradation reaches target complexity levels while preserving business behavior, and examine the limits of the approach, across three programs from distinct business domains. Results show the pipeline reliably produces syntactically valid programs and moves them toward realistic structural complexity. However, preserving business behavior is not always achieved by construction, and targeting structural metrics independently of business logic risks producing programs whose complexity does not reflect a plausible maintenance history. We discuss these limitations and outline a more realistic alternative as a direction for future work, generating legacy programs from…
PDF arXiv
arXiv cs.AISep 21
Epi-Logic: A Conceptual Framework for Epistemic Runtime Control, Schema Validity Checking, and Controlled Accommodation in Autonomous AI Agents
Boris Wetzk
Autonomous AI agents are increasingly deployed in areas where wrong decisions are hard to reverse. This paper examines schema mismatch: the condition in which an agent operates within an interpretive frame that no longer applies to the current context. Outputs produced under such a mismatch can appear internally consistent, linguistically plausible, and largely factually correct; output-quality metrics alone therefore capture the underlying loss of validity only partially. The paper introduces Epi-Logic, a conceptual framework for epistemic runtime control. It couples the detection of schema dissonance, a graduated reduction of autonomy, and the auditable switch to a validated schema. A schema is formalised as a tuple of variable space, expectation model, validity conditions, axioms, and metadata. The Epi-Score aggregates seven graded dimensions of epistemic dissonance; the temporal validity dimension D8, violations of the validity conditions G, and axiom violations are carried as separate categorical paths that are not offset against the aggregate. The architecture rests on a checking asymmetry: formalised validity conditions can be checked at runtime, whereas the correctness of…
PDF arXiv

Papers

Recent papers

All papers in Research

arXiv cs.AISep 21
Lean Pool: An AI-Maintained Archive of Formalized Mathematics
Vasily Ilin
Lean Pool is a repository of formalized mathematics. It is grown, maintained and optimized by AI agents.
PDF arXiv
arXiv cs.CLSep 21
Human-LLM Deliberation as Interactive Proof: Conditions for Verifiability Without Transparency
Baotong Zhang, Dean Foster, João Sedoc
When an LLM supplies an argument a user could not produce, how can the user decide whether to accept its claim? Inspired by interactive proofs, we pair a computationally unbounded LLM prover with a resource-bounded human verifier. The verifier checks requested details without seeing the LLM's internals; passed checks accumulate evidence toward an acceptance threshold. Given task-supplied bounds on false passes and human checking errors, we prove anytime-valid soundness against adaptive provers: the probability of ever accepting a false claim is at most a chosen error level. For a true claim, acceptance within the budget has probability at least the product of per-round probabilities that the LLM responds adequately and the human checks correctly, provided the threshold is reachable in time. Each additional check guards against false claims but gives a true claim another chance to fail. Which review lengths meet both error targets depends on the verifier's effort, cognitive load, expertise, and fatigue. With equal resources, we identify conditions under which the bounds certify a review by local checks but not a one-shot check. In LLM-only simulations on GPQA Diamond and…
PDF arXiv
arXiv cs.AISep 21
Silent Failures in Agent-Tool Interaction: An Audit of ToolUniverse
Devansh Singh, Shreya Gopalan, Sundaraparipurnan Narayanan
Agentic AI systems are increasingly adopting automated pipelines that integrate multiple tools. While prior research have studied task success and task completion of such agentic systems, research on agent to tool interaction chain via a wrapper and API, specifically in biological agentic workflows, remains limited. This study investigates specific failures in agent to tool interaction where a tool invocation appears successful but some or all of the information or functionality from the tool via its API / wrapper is incomplete or missing and there are no communications to the user or the agent regarding the missing information. We call these 'silent failures' because the user or the agents are not aware that such a failure has occurred. For the purpose of this study, we developed an audit mechanism to identify such silent failures in agent to tool interaction chains, by examining 15 scientific tools (and their associated API documentation and tool documentations) integrated within the ToolUniverse environment (ToolUniverse serves as our experimental environment rather than the object of the study itself). We structure our study around 7 failure locus, characterising where the…
PDF arXiv
arXiv cs.AISep 21
Convex AI Compositionality and the Governance of AI System Populations
Andrea Ferrario
AI governance increasingly requires providers and public authorities to reason about multiple AI instantiations, alternative versions, and deployment configurations of multiple AI systems. Yet current regulation remains predominantly single-system-centric, acknowledging such multiplicity only sparsely without treating collections of related AI systems as governance objects. This creates an AI population governance problem: determining which instantiations can be meaningfully considered together and how their changing configurations can be represented and monitored. The first requirement has recently been addressed through trustworthiness-based accounts of AI identity. We address the second by introducing convex AI compositionality: a formal representation of the configurations generated by finite AI system populations that uses convex spaces. The core idea is that convex compositions of the operational states that a population of AI system instantiations may occupy over time are compatible with lifecycle reachability across the population and can preserve the formal identity relations between these systems. Well-known statistical and geometric constructions, such as weighted…
PDF arXiv
arXiv cs.SESep 21
Spec2COBOLRot: An Agentic-AI Degradation Loop for Realistic COBOL Corpus Generation
Jean-Baptiste Espinasse, Djamel Eddine Khelladi, Mathieu Acher
COBOL remains widely deployed, yet representative corpora reflecting real production code are rarely available, limiting rigorous benchmarking of modernization approaches. We propose a systematic agentic AI pipeline for generating realistic COBOL programs, combining specification-driven generation with iterative degradation guided by patterns and complexity targets extracted from real production code. Here, realism is understood as structural fidelity to production code as captured by our metrics. We evaluate whether degradation reaches target complexity levels while preserving business behavior, and examine the limits of the approach, across three programs from distinct business domains. Results show the pipeline reliably produces syntactically valid programs and moves them toward realistic structural complexity. However, preserving business behavior is not always achieved by construction, and targeting structural metrics independently of business logic risks producing programs whose complexity does not reflect a plausible maintenance history. We discuss these limitations and outline a more realistic alternative as a direction for future work, generating legacy programs from…
PDF arXiv
arXiv cs.AISep 21
Epi-Logic: A Conceptual Framework for Epistemic Runtime Control, Schema Validity Checking, and Controlled Accommodation in Autonomous AI Agents
Boris Wetzk
Autonomous AI agents are increasingly deployed in areas where wrong decisions are hard to reverse. This paper examines schema mismatch: the condition in which an agent operates within an interpretive frame that no longer applies to the current context. Outputs produced under such a mismatch can appear internally consistent, linguistically plausible, and largely factually correct; output-quality metrics alone therefore capture the underlying loss of validity only partially. The paper introduces Epi-Logic, a conceptual framework for epistemic runtime control. It couples the detection of schema dissonance, a graduated reduction of autonomy, and the auditable switch to a validated schema. A schema is formalised as a tuple of variable space, expectation model, validity conditions, axioms, and metadata. The Epi-Score aggregates seven graded dimensions of epistemic dissonance; the temporal validity dimension D8, violations of the validity conditions G, and axiom violations are carried as separate categorical paths that are not offset against the aggregate. The architecture rests on a checking asymmetry: formalised validity conditions can be checked at runtime, whereas the correctness of…
PDF arXiv

Topics in this field

indirect prompt injection · prompt injection · ai agents480 papers43 this week
code generation · language models llms · compiler207 papers39 this week
agents · tool · multi agent486 papers37 this week
coding agents · coding agent · code generation290 papers35 this week
specification generation · formal specifications · formalization121 papers25 this week
proof assistant · proof search · theorem provers127 papers19 this week

Active contributors · 90 days

Activity · 12 weeks

Fields · AI · updated 49m ago

Agents & indirect prompt injection

Found by the topic model: agents, indirect prompt injection, code generation.

206papers this week−24.3% vs last week

All papers · 1.8k With code · 44

Newest Most relevant

1.1k papers this month

Open problems in Agents & indirect prompt injection

Lack of benchmarks for agent harness reliability2 papers raise it · 2 address it
Existing benchmarks focus on model capabilities rather than the reliability of the harnesses that support agent operation, leaving a gap in evaluating critical infrastructure.
- Trial by Trace: Benchmarking AI Agent Harnesses for Reliability and Failure Attributionraises · addresses
- Doc2Harness: Benchmarking LLMs on Long-Horizon Harness Engineering of Coding Agentsraises · addresses
Agents fail to meet real-world requirements3 papers raise it · 3 address it
Coding agents often produce outputs that appear correct but do not fully satisfy the intended requirements, especially in long-horizon or cross-repository tasks.
- AgentSWE: Can Coding Agents Build the Agent You Actually Want?raises · addresses
- CrossSWE: Benchmarking Cross-Repository Joint Development for Coding Agentsraises · addresses
- Doc2Harness: Benchmarking LLMs on Long-Horizon Harness Engineering of Coding Agentsraises · addresses
Security of agent-generated code1 paper raises it · 1 addresses it
Agents frequently produce functional but insecure code, with high rates of vulnerabilities even when functional requirements are met.
- Last Secure Code Benchmark: Is the Code You Vibe Secure?raises · addresses
Mosaic-style confidentiality leaks in agents1 paper raises it · 1 addresses it
Agents can inadvertently disclose sensitive information by combining non-sensitive data from multiple sources, bypassing safeguards.
- Agent Mosaic: Recognizing Confidentiality Leaks in Agentic LLM Systemsraises · addresses
Trustworthy recursive self-improvement1 paper raises it · 1 addresses it
Self-improving agents risk flawed evaluations, capability degradation, or constraint violations, making it hard to ensure trustworthy and sustained improvement.
- Trustworthy Recursive Self-Improvement in the Era of Large Models: Foundation, Risks, Evaluation and Open Challengesraises · addresses

Named by an AI from these papers' abstracts.