Papers

Field

Top tools here

Open problems in Agents & indirect prompt injection

Lack of benchmarks for agent harness reliability2 papers raise it · 2 address it
Existing benchmarks focus on model capabilities rather than the reliability of the harnesses that support agent operation, leaving a gap in evaluating critical infrastructure.
- Trial by Trace: Benchmarking AI Agent Harnesses for Reliability and Failure Attributionraises · addresses
- Doc2Harness: Benchmarking LLMs on Long-Horizon Harness Engineering of Coding Agentsraises · addresses
Agents fail to meet real-world requirements3 papers raise it · 3 address it
Coding agents often produce outputs that appear correct but do not fully satisfy the intended requirements, especially in long-horizon or cross-repository tasks.
- AgentSWE: Can Coding Agents Build the Agent You Actually Want?raises · addresses
- CrossSWE: Benchmarking Cross-Repository Joint Development for Coding Agentsraises · addresses
- Doc2Harness: Benchmarking LLMs on Long-Horizon Harness Engineering of Coding Agentsraises · addresses
Security of agent-generated code1 paper raises it · 1 addresses it
Agents frequently produce functional but insecure code, with high rates of vulnerabilities even when functional requirements are met.
- Last Secure Code Benchmark: Is the Code You Vibe Secure?raises · addresses
Mosaic-style confidentiality leaks in agents1 paper raises it · 1 addresses it
Agents can inadvertently disclose sensitive information by combining non-sensitive data from multiple sources, bypassing safeguards.
- Agent Mosaic: Recognizing Confidentiality Leaks in Agentic LLM Systemsraises · addresses
Trustworthy recursive self-improvement1 paper raises it · 1 addresses it
Self-improving agents risk flawed evaluations, capability degradation, or constraint violations, making it hard to ensure trustworthy and sustained improvement.
- Trustworthy Recursive Self-Improvement in the Era of Large Models: Foundation, Risks, Evaluation and Open Challengesraises · addresses

Named by an AI from these papers' abstracts.

Open problems

Recently funded

Topics in this field

agents · tool · multi agent458 papers41 this week
indirect prompt injection · prompt injection · ai agents450 papers35 this week
coding agents · coding agent · code generation266 papers35 this week
code generation · language models llms · compiler179 papers24 this week
specification generation · formal specifications · formalization100 papers17 this week
proof assistant · proof search · theorem provers114 papers13 this week

Topics in this field

Active contributors · 90 days

Active contributors

Activity · 12 weeks

Activity

Recent papers

All papers in Research

arXiv cs.AIOct 3
Functionally Equivalent or Not? Graph-Grounded Differential Surrogate Execution for Code Equivalence
Amit Kachroo, Like Hui, Haitao Mao, Yuhao Zhang, Nguyen Vo
Determining whether two programs are functionally equivalent is central to code modernization, patch validation, refactoring, and code-generation evaluation. Yet the usual signals are incomplete: tests cover only finite inputs, textual similarity confuses implementation with behavior, and unconstrained LLM judgments are difficult to audit. Direct execution is often impossible when a program depends on an obsolete, licensed, unavailable, or unsafe environment. We introduce FEAgent, a selective equivalence assessor agent that combines typed program-graph evidence with differential surrogate execution. FEAgent first aligns public interfaces and behaviorally relevant graph anchors, then issues bounded queries over call-flow, control-flow, data-flow, type, import, and effect relations. Next, a branch-aware generator agent proposes discriminating inputs, and two blinded LLM surrogates independently predict source and target observables. Every claim and predicted divergence is recorded in an evidence ledger. A deterministic reconciler then returns EQUIVALENT, INEQUIVALENT, or UNCLEAR rather than forcing a verdict when paths are uncovered or evidence conflicts. We evaluate FEAgent on…
PDF arXiv
arXiv eess.SYOct 3
Latent Safety Filters: When a Lossy Encoder Admits a Transferable Certificate
Johannes Mootz, Zahra Nili Ahmadabadi, Reza Akhavian
Latent safety filters certify safety on a learned low-dimensional representation of the state, enabling constraints that resist analytic description. Because the encoder is lossy, a filter can report safe while the physical state is unsafe, with no detectable model error. Existing transfer conditions leave the effect of discarded safety information implicit. We ask when a lossy encoder admits a safety certificate that transfers to the physical system, and show the answer is governed by the detectability of the discarded safety-relevant dynamics. We construct a system whose latent model is exact and whose latent signals always report safe, while the physical state becomes arbitrarily unsafe. For this system no certificate exists and no monitor downstream of the encoder can detect the failure. When the discarded dynamics contract, a latent barrier certifies true safety up to two explicit margins, one for the latent-model error and one for the variation of safety across states the encoder cannot distinguish. In the linear case and under boundedness and non-degeneracy conditions, every calibrated barrier transfers with a finite margin when the safety-relevant subspace is detectable,…
PDF arXiv
arXiv cs.CROct 3
CyTReX: Explainable AI-Based Cybersecurity Threat Reasoning Framework for DER Networks
Damilola Popoola, Souradeep Bhattacharya, Manimaran Govindarasu
Distributed Energy Resource (DER) environments rely on network communication protocols to coordinate control commands, measurements, and device states across edge assets and cloud systems. Edge anomaly detection systems (ADS) monitor this traffic to identify deviations from normal communication behavior, flagging suspicious flows for further investigation. When the ADS flags abnormal network traffic, a single attack label is often insufficient for operational response: the label reports the detector's selected class but does not expose alternative threat interpretations that may warrant investigation. This paper presents Cybersecurity Threat Reasoning with Explainable Artificial Intelligence (CyTReX), an evidence-grounded threat reasoning framework for DER security that transforms network-level anomaly alerts into ranked, analyst-facing threat hypotheses designed to support Security Operations Center (SOC) triage and investigation. CyTReX constrains large language model (LLM) reasoning through a structured evidence packet, defined as a consolidated record of detection outputs, model explanations, and cyber threat intelligence (CTI) context. The evidence packet integrates…
PDF arXiv
arXiv cs.AIOct 3
Dense Neuro-Symbolic Reasoning in a Unified Geometry State
Ruoran Xu, Wending Gao, Haoyu Cheng, Xiaoqiang Kang, Qiufeng Wang
Geometry reasoning is naturally stateful: solving a problem repeatedly alternates between structural proposals and exact deductions. We formulate this process as dense neural-symbolic coupling, in which neural guidance and symbolic execution share a typed state and communicate through executable actions at every search step. Neural proposals contribute theorem instances, constructions, and algebraic bridges; the symbolic runtime applies registered rules, propagates exact constraints, and records provenance. A nested controller allocates computation first between neural and symbolic proposal sources and then among admitted actions. We instantiate the framework in OmniGeo, a single solver for plane, analytic, and solid geometry. With Claude Sonnet 4.6, OmniGeo reaches 94.2%, 88.5%, and 89.8% on FormalGeo7K, Conic10K, and SolidFGeo, respectively (90.8% macro average), and solves 21/30 IMO-AG-30 problems.
PDF arXiv
arXiv cs.LGOct 3
Self-Reflection Fine-Tuning: Enhancing Agent Security against Prompt Injection Attacks from Failure Experience
Zixuan Wang, Hao Li, Fengyu Gao, G. Edward Suh, Yi Zeng +3
Large language model (LLM) agents are increasingly deployed in tool-augmented environments, but their reliance on external inputs makes them highly vulnerable to prompt injection attacks that can hijack task objectives. Existing safety alignment methods rely on static expert trajectories or preference optimization, limiting their ability to generalize to adaptive attack patterns. In this work, we propose Self-Reflection Fine-Tuning (SRFT), a training framework that enables agents to improve robustness by learning from their own failure experiences under adversarial conditions. Instead of passively imitating expert behaviors, SRFT exposes the agent to compromised trajectories constructed via injected attacks, and leverages an expert model to generate structured self-reflection reasoning that contrasts unsafe and optimal actions. This reflective supervision teaches the agent to identify malicious instructions, reason about their consequences, and maintain alignment with the original user intent. We instantiate this framework in SR-Agent, built on Llama-3.1-8B-Instruct and Qwen3-8B, and evaluate it on both static and adaptive prompt injection benchmarks. Experimental results show…
PDF arXiv
arXiv cs.CYOct 2
Agent Reliability Profiles in Financial Services
Mike Hsu, Medha Bankhwal, Béatrice Moissinac, Kevin Werbach, Lukasz Szpruch +1
AI agents can take actions. At times, those actions can go beyond what is intended. Agent reliability can be defined as assurance that an agent will stay within intended bounds and operate within limits. Today, there is no shared framework or language for describing, validating, and benchmarking the reliability of agentic deployments in financial services. This makes it difficult for financial institutions, vendors, and regulators to assess and trust agents at scale, thus limiting the pace of development and adoption. A standardized, shared representation of agent reliability would fill the gap. This paper introduces the Agent Reliability Profile, a per-agent unit of assurance evidence for agent deployments in financial services. Each Profile records a bounded, falsifiable claim, this agentic system reliably functions within its operating boundary. We define "operating boundary" as an agent having; (1) a defined autonomy tier, (2) a defined operational design domain, (3) defined classes of action, and (4) a defined control envelope. Production assurance progresses through three levels while the Profile schema remains constant: a Profile Builder compiles a Level 1 Asserted Profile…
PDF arXiv

Papers

Topics in this field

agents · tool · multi agent458 papers41 this week
indirect prompt injection · prompt injection · ai agents450 papers35 this week
coding agents · coding agent · code generation266 papers35 this week
code generation · language models llms · compiler179 papers24 this week
specification generation · formal specifications · formalization100 papers17 this week
proof assistant · proof search · theorem provers114 papers13 this week

Recent papers

All papers in Research

arXiv cs.AIOct 3
Functionally Equivalent or Not? Graph-Grounded Differential Surrogate Execution for Code Equivalence
Amit Kachroo, Like Hui, Haitao Mao, Yuhao Zhang, Nguyen Vo
Determining whether two programs are functionally equivalent is central to code modernization, patch validation, refactoring, and code-generation evaluation. Yet the usual signals are incomplete: tests cover only finite inputs, textual similarity confuses implementation with behavior, and unconstrained LLM judgments are difficult to audit. Direct execution is often impossible when a program depends on an obsolete, licensed, unavailable, or unsafe environment. We introduce FEAgent, a selective equivalence assessor agent that combines typed program-graph evidence with differential surrogate execution. FEAgent first aligns public interfaces and behaviorally relevant graph anchors, then issues bounded queries over call-flow, control-flow, data-flow, type, import, and effect relations. Next, a branch-aware generator agent proposes discriminating inputs, and two blinded LLM surrogates independently predict source and target observables. Every claim and predicted divergence is recorded in an evidence ledger. A deterministic reconciler then returns EQUIVALENT, INEQUIVALENT, or UNCLEAR rather than forcing a verdict when paths are uncovered or evidence conflicts. We evaluate FEAgent on…
PDF arXiv
arXiv eess.SYOct 3
Latent Safety Filters: When a Lossy Encoder Admits a Transferable Certificate
Johannes Mootz, Zahra Nili Ahmadabadi, Reza Akhavian
Latent safety filters certify safety on a learned low-dimensional representation of the state, enabling constraints that resist analytic description. Because the encoder is lossy, a filter can report safe while the physical state is unsafe, with no detectable model error. Existing transfer conditions leave the effect of discarded safety information implicit. We ask when a lossy encoder admits a safety certificate that transfers to the physical system, and show the answer is governed by the detectability of the discarded safety-relevant dynamics. We construct a system whose latent model is exact and whose latent signals always report safe, while the physical state becomes arbitrarily unsafe. For this system no certificate exists and no monitor downstream of the encoder can detect the failure. When the discarded dynamics contract, a latent barrier certifies true safety up to two explicit margins, one for the latent-model error and one for the variation of safety across states the encoder cannot distinguish. In the linear case and under boundedness and non-degeneracy conditions, every calibrated barrier transfers with a finite margin when the safety-relevant subspace is detectable,…
PDF arXiv
arXiv cs.CROct 3
CyTReX: Explainable AI-Based Cybersecurity Threat Reasoning Framework for DER Networks
Damilola Popoola, Souradeep Bhattacharya, Manimaran Govindarasu
Distributed Energy Resource (DER) environments rely on network communication protocols to coordinate control commands, measurements, and device states across edge assets and cloud systems. Edge anomaly detection systems (ADS) monitor this traffic to identify deviations from normal communication behavior, flagging suspicious flows for further investigation. When the ADS flags abnormal network traffic, a single attack label is often insufficient for operational response: the label reports the detector's selected class but does not expose alternative threat interpretations that may warrant investigation. This paper presents Cybersecurity Threat Reasoning with Explainable Artificial Intelligence (CyTReX), an evidence-grounded threat reasoning framework for DER security that transforms network-level anomaly alerts into ranked, analyst-facing threat hypotheses designed to support Security Operations Center (SOC) triage and investigation. CyTReX constrains large language model (LLM) reasoning through a structured evidence packet, defined as a consolidated record of detection outputs, model explanations, and cyber threat intelligence (CTI) context. The evidence packet integrates…
PDF arXiv
arXiv cs.AIOct 3
Dense Neuro-Symbolic Reasoning in a Unified Geometry State
Ruoran Xu, Wending Gao, Haoyu Cheng, Xiaoqiang Kang, Qiufeng Wang
Geometry reasoning is naturally stateful: solving a problem repeatedly alternates between structural proposals and exact deductions. We formulate this process as dense neural-symbolic coupling, in which neural guidance and symbolic execution share a typed state and communicate through executable actions at every search step. Neural proposals contribute theorem instances, constructions, and algebraic bridges; the symbolic runtime applies registered rules, propagates exact constraints, and records provenance. A nested controller allocates computation first between neural and symbolic proposal sources and then among admitted actions. We instantiate the framework in OmniGeo, a single solver for plane, analytic, and solid geometry. With Claude Sonnet 4.6, OmniGeo reaches 94.2%, 88.5%, and 89.8% on FormalGeo7K, Conic10K, and SolidFGeo, respectively (90.8% macro average), and solves 21/30 IMO-AG-30 problems.
PDF arXiv
arXiv cs.LGOct 3
Self-Reflection Fine-Tuning: Enhancing Agent Security against Prompt Injection Attacks from Failure Experience
Zixuan Wang, Hao Li, Fengyu Gao, G. Edward Suh, Yi Zeng +3
Large language model (LLM) agents are increasingly deployed in tool-augmented environments, but their reliance on external inputs makes them highly vulnerable to prompt injection attacks that can hijack task objectives. Existing safety alignment methods rely on static expert trajectories or preference optimization, limiting their ability to generalize to adaptive attack patterns. In this work, we propose Self-Reflection Fine-Tuning (SRFT), a training framework that enables agents to improve robustness by learning from their own failure experiences under adversarial conditions. Instead of passively imitating expert behaviors, SRFT exposes the agent to compromised trajectories constructed via injected attacks, and leverages an expert model to generate structured self-reflection reasoning that contrasts unsafe and optimal actions. This reflective supervision teaches the agent to identify malicious instructions, reason about their consequences, and maintain alignment with the original user intent. We instantiate this framework in SR-Agent, built on Llama-3.1-8B-Instruct and Qwen3-8B, and evaluate it on both static and adaptive prompt injection benchmarks. Experimental results show…
PDF arXiv
arXiv cs.CYOct 2
Agent Reliability Profiles in Financial Services
Mike Hsu, Medha Bankhwal, Béatrice Moissinac, Kevin Werbach, Lukasz Szpruch +1
AI agents can take actions. At times, those actions can go beyond what is intended. Agent reliability can be defined as assurance that an agent will stay within intended bounds and operate within limits. Today, there is no shared framework or language for describing, validating, and benchmarking the reliability of agentic deployments in financial services. This makes it difficult for financial institutions, vendors, and regulators to assess and trust agents at scale, thus limiting the pace of development and adoption. A standardized, shared representation of agent reliability would fill the gap. This paper introduces the Agent Reliability Profile, a per-agent unit of assurance evidence for agent deployments in financial services. Each Profile records a bounded, falsifiable claim, this agentic system reliably functions within its operating boundary. We define "operating boundary" as an agent having; (1) a defined autonomy tier, (2) a defined operational design domain, (3) defined classes of action, and (4) a defined control envelope. Production assurance progresses through three levels while the Profile schema remains constant: a Profile Builder compiles a Level 1 Asserted Profile…
PDF arXiv

Active contributors · 90 days

Activity · 12 weeks

Fields · AI · updated 2m ago

Agents & indirect prompt injection

Found by the topic model: agents, indirect prompt injection, code generation.

178papers this week−37.1% vs last week

All papers · 1.7k With code · 42

Newest Most relevant

1k papers this month

Open problems in Agents & indirect prompt injection

Lack of benchmarks for agent harness reliability2 papers raise it · 2 address it
Existing benchmarks focus on model capabilities rather than the reliability of the harnesses that support agent operation, leaving a gap in evaluating critical infrastructure.
- Trial by Trace: Benchmarking AI Agent Harnesses for Reliability and Failure Attributionraises · addresses
- Doc2Harness: Benchmarking LLMs on Long-Horizon Harness Engineering of Coding Agentsraises · addresses
Agents fail to meet real-world requirements3 papers raise it · 3 address it
Coding agents often produce outputs that appear correct but do not fully satisfy the intended requirements, especially in long-horizon or cross-repository tasks.
- AgentSWE: Can Coding Agents Build the Agent You Actually Want?raises · addresses
- CrossSWE: Benchmarking Cross-Repository Joint Development for Coding Agentsraises · addresses
- Doc2Harness: Benchmarking LLMs on Long-Horizon Harness Engineering of Coding Agentsraises · addresses
Security of agent-generated code1 paper raises it · 1 addresses it
Agents frequently produce functional but insecure code, with high rates of vulnerabilities even when functional requirements are met.
- Last Secure Code Benchmark: Is the Code You Vibe Secure?raises · addresses
Mosaic-style confidentiality leaks in agents1 paper raises it · 1 addresses it
Agents can inadvertently disclose sensitive information by combining non-sensitive data from multiple sources, bypassing safeguards.
- Agent Mosaic: Recognizing Confidentiality Leaks in Agentic LLM Systemsraises · addresses
Trustworthy recursive self-improvement1 paper raises it · 1 addresses it
Self-improving agents risk flawed evaluations, capability degradation, or constraint violations, making it hard to ensure trustworthy and sustained improvement.
- Trustworthy Recursive Self-Improvement in the Era of Large Models: Foundation, Risks, Evaluation and Open Challengesraises · addresses

Named by an AI from these papers' abstracts.