You're pledging to donate if the project hits its minimum goal and gets approved. If not, your funds will be returned.
1. Introduction
Large language model (LLM) agents are systems that plan multi-step tasks, call external tools, browse the web,
and act on retrieved information with minimal human supervision which have moved from research prototypes
to production deployments across coding assistants, customer support, and enterprise workflow automation
within roughly two years. This shift toward autonomy has been accompanied by a corresponding shift in the
attack surface: rather than attacking a model directly through a user-typed prompt, adversaries increasingly plant
malicious instructions inside content the agent is expected to read as part of its normal operation — a web page,
a retrieved document, a tool's output, an email, a code comment. This class of attack, known as indirect prompt
injection, exploits the fact that LLM agents cannot reliably distinguish between trusted instructions from their
operator and untrusted data encountered during execution. Industry threat reporting for 2026 identifies agentic
systems as the fastest-growing category of AI-related security incident, and standards bodies including OWASP
and the U.S. National Institute of Standards and Technology (NIST) have, within the past several months,
published dedicated guidance acknowledging that existing defenses were not designed with autonomous,
context-consuming agents in mind. against indirect prompt injection specifically in the setting where an agent's use of external context is legitimate
and necessary — a setting current defenses handle poorly.
2. Background
A typical loop for an LLM agent is: Take in a task, reason about what to do next, then call a tool (such as a
search engine, code interpreter, email client, file system, etc), and then observe its output, then repeat until the
task is complete. The model is provided with both the operator's original instruction and the content returned by
the tools used in it, and as current LLMs lack a structural distinction between “instruction” and “data”, any text
that reaches the “instruction” channel could be interpreted as a new instruction. The indirect prompt injection
is all about this: if the attacker can't directly interact with the model, they can still manipulate the model by
inserting their prompt at a location where the agent draws them.
This was mainly considered as a filtering problem; identify suspicious text and remove it before the model.
More recent approaches to architecture try to deliver more guarantees: an explicit hierarchy of instructions
(Wallace et al., 2024), having an isolated “quarantined” LLM process untrusted data while a separate, privileged
LLM takes actions (Kim, Choi & Lee, 2025), or restructuring the agent so it has one “quarantined” LLM for
data processing and another privileged LLM for action-taking (Debenedetti et al., 2025). Evaluation has also
developed over these defenses: benchmark environments like AgentDojo (Debenedetti et al., 2024), InjecAgent
(Zhan et al., 2024), and WASP (Evtimov et al., 2026) now make it possible to assess attacks and defenses under
tool-assisted agent scenarios in controlled settings.
Nevertheless, although thus far impressive, a 2026 systematic review of the field revealed that existing defenses
and benchmarks have a structural blind spot: they are evaluated on tasks where the agent's ideal behavior is to
ignore untrusted content, but not on the more realistic and more common scenario, where the agent is authorized
indeed required to act on the content it retrieves at runtime (Wang et al., 2026). In such benchmarks, merely
suppressing an agent's willingness to use retrieved context will result in a good score, and violate the tasks
agents are meant to perform.
3. Problem Statement
The core challenge that this research tackles is the absence of defenses against indirect prompt injection that
have been demonstrated to withstand the most important scenario for actual deployment: context-dependent
agentic tasks, where the agent is required to read and follow instructions from external content that it does not
trust to be part of its task. In existing evaluations, no single defense is simultaneously high in both
trustworthiness (resistance to agent injection) and task utility (ability to complete the legitimate task) and low
in latency (time required to complete the task) (Wang et al., 2026). With this is an agent that does everything
1by consuming untrusted information from the outside world, such as a web -browsing assistant, an agent that
retrieves material from a remote source or a document processing pipeline, and no effort has been made to
validate it an agent that is common today but has been left without a validated method to achieve. Thus there is
a clear, current gap between the security current defenses claim and the security which is available to agents
who can't avoid using untrusted context.
4. Research Objectives
1. Characterize how the effectiveness of representative existing prompt-injection defenses changes on
context-dependent agentic tasks, relative to their reported performance on standard, context-suppressible
benchmarks.
2. Identify the mechanism(s) by which context-dependent tasks defeat or degrade current defenses —
whether through an inability to distinguish legitimate from illegitimate context use, utility/latency trade-
offs, or benchmark design gaps.
3. Design and implement a context-aware defense mechanism that explicitly attempts to distinguish an
agent's legitimate use of retrieved information from an attacker's attempt to hijack the agent's behaviour
through that same information.
4. Evaluate the proposed defense against existing baselines on trustworthiness, task utility, and latency, using
existing and, where necessary, extended benchmark tasks.
5. Research Questions
1. How much does the effectiveness of current indirect-prompt-injection defenses degrade when evaluated
on context-dependent agentic tasks compared to context-suppressible tasks?
2. What distinguishes cases where an agent's use of retrieved context is legitimate from cases reflecting
successful injection, in terms of features observable at inference time (e.g., instruction–content similarity,
action consequence, source provenance)?
3. Can a defense mechanism be designed that uses such features to permit legitimate context-dependent
reasoning while blocking or flagging injected instructions, without a large utility or latency penalty?
4. How does the proposed defense's trustworthiness–utility–latency trade-off compare to existing baseline
defenses across a range of task types and attack strategies?
There are no bids on this project.