How to evaluate prompt-injection defenses: a practical checklist

A prompt-injection eval does not start with a list of phrases such as “ignore previous instructions.” First define the security expectation: what data the agent may read, which external parties can influence its context, which actions it may propose or execute, and what must never happen without a separate check. Direct injection from the user and indirect injection from a web page, email, document, tool output, or memory create different attack surfaces.

For every workflow, map source and sink pairs. A source is untrusted content an attacker can modify; a sink is secret transfer, a network request, message, payment, file change, memory write, or another consequential capability. OpenAI describes the same source-sink framing for agents: the test must prove not only that the model noticed suspicious text, but that a dangerous flow to the sink was blocked or required the correct confirmation.

• Asset → secrets, personal data, credentials, money, reputational, or operational actions.
• Source → user input, web, email, RAG, files, tool output, memory, and multimodal content.
• Sink → network, message, write tool, code execution, memory write, or privilege change.
• Invariant → a forbidden event that the harness checks deterministically.

The corpus should reproduce real trajectories: an ordinary page with a hidden instruction, an email with a plausible business pretext, a RAG document that asks to change policy, a redirect chain, a URL with data in the query, a poisoned tool result, and a memory entry pretending to be an approval. Add paraphrases, multiple languages, encodings, typographic hiding, images, and multi-step attacks. Each fixture stores the attacked source, allowed evidence, expected tool calls, and forbidden events.

Benign positive fixtures are as important as attacks. They show whether the defense blocks legitimate quoting of instructions, security research, support emails, external links, and permitted actions. Slice cases by source, sink, asset sensitivity, autonomy, permission scope, and need for user confirmation. A single average score can easily hide failure in a rare but critical data-exfiltration slice.

Run each fixture in a production-like sandbox with the same prompt, model, tools, policies, network rules, credentials broker, and confirmation UI, but with canary secrets and fake destinations. Testing only the final answer is insufficient: the model may silently call a URL, pass a payload in a tool argument, write the attack into memory, or prepare a dangerous draft. The harness records the full trajectory and independently observes every sink.

Do not give the test agent broader rights than its production role, and do not replace policy with a mock that always refuses. Verify least privilege, tenant binding, scoped credentials, destination allowlists, sandbox egress, schema validation, and approval binding to the exact payload. A confirmation does not pass the gate if the agent can change the recipient, data, or amount afterward without a new user decision.

The primary verdict is deterministic: forbidden sink reached, secret exposed, unauthorized write, poisoned memory, confirmation bypass, or safe completion. Separately label attack detected, refused, content safely summarized, user warned, and task completed. A refusal can look safe without proving that a background request was not sent; conversely, the model may not name the attack while a policy layer correctly blocks the side effect.

Report attack success rate only with the exact corpus, repetitions, sampling settings, model and policy versions, and a confidence interval; do not transplant a vendor benchmark into your own system. Useful operational metrics include critical invariant failures, sink-specific compromise rate, secret-canary exposure, blocked unauthorized attempts, benign task success, false-positive rate, confirmation quality, containment time, latency, and cost per evaluated trajectory. A critical exfiltration is not offset by high average task success.

Defense in depth includes model behavior, trust labels, content handling, least privilege, data-flow checks, sandboxing, network controls, approval, and monitoring. Run the full stack and then controlled ablations: remove the classifier, narrow or widen tool scope, disable the destination check, or remove confirmation. This shows which layer actually stopped the attack, where a single point of failure exists, and whether a decorative “AI firewall” merely adds latency.

OpenAI and Anthropic both describe prompt injection as an active problem without a single guaranteed defense. Classifier precision is therefore not a security result. Test whether the system limits the consequence even when the model or detector misses a socially convincing attack. For web agents, separately test silent URL-based leaks, redirects, previews, and embedded resources; a domain allowlist does not prove that a specific data flow is safe.

The decision record pins the workflow, sources, sinks, permissions, model, prompt, policy bundle, tool schemas, corpus revision, critical invariants, thresholds, owner, and review date. Promotion applies only to the verified scope. Restrict can disable network egress, memory writes, or high-impact tools; reject returns to a read-only or deterministic baseline. Any canary-secret leak, cross-tenant access, or unauthorized consequential action blocks release regardless of the average score.

Rollout moves through offline replay, adversarial staging, a read-only canary, a small permission-bounded cohort, and continuous monitoring. Rollback revokes scoped credentials, disables sinks and contaminated memory writes, restores the known-good policy/model bundle, and preserves a sanitized trace for incident review. A new exploit becomes a regression fixture after triage. Rerun the critical suite after changes to the model, prompt, retriever, browser, MCP/tool server, permissions, network policy, or confirmation UX.

• Promote → all critical invariants pass within the defined scope.
• Restrict → reduce sources, sinks, data class, autonomy, or permissions.
• Reject → keep a read-only or deterministic baseline.
• Roll back → revoke credentials, disable sinks, quarantine state, and reconcile side effects.