Advanced•10–16 hours

Prompt Injection Source-Sink Lab

Build a red-team lab for direct and indirect prompt injection where risk is measured by the actual path from attacker-controlled sources to sensitive sinks and side effects, not only by instruction bypass.

prompt injectionsource-sink analysisdata exfiltrationtool authorizationsecurity evaluation

Scenario

Task

An AI assistant reads web pages, documents, and RAG context and can call tools. Test whether attacker-controlled content can make the system disclose data, invoke a dangerous tool, or change business state even when the model does not formally violate its system instruction.

Step-by-step execution

1. Build the source-sink model

Outcome: Every attacker-controlled input and consequential sink has an explicit trust boundary.

Tasks

→Inventory user, web, RAG, tool, and memory sources
→Mark secrets, write tools, and external egress sinks
→Link source → interpreter → sink paths
→Define where deterministic policy must stop an action

Checks

✓No source lacks a trust classification
✓Write or egress sinks are not protected only by a system prompt

2. Build an adaptive attack corpus

Outcome: Tests cover families of realistic attacks rather than one jailbreak prompt.

Tasks

→Add direct injection
→Add indirect web and document injection
→Add RAG poisoning
→Add tool-result and persistent-memory injection
→Add obfuscation and multi-step escalation

Checks

✓Every case has an attack goal and an expected safe outcome
✓The corpus is versioned and reproducible

3. Test containment, not only refusal

Outcome: Even manipulated model reasoning cannot cause a dangerous side effect.

Tasks

→Disable one control at a time
→Test allowlists, denylists, and schema validation
→Test exact-action approval
→Test network and URL egress limits
→Test secret isolation

Checks

✓Compromised reasoning does not inherit business authority
✓Every sensitive sink has an independent control

4. Introduce a release gate

Outcome: A critical source-to-sink regression blocks release automatically.

Tasks

→Split evaluations by severity
→Set hard blockers for data exfiltration and unauthorized writes
→Preserve minimized reproductions
→Add incident cases to the permanent regression suite

Checks

✓No aggregate score can hide a critical failure
✓Every production incident can become a reproducible test case

Acceptance criteria

A complete source-sink map exists
The corpus includes direct and indirect injection
Sensitive sinks have deterministic controls
Unauthorized write or egress is a hard blocker
Minimized regression cases and a residual-risk owner exist