1. Map system boundaries
Outcome: All data and authority flows are visible.
Tasks
- →Mark trusted and untrusted inputs
- →Identify secrets
- →Classify tools
Checks
- ✓Write capability is not hidden
Advanced•12–20 hours
AI Threat Model and Control Plan
Build an AI-system threat model covering abuse cases, controls, security tests, residual risk, and an incident playbook.
threat modelingprompt injectionagent permissionsred teamingincident response
Scenario
Task
An AI assistant uses RAG and tools with write access. Define trust boundaries, attack paths, and controls before production launch.
Step-by-step execution
2. Build abuse cases
Outcome: Threats are linked to real flows.
Tasks
- →Prompt injection
- →RAG poisoning
- →Data exfiltration
- →Tool abuse
- →Supply-chain compromise
Checks
- ✓Every threat has a precondition and impact
3. Define controls and tests
Outcome: Controls are verifiable.
Tasks
- →Least privilege
- →Approval gates
- →Sandboxing
- →Security evals
Checks
- ✓A control without test evidence is not considered ready
4. Run an incident drill
Outcome: The team can detect, contain, and recover.
Tasks
- →Trigger scenario
- →Kill switch
- →Rollback
- →Postmortem
Checks
- ✓Timestamps and evidence are recorded
Acceptance criteria
- Trust boundaries are complete
- Threat register is prioritized
- Controls have tests
- Residual risk has an owner
- Incident drill is complete
Assessment rubric
How the result is assessed
Passing score: 75/100 · Distinction: 92/100
System and trust boundaries
Data, model, tools, users, secrets, and external systems are fully mapped.
Insufficient
Authority or data flows remain hidden.
Competent
Core boundaries are explicit.
Strong
Data classification, tenant boundaries, and supply-chain lineage are documented.
Evidence required
- ✓ Link to an artifact or code
- ✓ README with decisions and trade-offs
- ✓ Evidence of completed checks
- ✓ Architecture/threat diagram
Threat register
Threats have preconditions, impact, likelihood, and an owner.
Insufficient
The list is generic and not tied to real flows.
Competent
Threats are contextualized and prioritized.
Strong
Attack trees, chained abuse cases, and residual risk are documented.
Evidence required
- ✓ Link to an artifact or code
- ✓ README with decisions and trade-offs
- ✓ Evidence of completed checks
- ✓ Prioritized threat register
Controls and security tests
Prevent, detect, and respond controls have test evidence.
Insufficient
Controls are declarative only.
Competent
Key controls are verified.
Strong
Automated security evals, policy as code, and a regression gate are implemented.
Evidence required
- ✓ Link to an artifact or code
- ✓ README with decisions and trade-offs
- ✓ Evidence of completed checks
- ✓ Control-test matrix
Incident readiness
Kill switch, containment, rollback, and postmortem procedures are verified.
Insufficient
The response plan has not been tested.
Competent
The drill is complete with evidence.
Strong
Timed recovery metrics and automated containment are implemented.
Evidence required
- ✓ Link to an artifact or code
- ✓ README with decisions and trade-offs
- ✓ Evidence of completed checks
- ✓ Incident drill report