Advanced•12–18 hours

Production Observability for an AI System

Build monitoring and an incident workflow for an AI workload with latency, cost, quality, fallback, and rollback evidence.

SLOtracingcost monitoringquality signalsincident responsefallbacks

Scenario

Task

An AI feature depends on an external model and internal retrieval. The team must detect degradation in latency, cost, error rate, and answer quality before users do.

Step-by-step execution

1. Define the signals

Outcome: The operational state of the AI feature is measurable across the full pipeline.

Tasks

→Add latency and error rate
→Measure token and cost usage
→Add retrieval and quality signals
→Correlate by request ID

Checks

✓An end-to-end trace exists
✓PII and prompts do not enter logs without an explicit policy

2. Build the SLO

Outcome: The team has explicit limits for acceptable degradation.

Tasks

→Define an availability SLO
→Add a latency SLO
→Define a quality proxy
→Calculate the error budget

Checks

✓Alerts are tied to user impact
✓A single transient error does not trigger an alert

3. Add the control plane

Outcome: The system can enter degraded mode safely.

Tasks

→Add rate limits
→Configure a fallback model
→Add queue backpressure
→Implement a kill switch

Checks

✓Fallback behavior is tested
✓The kill switch does not require a new deployment

4. Run an incident drill

Outcome: Recovery is backed by evidence.

Tasks

→Simulate a provider outage
→Simulate a cost spike
→Verify rollback
→Create a postmortem

Checks

✓The timeline can be reconstructed from telemetry
✓Recovery matches the runbook

Acceptance criteria

End-to-end observability exists
SLO and error budget are recorded
Cost and quality have alerts
Fallback and kill switch are tested
The incident drill ends with a postmortem

Assessment rubric

How the result is assessed

Passing score: 75/100 · Distinction: 92/100

Telemetry coverage

Model, retrieval, and application layers have end-to-end visibility.

25 points

Insufficient

Only application logs exist.

Competent

Latency, errors, cost, and quality are correlated.

Strong

Distributed tracing and a redaction policy are in place.

Evidence required

✓ Link to code or artifact
✓ README with decisions
✓ Test output or runtime evidence
✓ Trace and metrics map

SLO and alerts

Alerts reflect user impact and the error budget.

25 points

Insufficient

Alerts are noisy or lack an SLO.

Competent

SLOs and thresholds are defined.

Strong

Burn-rate alerts and capacity signals are automated.

Evidence required

✓ Link to code or artifact
✓ README with decisions
✓ Test output or runtime evidence
✓ SLO document and alert examples

Fallback and control plane

Rate limits, fallback, backpressure, and the kill switch are verified.

25 points

Insufficient

There is no safe degraded mode.

Competent

Fallback and the kill switch work.

Strong

Automated routing and a rollback policy are implemented.

Evidence required

✓ Link to code or artifact
✓ README with decisions
✓ Test output or runtime evidence
✓ Fallback test

Incident readiness

A failure drill includes a timeline, recovery, and postmortem.

25 points

Insufficient

The runbook is declarative only.

Competent

The drill was completed with evidence.

Strong

Recovery is automated and tested regularly.

Evidence required

✓ Link to code or artifact
✓ README with decisions
✓ Test output or runtime evidence
✓ Incident report