Skip to main content
Core•8–14 hours

AI Capacity, Cost & Chaos Lab

Stress-test an AI workload across concurrency, token/tool budgets, queues, backpressure, 429/5xx/timeouts, load shedding, degraded mode, and cost per successful verified task.

capacity planningautoscalingqueueingbackpressurefailure injectionAI unit economics

Scenario

Task

An AI endpoint is stable under demo load, but production traffic has bursty arrival rates, long contexts, parallel tool calls, and provider quotas. CPU autoscaling alone does not solve token throughput, queue delay, 429s, retry amplification, or cost explosions. Build a capacity model that measures successfully verified tasks rather than merely the number of running pods.

Step-by-step execution

1. Build the workload model

Outcome: Capacity planning reflects AI-specific demand instead of an average HTTP request count.

Tasks

  • →Measure arrival rate and burstiness
  • →Collect input/output token distributions
  • →Record tool-call fan-out and the slowest dependency
  • →Separate user-facing, batch, and high-priority workloads

Checks

  • ✓P95/P99 workload differs from the average case
  • ✓Unknown quota or provider limits are marked as risks
  • ✓Priority policy does not starve critical tasks

2. Set budgets and scaling signals

Outcome: Autoscaling reacts to saturation, queues, and token throughput rather than CPU alone.

Tasks

  • →Define the concurrency ceiling
  • →Set token/tool/time budgets
  • →Add queue age/depth and saturation signals
  • →Calculate cost per successful verified task

Checks

  • ✓Budget exhaustion terminates or degrades a task in a controlled way
  • ✓A scaling signal has a causal relationship to the bottleneck
  • ✓Failed or unsafe tasks are not counted as successful in the cost metric

3. Run chaos and load tests

Outcome: Known overload and dependency failures produce predictable system responses.

Tasks

  • →Inject 429/5xx/timeouts
  • →Delay a tool or retrieval dependency
  • →Create a burst of long-context requests
  • →Test retry amplification and queue recovery

Checks

  • ✓No infinite retries or unbounded queue growth
  • ✓Load tests do not bypass tenant or risk limits
  • ✓Recovery does not create a second spike through synchronized retries

4. Verify graceful degradation

Outcome: When capacity is scarce, the system reduces functionality rather than control.

Tasks

  • →Test load shedding
  • →Use a lower-cost/fallback model only for eligible tasks
  • →Disable non-essential tools
  • →Record recovery and normal-mode re-entry criteria

Checks

  • ✓Fallback passes the minimum eval contract
  • ✓Degraded mode does not increase autonomy
  • ✓Normal mode resumes only after dependencies stabilize

Acceptance criteria

  • The workload model includes burst, token, tool, queue, and provider-quota characteristics
  • Capacity policy has bounded concurrency/token/tool/time budgets and explicit autoscaling signals
  • The chaos suite covers 429, 5xx, timeout, slow dependency, quota exhaustion, and retry amplification
  • Load shedding/degraded mode exists with unchanged authority and privacy boundaries
  • Cost is measured as cost per successful verified task rather than cost per request