Build a regression gate for an LLM feature with deterministic checks, model graders, safety cases, and release thresholds.
Scenario
A team changes prompts and models regularly. It needs an automated gate that blocks a release when quality, safety, or structured-output stability regresses.
Outcome: Product risks are covered by explicit test classes.
Tasks
Checks
Outcome: Each property has an appropriate evaluation method.
Tasks
Checks
Outcome: Release decisions follow recorded rules.
Tasks
Checks
Outcome: Every change receives a transparent verdict.
Tasks
Checks
Assessment rubric
The dataset covers normal, edge, and adversarial scenarios.
Insufficient
Happy paths dominate.
Competent
Critical risks are covered.
Strong
Coverage is tied to production failures and the threat model.
Evidence required
Deterministic checks and model graders match the properties they evaluate.
Insufficient
One LLM judge evaluates everything.
Competent
Deterministic and model-based checks are separated.
Strong
Grader calibration and disagreement analysis are automated.
Evidence required
Baseline, thresholds, and blocking rules are versioned.
Insufficient
The verdict is subjective.
Competent
Thresholds and blocking conditions are explicit.
Strong
Risk-based exceptions and trend analysis are implemented.
Evidence required
The gate blocks weak releases and stores evidence.
Insufficient
The report does not affect release.
Competent
A failure makes the job fail.
Strong
Artifacts, diff reports, and flaky-eval controls are present.
Evidence required