Advanced•10–14 hours

Agent Side-Effect Test Oracle Lab

Test an agent workflow by its trajectory, permissions, tool calls, side effects, and authoritative system-of-record state—not merely by its final text.

agent testingtrajectory evaluationauthority testingidempotencystate reconciliation

Scenario

Task

An agent can read CRM data, create a task, and update status. Its textual response may look correct while a timeout after a write, duplicate delivery, or stale approval leaves an incorrect side effect. Build a test oracle that verifies the actual action history and authoritative final state instead of trusting the model narration.

Step-by-step execution

1. Define observable truth

Outcome: Every consequential step has a deterministic postcondition.

Tasks

→List allowed tool actions
→Define the authoritative system for every side effect
→Specify the idempotency-key contract
→Describe prohibited transitions

Checks

✓Final assistant text is not the only oracle
✓Every write action has an authoritative read-back

2. Record the expected trajectory

Outcome: The test knows the allowed sequence of state and tool transitions.

Tasks

→Describe the normal path
→Add approval and deny paths
→Add escalation and abstain paths
→Define stop conditions

Checks

✓An out-of-scope tool call fails even if the final answer is correct
✓Approval is bound to the exact action and version

3. Inject failures around side effects

Outcome: Retries neither create duplicates nor hide unknown state.

Tasks

→Inject timeout after accepted write
→Inject duplicate event delivery
→Inject stale precondition
→Revoke permission
→Inject partial downstream failure

Checks

✓UNKNOWN triggers reconciliation rather than blind retry
✓Duplicate delivery does not create a repeated side effect

4. Build the CI gate

Outcome: Critical authority or state regressions block release.

Tasks

→Separate text, trajectory, and outcome graders
→Set severity-aware thresholds
→Persist the trace artifact
→Add incident-derived permanent regression cases

Checks

✓A critical side-effect failure cannot be offset by a high text-quality score
✓The regression report contains the exact configuration fingerprint

Acceptance criteria

Authoritative final state is checked by a deterministic oracle
Trajectory has explicit allowed and prohibited transitions
Timeout-after-write passes a reconciliation test
A duplicate action is not executed twice
A critical permission or side-effect regression blocks release