How to evaluate AI tool calling: a practical checklist

A tool-use eval starts with the verifiable task outcome, not with whether the model called the expected function. Pin the initial state, user request, available tools, data and authority boundaries, permitted changes, completion criterion, and forbidden events. Otherwise, exact matching against a reference call rewards one trajectory even when another safe path reaches the same correct result.

Separate read, propose, and mutate tools. Finding a document, preparing a draft, and sending an email carry different risks and need different graders. For every fixture, version the prompt, model, tool catalog, schemas, policy, and sandbox snapshot. Real tasks should require ambiguity handling, clarification, multiple calls, and error recovery rather than merely repeating a function name from the request.

• Initial state → known records, permissions, clock, and external dependencies.
• Expected outcome → a verifiable terminal state or a correct abstention.
• Allowed trajectory → required invariants without forcing one route.
• Forbidden event → unauthorized read/write, leakage, duplicate action, or action without approval.

Build slices for correct selection, confusion between similar tools, a missed call, an unnecessary call, and cases where no tool is needed at all. Separately test unknown entities, missing required fields, ambiguous dates, the wrong tenant, stale identifiers, and requests outside authority. Valid JSON proves only syntax: a semantic grader must verify that arguments match intent, current state, and policy.

Tool precision and recall are useful only within a specific slice. High aggregate accuracy can hide a model that consistently chooses a dangerous mutating tool instead of a read-only check. Add negative cases where the right behavior is to ask a clarifying question, refuse, or escalate to a human. For parallel calls, verify that operations are independent and that correctness does not depend on a random completion order.

A mock that always returns success does not test tool calling. The eval environment must reproduce schema validation, authorization, latency, pagination, rate limits, partial results, timeouts, and side effects. For mutating tools, use an isolated database or record-replay environment with authoritative read-back. The harness should record the model proposal, policy verdict, actual request, tool response, and terminal state as distinct events.

Inject controlled failures: a 429 before execution, a timeout after commit, schema evolution, permission revocation between planning and execution, duplicate delivery, and stale state. After an uncertain result, the correct trajectory first reconciles through a read-only operation instead of retrying blindly. The idempotency key must remain stably bound to the operation intent; generating a new random key for each attempt does not prevent duplicates.

The outcome grader reads the system of record and verifies that the required state was reached. The trajectory grader checks required and forbidden events: authorization before a write, approval bound to the exact payload, no secrets in arguments, correct handling of tool errors, and stopping after completion. The side-effect grader counts duplicates, orphaned writes, wrong destinations, and changes outside scope. A polished final answer cannot compensate for a dangerous action in the trace.

Do not require a literal match for every step when several trajectories are valid. Exact order matters for security invariants—for example, policy lookup before a refund—but not for two independent read calls. A model grader can assess explanation quality or whether clarification was appropriate, while permissions, schemas, ledger balance, and final state should be checked deterministically. Human calibration remains necessary for ambiguous business criteria.

• Outcome → correct state, answer, or justified refusal.
• Trajectory → correct decisions, calls, policy gates, and error transitions.
• Integrity → no unnecessary, duplicate, or unauthorized side effect.
• Efficiency → calls, tokens, latency, and cost per successfully verified task.

Tool performance depends on more than the model. Names, descriptions, overlap, parameters, response format, number of available tools, and returned context all change agent behavior. Compare the candidate with a baseline on a frozen regression set and a separate held-out set. Ablating one factor at a time helps distinguish a schema improvement from a sampling shift or data leakage.

Report task success, critical policy violations, semantic argument validity, the tool-selection confusion matrix, unnecessary calls, recovery success, final-state mismatch, p95 latency, and cost per verified success. Add repeated runs and confidence intervals for stochastic agents. Do not transplant a vendor benchmark into your workflow: the production catalog, data distribution, permissions, and error surface are different.

The decision record pins the dataset, model, prompt, catalog, schemas, executor, policy, thresholds, owner, and rollback revision. A critical unauthorized write, cross-tenant read, secret exposure, approval bypass, or duplicate financial action blocks release regardless of average task success. A non-critical regression may narrow the catalog, disable parallel calls, move a tool back to read-only, or require human approval.

Rollout moves through an offline sandbox, shadow traffic without side effects, a read-only canary, approval-required writes, and only then bounded autonomy. Production telemetry uses the same event schema as the eval harness, with redaction and retention controls. After an incident, reconcile external state first, block the affected tool/version, and restore the known-good envelope; the sanitized trace then becomes a permanent regression fixture.