How to Evaluate AI Agents Before Production: Metrics, Test Cases and Release Gates

Why a good demo is not production evidence

A demo usually shows a selected prompt, a favorable environment and one successful run. Production contains incomplete requests, conflicting data, slow APIs, expired credentials, unexpected tool output, changing system state and users who do not phrase tasks as designers expect. A single success cannot estimate how reliably the agent will operate across that distribution.

Agent evaluation is also harder than ordinary response evaluation. Google ADK distinguishes the final response from the trajectory—the sequence of decisions and tool calls used to reach it. An agent can give a plausible answer after using the wrong customer record, or arrive at the right state through an unsafe or wasteful path. Both the process and the result matter.

• Capability asks whether the agent can solve a task
• Reliability asks how often it solves the task under repeated conditions
• Safety asks whether it stays inside authority and policy
• Resilience asks how it behaves when dependencies or evidence fail
• Efficiency asks whether the successful path meets latency and cost limits

Define the unit of evaluation

Choose a unit that matches the business promise. A turn is appropriate for a narrow classification. A conversation may be required for support resolution. A workflow may include several agents, tools and human approvals. A business outcome may only be observable after the underlying system reaches a verified state.

Do not score a workflow against a requirement that is invisible to the evaluator. If success means a CRM case is updated correctly, capture the before-and-after state. If success means a customer received the right refund, verify the transaction and policy—not just the agent's confirmation message.

• Input: user request, trigger or event
• Initial state: relevant records, permissions and environment
• Allowed actions: tools, data and approval boundaries
• Expected outcome: observable state or deliverable
• Forbidden outcome: unsafe, unauthorized or misleading result
• Evidence: logs, state checks, artifacts and reviewer rubric

Build an evaluation dataset that represents real work

Start with product requirements and the checks people already perform manually. Add resolved incidents, support tickets, known edge cases and examples where users corrected the agent. Anthropic recommends beginning early and notes that 20–50 simple tasks from real failures can be a useful initial set; mature systems need larger and harder suites as improvements become smaller.

Each task must be unambiguous enough that two qualified reviewers can reach the same verdict. Specify necessary context, tool availability, expected artifacts and acceptable variations. Keep a development set for iteration and a holdout set that is not used to tune prompts, policies or graders.

• Happy-path tasks in realistic proportions
• Incomplete and ambiguous requests
• Conflicting or stale evidence
• Permission and tenant-boundary cases
• Tool timeout, malformed output and partial failure
• Duplicate request and retry scenarios
• Adversarial or manipulative content
• High-impact cases that require refusal or approval

Separate capability and regression suites

Capability evaluations contain tasks the agent does not yet solve reliably. They show where improvement is possible and help compare designs, tools or models. A low initial score is not necessarily a defect if the suite is deliberately difficult.

Regression evaluations protect behaviors that already work. Anthropic recommends treating these as a near-perfect reliability bar: when a previously solved case fails after a prompt, model, tool or policy change, the release needs investigation. As difficult capability cases become stable, promote them into the regression suite.

• Capability suite: difficult frontier tasks and new features
• Regression suite: previously solved critical behavior
• Safety suite: prohibited actions and policy boundaries
• Resilience suite: dependency and data failures
• Cost suite: representative high-token and high-tool workloads
• Do not combine all suites into one average that hides a critical failure

Use three kinds of graders

Deterministic graders are the first choice when success is objectively observable. They can check database state, API responses, file structure, exact fields, numerical tolerances, tool parameters, policy decisions or the absence of a forbidden mutation. They are fast, reproducible and easy to debug, though overly rigid checks can reject valid alternatives.

Model-based graders help with open-ended qualities such as completeness, relevance, groundedness, tone and the quality of a multi-step trajectory. Use a specific rubric with examples and separate criteria rather than a vague instruction to rate the answer. Human domain experts remain necessary for ambiguous, high-impact and novel cases and for calibrating model judges.

• Code-based: state checks, schemas, invariants and exact outcomes
• Model-based: rubric scoring, pairwise comparison and groundedness
• Human: expert review, disagreement resolution and spot checks
• Use independent evidence; do not ask the agent to grade itself
• Record the grader version and rubric with every result

Evaluate task completion and final state

Task completion is the strictest business-facing measure: did the agent deliver every material requirement and leave the system in the correct state? Microsoft Foundry separates system evaluation from process evaluation and defines task completion around a usable result that meets the request.

Prefer end-state verification over judging a confident explanation. A scheduling agent passes when the correct slot is booked under the correct account and policy, not when it says the booking succeeded. A research agent passes when required claims are supported and the deliverable is complete, not when the prose sounds authoritative.

• All required outputs present
• Authoritative system reflects the expected state
• No prohibited side effect
• Result belongs to the correct user, tenant and record
• Evidence supports material claims
• Failure or uncertainty is communicated accurately

Evaluate trajectory and tool use

Inspect which tools the agent selected, the order of calls, parameters, outputs used and unnecessary actions. Google ADK describes this sequence as the trajectory. Microsoft lists complementary process measures including tool selection, tool-input accuracy, tool-call success and correct use of tool results.

Do not demand one exact path when several are valid. Define required, optional and forbidden steps or score the trajectory with a rubric. A shorter path is not automatically better if it skips verification; a longer path is not automatically safer if it repeats expensive calls or expands exposure to untrusted data.

• Correct tool selected for the operation
• Parameters valid, scoped and authorized
• Required verification performed before mutation
• Tool output interpreted without invention
• No redundant, circular or forbidden calls
• Retries bounded and idempotent
• Approval obtained immediately before protected action

Measure reliability with repeated trials

Models and agent workflows are non-deterministic, so a case that passes once may fail later. Run important cases multiple times under controlled conditions and retain trial-level results. Report the distribution rather than only the best run.

Anthropic distinguishes pass@k—the chance of at least one success in k attempts—from pass^k, the chance that every one of k attempts succeeds. pass@k is useful when several independent attempts are acceptable; pass^k better reflects experiences where users expect consistent success. For most customer-facing or action-taking agents, first-try success and repeated consistency are more meaningful than eventual success after many retries.

• First-attempt task success
• All-trials consistency for critical tasks
• Failure mode frequency by task family
• Variance across models, prompts and environments
• Confidence intervals when sample size supports them
• Never hide retries that increase cost or duplicate-action risk

Test safety as a release gate, not an average

Create cases for unauthorized access, cross-tenant identifiers, prompt injection, sensitive data, unsupported claims, prohibited destinations, privilege changes and high-impact actions without approval. Verify the enforced outcome at the tool gateway and system of record; a refusal sentence does not prove that an unsafe call was blocked.

Set zero-tolerance or explicitly risk-approved thresholds for critical failures. An agent with a high overall success rate should still fail release if it can expose another customer's data or execute an unapproved payment. Align test ownership and documentation with the organization's risk process; NIST's AIRC provides resources for testing, evaluation, verification and validation in support of the AI Risk Management Framework.

• Authorization and tenant isolation
• Data minimization and destination control
• Prompt-injection and untrusted-tool-output handling
• Approval and business-rule enforcement
• Safe behavior under missing evidence
• Kill switch, rollback and recovery
• Privacy-aware logs sufficient for investigation

Evaluate resilience and recovery

Agents fail at integration boundaries as often as at reasoning. Simulate timeouts, rate limits, expired tokens, malformed responses, stale data, partial writes, duplicate events and unavailable downstream systems. Confirm that the agent distinguishes a technical failure from a completed business action.

A safe workflow should stop or escalate when it cannot establish required evidence or authority. Test compensation and recovery for partially completed operations. Preserve correlation IDs and idempotency keys so evaluators can prove what happened and prevent a retry from creating a second effect.

• Bounded timeout and retry policy
• No success claim after an unknown or failed effect
• Idempotent repeated request
• Partial execution detected and reconciled
• Fallback or human escalation available
• Recovery produces an auditable final state

Track latency, cost and operational efficiency

Quality alone does not determine viability. Record end-to-end latency, time to first useful result, model tokens, tool calls, external API charges, retry count and human review time. Evaluate representative load and long-tail tasks because averages can conceal cases that exhaust budgets or service-level targets.

Compare cost per successfully completed task, not cost per model response. Include failed attempts, repeated calls, monitoring, human correction and infrastructure. A cheaper run that creates more rework may be worse than a more expensive run that finishes correctly on the first attempt.

• Median and tail latency
• Cost per attempted and completed task
• Tool calls and model invocations per success
• Human review and correction minutes
• Retry and fallback frequency
• Throughput under expected concurrency
• External rate-limit and budget adherence

Calibrate model-based judges

Before relying on a model judge, assemble examples with expert labels and measure agreement. Review false passes and false failures by criterion. Refine ambiguous rubrics, add counterexamples and retest on cases outside the calibration set. Avoid using the same model configuration as both agent and sole judge for subjective criteria when independence matters.

Monitor judge drift when the model, prompt or rubric changes. Keep deterministic checks authoritative for permissions, transactions and system state. A model judge can explain whether an answer appears grounded; it should not overrule evidence that the wrong record was changed.

• One criterion per score where practical
• Observable anchors for each rating
• Examples at pass, fail and boundary
• Expert-labeled calibration set
• Agreement and disagreement reviewed
• Judge prompt, model and version recorded
• Periodic human spot checks in production

Set release thresholds before seeing results

Define release policy in advance to reduce pressure to reinterpret failures. Use separate gates for task completion, critical regressions, safety, reliability, latency and cost. State which failures block release, which permit a limited canary and which create a time-bound remediation item.

Weight cases by business impact and realistic frequency, but keep rare catastrophic cases visible as hard gates. Compare a candidate with the current production version on the same dataset and environment. Require an explanation for statistically or operationally meaningful regressions, even when the total score improves.

• No critical authorization, privacy or unsafe-action failure
• Regression suite meets its required reliability bar
• Capability score improves without unacceptable tradeoffs
• Tail latency and cost stay within operating limits
• Known limitations documented in the product experience
• Rollback criteria and accountable approver named

Run evaluations in CI and before configuration changes

Treat prompts, model versions, tool descriptions, schemas, retrieval settings, memory policies and orchestration code as release inputs. Run a fast regression subset on every relevant change and the complete suite before production promotion. Keep the environment stable enough to make comparisons meaningful.

Store the dataset version, configuration, results, traces and artifacts together. When a test fails, engineers should be able to reconstruct the initial state, tool calls, policy decisions and final effect. A percentage without a trace is hard to debug.

• Fast deterministic checks on each change
• Full repeated suite before production
• Canary evaluation on production-shaped infrastructure
• Versioned task and grader definitions
• Trace and artifact retention appropriate to privacy rules
• Automatic comparison with the current release

Monitor production and feed failures back into evals

Offline evaluations cannot represent every user, dependency and data condition. In production, monitor verified task outcomes, policy denials, approval rejections, corrections, abandonment, tool failures, latency and cost. Sample traces for expert review while minimizing unnecessary sensitive data.

Create a disciplined feedback loop: identify a material failure, preserve a privacy-safe reproduction, define the correct outcome, add it to the development suite, fix the system, and then promote the case to regression coverage. Do not train directly on noisy user feedback without validating the underlying event and desired behavior.

• Outcome metrics linked to real system state
• Alerts for safety and reliability thresholds
• User correction and human override captured
• Incident-to-regression-test process
• Distribution shift and new task families reviewed
• Scheduled reevaluation after model or tool changes

A 30-day evaluation program

Days 1–5: define the agent's promise, critical tasks, prohibited outcomes and accountable owners. Select the evaluation unit and establish a baseline from the current workflow or production version.

Days 6–12: collect 20–50 representative cases, write unambiguous specifications and create deterministic state checks. Split development and holdout sets. Add safety and dependency-failure cases.

Days 13–18: create rubrics for subjective criteria, label a calibration sample with domain experts and measure judge agreement. Instrument traces, latency, token use, tool calls and final state.

Days 19–24: run repeated trials, analyze failure clusters and fix system-level causes. Compare the candidate with the baseline on identical cases and infrastructure.

Days 25–30: apply predeclared release gates, run a limited canary, verify monitoring and rollback, and record the release decision. Schedule continuous regression runs and a process for adding production failures.

• Deliverable: versioned evaluation specification
• Deliverable: development, holdout, regression and safety sets
• Deliverable: calibrated graders and traces
• Deliverable: baseline and candidate scorecard
• Deliverable: signed release, canary or stop decision

Production-readiness checklist

An agent is ready for a controlled production release only when success is observable, tests represent real work, critical outcomes have deterministic verification, subjective graders are calibrated, repeated trials meet reliability requirements and the workflow behaves safely under missing data and failed tools.

This framework is vendor-neutral operational guidance, not a certification, penetration test or legal opinion. Test depth and thresholds must reflect the agent's users, authority, data and potential impact. Platform evaluation features can accelerate implementation, but the organization remains responsible for defining success and verifying real-world outcomes.