Evals: Measuring AI Agents Before They Go Live

Learn the vocabulary and mechanics of evals, three grader types, pass@k versus pass^k, and how to build a first test set from real cases.

The customer service team at an insurance company built an AI Agent to triage complaints. They tried it on 5 sample cases, it passed them all, and everyone was excited. In the first week live, car claims were routed to the health insurance queue and customers waited for days. The problem was not only the agent. The team had no evals good enough to tell them whether it was ready.

What are evals

Evals (evaluations) are test sets that measure how well an AI Agent performs in a systematic, repeatable way you can compare every time you change a prompt, swap a model or add a tool. Unlike "I played with it and it felt good", the results come as numbers and evidence.

Anthropic's Demystifying evals for AI agents (January 2026) sets out the basic vocabulary:

• Task: one test with an input and success criteria
• Trial: one attempt. You need several, because AI output varies from run to run
• Grader: the logic that scores the attempt
• Transcript: the full record of a trial, including tool calls and intermediate reasoning
• Outcome: the final state of the real system, for example whether a booking actually exists in the database, not just whether the agent said it booked

Three kinds of grader

Code-based

Checked by code, for example verifying that the ticket was moved to the correct queue in the database. It is fast, cheap and gives the same result every time. The downside is that it can be too strict with answers that are correct but look different.

Model-based

Another LLM scores the output against a rubric. It is flexible and handles open-ended work, but results vary and need regular calibration against human graders. A real example is Anthropic's Research system, which used an LLM to score factual accuracy, citation accuracy, completeness, source quality and tool efficiency.

Human

Experts check the work themselves. This is the most accurate option and is used to calibrate model-based graders, but it is slow and expensive.

Example: one task for the insurance agent

The input is a customer message: "My car was rear-ended at the Asok intersection and I want to claim." A code-based grader checks that the ticket really sits in the motor claims queue in the system. A model-based grader checks that the reply is polite and lists all the documents the customer needs. Using both together gives you precision and flexibility.

pass@k versus pass^k: the number many teams overlook

pass@k is the chance that at least one of k attempts succeeds. pass^k is the chance that every one of k attempts succeeds. Anthropic's example: if each attempt succeeds 75% of the time, the chance that 3 attempts in a row all succeed falls to about 42%.

The pass^k metric was proposed in the τ-bench paper (Yao et al., 2024), which simulates customer service work where an agent must talk with users and follow policy. The paper reports that pass^8 for a gpt-4o based agent in the retail domain dropped below 25%. For work customers rely on every day, consistency matters more than succeeding "sometimes".

Capability evals and regression evals

• Capability: asks what the agent does well. It should start with low scores so there is room to improve
• Regression: asks whether it still does what it used to. It should sit near 100% to catch backsliding after changes

Capability tasks that pass consistently can graduate into the regression suite.

Watch out for

• Grading the path instead of the outcome: Anthropic warns that enforcing a fixed sequence of tool calls is too brittle, because agents often find valid approaches you did not anticipate. Grade what the agent actually achieved
• Trusting scores without reading transcripts: sometimes the agent was right and the grader was wrong
• Forgetting real people: even with automated evals, Anthropic found human testing necessary to catch edge cases the evals missed

A demo that passes 5 cases proves it can work. Evals prove it works every day.

How to get started

1. Collect 20 to 50 tasks from real cases, especially past failures, as Anthropic recommends
2. Write a checkable definition of "success" for each one, ideally checking the outcome in the real system
3. Run each task several times and look at both pass@k and pass^k
4. Read the transcript of every failed case and decide whether the agent or the grader was wrong
5. Run the regression suite before every new release

Eval design is a skill that applies across AI, QA and product roles. If you want to know where your experience could take you, talk to the AI Career Assistant, or read Why Agents Fail for ideas on what to put in your test set.