What is an Eval?
Why it looked better is not evidence
An eval is a test for a system that gives you a different answer every time you run it. Instead of demanding one exact answer, it runs a set of saved inputs, scores each result against checks you wrote in advance, and gives you a score you can put next to last week’s. Without one, the only evidence you have is the impression a single try left you with.
🧭 Part 22 of the 🤖 Agents course
TL;DR
Nobody passes a driving test by driving well once.
The examiner has a list. Mirror check, hill start, parallel park, each one pass or fail on its own. An eval is that list: specific checks that can only come back pass or fail, written down before the drive starts.
One good drive proves nothing. The same candidate does it again tomorrow and stalls on the hill. So the test is the same task, run again and again, and the score is how often it holds.
The examiner has to watch the road. He cannot grade from the driver’s account of the drive, which is the part everybody gets wrong and the part with the ugliest numbers behind it.
The list grows from crashes. Every failure you find in production becomes a new line on the sheet, so the same mistake can never ship twice.
You write it before you need it. A test written after the failure only proves you understood the failure. The point is catching the next one.
Your Usual Test Breaks the Moment a Model Answers
Your test suite runs on a contract: the same input gives the same output, so one green run is proof forever.
A language model breaks that.
Ask the same question twice and the wording moves. Give an agent a job that needs tools and today it books the flight in three steps, tomorrow in five, and now and then it books the wrong flight. Nothing is stable enough to compare against.
So the industry kept the shipping and quietly dropped the checking. You change a prompt, run it once, read the answer, and if it reads better than the last one it goes out. What is Agent Prompt Engineering? is the craft that runs with no feedback loop under it.
Why “It Looked Better” Is Not Evidence
You tweak the system prompt. You run your one test question. The answer comes back cleaner, so you commit and move on.
You have shipped a prompt that way. It felt like testing.
It was an experiment with a sample size of one, on an input you picked yourself, graded by whether you enjoyed reading the output.
The real problem is that nothing in it could ever have told you that you were wrong. Run the old prompt again and it might give you the cleaner answer this time. You would not know. You did not run it again.
τ-bench puts agents through real conversations with real tool calls. Its best agent scored above 60% asked once, and under 25% asked to get the same task right eight times running1. Same agent, same tasks, and the only thing that changed was how many times anyone checked.
The first number sells the demo. The second one is your support queue.
Nobody Can Tell By Looking, Including the Model
Your own sloppiness is the smaller problem.
A 2026 study pulled the step-by-step logs of agent runs where the agent claimed to be finished, then went and checked the job. On AppWorld, a benchmark where a coding agent marks its own homework, 75.8% of the runs reporting success had not done the job.
Then they tried to catch that the way your team would: hand the log to a language model and ask whether it really worked.
Five judges, five prompting strategies, the full task specification handed over.
Score them the obvious way. Show the judge one run that really worked and one that only said so, and ask which is which. Guessing scores 0.50. Right every time scores 1.0.
The best of those twenty-five setups scored 0.65. On the coding logs, 0.542.
The judges read confident closing language and counted how many actions the agent took. An agent that ends with “I’ve successfully updated your booking” reads as successful whether or not a booking was updated.
Then the researchers tried the dumbest grader available: count which words appear in the log, ignore what they mean, no model involved at all.
It scored 0.83 and 0.95. It caught four to eight times more fake successes without flagging any more of the runs that had actually worked. It ran 3,300 times faster.
That is the state of the art on “did it actually work”.
Your First Eval Is a Morning’s Work
Twenty cases in a JSON file. Checks in whatever test runner you already have. Run on every merge.
That is the whole starting position, and it is a morning’s work.
One rule makes those checks worth writing: verify the world itself. If your agent writes a file, check the file. If it cancels an order, read the order back. The agent’s own report of what it did is not evidence of anything. A model-graded check gets added later, for the one or two things no code can decide, and it gets added grudgingly.
The teams with the best evals are the ones who argued longest about what belonged on the list.
The Four Parts of an Eval
Every one of these is a judgment call, and most teams usually make them by accident.
1. Which cases. Take them from your logs. Cases you invent come out of your picture of the system. The system already satisfies that picture. So an eval built from imagination passes on day one and never fails again. Twenty to fifty is the range where one lucky run cannot move the score and you can still read every failure yourself.
2. How specific. Ask “was this a good summary” and there is no answer that counts as a no, so the check passes every time and teaches you nothing. Ask “does the summary contain the order number” and there is an answer, every time, and it is one of two. Write enough of the second kind and the suite stops being only a score: when the number drops, the checks that went from pass to fail are your bug report.
3. What does the grading. Cheapest thing that works: a string match, a pattern, a schema, an assertion on the function the agent called (What is Function Calling? covers those). Most checks worth writing are decidable by code, which is lucky, given how the judges did.
4. What you compare it against. A score on its own tells you nothing. It earns its keep sitting next to last week’s. Skip this part and one day you will ship a prompt tweak that fixes the case you were staring at and quietly breaks four you were not. Nothing will tell you. That is the point of the number.
The same four parts answer two different questions depending on when you ask them. Before a feature ships you are asking whether the thing can do the job at all: that is a capability eval, and it is what tells you to launch. Afterwards you are asking whether it still does. That is a regression eval. It runs after every change and tells you if the score dropped. The second set grows every time production surprises you.
Nobody Solved This With a Better Grader
Two things work here, and neither one is what the eval vendors are selling:
Run every case more than once, and count only the ones that pass every time. A case that works on the first try and fails on the third is not a pass, and running it once will never show you that. So run each of your twenty cases five or ten times, and score the ones that came back right on all of them. Sierra built τ-bench on exactly this, because they sell customer-service agents, and an agent that works four times out of five turns straight into their support queue.
Have something other than the agent confirm the work. In one telecom setting in the false-success study, a second system had to verify that the change had actually landed before a run counted as done. False successes there were 3%, against 75.8% where the agent marked its own homework. Your version of this is one line long: after the agent says it cancelled the order, go and read the order.
Both are cheap. Neither needs a model.
What Can Go Wrong (and What’s Overhyped)
The public benchmark is the trap that catches good teams. It scores a model against a task set the whole industry can download, which means the model may well have trained on it, and none of those tasks came from your traffic.
Use one to pick which model to buy. But a benchmark scores a model on its own, and what you ship is that model plus your prompts, your tools and your data. None of that is in the number.
Everything a script can check, a script should check.
The overhyped part is the tooling. Every observability vendor now ships an eval product, the dashboards are good, and the demo shows you a spider chart of six quality dimensions scored by a judge.
Not one of them writes your checks.
Deciding what “correct” means for your twenty saved cases is the entire job. It takes an afternoon of arguing with your own team, and no purchase order shortens it. What is LLM Observability? covers the layer underneath.
The One Thing to Remember
Your test suite asks questions about your code.
An eval asks one about you: can you say what correct means, precisely enough that a machine can tell you that you are wrong? Most teams cannot, and that is why they have no eval suite. The tooling was never the problem.
💬 What is the check you wish you had written before your last production surprise? Tell me in the comments.
Where to Next?
📖 Go Deeper: Why AI Agents Keep Failing in Production, the failures an eval suite exists to catch.
🔗 Go Simpler: What is an AI Agent?, the thing you are grading.
🔀 Related: What are AI Agent Guardrails?, stopping bad output at runtime rather than catching it at build time.
🔜 Friday: How Netflix Serves LLMs In-House. Four decisions behind running the full stack yourself, and the production gaps the design phase missed.
FAQ
How is an eval different from a unit test?
A unit test asserts equality and one run proves it forever. An eval cannot work that way, because the same input gives different output every time. It runs many cases, runs each of them repeatedly, and reports a rate. A single success means nothing on its own.
Can I just use an LLM to grade the outputs?
For checks no code can make, yes, carefully. On real agent logs the best judge anyone tested scored 0.65, and only 0.54 on coding logs, where a coin flip scores 0.50. They grade confident language instead of verified changes. Use deterministic graders wherever possible, phrase model-graded checks as yes-or-no questions, and require the grader to quote its evidence.
How many test cases do I need to start?
Twenty real cases from your logs beat two hundred invented ones. Invented cases test the system you imagined. Your logs hold the traffic you actually got. Add one case for every production failure you find, and the suite grows into the shape of your actual problem.
What does pass^k mean?
Pass^k is the share of tasks where all k independent attempts succeeded, as opposed to pass@1, the share where a single attempt did. Report pass^k. Nobody gets one attempt at your product.
τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains, arXiv (June 2024)
One distinction worth drawing out: for single-turn systems evals measure output quality, but for multi-step agents you also need to score the trajectory. Two runs can return the same correct final answer via very different tool-call sequences, and the riskier path only surfaces if you evaluate intermediate steps rather than just the terminal score. That makes the 'score you can put next to last week's' harder to interpret — a number went up, but did the agent get there more safely or less?