Evaluation in Production

Observability tells you what happened. Evaluation tells you whether it was good, and whether your next change will make things better or worse. Teams without evals steer by stories. Someone tries three prompts, the fix seems fine, and a week later something else is quietly broken. Evals are the unit tests of agent building. They are the biggest single difference between Level 1 and Level 2.

The eval pyramid

• Base, code checks. Is the schema valid? Are the required fields there? Do the tests pass? Do forbidden words stay out? Was the tool called with valid inputs? Did the state change as expected? These checks are cheap and fast, and they run on everything.
• Middle, scenario tests: a chosen set of realistic inputs with expected results or checklists, run on every change.
• AI as judge: a model grades outputs against a written rubric. It scales, but it must be checked against people.
• Top, human review: expert spot-checks, review queues and regular audits. This is expensive and authoritative. It is the source of truth for checking everything below it.

Where test sets come from

Start with 20 to 50 cases that you write by hand from real user needs, including the awkward ones. Then let live use feed you. Every bad trace becomes a new case. That includes a thumbs-down, an escalation, a human correction or an incident. Include the boring happy paths too. A set made only of failures will make every change look like a regression. Tag each case with its type and difficulty, so you can see which slice moved.

Grade outcomes, not only text

For agents that take actions, the final text is the least important result. Did the ticket really get the right status? Was the refund issued exactly once, for the right amount? Check the end state of the world, such as database rows, files and API calls made, against what should have happened. Also grade the path when it matters. Did it use only the allowed tools, stay within budget and ask for approval when it had to?

Measure consistency, not just one pass

Agents are not predictable, so run each case several times. It can solve it sometimes is pass@k: it succeeds at least once in k tries. That is a very different claim from it solves it every time, where all k tries succeed. Customer-facing agents live or die by the second claim. Report both, along with the spread. Treat a case that passes 3 times out of 5 as a failing case with a flaky cause to investigate.

AI judges: powerful and slippery

• Write a rubric with concrete criteria that are scored separately. Do not write rate the quality.
• Ask for the reasoning before the score, and keep scales small, such as pass or fail, or 1 to 5.
• Use a different model as judge, often a stronger one, than the model being tested. This reduces bias towards its own style.
• Watch out for known biases: a liking for longer answers, for the first option and for its own style.
• Calibrate. Have people label a sample, measure how often they agree with the judge, and check again from time to time.

You are grading an AI support agent's reply. Score each criterion 0 or 1 and explain briefly BEFORE scoring.

1. ACCURACY      - every factual claim is supported by the provided knowledge-base excerpts.
2. COMPLETENESS  - it answers every question the customer asked.
3. POLICY        - it did not promise refunds/credits outside policy P-12.
4. TONE          - professional, no blame, no jargon.
5. ACTION        - if an action was needed it chose the correct tool with correct arguments.

Return JSON: {"reasoning": "...", "accuracy":0|1, "completeness":0|1, "policy":0|1, "tone":0|1, "action":0|1}

Evals as a release gate

import json, sys

BASELINE = json.load(open("evals/baseline.json"))     # last released scores
results  = run_eval_suite(cases="evals/cases.jsonl", trials=3)

failures = []
if results["task_success"] < BASELINE["task_success"] - 0.02:
    failures.append(f"task_success regressed: {results['task_success']:.1%}")
if results["policy_violations"] > 0:
    failures.append("policy violations must be zero")
if results["cost_per_success"] > BASELINE["cost_per_success"] * 1.25:
    failures.append("cost per success up >25%")

if failures:
    print("EVAL GATE FAILED\n- " + "\n- ".join(failures)); sys.exit(1)
print("eval gate passed")

Online evaluation

• Sample live traffic and run judges in the background, to track quality over time and catch drift.
• Shadow mode: run a new version on real inputs without showing its output, then compare.
• Canary and A/B: send a small share of traffic to the new version and compare the outcome numbers.
• Quiet signals: retries, rewording, giving up, human overrides and escalations. They are noisy, but there are plenty of them.