Evaluation Interviews & Case Practice

This last lesson prepares you for the interview. Picture the hiring manager. They have shipped AI features and been burned by weak testing. They want someone who can bring order to quality. They will test your thinking about metrics, data, judges and release rules. This lesson shows the common questions, a way to answer them and what to put in your portfolio.

What the interviewer wants to see

• You define quality from the user's job, not from a metric you like.
• You build test sets from real data and you know their limits.
• You use code checks, judges and people in the right places.
• You find root causes with error analysis, not guesses.
• You connect evaluation to release decisions and live monitoring.
• You are honest about noise, bias and what you cannot measure.

Common questions and how to answer

Question: how would you evaluate a support chatbot? A strong answer starts with the user's job, then defines a rubric with correct, grounded and tone, plus must-never-happen rules. It builds a test set from real tickets with experts' answers, splits it into a development set and a hidden set, and scores search and answers separately. It uses code checks, a calibrated model judge and human review on samples. It sets a release gate and monitors live traffic.

Question: your model judge gives high scores, but users complain. What do you do? A strong answer suspects the judge. It compares the judge with human scores on a sample, looks at length and position bias, reads the complaints, adds them to the test set and fixes the rubric. It does not trust a number that disagrees with users.

Question: the new prompt raised the average by two points. Do you ship it? A strong answer asks about noise and slices. With a small test set, two points may be chance. It looks at case-level flips, must-never-happen cases, important groups and cost. It runs a shadow or canary before full release.

Question: how do you test an agent that sends emails? A strong answer uses a sandbox with fake accounts and test data. It sets the starting state the same way each time. It runs each case several times. It checks the final state and the path. It requires approval for risky actions. It never tests against real customers.

A method for open design questions

1. USER JOB      Who uses this, and what are they trying to get done?
2. QUALITY       Rubric: three to five dimensions, plus must-never-happen rules.
3. DATA          Where do cases come from? How many? How labeled? Hidden set?
4. CHECKS        Code rules first, then judges, then people. How will you calibrate?
5. PIPELINE      Score each step (search, tools, answer), then the whole.
6. ANALYSIS      How will you study failures and slice results?
7. GATE          What blocks a release? How do you roll out and roll back?
8. MONITOR       What do you watch live, and how do failures return to the test set?
9. LIMITS        What can't you measure? Where is the noise?

Show your work

• Build an evaluation harness for a small RAG app: a test set of 50 cases, code checks, a calibrated judge and a report.
• Write an error analysis with 50 labeled failures and the fix for the top cause, with before and after scores.
• Make a release gate document and a monitoring plan for a fictional product.
• Publish a short write-up: what you measured, what you learned and what you could not measure.

Questions to ask them

• How do you decide today whether a change is good enough to ship?
• Who owns the test sets, and how often are they updated?
• How do real failures get back into testing?
• What has gone wrong in production, and what changed afterwards?