18 min

Production Monitoring & Feedback Loops

Sample live traffic, watch for drift and turn real failures into new tests.

Lesson 9 of 10

0 done

In plain English

Evaluation does not end at launch. Sample live traffic on purpose, both random and targeted. Watch quality, safety, behavior, cost and feedback. Catch drift, and close the loop by adding real failures to the test set. Respect privacy.

Drift: The world moving away from what the system was tested on.
Sampling: Reviewing a chosen small part of live traffic.
Feedback signal: A user action that hints at quality, such as an edit or a retry.

Picture it

Close the loop after launch

Sample live traffic

Random and targeted

Find failures

Judges, people, feedback

Add them to the test set

With right answers

Fix and release

Through the gate

Then it goes back to step 1 and repeats

Launch is not the end of evaluation. Real users ask questions your test set never saw. Documents change. Models are updated. Quality can slip in ways you never planned for. Production monitoring is how you notice and respond. This lesson covers what to watch, how to sample live traffic and how to turn feedback into better tests.

What to watch

Quality: sampled live answers scored by judges and people.
Safety: must-never-happen incidents, refusals and flagged content.
Behavior: answer length, refusal rate, tool use rate and steps per task.
Cost and speed: tokens, cost per task and response time.
Use: how many people use the feature and for what.
Feedback: thumbs up and down, edits, retries and escalations.

Sample live traffic

You cannot review everything. Sample on purpose. Take a random sample each day for an unbiased picture. Add targeted samples: low-confidence answers, thumbs-down answers, long conversations, new topics and flagged content.

monitoring-plan.md

DAILY      Random sample of {{n}} live cases scored by the judge; alert if the pass rate drops
WEEKLY     {{n}} cases reviewed by a person; compare with the judge's scores
ALWAYS     Every must-never-happen flag goes to a person within {{time}}
DASHBOARD  Pass rate, must-never-happen count, cost per task, response time, refusal rate,
           thumbs-down rate, escalation rate, new topics
ALERTS     Any must-never-happen incident; pass rate below {{threshold}}; cost above {{limit}};
           sudden change in refusal or length
OWNER      {{name}} reads the dashboard every {{day}} and writes a short note

Drift

Drift means the world moves away from what the system was tested on. Users ask about a new product. A policy changes. Wording shifts. The model provider updates the model. Quality falls slowly. Watch for it with trend lines and with a regular review of new kinds of questions.

Feedback signals

Users tell you quality in many ways. A thumbs-down is clear. Quieter signals matter too: they edited the answer a lot, asked the same question again, abandoned the chat or asked for a person. Treat these as clues. Review them, since a click may mean many things.

Close the loop

Review flagged and sampled failures every week.
Add real failures to the test set, with the right answers.
Fix the cause, then run the gate before you release the fix.
Tell users and the support team when you fix something they reported.

Who reads the dashboard

A dashboard that nobody reads is decoration. Name an owner. Give them a regular time to look at it and to write two or three lines: what changed, why they think so and what happens next. Share the note with the product team. Over a few months these notes become a story of how the system's quality moved, and they make decisions easier.

Privacy and fairness in monitoring

Follow your privacy rules for stored conversations. Mask personal data where you can.
Limit who can read live data, and log who did.
Check quality across groups, such as languages, to see whether some users get worse service.
Keep logs only as long as you need them.

⚠Do not rely on thumbs-down alone. Most users never click it. Silent failures are common, so sample real conversations.

→Hold a short weekly meeting where the team reads ten bad conversations together. It is the cheapest quality program there is.

Key takeaways

✓Watch quality, safety, behavior, cost, use and feedback. Sample live traffic on purpose, both random and targeted.
✓Drift is slow slipping as the world changes. Use trends and regular reviews to catch it.
✓Feedback signals include edits, repeats and escalations, not only thumbs-down.
✓Close the loop: add real failures to the test set, fix, run the gate and release. Respect privacy.

Hands-on exercise

Write a monitoring plan

Choose an AI feature and fill in the monitoring plan with real numbers.
List five feedback signals you could collect, and what each might mean.
Design a weekly ten-conversation review: who attends, what they bring, what they decide.
Write two alerts, with who is told and what they do first.

Check yourself

Q1Why sample both randomly and on purpose?show answer

Random samples give an unbiased picture. Targeted samples find known risk areas such as thumbs-down or new topics.

Q2What is drift?show answer

The world moving away from what the system was tested on, such as new products, changed policies or a model update, so quality slips.

Q3Why is thumbs-down not enough?show answer

Most users never click it. Silent failures are common, so you must sample real conversations too.

← Regression Tests & Release Gates Evaluation Interviews & Case Practice →