Metrics & Feedback Loops

Measure use, quality, trust and value together, and turn feedback into the roadmap.

In plain English

Measure four groups together: use, quality, trust and value. Pair each metric with a guard so success stays honest. Log and sample real conversations, tag failures, and run a weekly review that feeds the roadmap and the test set. Write a stop rule at launch.

Acceptance rate: How often users keep the AI's output.
Guard metric: A second measure that stops one number being gamed.
Slice: A group of users or cases looked at on its own.

Picture it

The weekly learning loop

Read real conversations

A sample of 20

Tally failure types

Biggest first

Pick one or two fixes

With owners

Release through the gate

Then watch the metrics

Then it goes back to step 1 and repeats

After launch, the work shifts to learning. You need to know whether people use the feature, whether it is good, whether they trust it and whether it pays. A single number cannot answer all of this. This lesson shows a set of measures for AI products, how to read them together and how to turn feedback into a better roadmap.

Four groups of metrics

metric-groups.md

USE         Are people using it?
            activation (first successful use), weekly active users, tasks per user, retention

QUALITY     Is it good?
            pass rate on sampled cases, must-never-happen incidents, groundedness,
            refusal rate, error reports

TRUST       Do people rely on it the right amount?
            acceptance rate, edit rate, override rate, regenerate rate, escalation to a person,
            thumbs up and down, repeat use after an error

VALUE       Does it matter to the business?
            time saved, tasks completed, resolution rate, conversion, revenue, cost per task,
            support tickets avoided

Read them together

High use and low quality is a risk. People may be misled.
High quality and low use means people do not find or trust it.
High acceptance with high error means people accept too easily. They may be over-trusting.
Low acceptance with high quality means the design or expectations are off.
Rising cost with flat value means the economics are slipping.

Counter-metrics

Every metric can be gamed or can hurt something else. Pair each with a guard. If you push for faster replies, watch quality. If you push for fewer escalations, watch unresolved cases. If you push for more usage, watch cost. Guard metrics keep success honest.

Collect the right signals

Log requests, results, user actions, edits and ratings, with privacy rules applied.
Sample conversations for human review every week.
Tag failures by type, so patterns show up.
Collect user comments and support tickets, and link them to cases.
Track cost and speed per task and per user group.

From feedback to roadmap

feedback-loop.md

WEEKLY
- Read 20 sampled conversations with the team
- Update the failure tally: top types and counts
- Check the dashboard: use, quality, trust, value, cost
- Pick one or two fixes; assign owners

MONTHLY
- Review trends and slices (user group, language, task type)
- Update the test set with new real failures
- Decide: fix, retire, expand or stop
- Share a one-page note with leaders: what we learned, what we will do

EVERY CHANGE
- Run the test set and the release gate
- Watch the metrics after release

Slice the data

Averages hide stories. Break results down by user group, task type, language and new versus returning users. A feature may work well for one group and badly for another. The slices show where to invest.

Look at the journey, not only the totals

Totals hide how people behave. Follow a user from first visit to repeat use. Where do they drop off? Which step takes the longest? Which users try the feature once and never return? Watch a few sessions in person if you can. Pair the numbers with stories, and you will see what to fix first.

Know when to stop

Not every feature deserves to live. If use is low after good effort, quality is stuck below the bar or costs outrun value, decide to change course or retire it. Stopping frees the team for better bets. Write the stop rule at launch.

⚠Vanity numbers, such as total prompts sent, show activity and not value. Always tie metrics to a user or business result.

Key takeaways

✓Measure four groups: use, quality, trust and value, and read them together.
✓Pair each metric with a guard so success stays honest.
✓Log, sample and tag failures, and run a weekly review that feeds the roadmap and the test set.
✓Slice results by group, write a stop rule at launch and avoid vanity numbers.

Hands-on exercise

Build a metric set

Take an AI feature that answers customer questions.
Choose two metrics from each of the four groups, and add one guard metric for each pair.
Write what you would conclude if use is high, quality is high and acceptance is very high.
Write a weekly review agenda and a stop rule.

Check yourself

Q1What does a high acceptance rate with high error suggest?show answer

People may be accepting too easily, which means over-trust. Design and communication may need to encourage checking.

Q2Why add guard metrics?show answer

Pushing one metric can hurt another. A guard metric keeps success honest, such as watching quality when pushing for speed.

Q3Why write a stop rule at launch?show answer

It makes it easier to retire a weak feature, and frees the team for better bets.

Up next

Lesson 9: Working With Engineers, Legal & Leaders

← Previous Course overview