Defining Quality & Writing Rubrics

You cannot measure quality until you say what quality is. This sounds simple. In practice, teams disagree about it for weeks. A rubric ends the argument. It is a short written description of what a good answer must do. This lesson shows how to turn a vague wish into measurable criteria.

Start from the user's job

Ask what the user is trying to get done. A support assistant should resolve the question and be polite. A summarizer should keep the key facts and stay short. A data extractor should fill every field correctly. The job decides what quality means.

Pick the dimensions that matter

• Correct: the facts and the steps are right.
• Complete: nothing important is missing.
• Grounded: claims come from the given sources and can be traced.
• Safe: it avoids harmful, private or off-limits content.
• Format: it follows the required shape, length and tone.
• Helpful: it actually moves the user forward.

Choose three to five that matter for this task. Too many dimensions make scoring slow and unclear.

Write the rubric

A good rubric uses short, concrete statements. Two different reviewers should reach the same score most of the time. Use levels with examples, not just words like good or bad.

TASK: Answer a customer question from our help articles.

CORRECT (0-2)
  2 = Every fact and step matches the article
  1 = Mostly right, one minor error that would not cause harm
  0 = A wrong fact or step that could mislead the customer

GROUNDED (0-2)
  2 = Every claim is supported by a cited article
  1 = Some claims uncited but still supported
  0 = A claim that no source supports

TONE (0-1)
  1 = Polite, clear, no jargon      0 = Rude, confusing or too long

MUST-NEVER-HAPPEN (any one fails the case)
  - Reveals another customer's data
  - Promises a refund or price not in the article
  - Gives legal or medical advice

PASS = no must-never-happen failures AND Correct >= 1 AND Grounded >= 1

Separate rules from judgment

Some criteria are checkable by code. Is the output valid JSON? Is it under 100 words? Does it cite a source? Check these with rules. They are cheap, fast and exact. Other criteria need judgment, such as is the answer helpful. Use people or a model judge for those. Keep the two apart in your reports.

Handle trade-offs

Quality dimensions pull against each other. A longer answer may be more complete and less helpful. A strict safety filter may block good answers. Decide the priorities with the product owner and write them down. For example, safety first, then correctness, then tone.

Test the rubric itself

• Give 20 cases to two reviewers. Compare their scores.
• Where they disagree, the rubric is unclear. Add an example or a sharper rule.
• Repeat until they agree most of the time.
• Keep a short guide with solved examples for new reviewers.