Agent ops & evalsFree→ Claude / ChatGPT
LLM-as-Judge Rubric Writer
A calibrated rubric with anchors so judge scores are consistent.
What you get: A criterion-by-criterion rubric with scoring anchors and a judge prompt.
Write an LLM-as-judge rubric and judge prompt to grade outputs of this agent task: {{task}}. What a great output achieves: {{great_output}}. Common failure patterns: {{failure_patterns}}.
Provide: 4–6 independent criteria; for each a precise definition and scoring anchors (what 0, 1 and 2 look like with a short example of each); instructions to give reasoning BEFORE the score; instructions to use evidence from the supplied trace and tool results, not the plausibility of the final text; guidance to avoid length and position bias; a JSON output schema; and a calibration plan — how many human-labelled examples I need, how to measure agreement, and what agreement threshold means the judge is usable. Include the full judge prompt ready to paste.