AI Evaluation Engineer
Prove whether an AI system is good enough, and keep checking after launch.
Define quality with rubrics, build test sets, use code checks and model judges, evaluate RAG and agents, analyze errors, set release gates and monitor live quality. Ends with interview practice.
10 lessons · about 3 hours
Certificate of completion
Earn your AI Evaluation Engineer certificate
Verified edition
Mark every lesson in this course complete. You have finished 0 of 10. 10 to go.
See a sample certificate. Anyone can check a certificate with its code at /verify. It records course completion. It is not an exam or an accredited qualification.
Write the policy a deterministic guardrail layer will enforce.
Plan the eval gate, canary and rollback for a prompt or model change.
Interview-style review of an agent against the eight ops pillars.
Find the biggest savings in an agent's usage without hurting quality.
A blameless postmortem with timeline, root cause and action items.
Generate attack cases for your agent and the checks that detect success.
A calibrated rubric with anchors so judge scores are consistent.
Create diverse test cases including edge, adversarial and policy-sensitive inputs.
Find the first wrong step in a long agent trace and classify the root cause.
Pick the few service-level indicators and targets that matter for your agent.