Generative & Agentic AI
Agent Evaluation and Reliable AI Systems
G03 · Specialist
Entry requirements
G02 and O01 or equivalent deployment skills
What 48 hours means
- Lessons
- 36 h
- Capstone
- 8 h
- Assessment
- 2 h
- Remediation
- 2 h
Syllabus
Four modules, twelve lessons. Every lesson states what it covers and the evidence you produce for it.
Module 1
Test design
Evaluation contracts · Golden datasets · Trajectory analysis
G03Show lessonsHide lessons
G03.01 · 180 min
Evaluation contracts
Translate product promises into measurable criteria.
Evidence: A matrix maps every important promise to an observable test.
G03.02 · 180 min
Golden datasets
Build representative, adversarial and withheld scenario sets.
Evidence: Dataset provenance and leakage controls are documented.
G03.03 · 180 min
Trajectory analysis
Inspect intermediate tool calls and state transitions.
Module 2
Judging quality
Deterministic checks · Model-based judges · Nondeterminism
G03Show lessonsHide lessons
G03.04 · 180 min
Deterministic checks
Validate schemas, citations, permissions and business invariants.
Evidence:
Module 3
Reliability
Fault injection · Security evaluations · Cost and latency budgets
G03Show lessonsHide lessons
G03.07 · 180 min
Fault injection
Simulate tool timeouts, stale results and malformed responses.
Evidence:
Module 4
Release governance
Regression gates · Production feedback · Release defence
G03Show lessonsHide lessons
G03.10 · 180 min
Regression gates
Compare candidate and baseline on a fixed evaluation set.
Evidence:
Capstone
Create an evaluation and release gate for an agent service with adversarial scenarios, reproducible traces, human calibration and rollback evidence.
Lab environment
AI API sandbox and developer sandbox
Lab infrastructure is not yet provisioned, so this describes the environment the course is designed for rather than one you can launch today.