Free learning series

Agent Evaluation Fundamentals

Learn how to observe an agent, turn reviewed failures into eval definitions, and understand what a separate runner would need to test next. Perseval itself stops before execution.

18 lessons Concrete examples No prior observability experience required
Start the series →

Five scenarios, one evidence boundary

The lessons use concrete failure shapes, including the published sanitized timeout, optional-recovery, SpanLink-handoff, missing-telemetry, and baseline/repaired scenarios.

Agent run
→ OpenTelemetry trace
→ evidence-backed finding
→ reviewed eval definition
│ Perseval stops here
→ external or future runner
→ new result and grade

Chapter 1

Observe the agent

Learn what traces record, how observability differs from logging, and where OpenTelemetry fits.

  1. 01 What Is an AI Agent Trace? A trace records how an agent reached an answer, including model calls, tool use, handoffs, retries, and verification.
  2. 02 What Is Agent Observability? Agent observability connects individual executions to recurring behavior, evidence, and changes across builds.
  3. 03 What Is OpenTelemetry for AI Agents? OpenTelemetry provides a vendor-neutral way to create, transport, and interpret telemetry from agent runs.
  4. 04 Logs, Traces, and Evals: What Is the Difference? Logs record events, traces connect an execution, and evals test behavior against an explicit expectation.

Chapter 2

Define useful evals

Separate cases, suites, runners, graders, and results so each part has one clear job.

  1. 05 What Is an Agent Eval? An agent eval is a repeatable test that combines an input, expected behavior, execution evidence, and a grader.
  2. 06 Offline, Online, and Shadow Agent Evals Offline evals test controlled cases, online evals judge production behavior, and shadow evals observe a candidate without affecting users.
  3. 07 Eval Definitions, Cases, Suites, Runs, and Results A precise vocabulary makes agent evaluation reproducible and prevents a test definition from being confused with its execution.
  4. 08 What Is an Eval Runner? An eval runner invokes the agent for each case, captures the result and trace, and hands that evidence to graders.
  5. 09 Deterministic Graders vs LLM Judges Use deterministic checks for facts you can state precisely, and LLM judges for bounded semantic questions that preserve evidence and uncertainty.
  6. 10 What Is LLM-as-a-Judge? LLM-as-a-judge uses a language model to evaluate an agent result against a rubric, with evidence, uncertainty, and provenance kept explicit.
  7. 11 Ground Truth and Calibration for Agent Evals Ground truth records the best supported label for a case, while calibration measures whether graders and confidence estimates deserve trust.

Chapter 3

Rerun and replay

Understand regrading, re-execution, black-box testing, gray-box runs, and controlled replay.

  1. 12 Replay, Re-execution, and Regrading Are Not the Same Regrading reuses evidence, re-execution invokes the agent again, and controlled replay fixes selected dependencies.
  2. 13 Black-Box, Gray-Box, and White-Box Agent Testing The three modes differ by how much of the agent execution and its dependencies the evaluation system can observe or control.
  3. 14 What Is White-Box Replay for AI Agents? White-box replay reruns agent logic while controlling recorded model, tool, time, or state dependencies.
  4. 15 How to Compare Two Agent Runs A useful comparison preserves both run identities, aligns equivalent work, and highlights the first meaningful divergence.

Chapter 4

Define the next regression test

Turn real failures into representative, reviewable definitions without pretending they already ran.

  1. 16 How to Turn Production Agent Failures Into Regression Evals A practical workflow for moving from recurring trace evidence to a reviewed eval definition, with the execution boundary kept explicit.
  2. 17 Why One Failed Trace Is Not an Eval A failed trace is evidence from one execution. An eval requires a representative case, expected behavior, and a repeatable way to judge new runs.
  3. 18 From First Failure to First Eval Definition Follow a sanitized mutating timeout from trace evidence through human review to an accepted eval definition—the point where Perseval stops today.