Agent evaluation fundamentals

The three modes differ by how much of the agent execution and its dependencies the evaluation system can observe or control.

Regrading reuses evidence, re-execution invokes the agent again, and controlled replay fixes selected dependencies.

LLM-as-a-judge uses a language model to evaluate an agent result against a rubric, with evidence, uncertainty, and provenance kept explicit.

Use deterministic checks for facts you can state precisely, and LLM judges for bounded semantic questions that preserve evidence and uncertainty.

An eval runner invokes the agent for each case, captures the result and trace, and hands that evidence to graders.

A precise vocabulary makes agent evaluation reproducible and prevents a test definition from being confused with its execution.