Agent evaluation fundamentals

Offline evals test controlled cases, online evals judge production behavior, and shadow evals observe a candidate without affecting users.

An agent eval is a repeatable test that combines an input, expected behavior, execution evidence, and a grader.

Logs record events, traces connect an execution, and evals test behavior against an explicit expectation.

OpenTelemetry provides a vendor-neutral way to create, transport, and interpret telemetry from agent runs.

Agent observability connects individual executions to recurring behavior, evidence, and changes across builds.

A trace records how an agent reached an answer, including model calls, tool use, handoffs, retries, and verification.