From Production Traces to Regression Evals

The final answer is often the least informative part of an agent run. It can look plausible even when the trajectory contains a bad tool choice, an ignored constraint, or a recovery that only worked by accident.

A useful evaluation loop starts earlier. It begins with traces from real runs, groups recurring failure patterns, and lets a reviewer confirm the exact evidence before creating an eval definition.

The loop

1. Collect traces with explicit project, environment, build, and session identity.
2. Detect recurring failures across runs, not just isolated errors.
3. Inspect representative examples and the spans that support each diagnosis.
4. Accept, reject, or defer the finding.
5. Turn accepted patterns into reviewable eval definitions.

The important boundary is human review. A generated diagnosis is a lead, not ground truth. The reviewer should be able to see expected behavior, observed behavior, evidence, provenance, and counterexamples before accepting it.

Perseval is designed around this loop. It currently creates and reviews eval definitions. Executing accepted evals against an agent is a later milestone.

Why production evidence matters

Handwritten eval suites cover anticipated failures. Production traces reveal what actually breaks: tool loops, role confusion, missing context, malformed arguments, weak verification, and recoveries that hide the original defect.

When those patterns remain attached to their evidence, the path from incident to regression test stays auditable.