How to Turn Production Agent Failures Into Regression Evals

Production traces contain cases your team did not think to write. Turning them into evals creates a feedback loop between real failures and future releases.

The goal is not to convert every error span into a test. It is to identify meaningful recurring behavior, verify the evidence, and create a representative definition that can survive beyond one incident.

Step 1: collect trustworthy traces

Capture enough context to distinguish executions:

• project and environment;
• build and agent version;
• session identity;
• planner, tool, model, and verifier roles;
• events, links, status, and timestamps;
• bounded references to inputs and outputs.

Missing context should remain visible as a telemetry gap.

Step 2: detect a concrete failure

A finding should describe an observable contradiction or policy violation.

Weak:

Card workflow failed.

Better:

The response promised a replacement shipment although order_replacement returned declined.

The second diagnosis identifies expected behavior, observed behavior, and supporting evidence.

Step 3: group exact recurrence

Group findings that share the same normalized failure signature. Keep semantic similarity separate from exact recurrence.

This lets you answer:

• how many times the failure occurred;
• how many runs, sessions, and builds were affected;
• whether it is increasing or recovering;
• which examples are representative;
• whether a superficially similar problem has a different cause.

Step 4: review representative evidence

Inspect more than the first occurrence. Include:

• a canonical example;
• the newest unrecovered example;
• materially different builds or sessions;
• useful counterexamples;
• examples with telemetry gaps.

Accept, reject, or defer the diagnosis before generating an eval. Automated findings are leads, not ground truth.

Step 5: draft the eval definition

Generate a candidate containing:

• representative source cases;
• proposed expected behavior;
• a rubric;
• a proposed grader;
• evidence packet and provenance;
• telemetry gaps;
• a redacted runnable input when one can be derived safely.

The source trace should remain linked, but the definition should describe the durable behavior rather than one incidental span ID.

Step 6: review for overfitting

Ask:

• Does the expected behavior allow multiple valid responses?
• Is the case testing the failure or merely copying the original wording?
• Does the grader rely on stable facts?
• Are success, failure, and inconclusive states distinguishable?
• Are sensitive inputs redacted?

Only a human-reviewed candidate should become an accepted definition.

Step 7: materialize and execute cases

An accepted definition still needs runnable fixtures. Add variations that cover meaningful states, then execute the suite against an explicit agent target.

This step is not shipped in Perseval today. Perseval stops after definition review. A separate or future runner must materialize the case, invoke the agent, correlate a new trace, and retain attempt-level results.

Prefer gray-box execution when possible:

Accepted cases
→ eval runner
→ candidate agent
→ output + OTLP trace
→ graders

Step 8: verify the remediation

Compare baseline and candidate results:

• Did the original failure recover?
• Did representative cases pass?
• Where did the trajectories diverge?
• Were new failures introduced?
• Is required telemetry still present?

A recovered source case is encouraging. A regression decision should consider the declared suite and every required gate.

This verification is also outside Perseval’s current product boundary. Compare can inspect two already-recorded runs, but it does not run the candidate or certify a fix.

Keep the chain of evidence

The durable path is:

trace
→ finding
→ failure group
→ evidence packet
→ accepted definition
│  Perseval stops here today
→ future case and suite revision
→ external or future eval run
→ result and verification report

That chain turns a production incident into a test you can understand, review, rerun, and revise without losing its origin.