Why One Failed Trace Is Not an Eval

A failed trace is valuable evidence. Copying it directly into an eval suite can produce a brittle test that remembers one accident instead of protecting the underlying behavior.

A trace describes one execution

The original replacement-card trace includes:

• one customer’s wording;
• one account state;
• one ordering-tool response;
• one model version;
• one sequence of retries;
• one final response.

Some of those details caused the failure. Others were incidental.

An eval must decide which is which.

Overfitting to wording

Suppose the bad response was:

Your new card is on its way.

An exact-match grader that forbids only this sentence will miss:

The replacement has been dispatched.

The durable criterion is not a sentence. It is the claim that ordering succeeded when the tool did not confirm success.

Overfitting to topology

The original trace may contain two planner spans because of a retry. Requiring every future run to have exactly the same span sequence can reject valid improvements.

Test meaningful behavior:

• ordering must be confirmed before promising shipment;
• a declined result must follow a failure or escalation path;
• verification must occur before the final claim.

Allow implementation details to change when they preserve those requirements.

One example cannot show the boundary

You need positive and negative examples:

• confirmed order and accurate success response;
• declined order and honest failure response;
• transient tool error followed by confirmed retry;
• missing tool telemetry;
• ambiguous status requiring abstention or escalation.

These examples reveal whether the grader distinguishes acceptable behavior from the failure rather than simply recognizing one recorded trace.

Recurrence improves confidence, not truth

Seeing the same signature in ten runs helps establish impact and provides representative cases. It still does not prove the diagnosis is correct.

A shared signature can reflect:

• one root cause;
• several causes with the same visible symptom;
• an instrumentation defect;
• expected behavior incorrectly classified as failure.

Review evidence and counterexamples before turning recurrence into a regression rule.

Perseval’s exact groups preserve this distinction. An exact signature establishes deterministic membership, not ground truth. Optional feature-similarity cohorts are even weaker: they are navigation hints and cannot merge groups or change their evidence.

Preserve the source without copying it blindly

A good eval definition links back to the original trace and evidence packet. That provenance explains why the test exists.

The runnable case should contain only the minimum state needed to reproduce the relevant condition. Sensitive or irrelevant production data should be removed.

A useful transformation

Recorded incident
"This exact response was wrong in this exact run"

→ reviewed behavioral criterion
"Do not claim replacement shipment without confirmed ordering"

→ representative cases
confirmed, declined, retried, missing evidence

→ versioned graders
structural tool check + bounded semantic claim check

The trace remains evidence. The eval becomes a durable test of the behavior the evidence exposed.