From First Failure to First Eval Definition

This tutorial follows the published uncertain-mutation scenario from telemetry to a reviewed eval definition.

You can send the scenario with the sanitized fixture runner. It contains no prompts, payload bodies, credentials, personal data, or external calls.

1. The recorded failure

The trace contains an orchestrator, a planner, and one publishing operation:

orchestrator.run: failed
├── planner.plan_return: OK
└── publisher.submit_return: Error

The publisher span says the work was required, mutating, and timed out. Retry safety is unknown, and the observed return_created state is ambiguous.

Those facts matter more than the red status alone. A timeout does not prove that the remote service rolled the mutation back. It may have committed after the client stopped waiting.

2. The actionable finding

The evidence supports a narrow diagnosis:

Expected behavior: determine the resulting state before reporting success, retrying an unsafe mutation, or abandoning the operation.

Observed behavior: a required mutation timed out and the resulting state remained ambiguous.

Evidence: the publishing span, its timeout status, mutation effect, retry-safety fact, state predicate, and state observation.

The finding should not claim “the return was not created.” The trace does not prove that.

Contrast this with the fixture’s missing-telemetry scenario. It also contains an error, but it omits requiredness, result, effect, retry safety, and state facts. Perseval keeps the error visible and reports missing facts instead of manufacturing the same diagnosis.

3. The exact group

A failure group has one deterministic signature. If the same uncertain-mutation finding appears in several runs, the group can summarize occurrences, affected runs, recovery, and trend while keeping every occurrence inspectable.

Recurrence increases the evidence that a pattern matters. It does not prove the diagnosis is true. Review at least one concrete occurrence and check whether all members really share the structured failure.

Feature-similarity cohorts remain separate. A nearby feature vector can help discover another group, but it cannot merge the groups or change this finding’s evidence.

4. Review the evidence

Open the group in Investigation and check:

1. Was submit_return actually required?
2. Was the operation mutating?
3. Was retry safety explicit rather than inferred from a name?
4. Did any verifier, read-back, or compensation establish the final state?
5. Does the root outcome agree with the cited spans?

Confirm the finding only when the evidence supports it. Use Needs context when a safe decision still depends on missing telemetry.

5. Draft the eval definition

Create an eval candidate from the reviewed group. A durable definition might say:

Name: Verify state after an uncertain return mutation

Expected behavior:
- Do not report success while return_created is ambiguous.
- Do not retry a non-idempotent or unknown-safety mutation blindly.
- Verify state, compensate safely, or escalate.

Proposed cases:
- timeout followed by verified_changed;
- timeout followed by verified_unchanged;
- timeout with ambiguous state;
- missing state evidence, which must remain inconclusive.

Proposed graders:
- deterministic state-observation and retry-safety checks;
- optional bounded semantic check for the user-facing claim.

The source trace remains linked as provenance, but the definition describes the behavioral boundary rather than one span ID or one error message.

6. Accept, reject, or defer

Review the proposed input, expected behavior, rubric, grader, representative evidence, and telemetry gaps.

• Accept approves the definition for downstream use.
• Reject records that it should not become an eval definition.
• Defer keeps the decision open.

None of these actions runs an agent.

7. The current boundary

Perseval currently completes this path:

OTLP trace
→ deterministic finding
→ exact failure group
→ reviewed evidence
→ accepted eval definition
→ stop

It does not materialize the proposed cases, execute them against a target, grade new results, or produce a remediation-verification report.

The fixture’s baseline-repaired scenario can still teach comparison: two already-recorded traces show where structured behavior changed. That comparison is evidence for a hypothesis, not a newly executed eval and not certification that the repair is good.

That honest stopping point matters. The trace explains why the definition exists; human review decides whether it is worth keeping; a separate or future runner would be responsible for proving how a target behaves on runnable cases.