How to Compare Two Agent Runs

Comparing two agent runs is not the same as diffing two log files. Agent executions can contain retries, optional steps, asynchronous work, and different span identifiers even when their behavior is equivalent.

A useful comparison asks where behavior first changed in a way that matters.

Preserve both identities

Every comparison should name:

• baseline and candidate trace IDs;
• revisions;
• project and environment;
• build, model, prompt, and tool versions;
• input or case identity.

Without those identities, the result cannot be reproduced and unrelated runs may appear comparable when they are not.

In Perseval, select exactly two finalized revisions from one project: baseline first, candidate second. Selection order defines the orientation.

Normalize before aligning

Raw span names and IDs are often unstable. Alignment works better with normalized execution features:

• agent role;
• operation category;
• tool or model identity;
• parent role;
• status;
• selected safe attributes;
• structural position.

Normalization should preserve meaningful differences rather than flattening everything into generic “LLM” and “tool” nodes.

Find the common prefix

The common prefix is the sequence of equivalent work before the runs diverge.

Baseline                     Candidate
planner.plan_release: OK     planner.plan_release: OK
run_release_tests: Error     run_release_tests: OK
verify_release: Error        verify_release: OK

This is the shape in the public baseline-repaired fixture. The first behavioral change is the test runner status. The root outcome also changes, but it summarizes the run; treating it as the cause would hide the useful divergence.

The later verifier change may be a consequence of the test result. Compare locates the earliest changed structured step after the equivalent prefix. A reviewer still has to test the causal story.

Not every difference is meaningful

Ignore or de-emphasize differences caused only by:

• new span identifiers;
• tiny timestamp changes;
• harmless wording variations;
• reordered independent spans;
• instrumentation version noise.

Preserve differences that affect:

• tool choice or arguments;
• status and recovery;
• agent ownership;
• verification;
• externally visible behavior;
• safety or policy constraints.

Bound the output

Large traces can contain tens of thousands of spans. A comparison UI should not render every aligned row immediately.

Return a bounded window around the first divergence while retaining counts for unmatched or truncated work. Allow the reviewer to request more context deliberately.

Compare more than shape

Structural alignment is one layer. A complete regression report may also compare:

• case-level grades;
• failure signatures;
• latency and token usage;
• telemetry completeness;
• recovered and newly failing cases;
• tool side effects;
• output artifacts.

No divergence is a result

If two normalized executions are equivalent, say so explicitly. Do not manufacture a difference because timestamps or IDs changed.

Likewise, a structural difference is not automatically a regression. A candidate may take a new path that is safer and more efficient. The comparison locates the change; the eval determines whether the resulting behavior is acceptable.

Perseval compares traces that already exist. It does not execute the candidate, rerun an accepted eval definition, or certify remediation.