Replay, Re-execution, and Regrading Are Not the Same

“Replay the eval” can mean at least three different operations. They have different costs and answer different questions.

Perseval does not currently perform any of these operations. It can inspect and compare traces that already exist and can create a reviewed eval definition. The operations below describe an external runner or future execution system.

Regrading

Regrading applies a new or updated grader to an existing result. The agent does not run again.

Existing input + existing output + existing trace
→ new grader
→ new grade

Use regrading to:

• test a revised rubric;
• compare two judge models;
• add a deterministic policy to historical runs;
• correct a grader bug;
• measure how a proposed criterion would classify past evidence.

Regrading is fast and reproducible, but it cannot show how the current agent behaves. It only changes the interpretation of recorded evidence.

Re-execution

Re-execution sends the same case through an agent target again.

Same case
→ current agent target
→ new output + new trace
→ grader

Use it to answer:

• Does the candidate build still reproduce the failure?
• Did the fix change the trajectory?
• Does the suite pass with the current model and tools?
• Did latency or tool usage regress?

Re-execution is realistic, but not necessarily deterministic. Models, retrieval results, tools, time, and external state can all change.

Controlled replay

Controlled replay re-executes some part of the system while fixing selected dependencies to recorded or synthetic values.

Same case
+ recorded tool responses
+ fixed model outputs or seed where possible
+ controlled initial state
→ new orchestration execution

Use it to isolate a layer:

• test a new verifier against the same planner and tool evidence;
• test orchestration changes with fixed model responses;
• reproduce a race or retry sequence;
• confirm that a parser now handles the recorded tool result.

Controlled replay improves reproducibility but reduces realism. A passing replay does not prove that current production dependencies will behave the same way.

Trace playback is different again

A UI may “replay” a trace by animating already recorded spans in timestamp order. That is visualization, not agent execution.

It can help explain timing and concurrency, but it produces no new agent result.

A replacement-card example

The original trace contains a declined order result followed by a response that promises shipment.

• Regrade: Apply a new claim-detection rubric to the existing response.
• Re-execute: Send the case to the candidate agent and call the real ordering test service.
• Controlled replay: Feed the recorded declined tool response to the candidate orchestration and observe whether it still promises shipment.
• Trace playback: Animate the original spans in the viewer.

These operations should have different buttons, result types, and provenance.

Which one should you use?

Precise language prevents a team from claiming that a fix passed “replay” when it only passed a new grader over old evidence.