· Perseval · Agent evaluation fundamentals · 3 min read
What Is White-Box Replay for AI Agents?
White-box replay reruns agent logic while controlling recorded model, tool, time, or state dependencies.
White-box replay reruns agent logic while controlling selected internal dependencies. Its purpose is to reproduce a behavior under known conditions, not to recreate production perfectly.
Perseval does not currently materialize replay fixtures or execute white-box replay. This lesson describes a testing technique that can consume a reviewed eval definition outside the current product.
What can be captured
A replay fixture may contain:
- the original user input;
- initial application state;
- retrieved documents;
- tool requests and recorded responses;
- model requests and recorded responses;
- clock values;
- random seeds where supported;
- expected ordering of important events.
The runner replaces live dependencies with fixture-backed versions and records a new execution trace.
What can be replayed
White-box replay does not need to replace everything.
Tool replay
Use recorded tool responses while allowing the model and orchestration to run again. This tests how the agent handles the same external outcome.
Model replay
Return recorded model responses while running current tool adapters and orchestration. This isolates changes outside the model.
Orchestration replay
Fix both model and tool responses to test routing, state transitions, retries, parsing, and verification logic.
Partial replay
Control only the dependency related to the failure and let the rest run normally. Partial replay is often more informative than trying to freeze the entire world.
Why it helps
Suppose the ordering tool returned a rare declined_pending_review status. The original agent treated it as success.
A white-box fixture can preserve that exact response. Every candidate build can then demonstrate how it handles the same condition without depending on the live service producing the rare state again.
The determinism trap
A perfectly repeatable replay can still be a poor predictor of production behavior.
It may hide:
- new model behavior;
- changed retrieval results;
- real latency and concurrency;
- tool schema drift;
- authentication or network failures;
- side effects that the fixture does not model.
Use replay to isolate a hypothesis. Follow it with a gray-box or black-box re-execution when the real integration matters.
Side effects and idempotency
Replaying a trace must not accidentally repeat a payment, card order, email, or destructive tool call.
A replay runner needs explicit policies:
- which tools must be mocked;
- which operations are safe to call;
- whether state is reset before every attempt;
- whether a tool request matches the recorded fixture;
- what happens when no fixture matches.
“Fall back to the live tool” is a dangerous default.
Replay is not checkpoint, fork, and resume
Checkpointing captures a running environment so it can resume from a previous point. White-box replay starts a controlled execution from declared fixtures and inputs.
A sandbox may implement replay using snapshots, but an evaluation system does not need to own that infrastructure. It can call a runner adapter that provides the required isolation.
Provenance for replay results
Record:
- source trace and fixture identity;
- which dependencies were replayed;
- which dependencies remained live;
- fixture schema and adapter versions;
- target build;
- mismatches between expected and actual requests;
- the new trace and grades.
The result should say “passed controlled tool replay,” not simply “passed production eval.”