· Perseval · Agent evaluation fundamentals · 2 min read
Black-Box, Gray-Box, and White-Box Agent Testing
The three modes differ by how much of the agent execution and its dependencies the evaluation system can observe or control.
Black-box, gray-box, and white-box testing describe how much the evaluator can observe or control inside the system under test.
They are not quality levels. Each mode answers different questions.
Perseval currently provides gray-box evidence collection and investigation through OTLP. It does not invoke black-box targets or control dependencies for white-box execution.
Black-box testing
Black-box testing supplies an input and observes the externally visible result.
Input → agent endpoint → outputThe evaluator may know latency and transport status, but it does not depend on internal traces or state.
Black-box tests are useful for:
- hosted agents you cannot instrument;
- user-visible output quality;
- public API compatibility;
- end-to-end availability.
They cannot reliably tell you which agent chose the wrong tool or whether a plausible answer came from a fragile trajectory.
Gray-box testing
Gray-box testing observes selected internals through a stable interface, usually telemetry, while the agent and its dependencies run normally.
Input → agent → output └── OTLP traceThe evaluator can inspect model calls, tools, agent roles, events, and execution topology without owning the agent’s implementation.
Gray-box tests are useful for:
- trajectory policies;
- tool-use verification;
- baseline versus candidate comparison;
- multi-agent handoffs;
- diagnosing why an output failed.
For most agent regression testing, this is the best default. OpenTelemetry provides the observation boundary without requiring a custom testing SDK.
White-box testing
White-box testing can inspect or control implementation-level state and dependencies.
It may substitute tool responses, intercept model calls, control time, inject application state, or call internal functions directly.
White-box tests are useful for:
- deterministic orchestration tests;
- parser and state-machine behavior;
- rare error paths;
- controlled replay;
- fault injection.
They require application-specific integration and may test a less realistic environment than production.
The comparison
| Mode | Sees output | Sees OTLP trace | Controls dependencies | Typical use |
|---|---|---|---|---|
| Black box | Yes | No | No | External behavior |
| Gray box | Yes | Yes | No | Behavior plus trajectory |
| White box | Yes | Usually | Yes | Isolation and controlled replay |
One case in three modes
The ordering tool returns declined.
Black box
Check whether the response falsely promises shipment. You may know the response is wrong but not why.
Gray box
Check the response and confirm through OTLP that order_replacement failed and the verifier was skipped.
White box
Inject the recorded declined response into the candidate orchestration and verify that it follows the failure branch without calling the live tool.
Combine modes instead of choosing one forever
A strong suite often uses all three:
- many fast white-box unit and orchestration tests;
- a representative gray-box regression suite;
- a smaller black-box end-to-end suite against the deployed interface.
The evaluation report should state which mode produced each result. A white-box pass and a black-box pass are both useful, but they do not prove the same thing.