Offline, Online, and Shadow Agent Evals

Agent evaluations can run before deployment, during production traffic, or alongside production without affecting the user. These modes answer different questions and carry different risks.

These are general evaluation modes, not current Perseval execution modes. Perseval can derive and review a definition from collected traces, but it does not run offline suites, launch shadow candidates, or grade live traffic.

Offline evals

An offline eval executes a versioned suite against a selected agent target in a controlled environment.

Versioned cases
→ candidate agent
→ outputs and traces
→ graders
→ release report

Offline evals are useful for:

• regression testing before release;
• comparing models, prompts, or builds;
• reproducing known failures;
• testing rare or unsafe situations;
• running deterministic fixtures repeatedly.

Their weakness is representativeness. A clean dataset may not capture the changing inputs, tool states, and user behavior found in production.

Online evals

An online eval grades behavior produced by real traffic. It may evaluate every eligible run or a sampled subset.

Online evals help answer:

• Are new failure patterns appearing?
• Did behavior change after deployment?
• How often does a criterion fail for real users?
• Are particular projects, sessions, or agent roles affected?

They also introduce harder constraints:

• sensitive production data;
• incomplete or delayed telemetry;
• uncertain ground truth;
• changing traffic distributions;
• cost and latency limits;
• the need to avoid blocking the user request.

An online judgment should not silently become a production action. Detection, review, alerting, and remediation are separate decisions.

Shadow evals

A shadow eval sends a copy of an eligible input to a candidate agent that cannot affect the live user or production state.

Production input
├── active agent → user response
└── shadow candidate → hidden result and trace → comparison

Shadowing tests a candidate on realistic traffic before promotion. It is useful when offline cases are too narrow but routing live users to the candidate would be risky.

The candidate must not execute destructive tools or create external side effects. Tool calls may need a read-only environment, fixture-backed responses, or explicit suppression.

Passive evaluation is different from shadow execution

An online evaluator can grade the trace produced by the active agent without running another agent. Shadow evaluation executes an additional candidate.

Passive online eval: existing production result → grader
Shadow eval: production input → additional candidate execution → grader

The second costs more and introduces another execution surface.

Use the same identities across modes

Every result should preserve:

• definition, case, and suite revision;
• target build and configuration;
• runner and grader versions;
• environment and execution mode;
• source trace or traffic-selection rule;
• whether tools were live, mocked, read-only, or suppressed.

This prevents a shadow result from being compared with an offline result as if their conditions were identical.

A practical progression

For the replacement-card criterion:

1. Offline: Run representative confirmed, declined, and retry cases before release.
2. Shadow: Send selected staging or production-like requests to the candidate with side effects disabled.
3. Online: Monitor the promoted agent for the same failure signature and review sampled evidence.

Each stage adds realism and operational risk. None replaces the others.

The useful rule

Use offline evals to develop and block known regressions, shadow evals to test candidates on realistic traffic safely, and online evals to detect behavior that controlled suites missed.