What Is an Eval Runner?

An eval runner is the component that executes cases against an agent. It prepares the input, invokes the selected target, captures its output and telemetry, and records one result for each attempt.

A runner is not a grader.

Current Perseval boundary: Perseval creates and reviews eval definitions, but does not ship an eval runner, materialize runnable fixtures, or produce verification reports. This lesson defines the runner that an external system or future integration would need.

Runner: invokes the agent and captures evidence
Grader: judges the captured result

The runner contract

A practical runner receives:

• an eval-run identifier;
• the case and its input fixture;
• the target build and configuration;
• timeout and cancellation settings;
• a trace-correlation context;
• references to allowed secrets or variables.

It returns:

• completion, error, timeout, or cancellation status;
• final output;
• structured artifacts;
• trace identifiers;
• execution metadata;
• sanitized error information.

A local command runner

The smallest general-purpose runner can start a local command and exchange JSON through standard input and output.

A runner might send:

{
  "eval_run_id": "er-0192",
  "case_id": "case-014",
  "input": {
    "message": "I lost my card and need a replacement."
  },
  "trace_endpoint": "http://127.0.0.1:4318/v1/traces"
}

The user’s adapter invokes the agent and returns:

{
  "status": "completed",
  "output": "I could not order the replacement yet.",
  "artifacts": {
    "replacement_ordered": false
  }
}

The agent emits OTLP normally, with eval_run_id and case_id attached to its telemetry. The evaluation system then correlates the response with the finalized trace.

Other runner adapters

The same contract can support:

• an HTTP endpoint;
• a Python or TypeScript test harness;
• a container;
• a continuous-integration job;
• a remote deployment;
• a specialized sandbox.

The evaluation product should not need application-specific agent logic. The adapter owns invocation. The evaluation system owns cases, provenance, results, and grading.

Runner responsibilities

A trustworthy runner should handle:

Isolation

Cases must not accidentally share mutable state unless the suite explicitly tests a session. Isolation can mean a fresh process, a reset fixture, a container, or an application-provided cleanup hook.

Concurrency

Parallel cases improve speed but can create rate limits or cross-case interference. Concurrency belongs in the recorded run configuration.

Timeouts and cancellation

A timed-out agent is not the same as a failed rubric. Execution status and behavioral grade must remain separate.

Secret handling

The runner should receive secret references or an explicit allowlist, not dump the user’s entire environment into every process.

Trace correlation

Every emitted span must be attributable to the correct run, case, and attempt. Missing trace evidence should produce an inconclusive trace-dependent grade.

What a runner should not do silently

A runner should not:

• alter the accepted eval definition;
• select a different target without recording it;
• retry and discard the first attempt;
• treat missing telemetry as success;
• accept an eval or authorize deployment;
• expose secrets in stored results.

Once the runner produces evidence, one or more graders decide what that evidence means.

Until Perseval ships this boundary, accepting a candidate means “this definition is approved for downstream use,” not “this test ran” or “this remediation passed.”