Eval Definitions, Cases, Suites, Runs, and Results

People often use “eval” to mean the test idea, the dataset, the execution, and the final score. That ambiguity makes systems difficult to reproduce.

It is more useful to separate five records.

Perseval currently creates and reviews the first record: an eval definition candidate. Cases, suites, runs, and results below describe the downstream evaluation system, not shipped Perseval execution.

1. Eval definition

The definition describes the behavior you want to test:

Name: Do not claim an unconfirmed replacement
Expected behavior: Shipment is promised only after ordering succeeds
Grader: replacement_confirmation_policy v1
Source: recurring failure group fg-018

The definition can exist before there is a runnable input. It may begin as a candidate generated from trace evidence and become accepted only after human review.

2. Eval case

A case is one concrete example of the definition:

{
  "customer_message": "I lost my card. Please send a replacement.",
  "account_state": "identity_verified",
  "order_tool_result": "declined"
}

One definition should usually have several cases. They can vary phrasing, account state, tool outcomes, and other meaningful conditions.

3. Eval suite

A suite is a versioned collection of cases and definitions selected for a purpose.

Examples include:

• card-support regressions;
• release-blocking safety checks;
• tool-recovery behavior;
• representative production failures from the last month.

Changing suite membership should create a new revision rather than rewriting history.

4. Eval run

A run executes one immutable suite revision against one explicit target.

Suite: card-support-regressions v3
Target: support-agent build c091bd
Model: selected-model-version
Environment: local-eval
Started: 2026-07-16 09:00 UTC

The run also records concurrency, timeouts, runner version, and any allowed environment configuration.

5. Eval result

A result belongs to one case attempt. It contains:

• output and structured artifacts;
• execution status;
• the correlated trace;
• grader results;
• evidence references;
• timing and provenance.

Retries should create additional attempts. They should not overwrite the first result.

Candidate is another important state

A candidate is a proposed definition that has not yet been accepted. It can carry:

• representative source findings;
• proposed input;
• expected behavior;
• rubric and grader;
• evidence packet;
• telemetry gaps;
• generator provenance.

Accepting a candidate approves the definition. It does not automatically execute it, activate a grader in production, or authorize a deployment.

Why immutability matters

Suppose the same suite reports 92% on Monday and 97% on Friday. That comparison is meaningless if someone silently edited the cases or rubric on Wednesday.

Reproducible evaluation needs immutable identities for:

definition revision
× case revision
× suite revision
× target build
× grader version
× runner version

You do not need to show that multiplication in every interface. The system must preserve it underneath.

A useful rule

When someone says “the eval passed,” ask:

Which suite revision ran against which target, using which graders, and where are the case-level results?

The next lesson explains the component responsible for producing those results: the eval runner.