Deterministic Graders vs LLM Judges

A grader evaluates an agent result. The best grader is not always the most sophisticated one. It is the one that tests the intended behavior with the least unnecessary ambiguity.

Deterministic graders

A deterministic grader produces the same result for the same normalized input and versioned configuration.

Common examples include:

• exact or normalized string comparison;
• JSON schema validation;
• numeric thresholds;
• required or forbidden tool calls;
• tool-result assertions;
• trace topology constraints;
• latency and retry limits.

For the replacement-card failure, a deterministic policy could state:

If tool.name == "order_replacement"
and tool.result.status != "confirmed"
then response must not claim that shipment is scheduled.

The tool part is precise. Determining whether free-form text “claims shipment” may still require structured output or a semantic classifier.

LLM judges

An LLM judge receives a bounded projection of the case, output, rubric, and evidence. It returns a structured judgment such as pass, fail, or abstain with cited evidence.

LLM judges help when:

• many phrasings can express the same behavior;
• correctness depends on context;
• a rubric is easier to state in language than code;
• partial credit or qualitative dimensions matter.

They introduce model variability, latency, cost, provider dependency, and the risk of unsupported explanations.

Use a ladder, not a rivalry

Start with the cheapest reliable layer:

1. Validate execution and telemetry completeness.
2. Apply deterministic structural checks.
3. Apply exact output or artifact checks.
4. Use an LLM judge only for the remaining semantic question.
5. Require human review for high-impact or uncertain decisions.

This reduces the amount of sensitive context sent to a model and makes failures easier to explain.

Evidence-bound judging

An LLM judge should not receive an unlimited raw trace and improvise a diagnosis. A safer input contains:

• the rubric and its version;
• a structured case summary;
• selected output fields;
• bounded evidence items with stable keys;
• declared telemetry gaps.

The response should cite only those keys. If the evidence is missing or contradictory, the judge should abstain.

Provenance is part of the result

Store:

• provider and model;
• rubric version and hash;
• projection version;
• grader prompt or policy version;
• timestamp;
• structured output;
• evidence references.

Without provenance, a model upgrade can change historical conclusions without explanation.

Agreement does not prove truth

Two graders agreeing can still be wrong. Calibration requires labeled examples that were not used to tune the grader.

Useful quality measures include:

• precision on actionable failures;
• false-positive rate;
• abstention rate;
• agreement with human review;
• stability across model versions;
• unresolved evidence citations.

The practical rule

Use deterministic graders wherever the expected behavior can be stated precisely. Add an LLM judge for bounded semantic interpretation, with visible provenance and an abstain path.

In Perseval today, deterministic findings and exact groups remain canonical. The optional hosted semantic judge receives structured-only behavior facts, is disabled by default, and cannot accept a finding or eval definition. No held-out study has shown that it improves finding quality, so its output is assistance to review, not a stronger class of truth.

The next lessons examine LLM-as-a-judge in detail and explain how ground truth and calibration determine whether either kind of grader deserves trust.