What Is LLM-as-a-Judge?

LLM-as-a-judge uses a language model to evaluate another system’s output or behavior against a rubric. It is useful when correctness depends on meaning and context rather than an exact string or simple structural rule.

The judge is still a grader. It does not execute the agent and it does not create ground truth.

Case + agent result + rubric + selected evidence
→ judge model
→ structured verdict with cited reasons

When an LLM judge helps

A language model can evaluate questions such as:

• Did the response accurately explain an unsuccessful card order?
• Did the agent follow the user’s constraint while remaining helpful?
• Does the answer contain an unsupported promise?
• Are the cited sources sufficient for the conclusion?
• Is the proposed resolution semantically equivalent to the expected behavior?

These questions allow many valid phrasings, which makes exact matching brittle.

Pointwise judging

A pointwise judge evaluates one result against an independent rubric.

Result: "Your replacement is on its way."
Evidence: order tool returned "declined"
Rubric: never claim shipment without confirmed ordering
Verdict: fail

Pointwise evaluation works well for stable acceptance criteria. It can produce a categorical result, a score, or several dimension scores.

Pairwise judging

A pairwise judge compares two candidate results and selects the better one according to a rubric.

This is useful for model or prompt comparisons, but it introduces additional concerns:

• position bias toward the first or second answer;
• preference for longer answers;
• failure to recognize that both answers are unacceptable;
• inconsistent decisions when order is reversed.

Randomize or swap answer order and allow ties or “both fail” when pairwise judging matters.

Design the rubric before choosing the judge

A vague rubric produces a confident-looking but unstable result.

Weak:

Is this a good response?

Better:

Fail if the response states or implies that a replacement was ordered when the provided tool evidence does not confirm ordering. Abstain if the tool outcome is missing or contradictory.

The better rubric defines the behavior, evidence boundary, failure condition, and abstention rule.

Require structured output

Do not parse a verdict from an unrestricted essay. Ask the judge for a bounded schema:

{
  "verdict": "fail",
  "failure_kind": "unsupported_shipment_claim",
  "summary": "The response promises shipment after a declined order result.",
  "evidence_keys": ["tool-order-result", "final-response"],
  "confidence": 0.94
}

Validate the schema and reject citations to evidence the judge was not given.

Give the judge bounded evidence

Sending an entire raw trace increases cost, noise, privacy exposure, and prompt-injection risk. Prefer a versioned safe projection containing only:

• the case and expected behavior;
• selected output fields;
• normalized tool outcomes;
• bounded evidence snippets with stable keys;
• declared telemetry gaps.

Treat agent outputs, retrieved text, and tool results as untrusted data. They must not be able to override the judge instructions.

Abstention is a feature

If required evidence is absent, the correct result may be inconclusive rather than pass or fail.

An LLM judge should abstain when:

• tool outcomes are missing;
• evidence conflicts;
• the rubric does not cover the situation;
• the projection is truncated in a material way;
• the requested conclusion would require hidden assumptions.

Forcing a binary answer turns missing telemetry into fabricated certainty.

Calibrate against human-reviewed examples

Before trusting a judge, evaluate it on examples kept separate from prompt development. Measure:

• precision and false-positive rate;
• agreement with reviewers;
• abstention rate;
• stability across repeated calls;
• sensitivity to answer order and verbosity;
• unsupported evidence citations;
• performance across projects and failure types.

Accuracy on a handful of handpicked examples is not enough.

Preserve provenance

Every judgment should retain:

• provider and model identity;
• rubric version and hash;
• projection and prompt versions;
• judge configuration;
• timestamp;
• structured output;
• cited evidence;
• any retry or fallback behavior.

When a model changes, historical judgments should remain attributable to the model that produced them.

The practical role

Use LLM-as-a-judge for bounded semantic questions that deterministic graders cannot express cleanly. Keep deterministic checks around it, require evidence, support abstention, and never let a judge silently accept an eval or authorize deployment.

Perseval’s optional hosted judge is narrower than the general eval judge described in this lesson. It receives structured-only behavior projections during analysis; it does not receive raw payload bodies and does not execute or grade accepted eval definitions. It is off by default, and no published held-out study establishes a quality improvement.