· Perseval · Product principles  · 2 min read

Exact Groups First, Optional AI Second

Keep failure identity deterministic, use similarity only for navigation, and treat hosted judgments as unproven assistance.

Three features that look like “grouping” prove different things:

  • an exact failure group says findings share one deterministic signature;
  • a feature-similarity cohort says findings are near one another in a bounded feature projection;
  • a hosted judge applies a versioned rubric to structured behavior facts.

Only the first establishes canonical failure identity in Perseval. Similarity and judging cannot merge exact groups, change their evidence, rewrite review decisions, or alter an eval candidate ID.

What runs locally

Exact deterministic analysis always runs first. Optional feature similarity projects safe finding features and runs seeded K-means locally. It is disabled by default, bounded, and useful only as a navigation aid.

A short distance is not a probability that two failures share a root cause. A cohort label is not a diagnosis.

What hosted features receive

Hosted features are separately opt-in. OpenAI embeddings receive a versioned safe finding projection. Optional labels receive bounded representative safe cases. The optional semantic judge receives structured-only behavior facts.

Raw payload blobs, prompts, messages, reasoning, source code, tool payloads, and full inputs and outputs stay outside those projections. A configured API key enables nothing by itself.

What is not proven

The shipped safeguards establish bounds, provenance, isolation, and deterministic fallback. They do not establish that hosted embeddings, labels, or judgments improve finding quality.

No held-out quality study has been published. Treat hosted output as reviewable assistance, not ground truth. If the provider fails, deterministic groups remain available.

That boundary keeps the investigation reproducible without pretending that optional AI has earned more authority than the evidence supports. See Feature similarity and hosted analysis for the current product contract.

Back to Blog

Related Posts

View All Posts »

Why One Failed Trace Is Not an Eval

A failed trace is evidence from one execution. An eval requires a representative case, expected behavior, and a repeatable way to judge new runs.

How to Compare Two Agent Runs

A useful comparison preserves both run identities, aligns equivalent work, and highlights the first meaningful divergence.