Calibrating LLM-as-a-Judge Against Experts

The most quoted result about LLM judges comes from the MT-Bench paper. On pairwise comparisons with ties excluded, GPT-4 agreed with human judges 85% of the time, more often than the humans agreed with each other (81%). It's a real result, and it gets stretched. It measured general chat quality on a public question set. It says nothing about whether a judge can tell a correct prior-authorization denial letter from a plausible wrong one under your medical policy.

A judge is a measuring instrument. Before anyone trusts an instrument's readings, it is calibrated against a reference, and it is calibrated again whenever something changes. For an LLM judge, the reference is your experts.

Know the biases before you measure

Three biases are documented well enough that a judge design should assume them.

Position. In the MT-Bench study, GPT-4 gave the same verdict after the two answers were swapped in only 65.0% of cases, and Claude-v1 in 23.8%. In a separate study, reordering responses alone let Vicuna-13B beat ChatGPT on 66 of 80 queries, with ChatGPT as the evaluator.

Length. Padding an answer with a repetitive list fooled Claude-v1 and GPT-3.5 judges 91.3% of the time. GPT-4 resisted far better, at 8.7%, which is the more useful lesson: susceptibility varies by model, so measure yours.

Self-preference. The same paper raises self-enhancement bias, the possibility that a judge favors answers it generated itself. Its data were too limited to settle the question, which is a reason to test for it, not to assume it away. If your agent and your judge run on the same model, compare its verdicts against a second judge on a sample.

The design responses are well known. Run pairwise comparisons in both orders and count only consistent verdicts. Prefer pointwise, binary criteria over "which is better." Give the judge the reference answer and the relevant policy excerpt, because it does not know your policy. Ask for the quoted evidence before the verdict. And send arithmetic, dates and schema checks to code, not to a judge.

Build a calibration set your experts own

Draw 200 items of real agent or model output, stratified by task type and including known failures. A set that's 95% good outputs can't measure how well the judge catches bad ones.
Label per criterion, not overall. "Cites the policy section that actually governs the denial: yes or no" calibrates. A 1-to-10 quality score does not.
Two experts label every item independently, using the same rubric the judge will use. Adjudicate disagreements and record expert-to-expert agreement. That is your ceiling. A judge can't be held to a standard your experts don't meet with each other.
Split 120 for development and 80 held out. Iterate on the judge prompt using the development items only. Touch the held-out items once per candidate judge.

Each record keeps everything needed to rerun the comparison later:

{
  "item_id": "cal-0147",
  "split": "heldout",
  "task": "denial-letter-review",
  "criterion": "cites_governing_policy_section",
  "output_sha256": "9c1e4f0a",
  "expert_labels": [{"rater": "rn-04", "label": "fail"}, {"rater": "md-02", "label": "fail"}],
  "adjudicated": "fail",
  "judge": {"model": "judge-model-2026-01", "prompt_version": "j-7", "order": "single", "label": "pass", "evidence": "per section 4.2"},
  "note": "Letter cites 4.2 (imaging) but the denial reason falls under 6.1 (step therapy)."
}

Measure agreement the right way

Percent agreement misleads when most outputs are good. Suppose experts pass 180 of 200 held-out outputs. A judge that passes everything agrees 90% of the time and has learned nothing. Here is a more realistic judge:

	Expert: pass	Expert: fail
Judge: pass	172	9
Judge: fail	8	11

Raw agreement is 91.5%, which sounds excellent. Cohen's kappa, which corrects for the agreement you'd expect by chance given those base rates, is about 0.52. The number that matters most is in the top-right cell: the judge passed 9 of the 20 outputs the experts failed. Its false-pass rate is 45%. Nearly half the bad outputs would sail through.

Report these four numbers for every judge, overall and per stratum:

Kappa against the adjudicated label (Krippendorff's alpha if you compare several raters).
False-pass rate: judge passes, experts fail.
False-fail rate: judge fails, experts pass.
Position consistency, for pairwise judges.

A workable acceptance rule: on the held-out split, judge-to-expert kappa within about 0.05 of expert-to-expert kappa, and a false-pass rate below a threshold set by what a missed failure costs. For compliance and safety criteria, that threshold is often low enough that the honest answer is a code check or a human reviewer, not a judge.

Some criteria are graded on a scale, such as a three-level completeness rating. For those, use quadratic-weighted kappa or Krippendorff's alpha at the ordinal level, so that confusing "complete" with "mostly complete" costs less than confusing it with "missing." And don't average several judges into one score until each has been calibrated on its own. Three judges with the same blind spot agree with each other and miss the same failures. A 2024 comparison of judge models also found a general tendency toward leniency, which shows up in exactly the cell that matters: false passes.

Correct the score you report

Even a calibrated judge has known error rates, and you can use them. If sensitivity is the share of truly good outputs the judge passes and specificity is the share of truly bad outputs it fails, the Rogan–Gladen correction recovers the true pass rate from the observed one:

true_pass_rate = (observed_pass_rate + specificity - 1) / (sensitivity + specificity - 1)

Take the judge from the table: sensitivity 0.96 (172 of 180), specificity 0.55 (11 of 20). If the agent's true pass rate is 60%, the judge reports about 75.6%. Correcting gives back 60%. Worse, the bias compresses improvements: when the true rate rises from 60% to 70%, the judge's reading moves only from about 75.6% to 80.7%. Teams read that as a small gain and abandon a change that worked.

The correction is only as good as your specificity estimate, and 20 expert-failed items give a wide interval. Label more failures. They are the scarce, informative class.

Recalibrate when anything changes

Calibration expires. Rerun it on the held-out set when:

The judge model or its version changes, including silent provider updates. Pin versions where you can.
The judge prompt or the rubric changes.
The system being judged changes enough to produce new kinds of failure.
The input mix shifts, for example a new line of business or a new document type.

Between those events, sample 50 fresh production items a month for expert labeling and track kappa and false-pass rate over time. Expect the rubric itself to move. Shankar and colleagues call this criteria drift: people need criteria to grade outputs, but grading outputs is how they discover their criteria. A judge calibrated in February against criteria the experts have since refined is measuring last quarter's idea of quality.

This is why our evaluation work prices judgments individually and keeps experts in the loop. The expert labels are the reference standard, and the annotation team that produces them is calibrated the same way, against hidden gold items.

A calibration checklist

Write criteria as binary, observable checks, one per judge call.
Build a 200-item calibration set with known failures, double-labeled by experts, split 120/80.
Record expert-to-expert agreement first. It sets the ceiling.
Test position consistency in both orders and length sensitivity with padded answers.
Report kappa, false-pass rate and false-fail rate per stratum, not just percent agreement.
Set false-pass thresholds by the cost of a miss, and move high-stakes criteria to code or people.
Correct reported pass rates with the judge's measured sensitivity and specificity.
Pin the judge version, and recalibrate on any change to the judge, rubric, system or input mix.
Sample 50 production items a month for expert labels, and watch for criteria drift.

If you are still assembling the items to judge, start with a golden set built from real failures.

Calibrating LLM-as-a-judge against your own experts