Rubrics That Reduce Expert Disagreement

When OpenAI built InstructGPT, its trained labelers agreed with each other 72.6% of the time on comparisons, and held-out labelers agreed 77.3% of the time. Roughly one pair in four was contested, and those were general-purpose instructions. Put two cardiologists, two coverage attorneys or two senior adjusters in front of the same pair of answers, and their agreement depends heavily on what exactly they were asked to judge.

That matters more now than it did then. Direct preference optimization turns each chosen-and-rejected pair straight into a training signal with a simple classification loss, with no separate reward model between noisy labels and the policy. Rubric-based rewards are carrying reinforcement learning into domains without checkable answers. Disagreement is no longer a footnote in the data card. It goes into the model.

Not all disagreement is noise

Before trying to reduce disagreement, sort it. In expert preference work we see four kinds:

Slips. Attention lapses, misread outputs, a click on the wrong button. Hidden gold items catch them.
Ambiguous instructions. Two experts read "more helpful" differently. A rubric fixes this.
Different weights. Both experts see the same facts, but one values brevity and the other completeness. Explicit priorities fix this.
Legitimate professional disagreement. Two defensible clinical approaches, two reasonable readings of a contract clause. This should be recorded, not trained away.

A good rubric drives the first three toward zero and makes the fourth visible. A rubric that pushes the fourth toward zero is teaching one expert's habits as if they were the standard.

Gate first, then score, then prefer

The rubric that works best for us is an ordered procedure, not a list of things to consider.

Step 1: gates. Hard constraints come first: a factual error against the source, unsafe advice, a policy violation, personal information that shouldn't be there. If one response fails a gate and the other doesn't, the pair is decided and style is irrelevant. If both fail, it's not a preference pair at all. Route it to be rewritten as a supervised example.

Step 2: atomic criteria. Next come yes-or-no checks, each observable in the text, each weighted. HealthBench is the public model for this: 262 physicians wrote conversation-specific rubrics totaling 48,562 unique criteria. The approach also works as a reward signal. In the Rubrics as Rewards paper, the best rubric-based method gave up to a 28% relative improvement on HealthBench-1k over simple Likert-based rewards.

Step 3: preference with a margin. Only then does the expert choose, on a five-point scale from "A much better" through "tie" to "B much better," and the rationale must cite gate or criterion IDs. A rationale that cites nothing is a signal that the rubric missed something the expert cares about.

Here is a compact version for adjuster file summaries at a property insurer:

task: adjuster-file-summary
rubric_version: 4
gates:
  - id: G1
    check: Every dollar amount and date matches the claim file
  - id: G2
    check: States no coverage decision unless the file records one
  - id: G3
    check: Includes no claimant health details beyond what the reserve needs
criteria:
  - id: C1
    weight: 3
    check: First sentence gives cause of loss and date of loss
    pass_example: "Wind damage to roof and gutters, DOL 08/29."
    fail_example: "Insured called regarding damage to the property."
  - id: C2
    weight: 3
    check: Lists every open item
  - id: C3
    weight: 2
    check: Names an owner for each open item
  - id: C4
    weight: 2
    check: Flags subrogation potential when a third party is named
preference:
  scale: [A_much, A_slight, tie, B_slight, B_much]
  rationale: must cite gate or criterion ids

Two details carry a lot of weight. Each criterion has a pass and a fail example drawn from real outputs, because abstract wording is where experts diverge. And length limits or formatting rules are checked by code before experts see the pair, so expert attention goes only to judgment.

Measure agreement per criterion

Use Krippendorff's alpha. It handles any number of raters, missing ratings, and nominal or ordinal scales, and it corrects for chance agreement. For data meant to support scholarly conclusions, Krippendorff suggests requiring α ≥ .800, with .667 as the floor where tentative conclusions are still acceptable. He is also clear that the bar should be set far higher when lives or legal decisions hang on the results, a description that fits a good deal of clinical, insurance and legal training data. Our own operating standard is alpha of at least 0.80 on categorical labels before a batch ships.

Compute alpha per criterion, not only overall. An overall number averages a well-written criterion with a broken one and hides both. Here is what two calibration rounds typically look like, as an illustrative composite rather than a single client's data:

Criterion	Round 1 alpha	Change made	Round 2 alpha
G1 Amounts and dates match	0.91	None	0.93
G2 No unrecorded coverage decision	0.58	Added three anchors showing implied decisions	0.84
C2/C3 Open items with owners	0.66	Split into "lists items" and "names owners"	0.88 / 0.81
Length under 150 words	0.97	Moved to an automatic check	Checked by code
Overall preference	0.52	Gate-first procedure introduced	0.74

Notice the last row. Preference agreement stays lower than criterion agreement even after the rubric improves, because the remaining disagreement is mostly the fourth kind. That's expected, and it's the reason to keep criterion-level labels alongside the preference.

Run calibration rounds before volume

A protocol that converges in two or three rounds:

Pilot 40 pairs with three experts working independently.
Compute alpha per criterion and list the items with disagreement.
Meet for an hour on those items only. Rewrite criteria and add anchors where the wording failed.
Run a fresh 40 pairs. Don't reuse the first set, because experts remember their answers.
Repeat until every criterion clears 0.80, then start volume labeling.

During volume labeling, keep the controls on: reviewers must stay at or above 95% on hidden gold items, and a second reviewer labels at least 10% of every batch. Recompute alpha weekly on the overlap. A drop usually means the input mix changed, or the rubric quietly did.

What to do with the disagreement that remains

Keep every vote. Store each rater's labels, not just the majority, so you can compute soft labels or filter later.
Train on decided pairs. For DPO, use pairs decided by a gate or a clear margin. Drop pairs where raters split across "A slightly better" and "B slightly better." Ties are not pairs.
Route persistent splits to a policy owner. Sometimes the answer is that both outputs are acceptable. Sometimes the split exposes a rule nobody wrote down, which is worth more than the label.
Version the rubric. Shankar and colleagues describe criteria drift: grading outputs is how people refine their criteria. When the rubric changes, relabel the affected items or segment the dataset by rubric version.

Expert preference data built this way is the core of our expert annotation work, staffed through our expert network of practitioners who have done the job being judged.

Checklist before the first thousand pairs

Sort past disagreements into slips, ambiguity, weighting and legitimate difference.
Write gates, atomic criteria and a margin scale, in that order, as a procedure.
Give every criterion a real pass example and a real fail example.
Move mechanical checks to code before experts see the pair.
Run 40-pair calibration rounds until each criterion reaches alpha 0.80.
Keep gold items and second review on through volume labeling.
Store every vote, train on decided pairs, and version the rubric.

The same rubric, once calibrated, is also the starting point for an LLM judge you can trust.

Rubrics that reduce expert disagreement in preference data