Formats
- JSONL or Parquet for text judgments; preference pairs as prompt, chosen, rejected
- COCO or YOLO for images; WebVTT or JSON segments for audio and video
- Exports into your labeling platform if you use one
Your model should learn from people who've done the job.
Clinicians, claims adjusters, attorneys, engineers, and underwriters score model outputs, compare pairs of answers, and annotate images, audio, and video against a rubric we build with you. Every batch ships with agreement statistics and the gold-question record of every reviewer who touched it.
Judgment is the scarce input
Human preference data has outweighed model size before. In the InstructGPT study, labelers preferred outputs from a 1.3B-parameter model trained on human demonstrations and rankings to those of the 175-billion-parameter GPT-3. The training signal came from people deciding which answer was better.
In specialized work, those people have to know the job. A general annotator can tell which answer is more polite. Only a utilization-review nurse can tell which one misreads the payer's criteria, and only an adjuster who has worked a hail season can tell which estimate summary forgot the roof. Preference data from non-experts teaches a model to sound right in your domain, which is worse than sounding unsure.
Our reviewers are employed, vetted US persons working under NDA, and under a business associate agreement when the data is health information. There is no anonymous crowd behind the numbers.
What we collect
| Judgment type | What the reviewer does | Used for |
|---|---|---|
| Rubric scores | Scores a response on defined criteria such as accuracy, completeness, safety, and tone, with a written rationale | Evaluation, reward models, filtering training data |
| Pairwise preferences | Picks the better of two responses, rates the margin, and says why | DPO, reward modeling, model selection |
| Corrections and rewrites | Edits a response into the one they would sign | Supervised demonstrations; the chosen side of a preference pair |
| Span and entity labels | Marks evidence, entities, or errors in text | Extraction, retrieval ground truth, error analysis |
| Image annotation | Boxes, polygons, and classifications on photos, scans, and drawings | Damage assessment, inspection, document layout |
| Speech and video | Reviews transcripts, labels speakers and events, and marks timestamped segments | Call analytics, safety review, speech model training |
How a rubric gets built
Most expert disagreement comes from an unclear question, not from unclear experts. The rubric is where we fix that.
Start from disagreement
Three experts judge the same 50 items with no rubric. Where they split shows exactly what the rubric has to settle.
Write criteria a second person can check
"Cites the governing policy section" rather than "is accurate." Each score point gets an anchored example, including a borderline one.
Pilot and measure
About 200 items are double-labeled. We compute Krippendorff's alpha per criterion and rewrite any criterion that falls short, instead of averaging the disagreement away.
Hide gold questions in the queue
Items with settled answers appear in every reviewer's queue unannounced. A reviewer who drops below 95% is retrained or taken off the batch, and their recent work is re-reviewed.
Adjudicate and version
A senior reviewer resolves disputed items. Every rubric change gets a version number, and items scored under the old version are re-checked where the change affects them.
What every batch is held to
Our own operating standards, measured on every batch and reported with the delivery.
α ≥ 0.80
Krippendorff's alpha
On categorical labels before a batch ships. Criteria below the line get rewritten.
≥ 95%
gold-question accuracy
Required of every reviewer, measured on hidden items with settled answers.
≥ 10%
second review
Of every batch is scored independently by a second reviewer.
0
anonymous crowdworkers
Onshore, employed US persons only; no offshore subcontracting.
Expert judgment at a glance
You provide
- Prompts and model responses to judge, or images, audio, or video to annotate
- The policies, guidelines, or standards that define a good answer in your work
- A subject-matter owner for rubric sign-off and escalations
We deliver
- A versioned rubric with anchored examples
- Judgments with rationales, reviewer IDs, timestamps, and rubric version
- An agreement report per criterion and per reviewer
- An adjudication log for every disputed item
Quality gates
- Krippendorff's alpha ≥ 0.80 on categorical labels in every batch
- Every reviewer ≥ 95% on hidden gold questions
- A second reviewer on at least 10% of every batch
- Rubric version recorded on every judgment
Turnaround
A 2,000-judgment pilot in two to three weeks, including rubric design
Team
An annotation lead, a rubric designer, and credentialed reviewers matched to your domain
Starts at
$4.80per judgment
Rubric scores and pairwise preferences from credentialed reviewers in your domain. Pilots start at 2,000 judgments.
Priced per judgment
Expert judgment
Starts at
$4.80per judgment
Rubric scores and pairwise preferences from credentialed reviewers in your domain. Pilots start at 2,000 judgments.
Request a scoped quoteThe rate depends on the credential and the time a judgment takes: a physician adjudicating a discharge summary costs more than a coder checking a code. Image, audio, and video work is quoted per asset or per hour after a sample. We work in your labeling tool or ours, inside your environment.
Questions about expert annotation
Why not use a crowd labeling platform?
For general tasks on non-sensitive data, a crowd can be the right call. For protected health information, privileged documents, or judgment that takes a license and years of practice, you need reviewers who are vetted, employed, accountable, and in the US. We compare the two in onshore vs. crowd.
Where do your reviewers come from?
From our expert network of working and recently retired practitioners. We verify licenses and experience, pay by the hour rather than by the item, and train every reviewer on your rubric before they score anything that counts.
Can our own staff do some of the reviewing?
Yes, and for some tasks they should. We design the rubric, run the gold questions and agreement measurement for everyone, and blend your reviewers with ours so the quality numbers cover the whole batch.
What happens when experts disagree?
We treat it as information. Persistent disagreement on a criterion usually means the rubric is ambiguous or the case is genuinely contested. We revise the rubric, adjudicate the item, and report contested cases rather than hide them in an average.
Do you annotate images, audio, and video, or only text?
All of them. Claims photos, drone and inspection imagery, scanned drawings, recorded calls, and safety video, using the same rubric, gold-question, and agreement process as text.
Keep reading
Bring us one messy data source and one question.
Tell us what the data is and what you want a model to do with it. A data engineer, not a salesperson, will reply within one business day with questions and a proposed first step.