Data rises one gate at a time, and nothing skips a lock.
Every dataset we build passes the same six gates in the same order: Source, Parse, Clean, Protect, Label, Prove. Each gate has an owner, a written exit check and a record that ships with the data. This page explains what happens at each one, the standards we hold ourselves to, and what you see every week.
Why gates instead of a pipeline diagram
Most data pipelines are drawn as arrows. Arrows hide the moment where bad data should have been stopped. A gate is different: a batch doesn't move to the next stage until it passes a written check, and when it fails, the failure is visible to you and recorded in the manifest.
The order matters too. You can't measure redaction on text you haven't parsed, and you can't label what you aren't allowed to use. Running the gates in sequence means each one inherits a known-good input from the one before it.
One gate breaks the order on purpose. Prove is the last gate a batch passes, but its main artifact, the evaluation set, is built in the first weeks of every project, before any training data exists.
The Six Gates
What happens at each gate, and what has to be true before data moves on.
1 · Source: inventory, owners, rights
We list every system and collection in scope, sample what is actually inside, and record the owner, volume, date range, format and sensitivity of each. Rights come first: who owns the records, what contracts or consents govern them, and which ones your counsel needs to review. Exit check: every source has an owner and a recorded permission, and anything unresolved is excluded, not assumed.
2 · Parse: make it machine-readable
Scanned PDFs, faxes, handwriting, tables, forms, email threads and audio become structured text with layout preserved where it carries meaning. We measure parsing quality on a hand-checked sample rather than trusting a tool's confidence score. Exit check: field- or character-level accuracy on the sample meets the threshold set for the use case, and unreadable material is counted and reported.
3 · Clean: dedupe, filter, decontaminate
Exact and near-duplicates come out, along with system-generated boilerplate, templates and records too thin to learn from. Anything that overlaps the evaluation set is removed from training candidates so before-and-after numbers stay honest. Exit check: duplicate rates and filter counts recorded, and zero training records within the similarity threshold of a test record.
4 · Protect: redact, pseudonymize, minimize
Personal, health, financial and other sensitive information is removed or replaced with consistent surrogates, and fields the task doesn't need are dropped. Recall is measured on a hand-labeled sample held out from rule tuning. Exit check: recall on a sample of at least 1,000 records meets the target agreed with your privacy team, with every miss reviewed.
5 · Label: expert annotation, preferences, demonstrations
Domain experts write examples, score outputs against a versioned rubric, compare pairs, or record how they complete multi-step tasks. Hidden gold questions check every reviewer, and a second reviewer covers part of every batch. Exit check: agreement and gold-question scores meet the standards below, field by field.
6 · Prove: evaluate, measure, document
We measure the model before and after training on the frozen evaluation set, using your experts' grades or a judge model calibrated against them, and write the datasheet and provenance manifest. Exit check: results reported on the same model, prompt and test set, and your data owner signs off on the release.
Evaluation comes first
In the first weeks of a project, before we write or label a single training example, we build the evaluation set with your experts: fifty to four hundred real cases, a rubric that says what a correct output must contain and what counts as an error, and grades on each case from the people who do the work.
That order does four things. It forces the task to be defined precisely enough to test. It gives a baseline: how your current process, the base model, or a vendor's tool performs today. It lets the Clean gate decontaminate training data against a set that already exists. And it calibrates reviewers, because disagreements about the rubric surface on fifty cases instead of five thousand.
The set is then frozen and hashed. It changes only with a new major version, so a number reported in March means the same thing in September. We wrote more about this in Fifty right examples.
The standards we hold ourselves to
These are our own operating targets, written into every statement of work. They are not industry benchmarks, and when a batch misses one, you see it in the weekly readout.
α ≥ 0.80
Krippendorff's alpha on categorical labels
Krippendorff suggests α ≥ .800 before data are relied on. We measure it per field before a batch ships, and a field below it goes back to the rubric, not into your dataset.
≥ 95%
Reviewer accuracy on hidden gold questions
A reviewer who falls below it stops labeling, and their recent work is re-reviewed rather than averaged in.
≥ 10%
Second review on every batch
At least one in ten items gets an independent second reviewer, which is what makes agreement measurable.
1,000+
Hand-labeled records to measure redaction recall
Held out from rule tuning and reported by identifier type, with every miss listed.
Every
Dataset versioned, with a datasheet and manifest
Frozen test splits, changelogs and a provenance manifest listing sources, licenses, transformations and reviewers.
30 days
Client data deleted after project close
Followed by a written deletion record that lists each location and the method used.
72 hours
Security incident notification
A contractual commitment to tell you about any security incident affecting your data. More on our security page.
How we staff a project
Small teams of employees, with domain experts brought in for judgment and separate reviewers checking the work.
Onshore employees
Data engineers, analysts and annotation leads are Benner House employees in the United States, working from Jackson or remotely. No anonymous crowd and no offshore subcontracting.
Domain experts
Nurses, physicians, adjusters, attorneys, mechanics and lineworkers from our expert network write and judge the examples. They are part-time employees who pass the same background checks and sign the same confidentiality terms as the rest of our staff.
Independent reviewers
Second reviews are done by someone other than the original labeler, and gold questions are written by the annotation lead, not the reviewers they test.
Named leads
Every project has a lead data engineer and an annotation lead, plus a privacy reviewer and an evaluation engineer where the work calls for them. You know who they are from the first day.
What you see every week
A short written readout and a working session with your team, in person or by video.
- The batch delivered, with its version tag and record counts in and out of each gate
- Agreement by field, and which fields are at risk of missing the threshold
- Gold-question scores by reviewer ID, and any reviewer removed or retrained
- Movement on the development set since the last batch
- Questions for your experts: ambiguous cases, rubric gaps, policy calls
- Issues found in the data, and what we did about each one
- Hours and spend against the fixed scope
- Any change in who has access to your environment
Questions about the method
Can we skip a gate?
No, but some gates are light. If your records are already digital text, Parse is a quick check. If the data is public, Protect is a scan to confirm nothing sensitive slipped in. Every gate still produces its record in the manifest.
What happens when a batch fails a gate?
It stops. We report the failure in the weekly readout, find the cause (a rubric gap, a parsing problem, a reviewer), fix it, and re-run the check. A missed gate can delay a delivery; it never ships quietly.
Who owns the rubric?
You do. We draft it with your experts, version it like code, and change it only with their agreement. Every label records the rubric version it followed.
Do you use models to label data?
For pre-labeling and drafting, often: it saves expert time. A model's output never becomes a final label without expert review, and where a judge model grades outputs, we measure its agreement with your experts first and report it.
How does this differ from a labeling platform?
A platform gives you tools and a workforce. We give you a measured result: a dataset that passed written checks, an evaluation showing what it changed, and the records to prove both. We'll work inside whatever platform you already use.
Related services and reading
Bring us one messy data source and one question.
Tell us what the data is and what you want a model to do with it. A data engineer, not a salesperson, will reply within one business day with questions and a proposed first step.