Formats
- Parquet or JSONL, partitioned the way your training jobs read
- Removal log as Parquet with record ID, rule, score, and cluster ID
- Report as PDF plus the notebook that generated it
Take out the copies, the junk, and the answers to the test before training starts.
Business data repeats itself: reply-all chains quoted ten times over, templated reports, the same policy in six versions. We remove exact and near duplicates, score quality against the task, filter what must not be learned, and check that none of your evaluation set leaked into training. Then we show you what changed.
Duplicates don't just waste compute. They change what a model learns.
A model trained on repeated text learns to repeat it. In a study of widely used language-model datasets, Lee and colleagues found that over 1% of unprompted model output was copied verbatim from the training data, and that one 61-word sentence appeared more than 60,000 times in a single public corpus. They also found train-test overlap in over 4% of standard validation sets, which means some reported scores were partly measuring memory.
Enterprise data repeats in its own ways. Adjusters paste the same template into every claim note. Support agents send the same macro thousands of times a week. A policy manual exists as the current version, three drafts, and a scan of a printout someone annotated. Fine-tune on that as-is and the model learns the template, not the judgment that varies around it.
Curation fixes the proportions before they become the model's habits, and it makes every later step cheaper, because you stop paying to embed, label, and train on the same content again.
The curation pass at a glance
You provide
- Up to 20 million records: documents, emails, tickets, transcripts, or an existing training set
- Every evaluation set and benchmark you report results against
- A short description of the target task, so quality is scored against what the model needs to learn
We deliver
- A deduplicated, filtered dataset, versioned
- A removal log with a reason code and matched cluster for every record taken out
- Quality scores per record, and the classifier that produced them
- A decontamination report listing every overlap with your evaluation sets
- A before-and-after report with counts, examples, and a small training comparison where scope allows
Quality gates
- Near-duplicate thresholds tuned on at least 500 hand-reviewed candidate pairs per source
- Quality classifier checked against reviewer labels before it is applied
- A sample of every filter's removals read by a person
- Decontamination run against every evaluation set in scope
Turnaround
Three to five weeks, depending on volume and modality
Team
A data engineer, an ML engineer, and two reviewers who read what the filters remove
Starts at
$9,200
Quality scoring, exact and near-duplicate removal, and evaluation decontamination for up to 20 million records, with a before-and-after report.
Six filters, and what each one catches
Each stage runs in your environment, and each logs exactly what it removed and why.
| Stage | Method | What it catches |
|---|---|---|
| Exact duplicates | Hashing after normalization: whitespace, case, signatures, and boilerplate footers stripped first. | The same PDF in three folders, re-sent attachments, re-exported tables. |
| Near duplicates | MinHash signatures over word shingles with locality-sensitive hashing, Jaccard threshold tuned per source. | Templated letters, quoted email chains, contract versions that differ by a date. |
| Semantic duplicates | Embedding clustering with a similarity cutoff, then sampling within each cluster. | The same answer reworded by different agents; paraphrased or translated copies. |
| Quality scoring | A small classifier trained on reviewer-labeled good and bad examples for your task, plus heuristics: length, symbol ratio, language ID, OCR noise. | Garbled OCR, auto-generated alerts, empty forms, emails that are only a signature. |
| Sensitive content | PII detectors and toxicity classifiers; flagged records go to redaction or removal under your policy. | Account numbers in free text, abusive messages in support logs. |
| Decontamination | Long n-gram overlap and embedding matches against every evaluation set, at document and passage level. | Golden-set questions, or their answers, sitting in the training data. |
Decontamination keeps your scores honest
If evaluation examples leak into training, the evaluation reports progress the model hasn't made. It happens more in enterprise work than in research, because the golden set and the training dump are usually cut from the same ticket system or claims archive. A question your team copied from a 2024 ticket into the evaluation set is probably also in the training export. We match at the passage level against every evaluation set you use and quarantine the training-side copy, so the number you report to leadership measures generalization, not recall.
After cleaning comes the mix
What survives curation still has to be balanced. If 70 percent of your support tickets are password resets, a model trained on all of them will be very good at password resets. We report the dataset's composition by source, task type, time period, and difficulty, then propose a mixture: downsample over-represented clusters, keep a representative from each, and upsample rare cases that matter, such as catastrophe claims, denied appeals, or safety incidents.
Where budget allows, we test two or three candidate mixtures with small training runs against your evaluation set, so the choice is made on measured results rather than intuition.
Priced by volume
Curation pass
Starts at
$9,200
Quality scoring, exact and near-duplicate removal, and evaluation decontamination for up to 20 million records, with a before-and-after report.
Request a scoped quoteAbove 20 million records, or for image, audio, or source-code corpora, we quote after a profiling run on a sample. Processing runs in your cloud, and we estimate the compute bill before the full run starts.
Questions about curation
Won't removing duplicates throw away real signal?
Sometimes repetition is the signal: a common case is common for a reason. That's why we downsample clusters instead of deleting them outright, keep a representative of each, and record cluster sizes so your team can reweight later if a training run needs it.
How do you choose the near-duplicate threshold?
Per source, never globally. Reviewers read candidate pairs across a range of similarity scores and mark which are true duplicates. We show you precision at each threshold and pick the cutoff together, because a contract archive and a chat log need different settings.
Do you use a large language model to score quality?
Sometimes as a labeler, to help produce training data for a small quality classifier, and only after checking it against our reviewers. Running a large model over 20 million records is rarely the cheapest way to get the same answer.
Can you curate data that has already been used to train a model?
Yes, and it often explains the model's behavior. Clusters of repeated templates, contamination with the test set, and lopsided proportions show up clearly in the report, and they usually point to the fix.
What does the before-and-after report show?
Record and token counts at each stage, removals by reason with examples, the composition of the final mix, and the decontamination findings. When scope includes it, a small fine-tune on the before and after sets, scored on your evaluation set.
Keep reading
Bring us one messy data source and one question.
Tell us what the data is and what you want a model to do with it. A data engineer, not a salesperson, will reply within one business day with questions and a proposed first step.