Near-Duplicate Detection at Scale

Duplicates cost more than disk. Lee and colleagues found that over 1% of the unprompted output of language models trained on standard datasets was copied verbatim from training data, and that a single 61-word sentence appeared more than 60,000 times in C4. Deduplicated training produced models that emitted memorized text ten times less often, and train-test overlap had affected over 4% of standard validation sets. The effect compounds: Kandpal and colleagues found that a sequence present 10 times is generated about 1,000 times more often than one present once.

Lee's team also noted what much of the near-duplicated text in C4 looked like: computer-generated pages, identical except for the names of places, businesses, products and dates. That is a fair description of a denial letter, a work-order template or a policy endorsement. Enterprise corpora are built from templates, which makes deduplication both more valuable and easier to get wrong.

Decide what counts as a duplicate, per task

Before choosing an algorithm, write down what a duplicate means for the model you're training.

Template letters. For a model that drafts letters, 40,000 copies of one denial letter with different names teach a single pattern 40,000 times; keep a few per template variant. For a model that extracts fields from those letters, the names, dates and amounts are the task. Deduplicating on template text would delete the training signal.
Email threads. Every reply quotes the thread below it. Broder's 1997 paper defined two measures for this reason: resemblance, the overlap of two documents relative to their union, and containment, the share of one document found inside another. A two-line reply inside a 40-message thread has low resemblance and near-total containment.
Versions. Contract drafts and redlines are near-duplicates by design. Keep the executed version, or keep the sequence if the model is learning how terms get negotiated.

Write the rule per source and per task, and keep the duplicate count for every survivor. Frequency is information, as the next sections show.

Three tools for three kinds of duplicate

Kind	Typical example	Tool	Relative cost
Exact, after normalization	The same PDF saved in two systems	SHA-256 of normalized text	One cheap pass
Lexical near-duplicate	Form letters, copied-forward notes, lightly edited boilerplate	MinHash with locality-sensitive hashing, then verification	Linear in records, plus candidate checks
Semantic duplicate	A rewritten FAQ answer, a translated policy, an OCR copy of a born-digital page	Embeddings with approximate nearest-neighbor search	GPU time plus an index

Run them cheapest first. Each stage shrinks the input to the next. Don't stop after the first: the builders of C4 and RealNews had both attempted deduplication, yet Lee's MinHash pass still flagged 3.04% and 13.63% of their training examples as near-duplicates.

MinHash, with the arithmetic

Shingle each document into word 5-grams, the choice in both Lee's work and FineWeb's. For each of k hash functions, keep the minimum hash over all shingles. The chance that two documents share a given minimum equals their Jaccard similarity, so the k minimums form a signature that estimates similarity without comparing full texts.

Comparing every signature with every other is still quadratic, so split each signature into b bands of r values. Two documents become candidates if any band matches exactly. The probability is 1 − (1 − s^r)^b for similarity s, an S-curve whose steep point sits near (1/b)^(1/r).

FineWeb used 112 hash functions in 14 bands of 8, which matches documents at 0.75 similarity with probability 77%. Lee's team used a 9,000-value signature and confirmed candidates with an edit-similarity check. The band shape is the real decision:

Bands × rows	Hashes	P(candidate) at s = 0.6	0.7	0.8	0.9
20 × 5	100	80%	97.5%	>99.9%	>99.9%
14 × 8	112	21%	56%	92%	>99.9%
9 × 13	117	1.2%	8.4%	40%	93%

Bias the curve slightly loose. False candidates cost verification time; a pair the bands never surface is lost for good. Then verify every candidate exactly, with Jaccard on the shingle sets or edit similarity.

Memory is modest. Fifty million documents with 112 32-bit values each is 22.4 GB of signatures, and banding maps cleanly onto a group-by in Spark or Ray: one key per band hash.

Two decisions remain. Clustering: connecting verified pairs with union-find chains A to B to C even when A and C differ, and with templates that chain can merge distinct letter types through intermediate variants. Check members against the cluster's representative, not just a neighbor. Which copy to keep: FineWeb kept one at random, which suits web text. In business data, keep by rule (the executed contract, the final note, the most complete record) and log the rule.

Global deduplication can remove the wrong things

More aggressive is not automatically better. FineWeb's authors first deduplicated globally across 96 Common Crawl snapshots and saw little improvement over not deduplicating at all. In one older snapshot, the 10% of data that survived turned out to be lower quality than the 90% removed: the removed text, which recurred in other crawls, was the better material, while what survived as unique carried more ads, keyword lists and badly formatted pages. Deduplicating each snapshot independently worked better.

The enterprise version is easy to picture. Standard operating procedures, clinical guidelines and policy manuals appear in every year's folder because people rely on them. Global deduplication reduces each to a single copy while every one-off fax cover sheet survives. Deduplicate within a source and time slice first, set cross-source policy deliberately, and keep the copy count as a feature you can use for weighting.

Embeddings for what MinHash can't see

Shingles miss paraphrases, translations and OCR noise. Embeddings catch them, and the payoff can be large: SemDeDup removed 50% of a LAION subset with minimal performance loss, and D4 reported a 20% training-efficiency gain at 6.7B parameters by selecting with embeddings on top of MinHash-deduplicated data.

Thresholds don't transfer between models. The model card for e5-base-v2 notes that its cosine scores fall mostly in the range 0.7 to 1.0, a side effect of training at low temperature, so a 0.9 cutoff means something different there than on another model. Calibrate on your own data: label about 300 pairs spread across the similarity range, then choose the threshold that gives the precision you need.

Keep it linear. Partition with k-means, or reuse the MinHash clusters, and search nearest neighbors within partitions using a library such as Faiss. A global all-pairs search over tens of millions of vectors is rarely necessary.

Decontamination is a different job

Deduplication is symmetric housekeeping inside the training set. Decontamination is directional: remove from training anything that overlaps an evaluation set, and never the reverse.

The GPT-3 authors used 13-gram overlap to strip benchmark text from training data, removed a 200-character window around each hit, and ignored 13-grams that appeared in more than 10 training documents because those were mostly common phrases and legal boilerplate. That ignore rule is necessary in business data, where every contract shares an indemnification clause. It is also where leaks hide, because evaluation items are often built from the same templates.

N-gram checks also miss paraphrase. Yang and colleagues showed that rephrased or translated test items pass n-gram filters, and with an LLM-based detector found that 8–18% of HumanEval overlapped with large pre-training sets. A layered check holds up better:

Exact substrings of 50 tokens or more, the threshold Lee's team used, plus 13-gram overlap with a boilerplate allowlist you review by hand.
Embedding nearest neighbors for every evaluation item against the full training set.
A human or model judge on the top candidates, with the decision logged per item.

Split evaluation data by entity and time as well, so a near-duplicate of a test case can't enter training through another customer's record. Our evaluation and red-teaming work runs this check before any before-and-after number is reported.

What a run report should say

Every pass should leave a report that a reviewer can check and the next run can compare against. An illustrative summary for a 50-million-record run:

{
  "run_id": "dedup-2026-07-01",
  "input_records": 50000000,
  "exact_removed": 6100000,
  "minhash": { "shingle": "word_5gram", "bands": 14, "rows": 8, "verify": "jaccard>=0.8" },
  "near_dup_clusters": 2310000,
  "near_dup_removed": 4870000,
  "semantic": { "model": "embed-model@rev-2026-05", "threshold": 0.92, "calibration_pairs": 300, "precision_at_threshold": 0.95 },
  "semantic_removed": 1120000,
  "kept_with_copy_count": true,
  "decontaminated_against": ["golden-claims-v6", "extraction-eval-v3"],
  "decontam_removed": 4120,
  "output_records": 37905880
}

The arithmetic should close (it does here), and the decontamination count should be small but rarely zero. Our curation and deduplication pass produces this report alongside a sample of removed records for a person to read.

Checklist

Define a duplicate per source and per task, including whether variable fields count.
Normalize, then remove exact duplicates first.
Pick MinHash bands and rows from the probability curve at your threshold, then verify every candidate exactly.
Use containment, not only resemblance, for threads and quoted text.
Check cluster members against a representative to stop chaining.
Keep survivors by a written rule and record their copy counts.
Deduplicate within source and time slice before going global.
Calibrate embedding thresholds on labeled pairs from your own data.
Decontaminate in layers against every evaluation set, and review the boilerplate allowlist by hand.
Publish a run report whose numbers add up, with a sample of removed records.

Near-duplicate detection at scale: MinHash, embeddings, and decontamination