AI-Ready Data: 12 Tests to Run First

Gartner predicts that through 2026, organizations will abandon 60% of AI projects that aren't supported by AI-ready data. Its survey found that 63% of organizations either lack the right data management practices for AI or aren't sure whether they have them. Those figures get quoted often. What gets quoted less is that "AI-ready" is not a property of a data lake. It is a property of data relative to one use case. The same claims archive can be ready for a triage classifier and useless for a reserving model.

So the tests below are run per use case, on samples, by people who can read the records. None of them needs a platform purchase, and most take an afternoon. Each ends in one of three verdicts: pass, fix (the gap has a cost you can estimate), or stop (the use case needs different data or a different design). Gartner's own description is that AI-ready data is not “one and done”, and these same tests become your monitoring after launch.

Before you start: one use case, one sample

Write the use case as one sentence with an input and an output: "Given a first notice of loss and the policy declarations page, assign one of 14 claim-handling tracks." If you can't write that sentence, stop there. No test will help.

Then draw the sample. For each source system, pull 200 records stratified by year and by the categories that matter (line of business, facility, region). A simple random sample over-represents the routine middle; stratifying makes sure the 2019 scans and the rare claim types show up. Keep the sample IDs, because every test below reuses them.

Tests 1–4: is the task actually in the data?

1. The expert-read test

Give two domain experts 50 sampled records and ask them to perform the task using only what is in the record. Count how often either one says "I'd need to look something up." If that happens on more than one record in five, the record alone does not carry the task. Fine-tuning won't fix that. The use case needs the missing context retrieved, or a different input.

2. The outcome test

Training and evaluation both need to know what happened next: the claim was paid or denied, the work order closed with a part replaced, the prior authorization was approved on appeal. Measure coverage (the share of records with a populated outcome) and fidelity (on 100 records, does the coded outcome match what the narrative says?). Curated benchmarks aren't clean either; one study estimated label errors averaging at least 3.3% across the test sets of ten widely used datasets. Operational codes entered under time pressure and rarely audited have no reason to be cleaner. Below about 90% fidelity, budget for relabeling.

3. The tail test

List the cases that cost money when the system gets them wrong: the coverage exclusion, the hazardous-materials exception, the pediatric dose. Count them in the sample and extrapolate to the full source. Thirty real examples per critical case is a practical floor for an evaluation set alone, and even at 30, a measured 80% carries a 95% confidence interval of roughly 63% to 91%. If the costly case appears 14 times in 90,000 records, that is a stop for fine-tuning on that case.

4. The freshness test

Find the dates when the process changed: a new claims system, a revised formulary, a tariff update, a new policy form. Tag every record with the regime it was created under, then compare outcome distributions before and after. If 60% of your examples predate the current procedure, the model will learn the old procedure with full confidence. The fix is usually cheap, filtering or weighting by regime, but only if someone wrote the dates down.

Tests 5–8: can a machine read it, and is it clean?

5. The parse test

Per source, measure the share of PDFs with no text layer, pages with tables, and pages with handwriting or stamps. Hand-transcribe 20 pages and compute character error rate against your OCR output, then check whether table cells landed in the right columns. Parse quality caps everything downstream, including retrieval, and averages hide it: report the worst document type, not the mean.

6. The duplication test

Hash records for exact duplicates, then run near-duplicate detection. MinHash over word 5-grams with a Jaccard threshold near 0.8 is a reasonable start. Report the share of records that sit in a cluster larger than one. Enterprise data is full of templates: email signatures, form letters, disclaimers, clinical notes copied forward from the last visit. In public web data, a single 61-word sentence appeared over 60,000 times in the C4 corpus, and models trained on deduplicated data emitted memorized text ten times less often. Boilerplate in a fine-tuning set teaches the model to produce boilerplate.

7. The split test

Decide how training and evaluation data will be separated before anyone trains. Split by entity, not by row: by claim, patient, customer or vessel, and ideally by time (train on 2023–2024, test on 2025). Then check whether any near-duplicate cluster from test 6 straddles the split. The same deduplication study found train-test overlap in over 4% of the validation sets of standard language-modeling datasets. Leakage doesn't make a model worse. It makes your measurement wrong.

8. The join test

Most valuable examples span systems: the call recording, the ticket, and the refund in the ERP. Take 100 records from the primary source and try to find their counterparts. Report the join rate and why joins fail: claim numbers reformatted between systems, account numbers typed into free text, merged customer records. Below 80%, the multi-source version of the use case is a data-engineering project before it is an AI project.

Tests 9–12: may you use it, and can you prove it worked?

9. The sensitivity test

Run a PII and PHI detector over the sample, then have a reviewer check 50 hits and 50 non-hits. Report density (identifiers per 1,000 words), categories, and the detector's precision and recall on your data. Density decides where processing may happen and what de-identification will cost. The detector's measured recall decides whether automation can be trusted at all.

10. The rights test

For each source, answer three questions: who created it, under what contract or terms, and does anything restrict its use for training? Vendor-supplied files, customer uploads and licensed content are where surprises live. Public data has the same problem: an audit of more than 1,800 text datasets found license omission rates of 70%+ and license error rates of 50%+ on widely used hosting sites. Record the answer per source with the evidence attached, and send it to counsel.

11. The ground-truth test

Can you build an evaluation set your experts would sign? Have two experts label the same 50 records independently and measure agreement. If they can't agree, the model has no target to hit. Our working threshold is a Krippendorff's alpha of 0.80 on categorical labels. Below that, fix the guidelines before labeling at volume.

12. The owner test

For each source, name the person who can answer "why is this field blank for most of 2021?" within a week. Every other test generates questions like that one, and data with no owner tends to stay unfixed.

Reading the results

Record every result in the same shape, so the scorecard can be compared quarter to quarter:

{
  "use_case": "fnol-track-assignment",
  "source": "claims-core-notes",
  "sample": {"n": 200, "strata": ["year", "line_of_business"], "seed": 4417},
  "tests": {
    "expert_read": {"result": "pass", "needs_lookup_rate": 0.12},
    "outcome": {"result": "fix", "coverage": 0.97, "fidelity": 0.84},
    "tail": {"result": "fix", "critical_case": "subrogation", "est_count": 410},
    "parse": {"result": "pass", "no_text_layer": 0.06, "cer_worst_type": 0.031},
    "duplication": {"result": "fix", "near_dup_share": 0.18}
  },
  "verdict": "fix",
  "owner": "claims-ops-data-steward",
  "run_date": "2026-01-09"
}

Then apply a few rules:

Test 1, 3 or 11 fails: don't fine-tune. The task, the hard cases, or the target is missing. Rework the use case.
Test 1 passes, but answers depend on reference material that changes monthly: that points to retrieval rather than fine-tuning, and tests 5 and 6 tell you how much parsing and cleanup it needs.
Test 2, 4, 6 or 7 fails: fixable. This is curation work with a cost you can estimate from the sample, and it is where curation and deduplication projects usually start.
Test 9 or 10 fails: pause and route the source to privacy and legal review before more engineering.
Test 12 fails: assign an owner before funding anything.

What to do this quarter

Pick the two use cases that have sponsors and write each as one input-output sentence.
Pull a stratified 200-record sample per source and keep the IDs.
Run tests 1, 2, 3 and 11 first. They are cheap and the most likely to stop a project early, which is the point.
Run the parse and duplication tests on the same sample and report numbers, not adjectives.
Write up rights and sensitivity findings per source, with evidence, for counsel.
Store results in one schema and rerun them quarterly. Movement in the outcome and freshness tests is your early warning.
For a quick first pass on these questions, the readiness scorecard takes about ten minutes.

AI-ready data: 12 tests to run before you fine-tune or build RAG