The label matters as much as the specimen.

Every dataset we deliver ships with a datasheet like this one. This is a complete sample for a fictional dataset, Claims Notes SFT v1.2. The figures are invented for illustration; the fields, the checks and the margin notes are what you would receive.

This is a sample

Claims Notes SFT v1.2 is fictional. No client data was used to produce this page, the carrier it describes does not exist, and every number is illustrative. The structure follows the datasheet proposal of Gebru et al. (2018), extended with the de-identification, evaluation and lineage sections our clients' governance teams ask for.

How to read it

Each row has three parts: the field, the value we would record, and a margin note explaining why the field is there. The notes are for this page only; a real datasheet carries the first two columns, plus links to the evaluation report and the full manifest.

Read it the way a skeptical reviewer would. The useful parts of a datasheet are the ones that make the dataset look worse: the excluded sources, the redaction misses, the label that failed its agreement check, the test-set contamination found in the previous version. A datasheet without any of those is usually incomplete, not perfect.

Part 1: What the dataset is

Motivation

Dataset: Claims Notes SFT v1.2 (sample)
Why it matters: Name and version travel together everywhere: in file paths, model cards and evaluation reports.
Purpose: Fine-tune a model that turns a property claim's adjuster diary notes into a structured summary (loss description, cause of loss, coverage questions, open items, reserve-relevant facts) for desk adjusters to review
Why it matters: A purpose narrow enough to test. "Claims AI" is not a purpose.
Requested by: The Carrier's claims operations and data science teams (fictional regional property carrier)
Why it matters: Tells a future reader whose decision the dataset was built to support.
Built by: Benner House, inside the Carrier's cloud tenant
Why it matters: Where the work happened matters as much as who did it.
Funding and conflicts: Funded by the Carrier under a fixed-scope statement of work. Benner House has no ties to any model vendor.
Why it matters: If a model vendor paid for a dataset, you'd want to know.

Composition

Examples: 5,800 (train 4,800 · validation 400 · test 600)
Why it matters: Counts per split, stated once and used consistently in every report.
Unit: One closed claim: its adjuster-written diary entries in, one reviewed summary out
Why it matters: Defines what a single example is, which decides how splits must be drawn.
Source pool: 61,240 closed claims with loss dates from January 2019 to December 2025
Why it matters: The sample only means something next to the population it was drawn from.
Line of business: Homeowners 4,060 (70%) · dwelling fire 690 (12%) · small commercial property 1,050 (18%)
Why it matters: Shows which kinds of claims the model has seen, and by omission which it hasn't.
Catastrophe claims: 1,972 (34%), against 22% of the book
Why it matters: Oversampled on purpose: surge-season notes are shorter and messier, and they matter most.
Hard cases: 522 reopened, 232 litigation or appraisal, 580 closed without payment
Why it matters: The cases that go wrong in production need to be in the training and test sets by design.
Lengths: Input median 1,850 tokens (95th percentile 9,400); summary median 310 tokens
Why it matters: Sets context-window requirements and flags truncation risk.
Language: English; 41 claims (0.7%) contain Spanish-language entries, kept and tagged
Why it matters: Small subgroups get their own tag so their results can be reported separately.
Split rule: Split by claim and by policy: no policy appears in more than one split. Test split frozen August 14, 2026, with its hash recorded.
Why it matters: Splitting by note instead of by claim would leak the answer into training.

Sources and rights

Source: Claims administration system, read-only replica, snapshot July 31, 2026: diary notes, cause-of-loss and status codes, paid amounts in six bands
Why it matters: Every source has a system, a snapshot date and an owner.
Permission: Carrier-owned data; internal model training only, under the master services agreement and a data-use memo approved by the Carrier's counsel
Why it matters: Rights are decided by your counsel. The datasheet records the decision and where the evidence lives.
Excluded: Recorded-statement transcripts (consent language varies); engineer reports and independent-adjuster report text (third-party terms under review, affecting 1,140 claims whose notes were kept); photos (out of scope for this version)
Why it matters: What was left out, and why, is often the most useful line in a datasheet.
Third-party content: None included after exclusions
Why it matters: A yes here would need its own license record.

Part 2: What was done to it

Preprocessing

Parsing: HTML stripped from diary entries; 214 entries with encoding errors repaired; timestamps converted to days since first notice of loss
Why it matters: Relative days keep the claim timeline without exposing real dates.
System entries: 38% of diary entries were system-generated (payment issued, letter generated, reserve changed); moved to structured events, removed from text
Why it matters: Boilerplate teaches a model to write boilerplate.
Filtering: 8,140 claims with fewer than three adjuster-written entries excluded
Why it matters: Too little input to summarize; including them rewards guessing.
Near-duplicates: 1,912 removed (MinHash, Jaccard similarity ≥ 0.85 on five-word shingles)
Why it matters: Copy-pasted notes inflate scores and overweight whichever adjuster pastes most.
Decontamination: 412 training candidates within the near-duplicate threshold of a test claim removed (new in v1.2)
Why it matters: Found during the v1.2 audit. It means v1.1's test results were somewhat optimistic, and the changelog says so.

De-identification

Method: Pseudonymization: consistent surrogate names within a claim; street addresses generalized to county; policy, claim and account numbers replaced with keyed hash tokens (key held by the Carrier); phone numbers, emails, VINs and plates removed; dates shifted per claim by 0–30 days with intervals preserved
Why it matters: Consistent surrogates keep "the insured" and "the contractor" distinguishable, which a summary needs.
Recall sample: 1,200 notes, hand-labeled by two annotators with adjudication, never used to tune the redaction rules
Why it matters: A sample used for tuning can't also measure. This one is held out, and it meets our 1,000-record minimum.
Recall: 99.3% overall (3,264 of 3,287 labeled identifiers). Names 99.6% · street addresses 98.9% · phone numbers 99.9% · policy and claim numbers 99.8% · VINs 98.4%
Why it matters: Recall is the number to ask for: it tells you what leaked. Report it by identifier type, not just overall.
Precision: 94.1%
Why it matters: Precision tells you how much harmless text was masked, which costs context but not privacy.
Misses: 23, all reviewed: 11 partial addresses, 7 VINs split across lines, 3 names in signatures with unusual spellings, 2 email fragments. Rules updated and the sample re-checked.
Why it matters: Every miss is listed in the full report, so your privacy officer can judge residual risk directly.

Labeling

Process: The base model drafts a summary; a licensed adjuster rewrites it to the rubric. 31% of drafts were rewritten from scratch.
Why it matters: Says how much of each target is human judgment and how much is model text.
Reviewers: 9 licensed property adjusters, 8 to 27 years of claims experience (median 14), identified by reviewer ID only
Why it matters: Who labeled matters as much as what. IDs let you trace any example to its reviewer without naming anyone.
Rubric: claims-summary-rubric v3: seven fields, each with definitions, examples and edge cases
Why it matters: The rubric is versioned like code; a label is only meaningful next to the rubric version it followed.
Second review: 870 examples (15%) reviewed by a second adjuster
Why it matters: Our minimum is 10% of every batch. Agreement can't be measured without overlap.
Agreement: Krippendorff's alpha on the overlap: cause of loss 0.86 · coverage-question flag 0.81 · open-items completeness 0.83 · litigation-risk flag 0.74
Why it matters: Reported per field, because an overall average hides the field that failed.
Field dropped: Litigation-risk flag removed from v1.2 targets: 0.74 is below our 0.80 threshold. Rubric revision planned for v1.3.
Why it matters: This is the threshold doing its job. A field experts can't agree on teaches noise.
Gold questions: 180 hidden gold items; reviewer mean 97.2%. Reviewer R-06 fell to 93.4% in batch 4, was removed, and their 214 examples were re-reviewed.
Why it matters: Reviewers must stay at or above 95% on gold questions. When one doesn't, their work is re-checked, not averaged in.
Rejected: 1,000 of 6,800 drafted examples rejected in review
Why it matters: A dataset with no rejections usually had no real review.

Part 3: Does it work, and where did it come from

Evaluation

Setup: Test split, 600 claims. Base: an open-weight 8B instruction model. Tuned: the same model with LoRA on the train split. Hyperparameters in the evaluation report.
Why it matters: The comparison isolates the data. Same model, same prompt, different training set.
Factual errors: Summaries with at least one unsupported statement: 18.7% base → 6.2% tuned. Adjuster-written summaries on a 200-claim subset: 3.5%.
Why it matters: The human reference shows how far the model still is from your best people.
Cause of loss: Accuracy 74.0% → 91.2%
Why it matters: The field that drives routing, so it gets its own line.
Coverage questions: Recall 0.58 → 0.83
Why it matters: Missing a coverage question is worse than raising an extra one, so recall is reported here.
Omissions: Reserve-relevant facts omitted: 22.5% → 9.1%
Why it matters: Omissions are harder to spot than errors and often more expensive.
Format: Valid structured output: 88.0% → 99.7%
Why it matters: A summary the claims system can't ingest is a failure, whatever it says.
Temporal holdout: 150 test claims from the 2025 season, after the training cutoff: factual errors 7.9% tuned
Why it matters: Checks whether the model holds up on claims newer than anything it trained on.
Judging: Adjuster panel graded a 200-claim subset; a judge model calibrated against the panel (Cohen's kappa 0.79 on factual errors) graded the other 400
Why it matters: If a model did the grading, you need to know how well it agrees with your experts.

Lineage manifest (excerpt)

Manifest ID: bh-manifest-claims-notes-sft-1.2.0
Why it matters: One ID ties the data, the datasheet and the evaluation report together.
Source hash: SHA-256 of the July 31, 2026 snapshot export
Why it matters: Lets anyone confirm later that the data came from this snapshot and no other.
Transformations: Nine steps across the Six Gates, each with tool name, version, parameters and record counts in and out
Why it matters: Every number in this datasheet can be traced to the step that produced it.
Reviewer roster: Reviewer IDs, roles, batches worked and gold-question scores
Why it matters: Accountability without exposing names.
Location: The Carrier's cloud tenant; no copy held by Benner House after close
Why it matters: Where the canonical version lives, so nobody trains on a stale copy.
Approvals: Annotation lead September 16, 2026; Carrier data owner September 18, 2026
Why it matters: A dataset ships when its owner signs, not when the vendor finishes.

Uses

Intended: Fine-tuning the Carrier's internal claim-summary models for desk adjusters; regression tests for new model versions on the test split
Why it matters: Narrow intended uses make out-of-scope uses easy to recognize.
Also permitted: Evaluating third-party summarization tools on the test split only; vendors never receive the training split
Why it matters: Lets you compare vendors on your own data without handing them your training set.
Out of scope: Coverage decisions or claim payments without adjuster review; fraud or special-investigation scoring; underwriting or pricing; auto, marine or liability lines; training any model for any party other than the Carrier
Why it matters: Written down so the next team doesn't stretch the dataset to a job it wasn't built or tested for.

Maintenance

Owner: The Carrier's claims data science lead (role, not a named person)
Why it matters: Datasets outlive projects. Someone has to own the next version.
Refresh: Quarterly, with newly closed claims, under a managed operations retainer
Why it matters: Claims practices change; so does the language in the notes.
Test split: Frozen until v2.0
Why it matters: Changing the test set between versions makes before-and-after numbers meaningless.
Version history: v1.0 May 2, 2026 (4,000 examples) · v1.1 July 1, 2026 (catastrophe oversampling) · v1.2 September 18, 2026 (decontamination, litigation flag dropped, R-06 re-review). v1.0 and v1.1 deprecated.
Why it matters: The changelog explains why the numbers moved.
Errata: Reported to the dataset owner; fixes ship as a patch version with a changelog entry
Why it matters: Gives people who find problems somewhere to send them.
Retention: Benner House working copies deleted within 30 days of project close, with a written deletion record
Why it matters: The datasheet records where copies are, and when the ones that shouldn't exist stopped existing.

The provenance manifest behind it

The datasheet is written for people. The manifest is written for machines and auditors: a structured record of every source, transformation, reviewer and approval, stored with the data. Here is a shortened excerpt from the same fictional dataset.

{
  "manifest_version": "1.0",
  "sample": true,
  "dataset": "claims-notes-sft",
  "version": "1.2.0",
  "released": "2026-09-18",
  "location": "client tenant: s3://carrier-ml/datasets/claims-notes-sft/1.2.0/",
  "sources": [
    {
      "id": "src-claims-diary",
      "system": "claims administration system, read-only replica",
      "snapshot": "2026-07-31",
      "sha256": "9f2c41d07be3a6e58c1f0a92d4b7e61c3a58f0e2d9b14c77a0e5f3c2b18d4e41",
      "records_in": 61240,
      "permission": "client-owned; internal model training only (MSA 7.2, data-use memo DU-14)",
      "excluded": ["recorded statements", "engineer reports", "independent adjuster report text", "photos"]
    }
  ],
  "transformations": [
    { "gate": "parse", "tool": "bh-notes-extract 3.4.1", "records_out": 61240 },
    { "gate": "clean", "step": "drop system-generated entries", "share_removed": 0.38 },
    { "gate": "clean", "step": "near-duplicate removal", "method": "minhash, jaccard >= 0.85, 5-word shingles", "removed": 1912 },
    { "gate": "clean", "step": "decontaminate against test split", "removed": 412 },
    { "gate": "protect", "policy": "deid-policy-claims-v4", "recall_sample": 1200, "recall": 0.993 },
    { "gate": "label", "rubric": "claims-summary-rubric-v3", "reviewers": 9, "second_review_share": 0.15 },
    { "gate": "prove", "test_split": "frozen 2026-08-14", "report": "eval-report-claims-notes-sft-1.2.pdf" }
  ],
  "reviewers": [
    { "id": "R-01", "role": "licensed property adjuster", "batches": [1, 2, 3, 4, 5] },
    { "id": "R-06", "role": "licensed property adjuster", "batches": [1, 2, 3], "note": "removed from batch 4; work re-reviewed" }
  ],
  "approvals": { "annotation_lead": "2026-09-16", "client_data_owner": "2026-09-18" },
  "deletion": { "benner_house_copies": "within 30 days of project close", "record": "issued at close" }
}

The full manifest lists every transformation, every reviewer and the record counts in and out of each step. Together with the datasheet, it is what you would hand an auditor, a regulator or a customer asking how your model was trained. For public training-data disclosures, we prepare a summary version with sensitive fields removed.

What to ask of any vendor's datasheet

Whether we build your data or someone else does, these are the questions a datasheet should already answer.

Is the source pool stated, so the sample can be compared with it?
Are excluded sources listed, with the reason for each?
Is redaction recall measured on a held-out, hand-labeled sample, and reported by identifier type?
Is agreement reported per field, and is there a threshold that a field can fail?
Are reviewers identified by role and ID, with their gold-question scores?
Is the test split frozen, hashed and checked for contamination?
Are before-and-after results reported on the same model, prompt and test set?
Are out-of-scope uses written down?
Does the changelog explain why numbers changed between versions?

Questions about datasheets

Do we get a datasheet with every dataset?

Yes. Every delivery includes a datasheet and a provenance manifest. A provenance package for a dataset built elsewhere starts at $5,400 per dataset.

Can we publish it?

Usually a version of it. Internal datasheets name systems, counts and exclusions you may not want public. For public disclosures, such as training-data summaries required by some state laws, we prepare a shorter version and your counsel decides what it says.

What other formats do you deliver?

Alongside the datasheet and manifest we deliver Croissant metadata, so the dataset can be loaded and described by standard tooling, and a source and license register.

What if a number in the datasheet looks bad?

It stays in. A datasheet exists to record what is true about the data, including the redaction misses and the field that failed its agreement check. Hiding those would make every other number on the page less believable.

Bring us one messy data source and one question.

Tell us what the data is and what you want a model to do with it. A data engineer, not a salesperson, will reply within one business day with questions and a proposed first step.

Start a project Take the readiness scorecard

This is a sample

How to read it

Part 1: What the dataset is

01 Motivation

02 Composition

03 Sources and rights

Part 2: What was done to it

01 Preprocessing

02 De-identification

03 Labeling

Part 3: Does it work, and where did it come from

01 Evaluation

02 Lineage manifest (excerpt)

03 Uses

04 Maintenance