The vocabulary of training data, translated.

A

Active learning

A labeling strategy in which the model chooses which examples humans label next, usually the ones it is least certain about. It spends expert time where it teaches the most, but it can under-sample rare cases the model doesn't know it is missing, so we pair it with stratified random samples.

Adjudication

Resolving a disagreement between annotators. A senior reviewer decides the final label and records why, and the reasoning often becomes a new example in the annotation guideline.

Agent trajectory

The full record of an AI agent working a task: the instruction, each step of reasoning or planning, every tool call with its arguments and result, and the outcome. Expert-reviewed trajectories become training data and test cases for agents. See Agent Trajectories & RL Environments.

Annotation guideline

The written instructions annotators follow: label definitions, decision rules, worked examples, and how to handle edge cases. Most disagreement between experts traces back to a gap in the guideline, so guidelines are versioned and revised between calibration rounds.

B

Business associate agreement (BAA)

A contract HIPAA requires when a covered entity, such as a hospital or health plan, has a vendor perform work involving protected health information on its behalf. It sets how the vendor may use, safeguard, and report on that information.

C

Calibration round

A session before production in which annotators label the same items, compare answers, and resolve differences, so that everyone reads the guideline the same way. Calibration is repeated whenever agreement drops or the guideline changes.

Chunking

Splitting documents into passages for retrieval. Chunk boundaries decide what a retriever can find: a chunk that separates a table from its header, or a clause from its defined terms, returns the right page and the wrong answer. Good chunking follows the document's structure rather than a fixed token count.

Cohen's kappa

An agreement statistic for exactly two annotators labeling the same items, corrected for the agreement expected by chance. With more than two annotators or missing labels, Krippendorff's alpha is the usual choice.

Croissant

A metadata format for machine learning datasets, developed through MLCommons, that describes a dataset's files, fields, and responsible-AI information in JSON-LD so tools can load and document it consistently. We deliver Croissant metadata with every dataset.

D

Data contamination

Overlap between training data and an evaluation set. It inflates scores because the model has, in effect, seen the test. Decontamination finds and removes exact and near-duplicate overlaps before training, not after the results look too good.

Datasheet (data card)

A structured document that travels with a dataset and records why it was created, what it contains, how it was collected and processed, its known gaps, and its intended uses. The idea comes from the paper "Datasheets for Datasets" by Gebru and colleagues. See our sample datasheet.

De-identification

Removing or transforming information so that records can no longer reasonably be linked to a person. Under HIPAA, health information counts as de-identified only through the Safe Harbor or Expert Determination methods; elsewhere, the standard is set by the applicable law or contract.

Deduplication

Removing exact and near-duplicate records. Duplicates over-weight whatever is repeated, usually templates and boilerplate, and they let copies of test items slip into training data.

Direct preference optimization (DPO)

A method for training a model directly on pairs of responses labeled preferred and rejected, without first training a separate reward model. It is only as good as the consistency of the preference labels it learns from.

Distillation

Training a smaller model on outputs from a larger one so the small model imitates the larger model's behavior on a task. Distilled data needs the same filtering and evaluation as human-written data, because it carries the larger model's mistakes with it.

E

Embedding

A list of numbers that represents a piece of text, an image, or audio so that items with similar meaning sit close together. Embeddings power semantic search, retrieval for RAG, clustering, and near-duplicate detection.

Evaluation set

A fixed, versioned set of inputs with expected outputs or scoring rubrics, kept out of training, used to measure whether a change to a model or dataset made things better. Every Benner House delivery is measured against one. Often shortened to eval set.

Expert Determination

One of HIPAA's two de-identification methods. A person with appropriate statistical and scientific expertise determines that the risk of identifying individuals from the data is very small and documents the methods and results. It can keep more detail in the data than Safe Harbor allows.

F

Few-shot prompting

Putting a handful of worked examples into a prompt to show a model the task. It needs no training, which makes it the baseline any fine-tuned model should beat.

Fine-tuning

Continuing to train an existing model on a smaller, task-specific dataset so it adopts a format, vocabulary, or behavior. For most enterprise tasks, the quality and coverage of that dataset matter more than the training recipe. See Fine-Tuning Dataset Engineering.

G

Gold question

An item with an agreed answer mixed, unannounced, into an annotator's queue. Accuracy on gold questions is a running check on each reviewer; our standard is 95% or better.

Golden set

A small, carefully adjudicated set of examples whose correct answers your experts agree on, used as the reference for evaluation. It is chosen to cover the hard and rare cases, not just the common ones. See Fifty right examples.

GraphRAG

A retrieval approach that extracts entities and relationships from documents into a knowledge graph and retrieves along those connections. It helps with questions that span many documents, such as every contract that references a particular supplier.

Ground truth

The answer accepted as correct for an example, established by the best available evidence: a recorded outcome, an adjudicated expert decision, or a measurement. For most business tasks, ground truth is a judgment, so how it was established should be documented.

H

Hallucination

A model output that is fluent and confident but unsupported by its sources or by fact, such as a policy clause that doesn't exist. It is measured with evaluation sets that have known answers and reduced with better retrieval, grounding checks, and training data.

I

Inter-annotator agreement

How consistently independent annotators assign the same label to the same item, corrected for chance. Low agreement signals an unclear guideline or a genuinely ambiguous task, and it limits how reliably any model trained or tested on those labels can be measured.

K

Krippendorff's alpha

An inter-annotator agreement statistic that works with any number of annotators, missing labels, and nominal, ordinal, or interval data. A value of 1 means perfect agreement and 0 means agreement no better than chance. We hold categorical labels to an alpha of at least 0.80 before a batch ships.

L

LLM-as-a-judge

Using a language model to score or compare outputs against a rubric. It makes evaluation cheap to scale, but it must be calibrated against expert scores on the same items before anyone trusts its numbers. See Calibrating LLM-as-a-judge.

M

MinHash

A technique that estimates how similar two documents are from short signatures of their overlapping word or character sequences. Combined with locality-sensitive hashing, it finds near-duplicates across millions of documents without comparing every pair. See Data Curation & Deduplication.

Model collapse

The degradation that occurs when models are trained repeatedly on data generated by earlier models: rare cases disappear first, and outputs drift toward bland, repetitive averages. Validating synthetic data against real distributions guards against it.

O

OCR

Optical character recognition: converting images of text, such as scanned pages and faxes, into machine-readable characters. Useful pipelines pair OCR with layout analysis so that tables, columns, checkboxes, and handwriting keep their structure.

P

PHI

Protected health information: under HIPAA, individually identifiable health information held or transmitted by a covered entity or its business associate, in any form or medium. It includes demographic, diagnosis, treatment, and payment information that can be tied to a person.

PII

Personally identifiable information: information that identifies a person directly, such as a name or Social Security number, or indirectly in combination, such as a birth date with a ZIP code. Its exact definition varies by law and by contract.

Preference data

Pairs or rankings of model responses labeled by which is better, often with a short rationale, used to train reward models or to optimize models directly with methods such as DPO. See Expert Annotation & Preference Data.

Provenance

The documented origin and history of data: where each record came from, under what rights or license, what was done to it, and by whom. Provenance is what lets you answer a regulator, a customer, or your own counsel when they ask what a model was trained on.

Provenance manifest

Our name for the machine-readable record that ships with every Benner House delivery: sources, licenses, transformations, reviewers, quality results, and version. See Data Provenance & AI Compliance.

Pseudonymization

Replacing direct identifiers with consistent substitutes, such as a token or a realistic stand-in name, so that records about the same person still link together. Whoever holds the key can reverse it, so pseudonymized data is generally still treated as personal data.

Q

Quasi-identifier

A field that doesn't identify a person by itself but can in combination with others: ZIP code, birth date, sex, occupation, a rare diagnosis. Much of the re-identification risk in a dataset sits in quasi-identifiers rather than in names.

R

RAG

Retrieval-augmented generation: a system that retrieves relevant passages from a document collection and gives them to a language model to ground its answer. Its accuracy is capped by parsing, chunking, and retrieval quality before the choice of model matters. See Document AI & RAG Data Pipelines.

Recall@k

The proportion of relevant passages that appear in a retriever's top k results, averaged over a test set of questions. For RAG, it is the first number to check: if the right passage isn't retrieved, the model can't use it.

Red-teaming

Deliberately trying to make a model fail by eliciting unsafe, wrong, biased, or confidential output with adversarial prompts and scenarios. Each finding becomes a new evaluation case and, often, new training data. See Evaluation, Benchmarks & Red-Teaming.

Redaction recall

The share of sensitive items a redaction process actually catches, measured against a hand-labeled sample. For privacy, recall matters more than precision: a missed name is a disclosure, while an extra redaction is an inconvenience. We measure it on at least 1,000 hand-labeled records.

Reward model

A model trained on human preference data to predict which of two responses people would prefer. It supplies the training signal in RLHF, so its blind spots tend to become the trained model's habits.

RL environment

A sandbox that reproduces a workflow, with its tools, seeded data, and a verifier that checks whether the task was done correctly, in which an agent can be trained by reinforcement learning or tested without touching production systems.

RLHF

Reinforcement learning from human feedback: training a model to produce responses people prefer, typically by training a reward model on preference data and then optimizing the model against it.

Rubric

A scoring guide that breaks the quality of an answer into named criteria, each with defined levels and examples, so that different experts score the same answer the same way. See Rubrics that reduce expert disagreement.

S

Safe Harbor

HIPAA's prescriptive de-identification method: remove 18 listed types of identifiers of the individual and of relatives, employers, and household members, and have no actual knowledge that what remains could identify the person. Names, dates more specific than the year, and most geography below the state level are on the list.

SFT

Supervised fine-tuning: training a model on prompt-and-response pairs that demonstrate the desired behavior. It is usually the first stage of adapting a model, before any preference training.

Six Gates

Benner House's delivery method, in order: Source, Parse, Clean, Protect, Label, Prove. Each gate has written exit criteria that a batch must meet before it moves on. See our approach.

Synthetic data

Data generated by a model or simulation rather than collected from real events. It fills gaps and protects privacy only when validated for fidelity to the real distribution, usefulness for the downstream task, and leakage of real records. See Synthetic Data Generation & Validation.

T

Tool use

A model's ability to call external functions, such as a search, a database query, or a claims-system API, by producing structured arguments and reading the results. Also called function calling. Agent training data records these calls step by step.

V

Vector index

A data structure that stores embeddings and quickly finds the ones nearest a query, usually approximately, using methods such as HNSW or IVF. Its settings trade recall against speed and memory, so they should be tuned against a retrieval test set.

Verifier

Code or a model that checks whether an output or an agent's actions are correct, such as confirming a claim was routed to the right queue or a calculated reserve matches policy. Verifiers make a workflow something you can train and score automatically.