Dataset Design for Small Domain Models

Gartner predicts that by 2027, organizations will use small, task-specific AI models at least 3× more than general-purpose LLMs. The reasons are practical. A model with 1 to 8 billion parameters is cheap to serve, fits inside your own tenant on modest GPUs, and can be fine-tuned without a lab budget.

Small models are also less forgiving. A very large model covers for a messy dataset with general competence. A 3B model learns what you show it, including the contradictions. For small domain models, the dataset is the main lever on accuracy. What follows is what moves it, for the tasks small models do well: extraction, classification, routing, and summarizing to a template.

Teach behavior; retrieve facts

The most expensive mistake is trying to fine-tune knowledge into a small model. Gekhman and colleagues found that fine-tuning examples introducing new knowledge are learned significantly slower than examples consistent with what the model already knows, and that as the model does learn them, its tendency to hallucinate increases linearly.

So split the work. Fine-tune the model to perform a task over context it is given: read this bill of lading and fill this schema; assign one of these 37 denial codes; answer from these retrieved passages or say the answer isn't there. Put facts that change, such as rates, formularies and contract terms, in retrieval. A simple rule decides most cases: if the correct output would change when a document changes, that document belongs in the prompt, and the training example should include it.

Scope narrowly

Small models do best on narrow, well-specified tasks. In LoRA Land, 4-bit LoRA fine-tunes across 10 base models and 31 tasks outperformed their base models by 34 points and GPT-4 by 10 points on average. Those were bounded tasks with clear outputs, the kind enterprises actually have.

Multi-task training works within a family, for example several extraction schemas over similar documents. Mixing open chat, extraction and classification in one 3B model forces trade-offs you'll see as regressions. On method: full fine-tuning learns weight perturbations of 10–100× higher rank than typical LoRA configurations, and the same study found LoRA learns less on code and math but forgets less outside the target domain. For a narrow task, LoRA is usually enough, and the forgetting protection is a feature. For a real domain shift, such as dense clinical shorthand or a language the base model barely saw, consider full fine-tuning or continued pretraining.

Select; don't accumulate

The literature on data selection says the same thing in several ways. LIMA was fine-tuned on 1,000 curated examples, and its answers were judged equivalent or preferred to GPT-4's in 43% of cases. AlpaGasus kept 9k examples from Alpaca's 52k, and GPT-4 graded it significantly better than the original. With LESS, a selected 5% of an instruction dataset often outperformed the full set. phi-1, at 1.3B parameters, reached 50.6% on HumanEval largely on the strength of textbook-quality training data.

For enterprise data, selection means:

Collapse templates. Cluster near-duplicates (form letters, copied-forward notes, standard dispositions) and cap each cluster, so the model doesn't learn that boilerplate is the answer.
Find contradictions. Within clusters of near-identical inputs, check that labels agree. Disagreeing labels on near-identical inputs are the single most damaging thing in a small-model dataset, because the model can't learn a rule that isn't there.
Up-weight hard cases. Run a baseline model over the pool and oversample the inputs it gets wrong, after an expert confirms the label is right.
Verify a sample. Experts review a stratified sample of the final set, not the raw pool.

In our fine-tuning dataset projects, this step takes more of the effort than any other, and most of it is curation and deduplication rather than new labeling.

Train on the inputs you'll actually see

If production inputs come through OCR, training inputs must come through the same OCR engine and version, with the same errors. If production emails arrive as forwarded chains with signatures and disclaimers, so should the training emails. A model trained on clean, hand-typed text and deployed on parser output will look fine in the notebook and fail in the queue.

The same goes for the wrapper. Use the exact chat template, system prompt and output schema in training that you will use at inference. Template mismatches are a common cause of a fine-tune that "didn't work."

Teach abstention explicitly

Small models fill gaps. Ask for a part number that isn't in the work order and a 3B model will produce one in the right format. The remedy is examples where the correct output is null, "not stated," or "escalate," at roughly the rate those cases occur in production, or a little higher:

{"messages": [
  {"role": "system", "content": "Extract to schema wo-v3. Use null for any field the text does not state. Never infer part numbers."},
  {"role": "user", "content": "WO 55120 | Unit 14 conveyor | Tech notes: replaced drive belt, adjusted tension. Bearing noise persists, recommend follow-up. Parts: belt (see PO)"},
  {"role": "assistant", "content": "{\"work_order\": \"55120\", \"asset\": \"Unit 14 conveyor\", \"actions\": [\"replaced drive belt\", \"adjusted tension\"], \"parts\": [{\"description\": \"drive belt\", \"part_number\": null}], \"follow_up_required\": true, \"follow_up_reason\": \"bearing noise persists\"}"}
],
 "meta": {"source": "cmms-export-2025q3", "split": "train", "template_cluster": "c-118", "verified_by": "maintenance-sme-02"}}

The metadata matters as much as the messages. Source and cluster IDs let you trace a bad behavior back to the examples that taught it.

Design the mix, then read the learning curve

A starting mix for a single-task fine-tune, to be adjusted against your evaluation set, not treated as a rule:

Component	Share	Purpose
Representative production examples	55–65%	Common cases at their real frequency
Hard cases	15–25%	Inputs a baseline gets wrong, rare classes raised to a floor
Abstain or escalate	5–15%	Missing fields, out-of-scope requests
Edge formats	about 5%	Very long inputs, unusual layouts, several entities in one record

Then run the learning-curve test before buying more labels. Train on 25%, 50% and 100% of the data and evaluate each on the same held-out set. If accuracy is still climbing steeply between 50% and 100%, more data of the same kind will help. If it has flattened, more volume won't. The problem is label quality, missing hard cases, or scope. Plot the curve per class, too.

An illustrative composite: a 37-code denial classifier reaches macro-F1 of 0.71 at 25% of 6,000 examples, 0.78 at 50%, and 0.80 at 100%. The overall curve is flat. The per-class curves show six rare codes with fewer than 20 examples each, scoring near zero. Forty expert-labeled examples per rare code is a cheaper fix than another 6,000 of everything.

Hold out by entity and time

Evaluate on records the model can't have memorized: different customers, assets or claims than the training set, and ideally a later time period. Check near-duplicate clusters across the split. Compare the fine-tune against the base model and a large general model given the same prompt and retrieval, on a golden set your experts trust. If the small model doesn't win on the measures that matter, it isn't ready, however good the training loss looks.

Failure modes

Unverified distillation. Training on a larger model's outputs without checking them copies its errors at scale.
Majority-class collapse. With a skewed label distribution and no hard-case weighting, the model learns the top three classes.
Renamed codes. Label sets that changed over the years teach two names for one thing. Map them before training.
Leakage through templates. The same form letter on both sides of the split inflates every metric.

What to do this quarter

Write the task as input, output schema and abstention rule, and move changing facts into retrieval.
Route training inputs through the production parser and the production prompt template.
Deduplicate, cap template clusters and resolve contradictory labels before training anything.
Build the mix with explicit hard-case and abstention shares, and keep source metadata on every example.
Run the 25/50/100 learning curve, per class, before commissioning more labeling.
Hold out by entity and time, and compare against the base model and a large model on the same golden set.

Dataset design for 1B–8B domain models: what actually moves accuracy