Embedding 50 million documents on a budget: batching, GPUs, and cost math

Embedding a large corpus looks like a GPU problem and gets budgeted like one. With a little planning, compute is the smaller line item. The expensive mistakes sit elsewhere: re-embedding everything because the chunker or the model changed, paying for padding, keeping float32 vectors in memory when int8 would do, and choosing a 7-billion-parameter embedder the retrieval task didn't need.

Below is the arithmetic for 50 million documents with every assumption stated, so you can replace ours with yours. GPU and API prices vary by contract and change often, so we compute GPU-hours and token counts and leave the rate as a variable you fill in.

Step 1: count tokens, not documents

Documents are the wrong unit. The model sees chunks, and the bill follows tokens.

Two things are already visible. Overlap adds roughly 14% to long documents (64 of every 448 new tokens are repeats). And every duplicate removed before this step is compute you never buy, which is why near-duplicate detection comes first.

Step 2: FLOPs per token, by model size

A transformer forward pass costs about 2 FLOPs per parameter per token, plus an attention term that grows with sequence length, roughly 4 × layers × sequence length × width per token. At our mean chunk of 395 tokens:

The 7B embedder costs about 22 times the large encoder per token. It has to earn that on your queries, and benchmarks won't tell you whether it does: the original MTEB study found across 58 datasets that no single embedding method dominated every task. Before the full run, embed a 1% sample with two or three candidates and score them on 200 real queries with passages your experts have judged.

Step 3: GPU-hours from the spec sheet, honestly

Read the spec sheet carefully. NVIDIA lists the L4 at 242 TFLOPS of FP16 tensor throughput and the H100 SXM at 1,979 TFLOPS of BF16, but both figures assume structured sparsity. Embedding inference doesn't use it, so plan from the dense numbers: about 121 and about 990 TFLOPS. Then assume you sustain 35% of dense peak. That is a planning assumption, not a promise; measure it on 100,000 real chunks before committing.

Your cost is GPU-hours times your hourly rate. A few rules fall out of the table:

• The large encoder on L4s is a two-day job. At 35% utilization one L4 processes about 66,000 tokens, or 167 chunks, per second. Eight of them finish in about 31 hours.
• Pick hardware by ratio, not by name. The H100's dense peak is about 8.2 times the L4's, so at equal utilization the H100 is cheaper per embedding only if its hourly rate is under about 8.2 times the L4's. The L4's modest memory bandwidth can hold its utilization below the H100's, so measure both.
• Hosted APIs scale with tokens. At 59,200 million tokens, every $0.01 of price per million tokens adds $592 to the run, and the whole amount again for each re-embedding.

If your measured utilization comes in at 20% instead of 35%, multiply the GPU-hours by 1.75. The structure of the estimate doesn't change.

Step 4: stop paying for padding

GPUs process a batch at the length of its longest member. With random batching, a third of our chunks are short tails, but nearly every batch contains a full 512-token window, so nearly every chunk is computed at 512 tokens. That's 76.8 billion token positions for 59.2 billion real tokens: about 30% more compute than the work requires.

Sort chunks by length and build batches by token budget, say 64,000 tokens per batch, rather than by count. Padding falls to a few percent, and short sequences also skip most of the attention cost, which grows with the square of length. On the large-encoder run, that's roughly 70 L4-hours saved by a sort.

Keep the GPUs fed. Parsing scanned PDFs and running OCR is usually slower than embedding, so run it as a separate stage that writes chunked text to Parquet shards. The embedding job then reads shards, tokenizes on CPU workers and does nothing but matrix multiplication on the GPU.

Step 5: storage is the bill that recurs

Compute is paid once. Vectors sit in memory every month. For 150 million chunks:

Quantization is cheap in quality for some models and expensive for others. In a Hugging Face benchmark, mxbai-embed-large-v1 kept 97% of its float32 retrieval score at int8 and 96.45% with binary vectors plus float32 rescoring. e5-base-v2 kept only 74.77% under the same binary setup, which the authors attribute to that model using only part of its embedding space. Truncating dimensions works the same way: Matryoshka training produced embeddings up to 14× smaller at the same accuracy in its authors' ImageNet classification tests, but only models trained that way tolerate truncation.

Two more numbers belong in the plan. An HNSW graph built with M = 32, a common setting, keeps about 64 neighbor IDs per vector at its base layer, roughly 256 bytes each, or 38 GB here, which is more than the binary vectors themselves. And the chunk text you return to users runs about 240 GB at roughly 4 bytes per token. Keep float32 vectors on object storage so you can rescore or re-quantize later without re-embedding.

Step 6: don't pay twice

Re-embedding is the most expensive line in most overruns, because it repeats Steps 2 to 4 in full. The usual causes are preventable.

• Read the model card before the run. e5-base-v2 expects inputs prefixed with "query: " or "passage: " and degrade without them. all-MiniLM-L6-v2 silently truncates input longer than 256 word pieces, so a 512-token chunker feeds it text it never sees. Chunk to the model's limit.
• Freeze decisions on a pilot. Model, chunker, dimensions, quantization and prefixes should all be settled on the 1% sample and the 200-query evaluation, not discovered at 40 million chunks.
• Address embeddings by content. Key every vector by chunk hash, chunker version, model revision and prefix, so an incremental update embeds only what changed.
• Shard and checkpoint. Write in shards of about a million chunks with idempotent writes, so a preempted node costs minutes, not the run.

A record that makes all of this enforceable:

{
  "chunk_id": "doc-00418823#c0003",
  "chunk_sha256": "9f2c41d07be3a8e65c1f0b94d2a7e3c8f61b5d09a4e27c3b8d1f6a0e95c4b712",
  "chunker": "section-512-64@v4",
  "model": "embed-large@rev-2026-06-12",
  "input_prefix": "passage: ",
  "dims": 1024,
  "dtype": "int8",
  "normalized": true,
  "tokens": 497,
  "source_doc_version": "2026-07-28T14:03:00Z"
}

Our computational data processing runs follow this pattern inside the client's own cloud account, and the same records feed the retrieval indexes we build in document AI and RAG pipelines.

Checklist before the full run

• Count tokens after parsing, deduplication and chunking, not documents.
• Compute FLOPs per token for each candidate model and compare them with retrieval quality on your own judged queries.
• Plan GPU-hours from dense, not sparse, peak throughput and a measured utilization.
• Batch by token budget after sorting by length.
• Separate parsing and OCR from embedding, with shards in between.
• Choose dimensions and quantization from a test on your model, and keep float32 copies on cheap storage.
• Budget index overhead and chunk text alongside the vectors.
• Follow the model card's prefixes and length limit, and chunk to that limit.
• Key every vector by content hash and pipeline versions, and checkpoint in shards.

Sources

1. Muennighoff et al., MTEB: Massive Text Embedding Benchmark (2022)
2. NVIDIA, L4 Tensor Core GPU product specifications
3. NVIDIA, H100 Tensor Core GPU product specifications
4. Shakir, Aarsen and Lee, Binary and Scalar Embedding Quantization for Significantly Faster & Cheaper Retrieval, Hugging Face (2024)
5. Kusupati et al., Matryoshka Representation Learning (2022)
6. sentence-transformers/all-MiniLM-L6-v2 model card, Hugging Face
7. intfloat/e5-base-v2 model card (query and passage prefixes, 512-token limit)
8. Douze et al., The Faiss library (2024)