Formats
- Parsed documents as JSON and Markdown with page coordinates
- Chunks and metadata as Parquet
- Index in the vector store or search engine you already run
- Report as PDF, with the full test set as CSV
Turn the scanned binders and SharePoint folders into retrieval you can measure.
We parse documents the way a careful reader would, by layout, tables, forms, and handwriting rather than as a stream of raw text. Then we chunk by the document's own structure, carry permissions through to the index, and prove retrieval quality with a test set your experts wrote.
Most RAG failures happen before the first embedding.
A retrieval system can only return what its parser understood. A two-column policy form, flattened by basic OCR, interleaves its columns line by line. A rate table loses its headers and becomes a run of numbers. A clause gets split from the exception in the next paragraph. The model then answers from the broken chunk, fluently and with a citation, and nobody notices until an adjuster or a nurse does.
Small changes at the document layer move results more than most model swaps. Anthropic reported that prepending each chunk with a short, document-aware context before embedding and keyword indexing cut top-20 retrieval failures by 49%, and by 67% once a reranker was added. The broader lesson holds for any stack: a chunk retrieves well when it knows where it came from.
So most of a pilot goes into the unglamorous part: layout, tables, forms, metadata, permissions. Then we measure retrieval on questions your own people ask.
The pilot at a glance
You provide
- One document family, such as claim files or vendor contracts, up to 50,000 pages
- Read access to where it lives: SharePoint, a document management system, a file share, or object storage
- Access control lists or group mappings for the documents in scope
- 50 to 150 real questions from the people who will use the system
We deliver
- Parsed documents with layout, reading order, and tables preserved
- Structure-aware chunks carrying document, section, page, version, and permission metadata
- Embeddings and a hybrid vector-plus-keyword index in your environment
- A retrieval test set and an accuracy report with recall@k and answer quality
Quality gates
- Low-confidence OCR pages routed to a second engine or a reviewer, never straight to the index
- Table extraction checked cell by cell on a sample of at least 200 tables
- Retrieval test set written or approved by your subject-matter experts
- Permission filters tested with accounts that should and should not see each document
Turnaround
Six weeks from access to accuracy report
Team
A document AI engineer, a retrieval engineer, an annotation lead, and a domain reviewer
Starts at
$18,500
One document family, up to 50,000 pages parsed, chunked, enriched and indexed, with a retrieval test set and an accuracy report.
What happens to a page between the scanner and the index
Each stage exists because skipping it causes a specific, recognizable failure.
| Stage | What we do | What goes wrong without it |
|---|---|---|
| Layout analysis | Detect columns, headers, footers, reading order, and regions: text, table, figure, form field, signature, stamp. | Two-column pages interleave. Running headers repeat in every chunk and crowd out the content. |
| OCR with confidence routing | OCR each region, score confidence per word, and route pages below threshold to a second engine or a human transcriber. | Misread policy numbers, dosages, and dollar amounts enter the index silently. |
| Table structure recovery | Rebuild rows, columns, merged cells, and header hierarchy. Store each table as structured data and as readable text. | A rate schedule becomes a list of numbers, and the model quotes the wrong row. |
| Forms and handwriting | Extract key-value pairs and checkbox states. Transcribe handwriting, with a reviewer on low-confidence fields. | The handwritten note in the margin, often the most important line on the page, disappears. |
| Structure-aware chunking | Split on sections, clauses, and list items. Keep tables whole. Attach the document title and parent headings to every chunk. | A clause is separated from its exception, and a chunk reads "see above" with nothing above it. |
| Metadata and ACL propagation | Carry document type, effective dates, version, source path, and the access list to every chunk. | Users retrieve documents they can't open in the source system, and superseded versions outrank current ones. |
| Embedding and hybrid retrieval | Embed chunks, index keywords with BM25 alongside, and add a reranker where the test set shows it pays. | Exact identifiers such as claim numbers, part numbers, and CPT codes fail to match on vectors alone. |
How we prove retrieval works
Retrieval quality gets its own measurement, separate from the quality of the generated answer, so you know which layer to fix.
Collect real questions
We pull questions from users, tickets, emails, and search logs. Fifty to 150 is enough to start, if they include the hard cases.
Mark the evidence
Your experts, or our reviewers with your sign-off, mark the passages that answer each question in the source documents. That marked evidence is the ground truth for recall@k.
Score retrieval and answers separately
We report recall@5, recall@20, and mean reciprocal rank for retrieval, then score generated answers for correctness and faithfulness to the retrieved text. A wrong answer with the right passage retrieved is a different problem from a wrong answer with nothing useful retrieved.
Test the edges
The set includes questions with no answer in the corpus, where the right behavior is to say so; questions a user isn't permitted to see answered; and questions where an old version of a document disagrees with the current one.
Hand over the test
The test set and scoring scripts live in your repository and run in your CI, so a new parser, embedding model, or chunk size is measured before it ships.
When a knowledge graph earns its cost
GraphRAG and related approaches extract entities and relationships into a graph and retrieve over it. They help when questions cross many documents or follow relationships: which vendors had both a late delivery and a quality hold last year, or which amendments changed the indemnity terms in contracts with the same counterparty. They also cost more to build and keep current, because entity extraction, resolution, and refresh each need their own pipeline and checks.
We build a graph when the retrieval test set shows multi-hop questions failing on hybrid retrieval, and not before. For most lookups, such as what section 7 says or what the deductible is on this policy, a well-chunked hybrid index with a reranker wins on cost and freshness.
Start with one document family
Document AI pilot
Starts at
$18,500
One document family, up to 50,000 pages parsed, chunked, enriched and indexed, with a retrieval test set and an accuracy report.
Request a scoped quoteAdditional document families, larger corpora, and incremental refresh pipelines are quoted from the per-page cost the pilot measures. All processing runs in your cloud tenant or on your premises.
Questions about document pipelines
Do we need a particular vector database or model?
No. We have no ties to any model or platform vendor. We build on the search and storage you already run where it can do the job, and if it can't, we show the test results that say so before recommending a change.
How do you handle handwriting and bad scans?
We measure them instead of hoping. Every page gets an OCR confidence score; pages below threshold go to a second engine and then to a human transcriber if needed. The accuracy report states what share of pages needed human help, which tells you what a full rollout will cost.
How do you stop users from retrieving documents they shouldn't see?
Permissions travel with every chunk, and retrieval filters on the user's groups before ranking. We test that filter with real account types, including users who should see nothing, and we re-sync permissions when they change in the source system.
Should we fine-tune a model instead of building RAG?
They solve different problems. Retrieval is for facts that change and answers that need a citation. Fine-tuning is for format, vocabulary, and decision patterns. Many production systems use both: a small fine-tuned model that reads well-retrieved context.
What happens when documents are added, edited, or deleted?
The pipeline runs incrementally. It detects new, changed, and deleted files, re-parses and re-embeds only what changed, and removes chunks from deleted or superseded versions so they stop appearing in answers.
Will this scale beyond the pilot's 50,000 pages?
Yes. The same pipeline runs on distributed workers for millions of pages, priced from the pilot's measured throughput. See computational data processing.
Keep reading
Bring us one messy data source and one question.
Tell us what the data is and what you want a model to do with it. A data engineer, not a salesperson, will reply within one business day with questions and a proposed first step.