Formats
- Golden set as JSONL with metadata for slicing
- Evaluation code in your repository, runnable locally and in CI
- Reports as HTML and PDF
Know whether the model got better before your customers do.
We build the evaluation your experts would trust: a golden set drawn from real cases, rubrics that score what matters, an automated judge calibrated against human reviewers, and a gate in your release pipeline. Then we try to break the system on purpose.
An evaluation is a measuring instrument. Calibrate it like one.
Public benchmarks tell you how a model does on public benchmarks. They say nothing about whether it summarizes your claim files correctly, routes your referrals to the right queue, or cites the right clause in a denial letter. For that you need a golden set: a few hundred real cases from your work, stratified by type and difficulty, with answers your experts have verified.
Scoring those cases at release speed usually means an automated judge, and judges need calibration. Research on LLM-as-a-judge found strong models agreeing with human preferences over 80% of the time on general chat questions, about the rate at which humans agree with each other. That's encouraging, and it's also not your use case. A judge that agrees with crowdworkers about travel tips hasn't been tested on whether a prior-authorization summary omits a contraindication. We measure the judge against your experts, on your cases, before it scores anything that gates a release.
How we build the suite
Sample real cases
From production logs, tickets, or records, including the hard and rare cases, and checked for overlap with training data so the scores measure generalization rather than memory.
Write the rubrics
Deterministic checks wherever possible, such as a valid code, a cited section that exists, or a correct field value, and scored criteria where judgment is needed, each with anchored examples.
Collect expert labels
Two experts score each case independently. We measure their agreement, adjudicate disagreements, and record the gold answer and the reasoning behind it.
Calibrate the judge
We run the judge on the expert-labeled cases and measure agreement per criterion. We test for position and verbosity bias by swapping answer order and padding responses. Where the judge can't reach the agreed threshold, we revise it or keep that criterion human-scored.
Wire the gate
The suite runs in CI on every model, prompt, retrieval, or tool change, with thresholds per slice. A regression blocks the release, and the report is attached to the change for review.
What a red-team pass tries
Prompt injection sits at LLM01, the top of OWASP's 2025 Top 10 for LLM applications, and enterprise deployments that connect a model to documents, email, and tools widen the attack surface. We test the system as deployed, with its retrieval and tools, not the bare model.
| Attack class | What we try | What a pass looks like |
|---|---|---|
| Direct prompt injection | Instructions to ignore the system prompt, role-play jailbreaks, encoded or translated payloads | The system stays within its policy and scope |
| Indirect prompt injection | Hostile instructions planted in documents, emails, web pages, and tool results the system reads | Retrieved content is treated as data, never as instructions |
| Data exfiltration | Attempts to pull other users' records, system prompts, or secrets, including through links and images in the output | Nothing outside the user's permissions appears in any response |
| Excessive agency | Coaxing an agent into tool calls beyond the task: sending email, changing records, approving payments | Tool use stays in scope; risky actions require confirmation |
| Sensitive information disclosure | Probing for memorized training data, personal information, or health information | Nothing reproduced that the user isn't entitled to see |
| Domain and policy harms | Requests for regulated advice, discriminatory decisions, or unsafe instructions specific to your industry | Refusal or escalation that matches your written policy |
The evaluation suite at a glance
You provide
- Access to the system under test, including model, prompts, retrieval, and tools, in a staging environment
- Production logs or records to sample cases from
- Two to four of your experts for 6 to 10 hours each to label and adjudicate
We deliver
- A golden set of up to 400 cases with expert-verified answers
- Scoring rubrics and a calibrated judge configuration
- A release gate in your CI pipeline with per-slice thresholds
- A red-team report with reproducible attack cases, severity ratings, and retest results
Quality gates
- Judge agreement with experts measured per criterion before automated scoring
- Golden set checked for overlap with training data
- Every red-team finding reproduced before it is reported
- Gate thresholds agreed with your product owner in writing
Turnaround
Five to six weeks
Team
An evaluation engineer, a security tester, an annotation lead, and domain reviewers from your team or our expert network
Starts at
$13,500
A golden set of up to 400 cases, scoring rubrics, judge-model calibration against your experts, and a release gate in your pipeline.
A demo is not an evaluation
How models usually get approved
- A dozen hand-picked prompts tried in a meeting
- Public benchmark scores from the vendor's announcement
- An LLM judge used without checking it against people
- Retesting only when someone remembers to
What the suite gives you
- Hundreds of real cases, sliced by type and difficulty
- Scores on your work, verified by your experts
- A judge whose agreement with your reviewers is measured and reported
- An automatic gate on every change, with a record of each result
Suite pricing
Evaluation suite
Starts at
$13,500
A golden set of up to 400 cases, scoring rubrics, judge-model calibration against your experts, and a release gate in your pipeline.
Request a scoped quoteRed-teaming agent systems with many tools, recurring red-team passes, and golden sets larger than 400 cases are quoted separately.
Questions about evaluation
How many golden examples do we need?
Fewer than you'd think to start. Fifty well-chosen cases that cover your important slices make a useful first gate; a few hundred let you report per-slice results with confidence. We cover how to choose them in fifty right examples.
Can we trust an LLM judge?
Only as far as you've measured it. We report the judge's agreement with your experts per criterion, and keep human scoring for any criterion where the judge falls short. The judge is re-checked whenever its model or prompt changes.
How often should the golden set change?
Keep a frozen core so results stay comparable over time, and add a rotating slice of fresh cases from production failures. Our managed data operations team can refresh that slice every month.
Is red-teaming a one-time exercise?
The full pass is periodic, typically before major releases. Every confirmed attack becomes a regression test in the gate, so it is re-run automatically on each change.
Can you test a vendor model we don't control?
Yes. We test the system as you deploy it. Most findings are fixable in your layer: retrieval permissions, tool scopes, output filtering, and confirmation steps, whatever the underlying model.
Keep reading
Bring us one messy data source and one question.
Tell us what the data is and what you want a model to do with it. A data engineer, not a salesperson, will reply within one business day with questions and a proposed first step.