How to Build a Golden Set for an AI Agent

The airline domain of τ-bench, one of the most cited benchmarks for tool-using agents, has 50 tasks. When the benchmark was published, gpt-4o's pass^1 on it was 35.2%. Fifty tasks were enough to show the field something it needed to see. In January, Anthropic's engineering team made the same point for teams building their own agents: 20–50 simple tasks drawn from real failures is a great start.

The count is the easy part. The hard parts are choosing the fifty, writing each one so it can be graded without argument, and keeping the set honest as the agent, its tools and your policies change.

What fifty tasks can and can't tell you

Start with the statistics, because they decide how you use the set. If an agent passes 40 of 50 tasks, the 95% confidence interval on its true pass rate runs from roughly 67% to 89%. Fifty tasks will catch a broken release. They won't tell you whether a prompt change moved you from 80% to 83%.

Two techniques get more out of a small set.

Compare versions on the same tasks. Run the old and new agent on all fifty and count only the tasks whose result changed. If the new version fixes 10 and breaks 1, a two-sided sign test gives p ≈ 0.012, which is a real improvement. If it fixes 6 and breaks 3, p ≈ 0.51, which is noise, however good the headline number looks.

Run every task several times. Agents are not deterministic, and a task that passes once may fail on the next run. τ-bench introduced pass^k, the probability that all k trials of a task succeed, and found the strongest function-calling agents had a pass^8 under 25% in its retail domain. With n trials and c successes per task, pass^k is the mean over tasks of C(c, k) / C(n, k). Fifty tasks at eight trials each is 400 episodes, which is cheap next to one bad week in production. Report pass^1 and pass^k side by side. A task that passes 5 of 8 times is not passing.

Where the fifty come from

A golden set built from whatever was easy to write will be all happy paths. Build it from slices on purpose:

Slice	Tasks	Where to find them	What it catches
Real failures	20	Escalations, complaints, QA reviews, incident tickets	The mistakes you have already paid for
High-volume paths	12	The top intents by volume	Regressions on the work that matters most
Policy edge cases	10	Policy owners: thresholds, exceptions, approvals	Rule violations that cost money
Must refuse or escalate	5	Compliance and security teams	Actions the agent must never take alone
Ambiguous requests	3	Real transcripts	Whether the agent asks before acting

The failure slice is the most valuable and the hardest to collect, because failures are scattered across ticket queues and QA spreadsheets. Start a shared intake now, even before the set exists.

Anatomy of a golden task

A good task specifies a starting world, a goal, and a way to check the result that two people would apply the same way. Here is one from a policy-servicing agent at an insurer:

{
  "id": "gs-031",
  "version": 3,
  "slice": "policy_edge",
  "origin": {"type": "escalation", "ref": "QA-2025-11-0412", "redacted": true},
  "user": {
    "persona": "Policyholder with an auto and home bundle who was charged twice in November",
    "opening": "I got charged twice for November and I want the money back today.",
    "hidden_facts": ["The second payment was made by a spouse over the phone", "Wants the refund on the card, not a credit"]
  },
  "initial_state": "fixtures/gs-031/db.sql",
  "tools": ["lookup_policy", "list_payments", "issue_refund", "create_task", "send_message"],
  "expected_state": {
    "refunds": [{"payment_id": "PMT-88213", "amount": 412.18, "method": "original_card"}]
  },
  "forbidden": [
    {"tool": "issue_refund", "when": "amount > 412.18"},
    {"tool": "issue_refund", "when": "calls > 1"}
  ],
  "response_rubric": [
    "States the refund amount and the card it returns to",
    "Gives the posting window from billing policy BIL-7",
    "Does not promise same-day funds"
  ],
  "reference_solution": "transcripts/gs-031/reference.jsonl",
  "policy_refs": ["BIL-7 rev 2025-09"],
  "signed_off_by": ["billing-policy-owner", "senior-service-reviewer"]
}

Grade it in layers. First, check the end state: τ-bench does this by comparing the database at the end of the conversation with an annotated goal state, and it is the most objective check you have. Second, scan the trajectory for forbidden actions. Third, grade the message against the rubric with an expert or a judge calibrated against experts.

Resist the urge to assert the exact sequence of tool calls. Anthropic's guide puts it plainly: it is often better to grade what the agent produced, not the path it took. The forbidden list covers the path constraints that actually matter, such as one refund and never more than was paid, without failing an agent for looking up the policy before the payments.

The reference solution matters more than it looks. A recorded run that passes every grader proves the task is solvable in your environment and that the graders accept a correct answer. Tasks without one tend to be quietly broken.

Experts write the answers

Engineers can build the fixtures. The expected outcomes should come from the people who do the work: the senior claims examiner, the billing supervisor, the utilization-review nurse. Have two of them resolve each task independently before comparing notes. In our evaluation and red-teaming projects, this double resolution is the step clients are most tempted to skip and the one that finds the most problems.

Their disagreements are findings, not friction. They usually expose a policy that is ambiguous or a step that lives only in someone's head. The public record shows how common this is. When OpenAI had 93 professional developers screen 1,699 SWE-bench samples, 38.3% were flagged for underspecified problem statements, and 68.3% were filtered out in all to create SWE-bench Verified. That benchmark was built from real GitHub issues, and it still needed heavy triage. Your first draft will too.

When a task turns out to be ambiguous, choose one of two fixes. Make the policy explicit and update the task, or make "ask a clarifying question" the expected behavior. Don't leave it in the set as an unwinnable task that silently drags the score down.

Keeping the set honest

Version tasks against policy. Pin each task to the policy revision it encodes. When the policy changes, the task changes, and scores from before and after aren't comparable.
Keep it out of training. Golden tasks must never appear in prompts, few-shot examples or fine-tuning data. Hash the task text and check every training export against the hashes.
Retire saturated tasks. A task that passes 8 of 8 for three releases moves to the regression suite, and a fresh production failure takes its place.
Refresh from the failure stream monthly. Every new escalation is a candidate task.
Read transcripts. A grader that passes the agent for the wrong reason is invisible in aggregate scores. Someone should read a sample of passing and failing runs after every grader change.

This is the Prove gate of our approach: every dataset and model change ships with a before-and-after measurement on a set like this one, built with the client's experts.

The first month

Week 1: open a shared intake for agent failures and pull the last 90 days of escalations.
Week 1: get the top intents by volume from your service or ticketing data.
Week 2: sit with policy owners for two hours and list the thresholds, exceptions and approvals that cost money when broken.
Week 2: draft 50 tasks across the five slices, each with a starting state, an expected end state, a forbidden list and a short rubric.
Week 3: have two experts resolve every task independently, then reconcile and record the reasoning.
Week 3: record a reference solution for each task and confirm every grader accepts it.
Week 4: run the current agent eight times per task, report pass^1 and pass^8, and read twenty transcripts before you trust either number.

Fifty right examples: how to build a golden set for an AI agent