Building the eval stack in the right order

TL;DR

The full eval and observability stack — golden sets, regression gates, calibrated LLM-as-judge grading, full production tracing, drift detection, trajectory evals for agents — is a lot to build at once, and trying to build all of it before shipping anything is its own kind of failure. There's a real order to this, and it starts smaller than most roadmaps assume: read real traces before building anything to grade them automatically, turn the recurring failures you find into a small golden set, gate changes on it, then add calibrated automated judging to scale past what a human can review by hand, and only then invest in full production observability as the system's traffic and stakes grow. Each stage earns the next one. Skipping ahead produces expensive tooling nobody trusts, because nobody did the reading that would have told them what to measure.

🎯 For the product leader

Why it matters — Most eval investments fail not because the techniques don't work, but because they were built in the wrong order — sophisticated scoring infrastructure for failure modes nobody had actually identified yet.

What it changes in your decisions — Where the team spends its first weeks of eval investment, and what "we have evals" is actually allowed to mean at each stage of a feature's maturity.

Ask yourself — "Has anyone on this team actually read fifty real traces end to end, or did we go straight to building a scoring pipeline?"

Risk if ignored — A team builds an impressive-looking eval dashboard that measures the wrong things, because nobody did the unglamorous work of reading real failures before deciding what to score.

The mental model: each stage earns the next

Each box only pays off once the one before it exists. A golden set built without first reading real traces measures whatever the team guessed mattered, not what actually fails. A calibrated judge built before a golden set has nothing representative to calibrate against. Full production observability, built before anyone has a working regression gate, produces a beautiful dashboard tracking a system nobody can yet tell is regressing or improving. The order is the discipline — not any single stage in isolation.

Where to actually start: reading, not scoring

The first real step is qualitative, not a metrics project: pull real traces — a random sample, plus anything a user flagged — and read them one by one, labeling what went wrong in plain language, in the team's own words, without forcing the labels into a taxonomy yet. Only once a pile of these free-form labels exists does clustering them into a handful of named, recurring failure modes make sense. Hamel Husain's evals FAQ gives a working size: read at least 30 traces yourself first, then at least 100 overall, and keep going while new failure types still appear. This unglamorous reading is what tells a team what's actually worth building an automated evaluator for, and it's also the step most commonly skipped in the rush to have a dashboard to show. The full method — what the discipline calls open coding and axial coding — is developed in complete depth in Evals.

Growing from a golden set to a full production practice

Once a handful of recurring failure modes are named, they become a small, version- controlled golden set: real inputs, paired with what a good output looks like, that a team can run before every change ships and gate the release on. From there, the stack grows to match the system's stakes and scale, not before it: LLM-as-judge grading takes over from all-human review once volume outgrows what people can read by hand, calibrated against those same human judgments so it can be trusted; full tracing and drift detection earn their cost once the system is live and quality can decay in ways no single deploy caused; and if the system is agentic, trajectory evals — grading the path a run took, not just its final answer — become necessary once a single mistake early in a long run can compound into a failure that only shows up at the end. All of this is developed in complete engineering depth in Observability and Reliability & evals. The mature end state — where no change ships without a score, because the eval set has effectively become the spec — is the operating discipline developed in TPM for AI products.

Choices that make the stack trustworthy

• Prefer pass/fail over 1-5 scales. Binary labels are easier to agree on and easier to validate. To track gradual progress, use several specific pass/fail checks, not one vague score. OpenAI's guide also says models grade pass/fail and pairwise choices more reliably than open-ended scores.
• One judge per failure mode. Split the human labels into train, dev and a held-out test set. Measure the judge's true positive rate (it catches real failures) and true negative rate (it passes good outputs). Track both, because a judge that flags everything has a perfect first number and a useless second one.
• Know the judge's biases. The MT-Bench paper (Zheng et al., 2023) documents position, verbosity and self-enhancement bias, and found that strong judges agree with human preferences over 80% of the time. A later study (Panickssery et al., 2024) found that models which better recognize their own output favor it more. Do not let a model family judge itself without a human check.
• Split the suite in two. Anthropic separates capability evals, which start with a low pass rate and give the team a hill to climb, from regression evals, which should sit near 100%. A capability eval that the system now passes "graduates" into the regression suite. When every task passes, the eval is saturated and no longer shows progress.
• Gate on a pass-all rate. An example passes only if it passes every check. Report one pass-all rate for checks that block a release and a second for issues the team accepts.
• Grade the outcome, not one exact path. Anthropic warns that rigid step-sequence checks punish valid solutions. For agents, grade the result, then add budgets (steps, cost, time) and look at the first wrong step in failing traces.
• Keep reading after launch. Re-run error analysis on a fresh sample of 100 or more traces every few weeks, and always after an incident. Keep some random traces in every batch so you still see failures your alerts miss.
• Treat synthetic test data with care. It can widen coverage, but it cannot show how common a failure is in production. Check it against real traces as soon as you have them.

Failure modes

• Dashboards before reading — investing in scoring infrastructure before anyone spent the hours reading real traces that would have revealed what to measure.
• A judge with nothing to calibrate against — standing up automated LLM grading before a human-reviewed golden set exists to check it against.
• Observability before a working gate — full production tracing built before there's even a regression suite to tell whether a change should have shipped.
• A judge nobody re-checks — the judge was validated once, then the model, prompt or traffic changed and its agreement with humans quietly fell.
• Trajectory evals skipped on an agentic system — grading only a long-running agent's final answer, missing the early step that actually caused the failure.

Practitioner checklist

• Has the team read real production traces and labeled failures in plain language, before building anything to score them automatically?
• Does a golden set exist that came from those real failure modes, not from a guess at what might matter?
• Is any automated judge calibrated against human-labeled examples before it's trusted to grade at scale?
• Are the judge's true positive and true negative rates measured on a held-out set of human labels, and re-measured when the model or prompt changes?
• Is there a regression suite near 100% that gates release, separate from capability evals that still have room to improve?
• If the system is agentic, does anything grade the path a run took, not just whether it reached the right answer?

Related lessons

• Why eval investment is the job — the case for making this investment before a team feels ready for it.
• Evals — the full depth on error analysis, golden sets, regression tests, and LLM-as-judge.
• Observability — the full depth on traces, spans, and drift detection in production.
• Reliability & evals — the full depth on trajectory evals for agents.