Evals & observability

TL;DR

Evals are the test suite for a non-deterministic system. You can't assert exact equality, so you build different tools instead. Golden sets are curated inputs with known-good outputs. Regression tests run those golden sets in CI to block quality drops. Adversarial tests cover the ways you expect the system to break. LLM-as-judge scores open-ended outputs, calibrated against human evals. Without evals you're flying blind: every prompt tweak, model swap, or quantization change becomes a coin flip, and regressions ship silently.

🎯 For the AI-native PM

Why it matters — Evals are the difference between "we think it got better" and "we know." For an AI-native PM, the eval set is the product spec. It encodes what "good" actually means.

What it changes in your decisions — Your release gates, how you compare model options, and how you prove improvement to stakeholders.

Ask your eng team — "What's our regression eval, and does it run before every model or prompt change ships?"

Product risk if ignored — A model update silently regresses 5% of cases and you hear about it from users, not from CI.

Mental model

Traditional tests assert f(x) == expected. LLM outputs are variable and open-ended, so exact-match doesn't work. Instead you grade quality over a representative distribution:

• For closed tasks (classification, extraction, structured output) you can use exact/structural checks.
• For open tasks (summaries, answers, code) you grade against a rubric with a judge (model or human). Track aggregate scores and regressions, not single outputs.

Evals turn "it feels better" into a number you can gate on.

Error analysis — look at your traces first

Before you choose metrics or build graders, practice error analysis. This discipline, borrowed from qualitative research, turns raw production behavior into an eval suite that measures your failures, not generic ones:

1. Create a dataset. Pull real traces: random samples plus anything flagged by users, cheap screens, or your own suspicion.
2. Open coding. Read traces one by one and label failures free-form, in your own words — "ignored the date constraint," "cited the wrong doc." Don't build a taxonomy yet.
3. Axial coding. Cluster those labels into a failure taxonomy: the five or ten recurring modes that actually matter.
4. Iterate. Build automated evaluators only for the recurring, high-impact modes. Re-run the analysis on fresh production traces regularly, because the taxonomy rots as users find new ways to use — and break — the product.

This is also the minimum viable eval setup. A spreadsheet of traces, pass/fail labels, and notes beats a metrics dashboard you haven't validated. Most teams over-invest in scoring infrastructure and under-invest in the hours of reading traces that tell you what's worth scoring.

Golden sets — the backbone

A golden set is a curated, version-controlled collection of inputs paired with known-good outputs or acceptance criteria.

• Representative. Mirror real production traffic — common cases and the long tail, per feature and per tenant segment where behavior differs.
• Labeled with intent. Store why an output is correct (criteria/rubric), not just the string, so judges and humans can grade consistently.
• Living. Grow it from production. Every incident and bug becomes a new golden case, so the same regression can't recur. This is how evals compound in value.
• Stratified. Tag cases by capability (math, extraction, refusal, multi-hop) so you see where quality moves, not just an aggregate.

Regression tests — gate every change

Run the golden set automatically before prompt, model, retrieval, or infra changes ship, and block the change on score drops.

• Catches the silent killers: a model upgrade that helps 95% and breaks 5%, a prompt edit with side effects, a quantization step that hurts strict-format tasks.
• Report per-stratum deltas, not one number. A flat average can hide a category falling off a cliff.
• This is the CI gate referenced throughout the curriculum (structured output, retrieval evals, routing portability).

Adversarial tests — break it on purpose

Beyond "does it work," test "how does it fail":

• Prompt injection / jailbreaks — does retrieved or user content hijack behavior? (see safety engineering).
• Edge inputs — empty, huge, malformed, multilingual, contradictory, out-of-scope.
• Hallucination bait — questions whose answer isn't in context. Does it say "I don't know," or does it fabricate an answer? (grounding).
• Format stress — does structured output stay valid under weird inputs?
• Tool misuse — does it call dangerous tools when goaded? Adversarial cases join the golden set and run in regression.

LLM-as-judge — scoring at scale

Use a model to grade outputs against a rubric — correctness, grounding, helpfulness, style. It's powerful, but it has failure modes you must control:

• Calibrate against humans. Establish agreement between judge and human labels before you trust it, and re-check periodically.
• Known biases: position bias (favoring the first option), verbosity bias (longer = better), self-preference (favoring its own family), and leniency. Mitigate them with randomized order, pairwise comparison, clear rubrics, and reference answers.
• Prefer comparative grading (A vs. B) over absolute scores where possible. It's more reliable.
• Don't judge with the same model under test for safety-critical evals. A separate judge avoids blind spots.
• The judge is itself a system to evaluate — its accuracy is a metric you monitor.

Human evals — the ground truth

The anchor everything else calibrates to.

• Use for: establishing rubrics, calibrating judges, ambiguous/subjective quality, high-stakes decisions, and sampling production for blind spots.
• Make it rigorous: clear guidelines, multiple raters, inter-rater agreement, blind/randomized presentation to avoid bias.
• Expensive and slow, so spend it where it matters most and let calibrated judges scale the rest.

A vocabulary note for eng conversations: public agent benchmarks became the industry's shared shorthand. SWE-bench Verified covers real GitHub issues resolved end-to-end. GAIA covers general assistant tasks. OSWorld covers computer use. tau-bench covers tool-using customer-service agents. Terminal-Bench covers command-line work. Know what each measures, so "we're at X%" means something to you. Know the caveat too: public benchmarks saturate and leak into training data. That's why serious teams treat them as market signals and run private evals (this lesson) as the real gate. On the observability side, OpenTelemetry's GenAI semantic conventions are standardizing trace formats. That means eval and observability tooling is becoming portable across vendors, not a lock-in decision.

Grading choices that keep evals honest

• Binary beats Likert. Prefer pass/fail judgments over 1–5 scales. Forced decisions are more consistent across annotators, easy to aggregate, and map directly to a launch bar. A 3.7 average hides exactly the disagreement a pass/fail split exposes.
• Skip generic similarity metrics. BERTScore, ROUGE, and off-the-shelf "quality" metric packs correlate poorly with what your users consider good. A custom evaluator per failure mode, built from your error analysis, is worth a dozen ready-made scores.
• Don't automate everything. An evaluator for every failure mode you've ever seen becomes maintenance debt. Automate the recurring, high-impact modes, and keep the long tail in periodic human review.
• Annotation is product judgment — keep it in-house. Outsourced labelers apply their judgment of good, not yours, and the learning — what's failing and why — leaves with the vendor. PM and eng should label traces together. Disagreements between them are requirement discoveries, not noise. A minimal custom review interface (render the trace intelligently, keyboard shortcuts, cluster/filter) pays for itself in review throughput.

How the layers fit

human evals ──calibrate──▶ LLM-as-judge ──scores──▶ golden set ──runs in──▶ regression CI gate
     ▲                                                   ▲
     └──────────── production sampling & incidents ──────┘  (adversarial cases added here)

Tradeoffs

Failure modes

• No regression gate — silent quality drop after a model/prompt change.
• Unrepresentative golden set — green evals, unhappy users. The set doesn't match reality.
• Trusting an uncalibrated judge — confident, biased scores.
• Aggregate hides a cliff — average flat while a category collapses. Stratify to catch it.
• Stale evals — the set rots as the product evolves. Prune and grow it.

Practitioner checklist

• Is there a version-controlled, representative, stratified golden set?
• Do regression evals gate prompt/model/retrieval/infra changes in CI?
• Do adversarial cases (injection, hallucination bait, format stress) run too?
• Is any LLM judge calibrated against humans and checked for bias?
• Do incidents and bugs get added back as permanent eval cases?
• Do you report per-capability deltas, not just one average?

Related lessons

• Observability
• Retrieval evals
• Structured output
• Quantization formats
• Production failure modes

TL;DR

Evals tell you if the system is correct before you ship. Observability tells you what it's doing in production. LLM observability is a first-class discipline because these systems fail silently and non-deterministically. You need end-to-end traces made of spans, with tokens, latency (split into TTFT/TPOT), errors, quality, and cost on every step. You also need drift detection to catch the slow regressions that no deploy caused. If you can't replay exactly what happened on a bad request, you can't fix it.

🎯 For the AI-native PM

Why it matters — AI fails silently. Wrong answers look exactly like right ones. Without traces, tokens, latency, and drift, you're operating blind, and you learn about problems from churn.

What it changes in your decisions — What you instrument, your incident-response plan, and how you detect slow quality decay.

Ask your eng team — "Can we pull up exactly what happened on a specific bad request?"

Product risk if ignored — Quality drifts down for weeks with no alarm, you can't debug complaints, and you're flying blind.

Mental model

A single LLM request is a distributed transaction across many components: retrieval, rerank, model calls, tools, validation, repair. Borrow distributed-tracing ideas, but add the LLM-specific dimensions:

TRACE: "answer support ticket #4821"  (tenant=acme, feature=support_copilot)
 ├─ span: retrieve         42ms   chunks=8  recall_proxy=...
 ├─ span: rerank           18ms   kept=3
 ├─ span: model.generate  910ms   ttft=210ms tpot=14ms in=3,100 out=180 tok  $0.0042
 │    ├─ prompt (full context, redacted as needed)
 │    └─ completion + finish_reason
 ├─ span: validate         2ms    repairs=0
 └─ span: tool.create_note 75ms   idempotency_key=... status=ok
TOTAL 1.05s   $0.0046   status=ok

The trace is the unit of debugging. The span is the unit of measurement.

What to capture

Traces & spans

• One trace per request, and one span per meaningful step: retrieve, rerank, each model call, each tool, validate, repair, route decision.
• Record inputs/outputs with privacy controls. Prompts and completions are gold for debugging, but they may contain PII. Redact or tokenize them, and respect tenant boundaries.
• Capture the decisions: which model/route was chosen, why fallback fired, which budget terminated an agent.

Tokens

• Input vs. output tokens per call. They cost differently and behave differently — see prefill vs. decode.
• Prompt-cache hit/miss (a cheap cost lever — see caching).
• Tokens are the raw material for cost attribution.

Latency

• Split it. Track TTFT (≈ prefill) and TPOT/inter-token (≈ decode) separately. A single "latency" number hides which phase is slow.
• Queueing time vs. compute time; end-to-end including retrieval/tools.
• Report distributions (p50/p95/p99), never just the mean. Tails are where SLOs and preemption spikes live.

Errors

• Provider errors (timeout, rate-limit, 5xx), validation/parse failures, repair exhaustion, tool failures, budget terminations, refusals.
• The hard part: silent errors. A grounded-looking hallucination throws no error. Approximate quality online with proxy signals: validation pass rate, grounding spot-checks, judge sampling, and user thumbs/edits/retries.

Drift

The regression that no deploy caused:

• Input drift — query mix, length, language, new topics shift over time.
• Output/quality drift — judge scores, refusal rate, format-validity, grounding trending down.
• Model drift — the provider updated weights underneath you, and behavior changes with no code change.
• Data/retrieval drift — corpus grows/ages; recall decays. Detect drift by tracking these distributions over time and alerting on shifts. Also re-run evals on a schedule, not only at deploy.

From signals to operations

• Dashboards & SLOs: latency (p95/p99 TTFT/TPOT), error rate, token/cost per request, cache-hit rate, eval scores — per feature and tenant.
• Alerts: on error spikes, cost spikes, cache-hit collapse, drift, budget-hit rate.
• Trace-level debugging: jump from an alert or a user complaint to the exact trace, with the full chain reconstructed.
• Feed the loop. Sampled production traces become new eval cases, and observed failures become adversarial tests. Observability and evals form a cycle, not two silos.

Guardrails vs. evaluators

Two production roles that get conflated because both "check outputs":

• Guardrails run inline, per request, and can block the output. They check for PII, policy violations, bad format, and obvious hallucination signals before the user sees the output. They must be fast and cheap enough to sit on the critical path, and they fail closed for the worst categories.
• Evaluators run offline — in CI to gate changes, and over sampled production traffic to measure quality trends. They can be slow, expensive, and thorough, because no user is waiting.

This distinction disciplines your architecture. An evaluator too slow or too expensive to run on every request is not a guardrail. A guardrail lightweight enough for the hot path is usually too crude to be your quality measurement. Using an evaluator to auto-correct outputs in production — grade, then regenerate on failure — is possible, but it pays latency and cost on every retry. Reserve it for high-stakes surfaces, and log every correction as an eval case.

Tradeoffs

Sample intelligently: trace everything cheaply (metrics), retain full payloads for a sample + all errors, and judge-score a representative slice.

Failure modes

• Mean-only latency — hides a terrible p99; users feel the tail.
• No silent-error signal — hallucinations and quality drops invisible until users churn.
• Undetected drift — slow decay over weeks with no alarm because nothing was trended.
• PII in logs — debugging convenience becomes a compliance/leakage incident.
• Untagged traces — can't slice by feature/tenant, so you can't localize a problem or attribute cost.

Practitioner checklist

• Is every request a trace with per-step spans (retrieve, model, tool, validate)?
• Do you record input/output tokens, cache hits, TTFT, TPOT, cost per call?
• Are latency and cost reported as p50/p95/p99 distributions?
• Do you capture validation failures, repairs, refusals, and budget hits?
• Is there an online proxy for quality (judge sampling, user signals)?
• Do you trend input/output/model/data drift and alert on shifts?
• Are traces tagged by feature and tenant, with PII redaction?
• Do production traces feed back into the eval set?

Related lessons

• Evals
• Cost attribution
• Prefill vs. decode
• Agent guardrails
• Production failure modes

TL;DR

"Our LLM bill is $80k/month" is not an actionable number. To optimize, control, and price an AI product, you must attribute cost along the dimensions you make decisions on: which feature, which workflow, which tenant, which user journey is spending the tokens. Per-model spend is what the provider invoices. Per-feature/tenant spend tells you what to cache, route, cap, or bill. Cost attribution is a product and business capability, not just a finance line item.

🎯 For the AI-native PM

Why it matters — "What does this AI feature cost, and who's driving the spend?" is a core PM question. Per-tenant and per-feature cost is the foundation of pricing, packaging, and margin.

What it changes in your decisions — Pricing and plan limits, which tenants are unprofitable, and where optimization actually pays off.

Ask your eng team — "Can we see cost per feature and per tenant — not just the total bill?"

Product risk if ignored — You price blind, a few power users quietly destroy margin, and you can't tell which feature to fix.

Mental model

Tokens are the currency, and every token is generated for a reason: a feature serving a workflow for a tenant during a user journey. If your traces carry those tags, cost rolls up along any of them:

$ total
 ├─ by feature      (support_copilot vs. search vs. summarize)
 ├─ by workflow      (onboarding flow vs. report generation)
 ├─ by tenant        (acme: $9k, globex: $40 — wildly uneven)
 ├─ by user journey  (first 3 messages cost 5× the rest)
 └─ by model/route   (what the invoice shows)

The provider gives you only the last row. You have to build the rest.

Why per-model is not enough

• You can't optimize what you can't localize. A 30% cost cut might hide in which feature's prompt is bloated, or which workflow loops too much. Neither is visible at the model level.
• Tenants are wildly uneven. In multi-tenant SaaS, a tiny fraction of tenants often drives most spend. Without per-tenant cost, you can't find unprofitable accounts, set quotas, or price fairly.
• Unit economics need a denominator. "Cost per resolved ticket," "per generated report," "per active user" — these require attribution to the unit, not the model.
• Runaway detection. A runaway agent or a cache-busting prefix shows up as a feature/tenant cost spike, but only if cost is sliced that way.

What to capture (per call, then roll up)

On every model/tool span, record:

• Tokens: input and output separately (priced differently), plus cache-read/cache-write tokens (cached input is much cheaper — prompt caching).
• Model & route chosen (and whether a fallback/escalation fired — escalations are a cost driver).
• Dimensional tags: feature, workflow step, tenant, user, request/journey id.
• Derived cost: apply the price book (per-model, per-token-type) to the tokens.

These are the same tags on your traces. Cost attribution is an aggregation over instrumented spans, not a separate system.

Dimensions that matter

From attribution to control

Attribution is the input. Control is the point:

• Budgets & quotas per tenant/feature; alert and throttle on breach (ties to agent budgets).
• Routing decisions. Push cheap traffic to small models, and reserve big models for hard cases (routing). Measure this by cost-per-route.
• Caching ROI. Quantify savings from prompt/semantic caching, and protect cache-hit rate as a cost metric.
• Context discipline. Target the bloated prompts attribution exposes (context engineering).
• Compression & summarization. Prompt-compression tools (LLMLingua-class, up to ~20x on long prompts with little quality loss) and summarizing multi-turn conversation history shrink exactly the token lines attribution flags first.
• Serving efficiency. For self-hosted models, the runtime itself is a lever. Optimized inference engines (vLLM, TGI) and inference-specialized hardware change the per-token price without touching the product.
• Pricing & packaging. Set plan limits and prices from real unit economics, not guesses.

Tradeoffs

• Tag granularity vs. overhead — more dimensions mean richer slicing but more instrumentation and cardinality cost. Tag the dimensions you actually decide on.
• Real-time vs. batch — live cost dashboards enable fast reaction but cost more to run than daily rollups. Match the choice to how fast you need to act.
• Cost vs. quality vs. latency — the cheapest route isn't always acceptable. Attribution lets you optimize cost within quality/latency SLOs, the theme of stack tradeoffs.

Failure modes

• Flat bill, no slices — a cost spike, no idea which feature/tenant caused it.
• Ignoring cache-read pricing — over-reporting cost or missing the value of caching.
• Input/output lumped together — misreads where spend really is (long context vs. long generation need different fixes).
• No per-tenant view — unprofitable accounts hide; pricing is guesswork.
• Attribution without action — pretty dashboards, no budgets/routing changes.

Practitioner checklist

• Do spans record input/output (and cache-read/write) tokens separately?
• Is every call tagged with feature, workflow, tenant, and journey?
• Can you produce cost per tenant and per unit (ticket, report, active user)?
• Do you track cost-per-route and escalation rate?
• Are there per-tenant/feature budgets with alerts and throttling?
• Is prompt-cache hit rate tracked as a cost metric?
• Does attribution actually drive routing/caching/pricing decisions?

Related lessons

• Observability
• Prompt vs. semantic caching
• Model routing
• Multi-tenant isolation
• Inference-stack tradeoffs