Evaluating context quality
TL;DR
Eval-driven development grades whether an AI feature's output is good. That's necessary and not sufficient: a failing output score doesn't tell you whether the model reasoned badly or was simply never shown what it needed. Context evals answer the narrower, faster question — did the pipeline supply the right instructions, the right retrieved facts, the right memory, the right live state — before you spend a debugging session blaming the model. Separating "was the context right?" from "was the answer right?" turns a vague quality regression into a five-minute check instead of a week of prompt archaeology.
🎯 For the product leader
Why it matters — Most AI debugging time is spent re-reading model output looking for a pattern, when the faster diagnostic is checking what the model was shown. A context eval turns that check into a repeatable score instead of a guess.
What it changes in your decisions — You ask "did retrieval find the right document?" as a separate, gradable question from "was the final answer good?" — and you fund whichever one is actually failing, instead of tuning the prompt for a retrieval problem.
Ask yourself — "The last time this feature failed, did we check whether the context was right before we started rewriting the prompt?"
Risk if ignored — Weeks spent tuning a prompt against a retrieval bug, because nobody separated the two questions before debugging started.
The mental model: two separate scores, not one
Two separate scores, not one · was the RIGHT input assembled, vs. was the answer good
A failing output score with a passing context score points at the model. A failing context score points upstream.
was the right input assembled?
was the answer good?
Fix retrieval, memory, or instructions
No amount of prompt tuning fixes an upstream miss.
Fix the model, prompt, or reasoning
An isolated, provable model or prompt problem.
A failing output score with a passing context score points at the model and the prompt. A failing output score with a failing context score points upstream, at retrieval, memory, or instructions — and no amount of prompt tuning fixes an upstream miss.
Building a context eval set
Reuse the sourcing discipline from eval-driven development, aimed one layer earlier:
- Retrieval precision/recall — for a graded set of real queries, did the retrieval step return the document that actually answers it? Grade this independently of what the model does with it. (Full retrieval-quality mechanics in RAG architecture.)
- Memory correctness — for a graded set of returning-user scenarios, did the right prior fact get surfaced, and only to the right audience? See Memory & context for what "right" means per memory type.
- Instruction currency — spot-check, on a schedule, whether the encoded policy matches the current real policy — not whether it matches what was true when the prompt was written.
- Live-state freshness — for time-sensitive answers, check the age of the tool data behind a sample of real responses, not just whether the tool call succeeded.
- Noise and size — for the same graded queries, what share of the supplied context was relevant, and how many tokens did the request carry? A context that contains the right fact plus a great deal of irrelevant text can still fail, because the fact is buried. It also costs more. Track relevant share and total size next to accuracy.
- Conflicts — check whether two supplied sources disagree (an old policy page and a new one). The model cannot know which one you meant.
A feature that passes every context check but still fails its output eval has a model or prompt problem, isolated and provable. A feature that fails any of them has found its actual bug, before a single line of the prompt gets touched.
When you change the model, rerun both sets. A newer model may handle long context better or worse than the old one, and it may treat a noisy context differently. The upgrade steps are in Prompts in production.
The five-minute triage
When a specific answer is wrong, before touching the prompt:
- Pull the exact context the model was given for that call (see the pipeline audit).
- Ask: was the needed fact actually in there? If no — it's a retrieval or memory miss, not a model miss.
- Ask: was it in there, but the model ignored or misused it? Check whether it was buried among a lot of irrelevant text or contradicted by another source. Only if the context was clean is it a model or prompt problem worth iterating on.
Skipping step 1 is the single most common reason prompt-tuning sessions run long without progress.
Failure modes
- Output-only evals — grading only the final answer, so a retrieval regression and a model regression look identical and get debugged the same (slow) way.
- Debugging by rewriting the prompt first — reaching for prompt changes before checking whether the context was even right, burning cycles on the wrong layer.
- A context eval that's never rerun — building the checks once at launch, then letting them go stale exactly like the pipeline they were meant to catch drifting.
Practitioner checklist
- Do we grade context quality (retrieval, memory, instructions, live state) separately from output quality, or only the final answer?
- For our last major "the AI got worse" incident, did we check context quality before assuming a model or prompt regression?
- Are context evals rerun on a schedule, the same way output evals are?
- Do we measure how much of the supplied context was relevant, not only whether the right fact was present?