RAG & retrieval

TL;DR

RAG is a pipeline, and quality is bounded by its weakest stage. Chunking decides what a retrievable unit is. Embeddings decide how meaning is matched. Hybrid search combines keyword and vector recall. Reranking sharpens precision before the prompt. Freshness keeps the index from lying. The generator can only answer from what retrieval surfaces, so most "the LLM is wrong" RAG bugs are actually retrieval bugs. Optimize retrieval first, generation second.

🎯 For the AI-native PM

Why it matters — Most "the AI gave a wrong answer" bugs are actually retrieval bugs, not model bugs. RAG quality is where grounded, current, citeable answers come from — the core of enterprise trust.

What it changes in your decisions — What data you invest in indexing, your freshness SLAs, and whether you can promise citations.

Ask your eng team — "When an answer is wrong, is it because we never retrieved the right source in the first place?"

Product risk if ignored — You blame the model and tune prompts for months while the real defect — retrieval — goes unfixed.

The pipeline

INDEX TIME:   documents ──▶ chunk ──▶ embed ──▶ vector store (+ keyword index)
                                  └──▶ keep fresh (re-index on change)

QUERY TIME:   query ──▶ (rewrite) ──▶ hybrid search (dense + lexical)
                    ──▶ rerank top-N ──▶ trim to budget ──▶ context + cite ──▶ generate

This is context engineering in motion: retrieval is how the right information gets into the window.

Chunking — define the retrievable unit

• Why it matters: chunks that are too big dilute relevance and waste tokens; chunks that are too small lose the context needed to answer. Chunk size is a recall/precision dial.
• Strategies: fixed-size with overlap is simple and robust. Structure-aware chunking splits on headings, sections, or functions, and is usually better because it respects meaning. Semantic chunking splits where the topic shifts.
• Match chunk size to task shape: fixed-output tasks — extract a field, answer a pointed question — tolerate large chunks, since the model finds the needle. Expansive-output tasks — summarize, synthesize across sources — want smaller chunks, so retrieval can compose coverage from many places instead of drowning in a few.
• Carry metadata on every chunk: source id, title, section, timestamp, tenant, permissions. Metadata powers citations, freshness, and tenant filtering.
• Tabular/structured data often shouldn't be free-text chunked at all. Consider text-to-SQL or structured retrieval instead.

Embeddings — how meaning is matched

• Embeddings map text to vectors, so semantic nearness equals vector nearness. The choice of model sets your semantic ceiling.
• Match query and document embeddings to the same model and space. Domain fit matters — a general embedder may miss jargon, code, or multilingual content, so sometimes a domain-tuned embedder is the highest-leverage change.
• Asymmetric search: short queries versus long passages benefit from models trained for that, such as query/passage encoders.
• Cost and latency: embedding dimension and model size trade retrieval quality against storage and query speed. Re-embedding the whole corpus on a model change is a real migration cost, so version your embeddings.

Hybrid search — recall from two angles

Dense (vector) search captures meaning but can miss exact terms — IDs, error codes, rare names, acronyms. Lexical search (BM25/keyword) nails exact matches but misses paraphrase. Hybrid search runs both and fuses the results, for example with Reciprocal Rank Fusion or weighted scores.

• Hybrid almost always beats either alone, especially for technical or enterprise corpora full of exact identifiers.
• Add metadata filters — tenant, date, doc type, ACL — as hard constraints before ranking. This helps both relevance and isolation/security.

Reranking — precision before the prompt

First-stage retrieval optimizes recall by casting a wide net, say the top

50. A cross-encoder reranker then scores each (query, chunk) pair jointly for true relevance and keeps the top few.

• Why: you cheaply pull many candidates, then spend a precise model on a short list. This is usually the single biggest precision win in a RAG system.
• Effect on the generator: fewer, better chunks reduce distraction and lost-in-the-middle failures, and cut prefill cost.
• Cost: an extra model call and latency, but on a small candidate set. It often lets you shrink the context you send, paying for itself.

Freshness — keep the index honest

A retrieval index is a cache of your data, and it goes stale the moment the source changes.

• Incremental indexing / CDC: re-embed and upsert on create, update, or delete. Don't full-rebuild nightly if data changes continuously.
• Deletions and tombstones: removed source docs must leave the index, or you'll cite deleted or retracted content.
• Recency signals: timestamp chunks and let ranking prefer recent versions. Expire or down-weight stale ones.
• Freshness failures mirror semantic-cache staleness: a confident answer from outdated data.

Generation & attribution

• Pass reranked, trimmed chunks with source tags and instruct the model to answer from the context and cite sources. This enables grounding and citation evals.
• Handle "not in the context" explicitly: the model should say it doesn't know rather than fill the gap from parametric memory — a hallucination guard.

Tradeoffs

Failure modes

• Right answer never retrieved — a chunking, embedding, or recall problem. No prompt fix helps. Measure retrieval recall.
• Answer buried in noise — too many chunks with no reranking, so distraction degrades the answer.
• Stale/deleted content cited — a freshness or deletion gap.
• Exact-match misses — dense-only search can't find an error code. Add lexical search.
• Cross-tenant leakage — a missing ACL or tenant filter returns another customer's docs. See isolation.

Practitioner checklist

• Is chunking structure-aware with rich metadata (source, time, tenant, ACL)?
• Do query and document embeddings share a model/space, fit to your domain?
• Is search hybrid (dense + lexical) with metadata pre-filters?
• Is there a reranking stage feeding a small, trimmed context?
• Is the index updated incrementally, with deletions honored?
• Does the prompt enforce cite-from-context and "say I don't know"?
• Do you measure retrieval quality separately from answer quality?

Related lessons

• Retrieval evals
• Context engineering
• Prompt vs. semantic caching
• Multi-tenant isolation
• Fine-tuning vs. ICL vs. RAG vs. distillation
• Why RAG — this same pipeline at product-decision altitude, plus vector-database indexing and retrieval-quality tuning in their own dedicated lessons.

TL;DR

A RAG system has two failure surfaces: did we retrieve the right thing? and did the model use it faithfully? You must measure them separately, or you'll tune the prompt to fix a retrieval bug, or the reverse. Retrieval is judged by recall and precision. Generation-over-retrieval is judged by grounding (faithfulness), attribution (claims traced to sources), and citation quality. End-to-end answer quality alone hides which half is broken.

🎯 For the AI-native PM

Why it matters — You can't claim "grounded" or "accurate" unless you measure grounding and attribution separately. This turns a marketing claim into a number you can defend to customers and legal.

What it changes in your decisions — The quality bar you commit to, how you report accuracy externally, and any citation guarantees.

Ask your eng team — "How do we measure that answers are actually supported by sources, instead of plausibly made up?"

Product risk if ignored — You market accuracy you can't substantiate; hallucinated citations become a credibility — and legal — problem.

Why split the evaluation

query ──▶ [ retriever ] ──chunks──▶ [ generator ] ──▶ answer
            measure here:                measure here:
            recall, precision            grounding, attribution, citations

If end-to-end accuracy is low, the split tells you where to invest: better chunking, embeddings, and reranking, or better prompting and grounding constraints. Without it, you're guessing. This mirrors the general evals discipline applied to retrieval.

(Every long-context model release re-raises "is RAG dead?" The answer stays no. As long as knowledge is larger than the window, changes faster than retraining, or is permissioned per user, retrieval is how freshness, access control, and cost control happen. What the question should prompt is exactly this lesson: measure your retrieval separately, so you know what it's contributing.)

Retrieval metrics

You need labeled data: queries paired with the documents or chunks that should be retrieved — a golden set for retrieval.

• Recall@k — of all relevant chunks, how many appear in the top k? This is the ceiling on the whole system: if the right chunk isn't retrieved, no prompt can fix the answer. It's usually the first metric to optimize.
• Precision@k — of the top k retrieved, how many are actually relevant? Low precision means the context is noisy, the generator gets distracted, and you pay for junk tokens.
• MRR / nDCG — rank-aware metrics: is the relevant chunk near the top, not just present? These matter because of lost-in-the-middle, and because reranking is judged on ordering.
• Context recall vs. context precision — recall asks did we get enough to answer; precision asks how much of what we got was needed. This is the classic recall/precision tension.

Use these to tune chunk size, k, hybrid weighting, and reranking directly — each metric points at a specific stage.

Generation-over-retrieval metrics

Even with perfect retrieval, the model can ignore or misuse the context.

• Grounding / faithfulness — is every claim in the answer supported by the retrieved context, not invented from parametric memory? Low grounding means hallucination despite good retrieval. Measure it by checking each answer claim against the provided chunks, often via an LLM-as-judge or NLI/entailment scoring.
• Attribution — can each claim be traced to the specific source that supports it? This is stronger than grounding: not just "supported somewhere" but "supported by this citation."
• Citation quality — are the citations (a) present where claims need them, (b) correct — the cited source actually supports the claim, not a hallucinated or mismatched reference — and (c) complete, with no unsupported claims left uncited?
• Answer relevance / completeness — does the answer actually address the question, using the retrieved material?

A useful framing is the "RAG triad": context relevance (retrieval), groundedness (answer ⊆ context), and answer relevance (answer ⊆ question). All three must hold.

How to actually run these

1. Build a retrieval golden set — representative queries with labeled relevant chunks. Include hard cases: exact-identifier lookups, paraphrases, multi-hop, "answer not in corpus."
2. Score retrieval offline (recall@k, precision@k, nDCG) on every index, chunking, embedding, or reranker change. It's cheap, deterministic, and fast to iterate.
3. Score grounding and attribution with an LLM judge against a rubric, spot-validated by humans. Calibrate the judge before trusting it.
4. Track freshness explicitly — include time-sensitive queries whose correct answer changes, to catch stale-index regressions.
5. Gate changes in CI so a chunking tweak that quietly drops recall can't ship.

Tradeoffs & what each metric pushes

Don't chase recall to 100%. Beyond "enough," extra chunks hurt precision, grounding, and cost. The objective is the right context, not the most.

Failure modes

• Grading only end-to-end — you can't tell whether retrieval or generation is the problem, so you fix the wrong one.
• Hallucinated citations — the answer cites a source that doesn't support the claim. It reads authoritative but is wrong. A citation-correctness eval catches it.
• Recall regressions hidden by the LLM — a strong model papers over weak retrieval using parametric knowledge, inflating accuracy while grounding silently drops.
• Stale golden set — labels drift from the live corpus. Refresh them.
• Untrusted judge — an uncalibrated LLM judge for grounding gives false confidence.

Practitioner checklist

• Do you measure retrieval (recall/precision/nDCG) separately from answers?
• Is there a labeled retrieval golden set, including "answer not in corpus"?
• Do you score grounding and attribution, not just answer correctness?
• Are citations checked for presence, correctness, and completeness?
• Is your LLM grounding-judge calibrated against human labels?
• Do retrieval evals gate index/chunking/embedding/reranker changes in CI?
• Do time-sensitive queries guard against freshness regressions?

Related lessons

• RAG architecture
• Evals
• Context engineering
• Observability
• Production failure modes