Strategy & tradeoffs

TL;DR

These four are not competitors; they solve different problems. In-context learning (ICL) shapes behavior via the prompt. It needs zero infra and gives instant iteration, but it pays tokens on every call. RAG injects knowledge at query time. It's best for facts that change or are too large to memorize. Fine-tuning bakes in behavior, format, and style. It's best when the prompt can't reliably get there. Distillation produces a smaller, cheaper model for a narrow, high-volume task. The expensive mistake is reaching for fine-tuning when you have a knowledge problem (use RAG instead) or a prompt problem (use ICL instead).

🎯 For the AI-native PM

Why it matters — Teams burn entire quarters fine-tuning when they actually had a retrieval problem (or vice versa). Picking the right approach is the highest-leverage early product decision you'll make.

What it changes in your decisions — Your build approach and timeline, your data investment, and whether fine-tuning is even the right project.

Ask your eng team — "Is this a knowledge problem (use RAG) or a behavior problem (maybe fine-tune)?"

Product risk if ignored — A quarter is spent fine-tuning a model that's already stale, when RAG would have been cheaper, fresher, and citeable.

The key distinction: knowledge vs. behavior

Need the model to KNOW something it doesn't?      → RAG (or ICL for small/static facts)
Need the model to BEHAVE a certain way reliably?  → Fine-tuning (or ICL if prompt suffices)
Need a cheaper/faster model for ONE narrow task?  → Distillation
Just need to steer it, fast, with no infra?       → In-context learning

Most "should we fine-tune?" questions dissolve once you ask: is this a knowledge gap or a behavior gap? Fine-tuning teaches behavior, not facts. It's a poor and stale way to store knowledge that RAG handles live.

The four approaches

In-context learning (ICL)

Instructions and few-shot examples in the context, with no weight changes.

• Best for: fast iteration, prototypes, tasks a capable model can do when shown how, and low or medium volume.
• Costs: every example is paid for on every call — tokens, latency, prefill. It's limited by the context window, and it can't teach truly novel behavior.
• Wrong when: the prompt is huge and repeated at scale (fine-tune or distill to amortize the cost), or you need knowledge that changes (use RAG).

RAG

Retrieve relevant data at query time and ground the answer in it (RAG architecture).

• Best for: large, changing, or proprietary knowledge; needing citations/attribution; freshness; reducing hallucination on facts.
• Costs: a whole pipeline to build and operate — chunking, embeddings, indexing, reranking, freshness. Retrieval quality caps answer quality, and you pay per-call latency and token cost.
• Wrong when: the problem is behavior, format, or tone (retrieval won't fix it), or the knowledge is tiny and static (just put it in the prompt).

Fine-tuning

Update weights, often with a parameter-efficient method like LoRA, on task examples.

• Best for: consistent format, style, or behavior the prompt can't reliably get; encoding a domain skill; shortening prompts by moving few-shots into weights for cheaper inference; a narrow task done at high volume.
• Costs: data curation and a training pipeline; re-training on drift; risks like catastrophic forgetting and overfitting; a model artifact to version, evaluate, and serve; slow iteration compared with editing a prompt.
• Wrong when: you need fresh facts — it's frozen at training time, so use RAG instead — you have little data, or a prompt change would have sufficed. Fine-tuning to "add knowledge" is the classic anti-pattern: stale, expensive, and unattributable.

Distillation

Train a small student to mimic a large teacher (details).

• Best for: a proven, narrow, high-volume task where a big model works but is too slow or expensive; squeezing steady-state cost and latency.
• Costs: teacher-generated data and training; a lower quality ceiling; lost generality that makes it brittle off-distribution; a pipeline to maintain.
• Wrong when: the task is broad or evolving, volume is low (not worth it), or you haven't yet nailed the task with a big model — distill after it works.

Decision guide

1. Start with ICL. It's the cheapest to try and often enough. Establish evals here.
2. Knowledge gap? Use RAG — facts that are large, proprietary, or changing, and that need citations.
3. Behavior gap the prompt can't close, after honest prompt effort? Fine-tune — for format, style, or skill consistency, and also to shrink expensive long prompts at scale.
4. Proven narrow task, high volume, cost/latency-bound? Distill, then optionally quantize the student.
5. They compose. The mature stack is often a model (possibly fine-tuned or distilled), with RAG for knowledge, plus ICL for per-request steering, chosen per subproblem, not globally.

The same triad shows up inside agents, where Google's Agents whitepaper calls it targeted learning: teaching a model when and how to use its tools. In-context learning puts tools and few-shot examples in the prompt, ReAct-style. Retrieval-based in-context learning dynamically populates the prompt with the most relevant examples from an external store — RAG applied to behavior, not just facts. And fine-tuning trains on a corpus of tool-use examples before inference. It's a different problem with an identical decision structure: start in the prompt, retrieve when it must scale, train when it must be innate.

Comparison

Failure modes (using the wrong tool)

• Fine-tuning for knowledge — stale, expensive, and uncited; this should have been RAG.
• RAG for a behavior problem — retrieval is perfect, but output is still the wrong format; this needed prompting or fine-tuning.
• Premature fine-tuning/distillation — you committed to an artifact before the task was even solved with a big model and good evals.
• ICL at scale — a 6k-token prompt repeated millions of times should have been amortized into weights. Cost attribution reveals it.
• No evals first — you can't tell whether the new approach actually helped.

Practitioner checklist

• Have you classified the problem as knowledge vs. behavior vs. cost?
• Did you exhaust ICL (and honest prompt iteration) before training anything?
• Is fresh/changing knowledge handled by RAG, not baked into weights?
• Do you have evals to prove the chosen approach beats the baseline?
• For fine-tune/distill: do you have a plan for drift and re-training?
• Are you composing approaches per subproblem rather than picking one globally?

Related lessons

• RAG architecture
• Context engineering
• Speculative decoding vs. quantization vs. distillation
• Evals
• Inference-stack tradeoffs

TL;DR

Almost every decision in an LLM system is a move in a four-way tradeoff between latency, quality, cost, and reliability. You rarely improve one without spending another. Mature AI engineering means making these tradeoffs deliberately and per-use-case, driven by explicit SLOs and budgets, instead of accepting whatever defaults give you. This lesson is the map that connects every other lesson in the repository to the axis it moves.

🎯 For the AI-native PM

Why it matters — Latency, quality, cost, reliability — you cannot maximize all four, and pretending otherwise is how AI roadmaps go off the rails. This is the PM's core tradeoff framework.

What it changes in your decisions — Your SLOs per use case, what you choose to optimize vs. protect, and how you set realistic targets.

Ask your eng team — "Which of latency / quality / cost / reliability are we choosing to trade here, and in exchange for what?"

Product risk if ignored — "Make it faster, cheaper, and better" with no target means the team can't tell when they're done or when they've gone too far.

The four axes

• Latency — TTFT and TPOT (prefill vs. decode), end-to-end including retrieval and tools. Judge it at p95/p99, not the mean.
• Quality — correctness, grounding, format validity, helpfulness. Measure it with evals, not vibes.
• Cost — $/request and per-unit economics, attributed by feature/tenant.
• Reliability — availability, graceful degradation, consistency, isolation. This is the worst-case behavior, not the average.

You cannot maximize all four. The job is to hit your required level on each and optimize the rest.

Same lever, different axis — the whole stack

Each technique elsewhere in this repo is a trade among the four:

Read this table as the unifying thread: there is no globally "best" configuration, only the best one for a given SLO.

The classic tensions

• Latency ↔ Quality — a bigger model or more retrieval/reasoning is better but slower. Cascades and routing let you spend latency only on hard requests.
• Cost ↔ Quality — a cheaper, smaller, or quantized model cuts cost and may cut quality. Keep it within a quality floor enforced by evals.
• Latency ↔ Cost (throughput) — batching lowers $/token but raises per-user TPOT. Pick the operating point from your SLO.
• Reliability ↔ Cost/Latency — retries, fallbacks, multi-provider setups, and isolation all add cost and latency to buy resilience.
• Quality ↔ Reliability — the most capable single model may be less available than a routed multi-provider setup that's slightly weaker but never down.

How to make the tradeoff deliberately

1. Set SLOs/budgets per use case. Interactive chat, an overnight batch job, and a safety-critical workflow are not the same. Define required latency, quality floor, cost ceiling, and availability separately for each.
2. Measure all four. Use observability for latency, cost, and reliability, and evals for quality. You can't trade what you don't measure.
3. Optimize the slack, protect the floor. Improve the unconstrained axes without breaching the required level on the others. For example, cut cost via quantization only if evals stay above the quality floor.
4. Differentiate by request. Route easy traffic cheap and fast, and hard traffic to a capable model. Don't pay worst-case cost for the average request.
5. Revisit continuously. Models, prices, and traffic change, so today's optimal point drifts — re-evaluate as part of operations. Four forces are falling or rising in your favor simultaneously: training costs are falling, per-token inference costs are falling, tokens-per-second is rising, and usable context windows are growing. That means every "not economical" and "too slow" verdict has an expiry date. Keep a list of the features that just missed the bar, and re-run the numbers on a schedule, because the dial-up era's "no" is often the broadband era's product. The canonical proof is the DeepSeek moment (January 2025): efficiency work (latent attention, FP8 training, mixture-of-experts) delivered near-frontier reasoning at a fraction of assumed cost, and the open-weight wave behind it (R1, Qwen3, Kimi K2) has kept frontier-adjacent capability within reach of commodity budgets ever since. Whole product categories that were "uneconomic" in 2024 stopped being so in one quarter.

Failure modes

• Optimizing one axis blindly — cost-cutting that quietly tanks quality, or a latency push that wrecks reliability.
• One config for all requests — paying premium cost and latency on easy cases, or starving hard ones.
• No SLOs — "make it faster, cheaper, better" with no target means you can't tell when you're done or when you've gone too far.
• Mean-driven decisions — tuning to averages while p99 latency and worst-case reliability, the things users actually feel, degrade.

Practitioner checklist

• Do you have explicit latency, quality, cost, and reliability targets per use case?
• Do you measure all four (evals + observability), at the tail, not just the mean?
• Does each optimization protect the floor on the axes it isn't improving?
• Do you route/differentiate by request difficulty instead of one global config?
• Do you re-evaluate the operating point as models, prices, and traffic change?

Related lessons

• Prefill vs. decode
• Model routing
• Cost attribution
• Evals
• Production failure modes

TL;DR

LLM systems fail in a recognizable handful of ways: hallucinated tool calls, malformed JSON, stale retrieval, runaway agents, and silent eval regressions, plus their cousins — cost spikes, cross-tenant leaks, prompt injection, degraded fallbacks. What they share is that they're quiet: the system keeps returning plausible-looking output while doing the wrong thing. This lesson is a field guide: each mode, why it happens, how to detect it, and which lesson hardens against it. Treat it as the integration test for everything else in the repository.

🎯 For the AI-native PM

Why it matters — These are the incidents that will actually page your team. Knowing the catalog lets you put prevention on the roadmap before the postmortem instead of after it.

What it changes in your decisions — Your pre-launch risk review, what you choose to monitor, and what goes in the runbook.

Ask your eng team — "Which of these five failure modes can happen to us today, and what's our detection for each?"

Product risk if ignored — You learn the catalog the expensive way, one outage at a time.

Why LLM failures are different

In a normal service, failures throw exceptions, 500s, timeouts. LLM failures often succeed loudly and are wrong quietly — a hallucination reads like a fact, a malformed plan parses far enough to act, a stale answer looks current. So the engineering response isn't just error handling; it's validation, bounding, grounding, and measurement that make silent failures visible and contained.

The catalog

1. Hallucinated tool calls

• What: the model invents a tool, a parameter, or a plausible-but-fake argument — a non-existent id, an invalid enum — then "acts."
• Why: the system treats model output as a trusted command, and arguments aren't checked against the real system.
• Detect: tool-name/arg validation failures; downstream "not found" errors; traces of rejected calls.
• Engineer around it: strict tool contracts and argument validation; validate args against the system of record; give structured, actionable tool errors; add idempotency so a bad-then-retried call is safe.

2. Malformed JSON / structured output

• What: output doesn't parse or violates the schema. A markdown fence, a trailing comma, or a missing field takes down a workflow.
• Why: the system trusts raw generation with no validation, sometimes made worse by aggressive quantization hurting format adherence.
• Detect: parse/validation failure rate; repair rate trend.
• Engineer around it: constrained decoding, schema validation, bounded repair, and a fallback chain. Never JSON.parse without a schema. Track validity after model or quantization changes.

3. Stale retrieval

• What: RAG cites outdated, deleted, or superseded content, confidently wrong on facts that changed.
• Why: the index is a cache that drifted from the source, deletions weren't propagated, and there are no recency signals. This is the same shape as semantic-cache staleness.
• Detect: freshness evals with time-sensitive queries; recall/grounding trends; user "that's out of date" signals.
• Engineer around it: incremental indexing, deletion handling, and recency ranking (RAG freshness); cache TTLs and invalidation; "say I don't know" when context is absent.

4. Runaway agents

• What: an agent loops, thrashes a tool, or fans out, burning time and money, often circularly rather than infinitely.
• Why: there are no budgets and no termination or no-progress detection, and non-idempotent tools amplify the damage.
• Detect: budget-hit rate, steps-per-task distribution, cost spikes by feature/tenant, repeated identical actions.
• Engineer around it: loop, tool, token, cost, and time budgets, plus termination and no-progress detection; idempotent tools; per-tenant cost alerts.

5. Silent eval regressions

• What: quality drops with no code change — a provider model update, a prompt tweak with side effects, or drift in data or inputs — and nobody notices for weeks.
• Why: there's no regression gate, the team relies on "it looked fine," and aggregate metrics hide a category cliff.
• Detect: regression evals in CI; scheduled re-runs; per-stratum deltas; online quality proxies and drift monitors.
• Engineer around it: golden sets that gate every change; stratified reporting; production sampling that feeds back into evals.

The cousins (don't forget these)

The common cure

Every mode above is defeated by the same four habits, the thesis of this whole repository:

1. Treat model output as untrusted. Validate, bound, authorize.
2. Put brakes on everything. Use budgets, timeouts, fallbacks.
3. Ground and isolate. Cite from context; scope by tenant.
4. Measure relentlessly. Run evals before shipping and observability while running, and feed incidents back into both.

This is the difference between a demo and infrastructure.

Practitioner checklist (pre-launch failure review)

• Are tool calls validated so a hallucinated call can't act?
• Is every structured output schema-validated with bounded repair + fallback?
• Is the retrieval index fresh, with deletions handled and time-sensitive evals?
• Does every agent have enforced budgets and no-progress termination?
• Do regression evals gate changes, with scheduled re-runs and drift monitoring?
• Are cost, isolation, injection, outage, and latency-tail risks each covered?
• Does every incident become a new permanent eval case?

Related lessons

• Harness engineering
• Shipping as infrastructure, not demos
• Structured output
• Function calling
• Agent guardrails
• Evals · Observability
• Inference-stack tradeoffs