Inference internals

TL;DR

They share the word "caching" and almost nothing else. Prompt caching reuses the model's internal computation (the KV cache) for an exact shared prefix. It is lossless, and it changes only cost and latency, never the answer. Semantic caching returns a previously generated response when a new query is similar. It can skip the model entirely, but it risks returning a subtly wrong answer. One is a performance optimization. The other is a correctness gamble you must validate.

🎯 For the AI-native PM

Why it matters — Caching is one of your biggest levers on cost and latency. But semantic caching can serve a wrong or stale answer. That's a trust and even a compliance problem, not just a performance tweak.

What it changes in your decisions — Cost targets, whether to enable semantic caching for a given feature, and the SLAs you can stand behind.

Ask your eng team — "Are we caching responses across users, and how do we know we're not serving the wrong one?"

Product risk if ignored — A loosely-tuned cache serves one user's answer to another — a privacy incident dressed up as an optimization.

The two are not the same layer

                      ┌─────────────────────────────────────────┐
   request ──────────▶│ semantic cache?  (embed query, ANN match)│──hit──▶ stored response
                      └─────────────────────────────────────────┘         (NO model call)
                                       │ miss
                                       ▼
                      ┌─────────────────────────────────────────┐
                      │ model server: prompt cache reuses KV for │──▶ generate tokens
                      │ the matching prompt *prefix*             │
                      └─────────────────────────────────────────┘

Prompt (prefix / KV) caching

What it reuses: the computed key/value tensors for a prompt prefix that is byte-for-byte identical to one seen before. The prefill phase for that prefix is skipped; decode proceeds normally.

Why it matters: prefill is compute-heavy and scales with input length. If 2,000 tokens of system prompt and tool definitions are shared across every request, caching that prefix removes most of the prefill cost for every call after the first.

Properties:

• Lossless. The output distribution is unchanged. You compute the same thing; you just skip recomputing the shared part.
• Prefix-only and order-sensitive. The cache matches from the start of the prompt. One changed byte near the top — a timestamp, a user name — invalidates everything after it. This is why context engineering insists on a stable prefix.
• Time-limited. Cached entries expire; provider TTLs are short, often minutes. Self-hosted entries are subject to KV cache eviction under memory pressure.

Design rules:

• Put stable content — system prompt, policies, tool contracts, few-shot examples — first. Put volatile content, the user's query, last.
• Keep the prefix identical across requests. Don't put per-request strings up top.
• In multi-tenant systems, be deliberate: a shared prefix is fine, but never let cache reuse cross a trust boundary in a way that exposes data. See multi-tenant cache safety.

Semantic caching

What it reuses: a final response. The system embeds the query. If its nearest neighbor in a vector store falls within a similarity threshold, the system returns the stored answer and never calls the model.

Why it matters: this is the biggest possible win. You skip generation entirely, cutting latency to a lookup and cost to near zero for repeat-ish questions like FAQs and common support queries.

Properties & risks:

• Lossy and approximate. "Similar embedding" does not mean "same correct answer." What's my account balance? and What was my account balance last month? can sit close in embedding space and demand totally different answers.
• Staleness. A cached answer can be correct today and wrong tomorrow — prices, inventory, policy change. It needs TTLs and invalidation, which connects to retrieval freshness.
• Context-blind. Two users asking the same words may need different answers, because of different permissions, tenant, or locale. Caching across users without keying on context causes cross-user contamination.
• Threshold tuning is a precision/recall problem. Too loose, and you serve wrong answers confidently. Too tight, and you get almost no hits.

Tradeoffs at a glance

Failure modes

• Silent prefix busting — a logging timestamp injected at the top of the system prompt drops your prompt-cache hit rate to near zero and triples cost. Catch it with cost attribution and cache-hit metrics in observability.
• Confident wrong semantic hit — the threshold is too loose, so users get plausible, incorrect answers with no model call to catch it. Guard against this with conservative thresholds, context-aware cache keys, and evals.
• Stale semantic cache — yesterday's price served today. Add TTLs and event-based invalidation.

Practitioner checklist

• Is your prompt prefix stable enough to actually hit the prompt cache?
• Do you measure prompt-cache hit rate as a first-class metric?
• Does your semantic cache key include user/tenant/permission context?
• Do semantically cached entries have TTLs and invalidation hooks?
• Have you evaluated semantic-cache false-hit rate on a labeled set?

Related lessons

• KV cache management
• Prefill vs. decode
• Context engineering
• Multi-tenant isolation & cache safety
• Cost attribution

TL;DR

During generation, the model caches the key and value tensors of every token it has already processed. This way attention doesn't recompute the whole sequence at each step. This KV cache is what makes decode fast. It is usually the binding constraint on how many requests a GPU can serve at once. Managing it — sizing, reuse, eviction, fragmentation — is the core memory problem of LLM serving.

🎯 For the AI-native PM

Why it matters — This is the hidden reason you can't just add longer context and more concurrent users cheaply. It sets your capacity ceiling and your unit economics.

What it changes in your decisions — The context-length limits you expose in the product, your scaling plan, and your pricing tiers.

Ask your eng team — "What does doubling our max context length do to our cost and our capacity?"

Product risk if ignored — You promise long-context or high-concurrency features. They quietly blow up cost or fall over at scale.

Mental model

Attention at step t needs the keys and values of all tokens 0..t-1. Recomputing them every step would make generation quadratic. Instead, the server stores them:

KV cache size ≈ 2 (K and V)
              × num_layers
              × num_kv_heads × head_dim
              × sequence_length
              × bytes_per_element (dtype)
              × batch_size (sum of all sequences in flight)

The two things to feel in your gut:

1. It grows linearly with total tokens in flight (context length × concurrency).
2. It lives in scarce GPU HBM, competing with the model weights themselves.

A long-context, high-concurrency workload can need more memory for KV than for the model weights. That's why KV cache — not FLOPs — limits your batch size and throughput.

The levers

1. Reuse

Identical prefixes can share KV entries. This is exactly what prompt/prefix caching exploits. A stable system prompt shared across requests is computed once and reused. This saves both prefill compute and memory. Reuse requires the entries to still be resident, which ties reuse to eviction policy.

2. Eviction

When memory fills, something must go. Here are the policies and their costs:

• By completion — free a sequence's KV when it finishes (the baseline).
• LRU on cached prefixes — evict the least-recently-used shared prefix. A future request with that prefix pays full prefill again. This is a recompute, not a wrong answer.
• Preemption / swapping — under pressure, the server can pause a running sequence. It can then recompute the sequence's KV later, or swap it to CPU/host memory and back. Recompute trades compute for memory. Swapping trades PCIe bandwidth for memory. Both add tail latency.

3. Memory pressure & fragmentation

Naively, each sequence reserves a contiguous block sized for its maximum possible length. This wastes memory for short outputs and fragments the pool. Paged attention solves this by allocating the KV cache in fixed-size blocks that need not be contiguous — OS-style virtual memory for the cache. It nearly eliminates fragmentation and enables much higher concurrency. See paged attention.

4. Compression

• Smaller dtype — storing KV in FP8/INT8 halves or quarters cache size (a quality/memory tradeoff; see quantization formats).
• Grouped/Multi-Query Attention (GQA/MQA) — fewer KV heads than query heads, so the cache shrinks proportionally. Most modern models use GQA for exactly this reason.
• Multi-head Latent Attention (MLA) — a deeper compression than GQA: instead of shrinking the number of KV heads, compress the KV representation itself into a low-rank latent vector, then reconstruct it at attention time. DeepSeek-V2 and V3 made this the reference architecture for the technique. It buys a bigger cache reduction than GQA alone, at the cost of a materially more complex attention implementation — an architectural bet made at training time, not a serving-time knob.
• Sliding-window / local attention — bound the cache to the last N tokens.

Why this is the throughput story

Concurrency — how many requests you batch together — is capped by how much KV cache fits. More memory headroom means bigger batches, which means higher GPU utilization, which means lower cost-per-token. So every KV optimization (paging, GQA, FP8 KV, eviction policy) is really a throughput and cost optimization. This is the mechanism behind continuous batching.

Tradeoffs

Failure modes

• OOM under load — concurrency times context exceeded HBM, so requests get rejected or the server crashes. Mitigate with admission control and max-context limits.
• Tail latency from preemption — under pressure, long requests get swapped or recomputed, which spikes p99. You can see this only with per-span latency tracing.
• Cache thrash — too many distinct prefixes for the pool, so reuse never hits. Standardize prefixes (context engineering) to keep the working set small.
• Cross-tenant reuse hazard — sharing KV across trust boundaries can leak data if it isn't carefully scoped. See multi-tenant cache safety.

Practitioner checklist

• Have you computed KV memory for your max context × target concurrency?
• Does your model use GQA/MQA (and could FP8 KV buy headroom)?
• Is there admission control so you reject rather than OOM?
• Do you monitor preemption/swap rate and its effect on p99?
• Are shared prefixes scoped so reuse never crosses a tenant boundary?

Related lessons

• Prefill vs. decode
• Continuous batching & paged attention
• Prompt vs. semantic caching
• Quantization formats
• Multi-tenant isolation & cache safety

TL;DR

Generation has two phases with opposite hardware profiles. Prefill processes the entire prompt in parallel to build the KV cache. It is compute-bound, and its cost scales with input length. Decode then emits output tokens one at a time. Each step reads the whole model and KV cache, so decode is memory-bandwidth-bound, and its cost scales with output length. The two phases have different bottlenecks, so you optimize them with different techniques. Mix them up and you'll optimize the wrong thing.

🎯 For the AI-native PM

Why it matters — "It feels slow" has two different causes: time-to-first-token, or per-token speed. Each has a different fix. Knowing which one you have lets you fund the right latency work instead of guessing.

What it changes in your decisions — Your latency SLOs, UX choices (streaming, skeleton states), and which optimization you pay for.

Ask your eng team — "Is our latency problem time-to-first-token, or tokens-per-second?"

Product risk if ignored — You fund the wrong latency fix and the number you promised the exec doesn't move.

Mental model

PREFILL  (process N prompt tokens together)        DECODE (generate one token at a time)
  ─ one big parallel matmul over all input          ─ T sequential steps
  ─ saturates the GPU's compute (FLOPs)             ─ each step: load weights + KV, do tiny matmul
  ─ cost ∝ input length                              ─ cost ∝ output length
  ─ bottleneck: arithmetic throughput               ─ bottleneck: memory bandwidth

The two user-visible latency numbers map directly onto the phases:

• TTFT — Time To First Token ≈ prefill time (plus queueing). Input length and prompt-cache hits dominate it.
• TPOT / ITL — Time Per Output Token / Inter-Token Latency ≈ decode step time. Model size, memory bandwidth, and batch dynamics dominate it.

Total latency ≈ TTFT + (num_output_tokens × TPOT).

Why they're bound by different limits

• Prefill is compute-bound because the server processes all prompt tokens at once: large, dense matrix multiplies that keep the GPU's compute units busy. Doubling the prompt roughly doubles prefill work. The fix is to do less of it: shorten or compress the input (context engineering), or skip it with prefix caching.
• Decode is memory-bandwidth-bound because each single-token step must stream the entire set of model weights (and the growing KV cache) through the compute units to produce one token. The arithmetic per step is tiny; the bottleneck is moving bytes. The GPU sits underutilized on compute, which is exactly why batching helps decode so much — more sequences share each weight read.

How this changes what you optimize

Key consequences:

• Speculative decoding only helps decode, not prefill. It attacks the sequential token-by-token bottleneck. Don't expect it to fix a slow TTFT caused by a huge prompt.
• Quantization can help both, but for different reasons: it moves less data (which helps bandwidth-bound decode) and sometimes does faster math (which helps compute-bound prefill).
• Batching is mostly a decode/throughput lever. It improves tokens per second across many users, but it can slightly raise any single user's TPOT.
• Chunked prefill interleaves prefill of new requests with ongoing decode, so a giant prompt doesn't stall everyone else's token stream. It's a scheduling fix for the prefill/decode interference problem — for the more aggressive fix, running prefill and decode on separate GPU pools entirely, see prefill/decode disaggregation.

Workload shape matters

• Long input, short output — classification, extraction, a RAG answer over a big context — is prefill-dominated. Prompt caching and input compression are your biggest wins.
• Short input, long output — open-ended generation, long agent turns — is decode-dominated. Speculative decoding, smaller models, and batching matter most.
• Knowing which regime you're in tells you where to spend engineering effort. Measure TTFT and TPOT separately in observability — a single "latency" number hides which phase is the problem.

Failure modes

• Optimizing the wrong phase — adding speculative decoding to fix latency that is actually 90% prefill of a 30k-token prompt.
• Prefill stalls — one user's huge prompt monopolizes the GPU and spikes everyone else's TPOT. Fix this with chunked prefill or scheduler tuning.
• Reporting only average latency — this hides that TTFT is fine but TPOT is terrible, or the reverse. Always split the two.

Practitioner checklist

• Do you measure TTFT and TPOT as separate metrics?
• Do you know whether your workload is prefill- or decode-dominated?
• Is prompt caching enabled for long, shared prefixes (prefill win)?
• Are decode-side wins (spec decoding, batching) applied where output is long?
• Does a large prompt from one user degrade others' token stream?

Related lessons

• KV cache management
• Continuous batching & paged attention
• Speculative decoding vs. quantization vs. distillation
• Inference-stack tradeoffs

TL;DR

Two innovations made modern LLM serving economical. Continuous batching keeps the GPU full by adding and removing requests at the token level instead of waiting for a whole batch to finish. Paged attention manages the KV cache in fixed-size, non-contiguous blocks (like OS virtual memory), so memory isn't wasted or fragmented. That lets you fit more sequences and therefore batch more. Together they turn the bandwidth-bound decode phase into a high-throughput one.

🎯 For the AI-native PM

Why it matters — This is the throughput-vs-latency dial for self-hosted inference, and it sets the gross margin on your AI features more than almost anything else.

What it changes in your decisions — Build-vs-buy for inference, your margin model, and the latency-vs-cost point you choose to operate at.

Ask your eng team — "What's our cost per token at our latency target, and how much does relaxing latency a little save us?"

Product risk if ignored — The margins assumed in your business case never materialize because the serving stack was never tuned.

The problem they solve

Decode is memory-bandwidth-bound: each step streams the whole model through the compute units to produce one token, leaving compute units idle. The cure is batching — process many sequences per weight read so each expensive byte-movement serves many users. But two things get in the way:

1. Ragged completion. In a static batch, sequences finish at different times. With naive batching the whole batch waits for the slowest sequence, and finished slots sit idle → wasted GPU.
2. Memory waste. Reserving a contiguous max-length KV block per sequence wastes memory (most outputs are short) and fragments the pool, capping how many sequences fit.

Continuous (in-flight) batching

Instead of "form batch → run to completion → form next batch," the scheduler operates per decoding step:

every step:
  - run one decode step for all active sequences
  - any sequence that emitted EOS → evict, free its KV, return result
  - any newly arrived request → admit, schedule its prefill, add to the batch

Effects:

• GPU stays saturated — finished slots are immediately refilled by waiting requests. No idle bubbles.
• New requests don't wait for a batch boundary — they join almost immediately, so queueing latency drops.
• Throughput rises sharply at a given latency target compared to static batching.

The catch: prefill of a newly admitted request can momentarily interfere with ongoing decode (a TTFT-vs-TPOT tension). Chunked prefill mitigates this. It slices big prefills and interleaves them with decode steps, so no single large prompt stalls the token stream.

Paged attention

Borrowing virtual-memory ideas from operating systems: divide the KV cache into fixed-size blocks (pages). A sequence's KV is a list of blocks that need not be contiguous. A block table maps logical positions to physical blocks.

Benefits:

• Near-zero fragmentation — allocate blocks on demand as a sequence grows, instead of reserving its maximum up front. More sequences fit in the same HBM → bigger batches → higher throughput.
• Cheap sharing (copy-on-write). Sequences with a common prefix can share the physical blocks for that prefix, and only diverge when they differ. This is the mechanism behind efficient prefix caching and parallel sampling or beam search.
• Clean eviction/swap units. Blocks are natural granules to evict or swap to host memory under memory pressure.

How they reinforce each other

Paged attention raises how many sequences fit in memory. Continuous batching turns that headroom into sustained GPU utilization. The product is dramatically higher throughput — and therefore lower cost per token — at comparable latency. This is the core reason a well-configured open-source server (vLLM, TGI, TensorRT-LLM, SGLang) can serve far more traffic per GPU than a naive loop.

Prefill/decode disaggregation — separating the pools entirely

Chunked prefill mitigates prefill/decode interference on a shared pool of GPUs. The more aggressive fix removes the interference at the root: run prefill and decode on separate GPU pools entirely. A request's prefill runs on a compute-optimized pool, its KV cache transfers over a fast interconnect, and decode continues on a pool tuned for memory-bandwidth-bound work. Neither phase ever waits behind the other, because they were never sharing hardware to begin with.

This is now a mainstream production serving architecture (DistServe, Mooncake, NVIDIA Dynamo), not a research curiosity. The tradeoff is operational complexity — two pools to provision, monitor, and keep balanced, plus a KV-transfer hop that adds its own latency — against a ceiling chunked prefill alone can't reach: at high enough concurrency, no scheduling trick fully hides one phase's footprint from the other on shared hardware. Disaggregation is the lever teams reach for once that ceiling is the actual bottleneck, not before.

Tradeoffs

There is no free lunch: pushing batch size up improves cost and throughput but can raise tail latency for individual requests. The right operating point is an SLO decision, not a default.

Failure modes

• Throughput tuned, latency forgotten — batch sizes cranked for cost, p99 TPOT quietly blows the SLO. Track both.
• Prefill stalls — a 32k-token prompt admitted mid-flight freezes everyone's token stream for a moment; needs chunked prefill.
• KV exhaustion — high concurrency × long context overruns the block pool → preemption/swap and tail-latency spikes (see KV management).

Practitioner checklist

• Are you on a server that does continuous batching + paged attention?
• Have you set a max batch size / max-num-seqs tied to an SLO, not just "max"?
• Is chunked prefill enabled if you mix long prompts with latency-sensitive decode?
• Do you alert on KV-pool utilization and preemption rate?
• Do you track throughput and p99 TTFT/TPOT together?

Related lessons

• KV cache management
• Prefill vs. decode
• Prompt vs. semantic caching
• Inference-stack tradeoffs

TL;DR

Three ways to make inference faster or cheaper, with very different risk profiles. Speculative decoding speeds up decode with zero quality change — the output distribution is provably preserved. But it needs extra memory, and it helps only when a small draft model agrees often. Quantization shrinks the model to use less memory and bandwidth, which is cheaper and faster, with a bounded, measurable quality risk. Distillation trains a genuinely smaller model. It's the biggest speed/cost win, but it carries the largest and least reversible quality risk, plus real training cost. Pick by which resource you're short on and how much quality risk you can carry.

🎯 For the AI-native PM

Why it matters — These are the three knobs for "make it cheaper and faster," and each carries a different quality risk. Pick the wrong one, and it shows up as a silent quality regression your users feel.

What it changes in your decisions — Your cost-reduction roadmap, how much quality risk you'll accept, and the eval gates you require before rollout.

Ask your eng team — "Which of these are we using, and did quality hold on our eval set after we turned it on?"

Product risk if ignored — A cost-cutting change quietly degrades the experience and no one's eval catches it.

The three techniques

Speculative decoding — free speed, same answer

A small, fast draft model proposes the next k tokens. The large target model verifies all k in a single forward pass — parallel verification is cheap because the target is bandwidth-bound, not compute-bound. The server keeps accepted tokens; on the first rejection, it falls back to the target's own token. Because verification uses the target's true probabilities, the output distribution is identical to plain decode.

• Wins: lower TPOT / inter-token latency. It can be 1.5–3× faster on agreeable workloads.
• Costs: extra GPU memory for the draft model. The benefit depends entirely on the acceptance rate — if the draft rarely agrees, you pay overhead for little gain.
• Scope: decode only. It does nothing for prefill-dominated (long-prompt) latency.
• Variants: separate draft model, Medusa-style extra heads, n-gram or lookahead decoding, EAGLE.

Quantization — smaller model, bounded quality risk

Store, and sometimes compute, weights and activations in fewer bits — FP16 to INT8, FP8, or INT4. Moving less data directly helps the bandwidth-bound decode phase and frees KV/memory headroom for bigger batches.

• Wins: lower memory footprint, lower $/token, often lower latency, more concurrency.
• Costs: quality degradation that grows as bits shrink, and that varies by method and by what you quantize — weights, activations, or KV. 8-bit is usually near-lossless. 4-bit needs care. Below that, quality often falls off a cliff.
• Reversibility: high. It's a post-training transform you can dial back.
• Full detail and method comparison: Quantization formats.

Distillation — a new, smaller model

Train a small student to imitate a large teacher, matching its outputs or output distributions. The result is a permanently smaller, cheaper, faster model specialized to your task distribution.

• Wins: the largest steady-state speed/cost reduction. It's great for a narrow, high-volume task.
• Costs: upfront training effort and data. The quality ceiling is lower and depends on the teacher and data. Generality is lost — the student is good at what it was distilled for and can be brittle off-distribution.
• Reversibility: low. You've trained an artifact and built a pipeline around it.

Choosing between them

They are composable: a common production stack is a quantized model served with speculative decoding, and for a hot narrow task, a distilled student that is itself quantized. They attack different resources, so stacking compounds the wins.

Decision guide

1. Latency-bound, output-heavy, can't touch quality? Reach for speculative decoding first.
2. Memory/cost-bound, and can tolerate a small, measured quality dip? Use quantization — start at INT8/FP8, validate, and only go to INT4 if evals hold.
3. One narrow, very high-volume task where a big model is overkill? Use distillation, then quantize the student.
4. Always: gate every one of these behind your eval suite. Speculative decoding shouldn't move evals at all — that's a red flag if it does. Quantization and distillation will move evals, so you must measure by how much.

Failure modes

• Speculative decoding with a bad draft — a low acceptance rate makes it net-neutral or slower. Measure acceptance rate, not just "it's enabled."
• Quantizing past the cliff — INT4 on a task with tight numeric or format demands silently degrades. Only evals catch this, not eyeballing.
• Distilling on the wrong distribution — the student looks great offline but fails on the long tail of real traffic it wasn't distilled for.

Practitioner checklist

• Are you short on latency, memory, or steady-state cost? (picks the technique)
• For spec decoding: do you monitor draft acceptance rate?
• For quantization: did evals hold at the chosen bit-width?
• For distillation: does the student's eval set reflect production traffic?
• Are all three gated by regression evals before rollout?

Related lessons

• Quantization formats: INT8, INT4, FP8, AWQ, GPTQ
• Prefill vs. decode
• Fine-tuning vs. ICL vs. RAG vs. distillation
• Inference-stack tradeoffs
• Evals

TL;DR

Quantization stores numbers in fewer bits. This shrinks the model's memory footprint and moves less data per decode step, buying lower cost, more KV/batch headroom, and often lower latency. The format (INT8, FP8, INT4) sets the precision/range ceiling. The method (AWQ, GPTQ, and others) decides how cleverly you map full-precision weights into that budget. 8-bit is usually near-lossless. 4-bit is viable with a good method. Below that, quality typically collapses. Always confirm with evals — degradation is task-specific and invisible to the eye.

🎯 For the AI-native PM

Why it matters — "We quantized to save money" can be free — or it can wreck quality on exactly the hard tasks (math, code, strict JSON) your product depends on. It's a margin decision with a quality blast radius.

What it changes in your decisions — Whether to approve a quantization change, and which high-value task types to guard with evals.

Ask your eng team — "At what bit-width did our evals start to slip, and on which task types?"

Product risk if ignored — You bank the savings and inherit a subtle quality cliff on your highest-value workflows.

What "quantize" actually means

A full-precision weight is FP16/BF16 (16 bits). Quantization maps ranges of those values onto a smaller set of levels:

• Integer (INT8, INT4): pick a scale (and maybe zero-point) per tensor/channel/ group, then round weights to integers. Fewer bits = coarser grid = more rounding error.
• Float (FP8): keep a floating layout (e.g. E4M3 = 4 exponent + 3 mantissa bits, or E5M2). Floats preserve dynamic range far better than integers at the same bit count. That matters for the large outlier values that show up in activations.

What you quantize matters as much as the bit count:

• Weight-only (most common for INT4) — weights are small precision, math often still done in FP16. Great memory win, modest compute win, lower quality risk.
• Weight + activation (e.g. W8A8 INT8, or FP8 both) — this also speeds up the matmuls. But activations have outliers that are hard to quantize, and they are the usual source of quality loss.
• KV-cache quantization — store the KV cache in FP8/INT8 to roughly double or quadruple concurrency. This is a separate, valuable lever.

The formats

FP8 vs INT8 at the same 8 bits: FP8's exponent gives it the range to absorb activation outliers, so it often quantizes activations more gracefully. This is handy on hardware with native FP8 support. INT8 has the widest software and hardware support. The same relationship repeats one tier down: FP4 vs. INT4 at 4 bits, FP4 keeps a float layout's dynamic range where INT4's fixed grid struggles most, and it's now natively accelerated on Blackwell-class GPUs. The rule doesn't change with the tier — confirm gains with evals, not intuition about the format's name.

The methods (how to hit 4-bit without wrecking quality)

Naive round-to-nearest at 4-bit loses too much. Smarter post-training quantization (PTQ) methods use a small calibration dataset to minimize error:

• GPTQ — quantizes weights one column or group at a time. It uses approximate second-order (Hessian) information to compensate remaining weights for the error introduced, minimizing layer-wise output error. Results are strong for 4-bit weight-only quantization, but the method is sensitive to the calibration set.
• AWQ (Activation-aware Weight Quantization) — observes that a small fraction of weight channels (those multiplied by large activations) dominate quality. It scales to protect those salient channels and quantizes the rest aggressively. Robust at 4-bit, less sensitive to the calibration set, fast.
• Others you'll meet: SmoothQuant (shifts activation outliers into weights so W8A8 works), GGUF k-quants (llama.cpp's mixed-bit CPU/edge formats), bitsandbytes (NF4/INT8 for QLoRA-style training and easy loading), and SpinQuant/QuaRot (rotations that make 4-bit activations tractable).

Rule of thumb: the format sets the ceiling. The method determines how close to it you get. "INT4" with AWQ/GPTQ ≫ "INT4" with naive rounding.

When quantization hurts

Degradation is not uniform — it concentrates in specific places:

• Lower bits, more risk — INT8 and FP8 are usually safe. INT4 needs a good method. 3-bit or lower usually hurts.
• Activation quantization is riskier than weight-only because of outliers.
• Long-context / KV quantization can accumulate small per-token errors over a long sequence.
• Hard, precision-sensitive tasks suffer most: math/arithmetic, code, strict structured output / JSON, low-resource languages, and long multi-step reasoning. Easy classification/summarization barely notices.
• Small models have less redundancy to spare and degrade more than large ones at the same bit-width.
• Calibration mismatch — calibrating GPTQ/AWQ on data unlike production traffic bakes in error where you'll actually use the model.

Because it's task-specific, you cannot judge quantization by spot-checking a few prompts. You need a task-representative eval set, ideally including adversarial and format-strict cases. Run it at each candidate bit-width.

Tradeoffs

Practitioner checklist

• Start at INT8/FP8; only go INT4 if evals hold.
• Use AWQ or GPTQ (not naive rounding) for 4-bit.
• Calibrate on data that looks like production traffic.
• Run task-representative + adversarial evals at each bit-width.
• Pay special attention to math, code, and strict-JSON tasks.
• Consider KV-cache quantization as a separate concurrency lever.
• Confirm your hardware actually accelerates the chosen format (e.g. FP8 on Hopper+).

Related lessons

• Speculative decoding vs. quantization vs. distillation
• KV cache management
• Prefill vs. decode
• Evals
• Structured output