Transformers for the product and technology leader
The stack behind almost every language model: attention, the block, position and the cost of context, the KV cache, and how to read a model card as memory, compute and cost.
A transformer is one block repeated, so almost every question about it is a counting question: parameters, multiply-adds and cache.
Almost every language model you buy or build is a transformer. It is a stack of identical blocks, and each block does two things. Attention lets each token read from the others. A feed-forward layer then reworks each token on its own. That is the whole design. The rest is scale, and a few well-chosen tricks.
You do not need the mathematics to decide well. You need to count. How many parameters? How many are active for each token? How much memory does a request add? How does cost change when the input is ten times longer? This module teaches the parts, so you can turn a model's spec sheet into memory, compute and cost on one page.
Every lesson has two layers. The body is for the leader who decides. A short Under the hood section is for the engineer who builds. It points to code that runs on the Python standard library, with no install, and ends in assert lines. The code builds one small transformer, with random weights, and a counter for multiply-adds. The weights say nothing useful. Their job is to let every claim be counted and checked. The matrix multiplication underneath is the one from Tensors.
A note on scope. LLMs covers the model most products call: tokens, context, prompting and choosing. This module goes one level lower, into the machine. Inference internals covers serving in engineering depth. Here you get the arithmetic behind those lessons. The delta over llms/what-is-an-llm is the counted, runnable version: parameters, multiply-adds, the cache and the model card.
Where the depth lives
Each lesson summarizes an idea and points to the lesson that covers it in full.
| If you want the full depth on | Read |
|---|---|
| What an LLM is, and how to choose one | What an LLM actually is and Choosing a model |
| The context window as a product limit | The context window |
| Serving: prefill, decode, batching and the cache | Prefill vs. decode and KV cache management |
| Smaller number formats for weights and cache | Quantization formats |
| Why the multiplies dominate | Matrix multiplication: the workhorse |
| The training loop and gradients | Training: loss and gradient descent and Automatic differentiation |
| The same block applied to images | CNNs vs. vision transformers |
| What a model costs over its life | Cost optimization |
The knowledge graph
Six lessons, one loop
Count the parameters, the multiply-adds and the cache, then turn a spec sheet into a cost.
1The shape
layers · width · parametersFive numbers on the spec sheet give the weights.
2Attention
query · key · valueHow a token reads from other tokens.
3The block
heads · feed-forward · residualsWhat repeats, and where the weights live.
4Position and context
order · the n² costWhy long inputs cost more than they look.
5Generation
prefill · decode · KV cacheWhy memory, not weights, sets concurrency.
6Variants and the model card
grouped-query · experts · tilesReading total and active size.
The loop closes: a spec sheet becomes a serving cost, and a serving cost sends you back to the sheet.
Read it as a chain with one loop. You see the shape: five numbers on a spec sheet. Attention is the one step where tokens exchange information. The block pairs it with a feed-forward layer and keeps a running notebook, the residual stream. Position fixes order, and the cost of attention grows with the square of the length. Generation writes one token at a time, and a cache of keys and values keeps that affordable and sets how many requests fit on a machine. The variants change these costs. Reading a model card then closes the loop, from spec sheet back to the shape.
The lessons
- The shape of a transformer — five numbers, where the parameters live, and what a forward pass costs.
- Attention — queries, keys and values, the scale, the mask, and a head you can build by hand.
- The block: heads, the feed-forward layer and the residual stream — what one block does, and why the feed-forward layer is most of the weight.
- Position, and the cost of context — why order must be added, and when attention takes over the bill.
- Generating text and the KV cache — prefill, decode, and why memory per request limits serving.
- Variants, and reading a model card — grouped-query attention, mixtures of experts and tiled kernels.
Then the recap and real-world examples, with a self-test.
Run the code
Every example is in transformers/code/. Run them all with one command.
python3 transformers/code/test_examples.py
The files are lesson1_shape.py through lesson6_variants.py, built on mini_transformer.py. A passing file checks the direction of each claim, such as the cache giving the same tokens as recomputing. Run a file to see every quoted number printed.