Fine-tuning and post-training for the product and technology leader
How a base model becomes an assistant, what supervised tuning, LoRA and preference tuning change, why the dataset is the product, and how to test the break-even and gate the release.
Tuning is a small push on a model that already knows most of what it knows, so it changes behaviour well, adds facts badly, and needs a test and a break-even before you start.
A language model starts as a base model. It predicts the next token, and that is all. The assistant you use is that model after post-training: supervised fine-tuning on example answers, preference tuning on pairs of better and worse answers, and sometimes reinforcement learning against a check. Teams can add their own fine-tuning on top.
Fine-tuning is the most over-chosen option in AI roadmaps. It feels like owning the model. It is good at changing behaviour, such as format, tone and a narrow task. It is a poor way to add facts. It has side effects, and it has a fixed cost that only high volume repays.
This module teaches how it works, so you can decide. You learn what each stage changes, what a few examples can and cannot do, why LoRA made tuning cheap, how preference tuning goes wrong, why the dataset is the real product, and how to test the break-even and gate the release.
Every lesson has two layers. The body is for the leader who decides. A short Under the hood section is for the engineer who builds. It points to code that runs on the Python standard library, with no install, and ends in assert lines. The code trains a very small next-word model by gradient descent, so each effect can be measured: the behaviour learned, the damage to everything else, the adapter that can be removed, the reward that gets gamed. The numbers are real for the toy and illustrative for real models.
A note on scope. The decision between prompting, retrieval and tuning is covered in RAG vs. long context vs. fine-tuning and the strategy lesson. This module does not repeat it. It adds the mechanics, the data, the cost model and the checks.
Where the depth lives
Each lesson summarizes an idea and points to the lesson that covers it in full.
| If you want the full depth on | Read |
|---|---|
| Whether to prompt, retrieve or tune | RAG vs. long context vs. fine-tuning and Build, buy or fine-tune |
| The training loop and overfitting | Training: loss and gradient descent and Generalization |
| Memory for weights, gradients and optimizer state | Batching, memory and precision |
| What the weights are made of | The block and The shape of a transformer |
| Smaller number formats and distillation for serving | Quantization formats and Distillation |
| Evals, judges and monitoring | Evaluation and observability |
| Data poisoning and what tuning does for safety | AI security and guardrails |
| What a model costs over its life | Cost optimization |
The knowledge graph
A chain with a fan
The pipeline shows where tuning sits. Three kinds of work feed one decision.
1The pipeline
pretrain · SFT · preference · verifiableWhere tuning sits.
2Supervised tuning
data · speed · replay · labelsThe step most teams run.
3LoRA
a small patch on a frozen modelCuts the cost.
4Preference tuning
reward models · DPO · reward hackingShapes taste, invites gaming.
5The dataset
audit · leaks · synthetic dataDecides what any of it learns.
6Decide, cost, gate
break-even · release gateDoes it pay, and did it pass?
When the base model changes, the loop starts again.
Read it as a chain with a fan. The pipeline shows where tuning sits. Supervised tuning is the step most teams run, and it has four dials. LoRA cuts its cost. Preference tuning shapes taste and invites gaming. The dataset decides what any of them learns. All three feed one decision: whether it pays, and whether it passed the gate. When the base model changes, the loop starts again.
The lessons
- From base model to assistant: the post-training pipeline — what each stage uses and teaches, and what it breaks.
- Supervised fine-tuning — how much data, how fast, what to mix in, and how labels matter.
- LoRA and parameter-efficient tuning — a small, removable patch, and what it saves.
- Preference tuning and reward hacking — reward models, DPO, and why the proxy gets gamed.
- The dataset: quality, duplicates, contamination and synthetic data — the audit that comes before training.
- Deciding, costing and checking a fine-tune — break-even, distillation and the release gate.
Then the recap and real-world examples, with a self-test.
Run the code
Every example is in fine-tuning/code/. Run them all with one command.
python3 fine-tuning/code/test_examples.py
The files are lesson1_pipeline.py through lesson6_decide.py, built on toy_lm.py. A passing file checks the direction of each claim, such as general loss rising after narrow tuning, or dropping the adapter restoring the base model exactly. Run a file to see every quoted number printed.