Machine learning for the product and technology leader
What a learned system is, what it learns from, how it learns, how to trust its score, which kind of model to pick, and how to keep it working after launch.
A learned model is a bet that the past resembles the future. The costly failures sit in the data, the test and the lifecycle, not in the model.
Most AI products rest on one idea. A program improves at a task by learning from examples, not by following rules a person wrote. A spam filter, a fraud score, a price forecast and a large language model all work this way. They differ in scale, not in kind.
Product and technology leaders make decisions about this idea every week. Is this a rules problem or a learning problem? Is the data good enough? Is a 94% score real? Why did the model get worse after launch? Those decisions do not need the mathematics. They need a clear picture of the learning loop and of the five ways it fails.
This module gives that picture. Every lesson has two layers. The body is for the leader who decides. A short Under the hood section is for the engineer who builds. It shows real code that runs on the Python standard library alone, with no install, and ends in assert lines so the claims in the lesson are checked.
One running example. A subscription company wants to predict which customers will cancel in the next 30 days. About one customer in five does. The data is invented and generated by code with a fixed seed, so every number in the lessons can be reproduced. Each lesson meets the same dataset and learns one more way it can go wrong.
A note on scope. This module does not teach prompting, retrieval, fine-tuning, or how a transformer works. LLMs covers the model most products call, and its attention section explains the core of a transformer. This module stays one level lower and earlier. It answers the questions that come before and after any model: whether you need one, what it learns from, how it learns, how to trust its score, and how to keep it working. Deep networks get their own modules: Tensors is published, and convolutional networks follow.
Where the depth lives
Each lesson summarizes an idea and points to the lesson that covers it in full.
| If you want the full depth on | Read |
|---|---|
| Choosing between prompting, retrieval and fine-tuning for an LLM | Prompting vs. RAG vs. fine-tuning |
| Build, buy or fine-tune as a business decision | Build, buy, or fine-tune |
| What training and inference mean for a language model | What an LLM actually is |
| Precision and recall for search and retrieval | Retrieval quality |
| Building an eval set and watching an LLM feature | Evaluation and observability |
| Running an online experiment on the product number | Metrics and experimentation |
| What training and serving cost over a model's life | Cost optimization |
| Operating a system that acts, with limits and a kill switch | Running an agent in production |
The knowledge graph
From a decision to a system that keeps working
Read left to right, then follow the dashed arrow back: the world moves, so the loop runs again.
1Decide
Rules, a trained model or an LLM?
2Feed it
Features, labels, honest splits
3Teach it
Loss, gradient descent
3-4Generalize
Overfit or underfit?
5Trust it
Metrics, cutoffs, baselines
6Shape it
Linear, trees, networks
7Keep it working
Skew, drift, lifecycle
8Work with agents
Contract, checks, review
↺ retrain when the world moves — back to lesson 2
Read it as one loop with a feedback arrow. You decide that learning is the right tool. You feed it data that matches the future. It learns by repeated small corrections, and you check that it generalizes to cases it has not seen. You measure it against a baseline with a threshold that fits your costs. You choose a shape of model that fits the data. Then you keep it working, because the world moves and the loop has to run again.
The lessons
- What machine learning is, and when to use it — three parts of every learned system, four ways a machine learns, and a test for rules, a trained model or an LLM.
- Data: features, labels, splits and leakage — what the model learns from, and why a score that looks too good deserves a leak check.
- Training: loss and gradient descent — the loop that every model uses, and how to read a loss curve.
- Generalization: overfitting and the bias–variance tradeoff — why a model fails on new cases, and which of three fixes to buy.
- Measuring a model: metrics, thresholds and baselines — accuracy's traps, confusion matrices, cost-based cutoffs and calibration.
- Model families: linear models, trees and neural networks — what each can see, and backpropagation in one page.
- ML in production: drift, skew and the lifecycle — the five failure modes after launch, and the monitors that catch them.
- Learning and doing ML with Claude Code — how to run experiments through a coding agent without fooling yourself, and how to use the agent to learn this module.
Then the recap and real-world examples, with a self-test.
Run the code
Every example is in machine-learning/code/. Run them all with one command.
python3 machine-learning/code/test_examples.py
The files are lesson1_rules_vs_learning.py through lesson8_agent_guardrails.py, with shared pieces in data.py, tree.py and mlp.py. A passing file checks the direction of each claim, such as a deep tree scoring worse on new rows. Run a file to see every quoted number printed.