Learning path: AI Engineering Lead
Last reviewed: 2026-10 · Volatility: medium
TL;DR
This path is the engineer path in short form, plus what a lead adds: economics, architecture review, governance, and working with product. You read for judgement. You skim the build lessons, so you can ask your team the right questions, and you read the decision lessons in full.
Who it is for: an engineering manager or tech lead who runs a team that builds AI systems.
Outcome: You review AI architecture, set quality and safety standards, control cost, and work well with product.
Before you start: Years of engineering and some time running a team. You may not have shipped an AI feature yet.
Length: about 42.5 hours, across 5 stages. The time is a rough estimate (see Under the hood).
Terms: a large language model (LLM) is a model like those behind chat assistants; the plural is large language models (LLMs); machine learning (ML) is the practice of training models from data.
🎯 For the AI engineering lead
Why it matters — Your team will build what you can review. If you cannot read an architecture, an eval report or a threat model, you cannot lead the work.
What it changes in your decisions — You set standards for evals, safety and cost before the build. You approve designs by their failure handling.
Ask your team — "Show me how this fails, how we would notice, and what it costs per request."
Risk if ignored — The team ships fast and unsafe. Review happens after an incident.
The path at a glance
Each stage ends with a checkpoint. Move on when you can do what the checkpoint says.
| Stage | Focus | About |
|---|---|---|
| 1 | How models behave and what they cost | about 13 hours |
| 2 | Architecture you can review | about 16 hours |
| 3 | Quality, safety and access | about 6 hours |
| 4 | Systems at scale | about 2.5 hours |
| 5 | Work with product and run the team | about 5.5 hours |
Stage 1: How models behave and what they cost
Cost and latency come from model and serving choices. Know the levers. (about 13 hours)
- LLMs: 1. What an LLM actually is · 2. The context window · 3. Capabilities & the jagged frontier · 6. Choosing a model · 7. Prompting vs. RAG vs. fine-tuning
- Machine learning: 1. What machine learning is, and when to use it · 2. Data: features, labels, splits and leakage · 4. Generalization: overfitting and the bias–variance tradeoff · 5. Measuring a model: metrics, thresholds and baselines · 6. Model families: linear models, trees and neural networks · 7. ML in production: drift, skew and the lifecycle · 8. Learning and doing ML with Claude Code
- Tensors: 1. What a tensor is: shape, rank and dtype · 3. Matrix multiplication: the workhorse · 4. Batching, memory and precision
- Transformers: 1. The shape of a transformer · 4. Position, and the cost of context · 5. Generating text and the KV cache · 6. Variants, and reading a model card
- Fine-tuning and post-training: 1. From base model to assistant: the post-training pipeline · 3. LoRA and parameter-efficient tuning · 6. Deciding, costing and checking a fine-tune
- AI engineering: inference internals: Prefill vs. decode latency · Continuous batching & paged attention · Prompt caching vs. semantic caching
- AI engineering: reliable outputs: Model routing, graceful fallback logic, and degraded-mode UX
- AI engineering: evals & observability: Cost attribution per feature, workflow, tenant, and user journey — not just per model
- AI engineering: strategy & tradeoffs: Fine-tuning vs. in-context learning vs. RAG vs. distillation — and when each is the wrong tool · Latency, quality, cost, and reliability across the full inference stack
- Cost optimization: 1. The cost stack, and the build-vs-buy breakeven · 2. FinOps for AI: budgets, forecasting & the cost review
Ready to move on when: You can explain where a request's time and money go and name the three levers that change them.
Stage 2: Architecture you can review
Know the shapes: grounding, context, memory, agents, workflows. Skim the build lessons. (about 16 hours)
- RAG & vector databases: 1. When to use RAG, and when not to · 2. Why RAG? · 6. Retrieval quality · 7. RAG vs. long-context vs. fine-tuning · 9. Building RAG with Claude Code, pgvector and Pinecone
- Context engineering: 1. What is context engineering, for a product leader? · 3. The anatomy of a context pipeline · 5. Context governance at scale
- Memory & context: 1. Memory as a product decision · 5. When memory goes wrong
- Tool calling: 2. Tool contracts & reliability
- Knowledge graphs: 1. What is a knowledge graph? · 8. Knowledge graphs as a product
- AI agents: 1. What an agent is, and how much autonomy it needs · 3. When not to build an agent · 4. Running an agent in production
- Agentic workflows: 1. Choosing a workflow pattern · 2. Orchestrating more than one agent · 3. Making a workflow durable, and worth owning
- Harness engineering: Phase 01 — Foundations and the Loop (5 lessons, skim it)
- Harness engineering: Phase 06 — Permissions and Security (4 lessons, skim it)
- Harness engineering: Phase 07 — Planning and Subagents (4 lessons, skim it)
- Harness engineering: Phase 09 — Reliability, evals, and ops (5 lessons, skim it)
Ready to move on when: You can review a design doc for an agent: its loop, tools, permissions, budgets and failure handling.
Stage 3: Quality, safety and access
Set the standards. Evals, threat models and access control are yours to require. (about 6 hours)
- Evaluation & observability: 1. Why eval investment is the job · 2. Building the eval stack in the right order
- AI engineering: evals & observability: Evals: golden sets, regression tests, adversarial tests, LLM-as-judge, and human evals · LLM observability: traces, spans, tokens, latency, errors, and drift
- AI security & guardrails: 1. The threat model, and guardrails as architecture · 2. Red-teaming: testing your defenses · 3. Governance, audit & compliance
- Access control: 1. Authentication, authorization and the access-control model · 2. OAuth 2.0, OpenID Connect and tokens · 5. ReBAC and policy engines · 8. Access control for AI agents
- AI engineering: safety & multi-tenancy: Safety engineering: prompt injection defense, data leakage prevention, and permission boundaries · Multi-tenant isolation, cache safety, and cross-user context contamination prevention
- AI engineering: strategy & tradeoffs: Production failure modes & how to engineer around them
Ready to move on when: You can name the release gate for an AI feature, who owns it, and what evidence a buyer would ask for.
Stage 4: Systems at scale
AI features still run on ordinary systems. Hold the same bar for reliability. (about 2.5 hours)
- System design: 1. Foundations & framework · 2. Numbers, formulas and papers to know · 3. Traffic, caching, APIs and storage choice · 7. Real-time systems
- Technical product sense: 5. Reliability & failure · 8. The economics of infrastructure
Ready to move on when: You can run a design review for scale, failure and cost on any service in the stack.
Stage 5: Work with product and run the team
Turn standards into process: specs, launches, incidents and how product and engineering share ownership. (about 5.5 hours)
- Technical product management: 3. Specs, PRDs & RFCs · 4. Prioritization & roadmaps · 5. Working with engineering · 6. Metrics & experimentation · 7. Launches, rollouts & migrations · 8. Incidents & postmortems · 9. Technical product management for AI
- AI-assisted development: 1. When an agent writes the code: where the bottleneck goes · 2. Specs and the context file · 4. Control: permissions, modes and hooks · 5. Running at scale: sessions, subagents and fan-out · 6. Measuring and governing
- Product sense: 7. Product sense for AI products
Ready to move on when: You and your PM share one definition of done, one quality bar and one incident process.
Skip-ahead rules
- If you came from machine learning, skip the first stage's model lessons and read the cost lessons.
- If your team already runs evals in CI, read the evaluation lessons as a gap check.
- If you want hands-on depth, add the build lessons from the AI Engineer path.
- Flowable (process automation) is not on this path. Add it if your team builds approval or long-running workflows.
Worked example: a week-by-week plan
This example is invented, to show the method.
An AI engineering lead with three hours a week plans 5 weeks of work. Each week ends with something you can show. This plan is a fast tour: one activity per stage. At this pace the full path (about 42.5 hours) takes about 15 weeks.
| Week | Stage | What you do |
|---|---|---|
| 1 | Stage 1 | Pull one month of cost and latency numbers for your busiest model call. Find the biggest lever. |
| 2 | Stage 2 | Review one agent design doc against the loop, tools, permissions and budget. |
| 3 | Stage 3 | Write the release gate for one feature and name its owner. |
| 4 | Stage 4 | Run a failure review on one service. |
| 5 | Stage 5 | Agree a definition of done with your PM. |
Tradeoffs
- Judgement vs. hands-on skill. Skimming the build lessons keeps you fast. It also means you rely on your team's account of how things fail.
- Standards first vs. build first. Standards slow the first feature and speed the next ten.
- One path vs. two. Pair this path with the engineer path if you still write code.
Failure modes
- Approving by demo. Review by failure handling, not by the happy path.
- No owner for the eval set. An eval set with no owner decays.
- Treating safety as a security-team task. Product and engineering own the design that makes it hold.
- Skipping the economics. Cost surprises arrive after launch.
Practitioner checklist
- Do we have a written release gate for each AI feature?
- Does every agent design show permissions, budgets and a kill switch?
- Do we track cost and latency per request?
- Do we have an incident process that covers model failures?
- Do product and engineering share one quality bar?
Under the hood: how this path was built
Lesson links come from the track build configs, so a renamed lesson is caught by the link check. Time is the number of lessons times 25 minutes for a reading lesson, times 60 minutes for a harness lesson that you build, and 120 minutes to skim a harness phase. Add time for exercises. Modules marked "coming soon" are planned and not counted.
Related lessons
Sources
Lesson counts and titles come from the track overview pages and build configs in this repository. The time estimates are this course's rule of thumb, not measured results. The week plan is invented.