Practice: the agent spec and risk register

Last reviewed: 2026-10 · Volatility: fast

TL;DR

An agent needs a spec as much as a screen does, and the spec has different parts: the goal, the tools and their limits, the autonomy level, the stopping rules, how a human steps in, and what can go wrong. This lesson gives you three tools. An agent spec states what the agent is allowed to do. A risk register lists what can go wrong and what stops it. An approval design says how a human reviews actions without clicking "yes" by reflex.

Then you test the spec with scenarios. Write five bad days before you build, and check that the design handles each.

🎯 For the AI product leader

Why it matters — The spec is where "the agent can do things" becomes "the agent may do these things, up to these limits, and stops here". It is also what your security, legal and support teams need to review.

What it changes in your decisions — You approve an agent by its spec and risk register, not by a demo. You require a named owner for each risk. You set the first autonomy level low.

Ask your team — "Which of the five worst days in our register have we tested, and what did the agent do?"

Risk if ignored — A first incident that teaches you what the register would have listed in an afternoon.

The mental model: permission, limits, review

Three layers surround every agent action. Permission decides whether the action is allowed at all. Limits bound how much, how often and how fast. Review decides when a human looks before or after. A good spec fills in all three for each tool.

Mechanics

The agent spec

1. Goal and users. What job the agent does, and for whom.
2. Flow. Fixed steps (workflow) and open steps (agent), as a short list.
3. Tools. For each: what it does, class, autonomy level, limits, undo.
4. Context. What the agent sees, where it comes from and how long it lives.
5. Stopping rules. Maximum steps, spend and time; when to hand off.
6. Human roles. Who approves, who is notified, who owns the kill switch.
7. Logs. What is recorded, for how long, and who can read it.
8. Evals. The scenarios and pass rates the agent must meet before each autonomy level.

The risk register

List what can go wrong. For each risk, score likelihood and impact on a three-point scale, name the control, name an owner and say how you will test it. Include the agent-specific risks: repeated actions after a crash, runaway loops, instructions hidden in data, over-broad credentials, and silent failures.

Approval design

Approval only protects if people read what they approve. Show a preview of the action in plain words, with the amount and the target. Show the reason and the source. Make the safe choice easy. Do not ask for approval on low-risk actions, or users learn to click through. Track the approval rate and the edit rate: a 100% approval rate on a risky action means either the agent is very good or nobody is looking.

Five bad days

Before you build, write five scenarios that would hurt. For example: the agent refunds the wrong customer; a crash repeats a refund; a customer's message tells the agent to ignore its rules; a tool returns an error that the agent misreads as success; the cost per ticket triples overnight. For each, write what the design does. If the answer is "nothing", fix the design.

Raising autonomy

Write the condition for each step up the ladder. For example: after 500 approved refunds with an edit rate under 2% and zero severe errors, allow automatic refunds under $30, with notification. Review the data, not the mood.

Tradeoffs and decisions

• Detailed spec vs. speed. A short spec is quick and leaves risks hidden.
• Approval everywhere vs. only where needed. Too many prompts teach people to ignore them.
• Tight limits vs. usefulness. A limit that never triggers is probably too loose. One that always triggers is too tight.
• Register as a document vs. as tests. A register nobody tests is decoration.

Failure modes

• A register nobody owns.
• Approval theatre. A prompt that always says yes.
• Limits in prose only. Not enforced in code.
• Untested bad days. The scenarios exist on paper.
• Autonomy by calendar. Raised after a date, not after evidence.
• Hidden tool power. A tool that can do more than its description says.

Under the hood

Three templates.

AGENT SPEC
Goal and users:
Flow: fixed steps ___ / open steps ___
Tools: name | does | class | level | limit | approval | undo
Context: sees ___ from ___, kept for ___
Stop rules: steps ___ spend ___ time ___ hand off when ___
Humans: approver ___ notified ___ kill-switch owner ___
Logs: what ___ how long ___ who reads ___
Evals before each level: scenario | pass rate needed

RISK REGISTER
# | Risk | Likelihood (L/M/H) | Impact (L/M/H) | Control | Owner | How tested | Status

APPROVAL CARD
Action in plain words (who, what, how much):
Why the agent proposed it, and the source:
Safe choice (default):     Undo available? yes / no

Worked example: Relay's refund agent

Relay is invented. All details are made up for teaching.

Spec in brief. The goal is to prepare refunds on eligible tickets. The flow is fixed: look up the order, check the policy, draft the reply and propose the refund. Tools: lookup_order (read), draft_reply (draft), issue_refund (act, irreversible, level 2, cap $30 for later levels) and send_message (act, level 2). Stops: 8 steps, $0.50 spend, 60 seconds. Humans: the support agent approves, the team lead owns the kill switch.

Risk register (three of nine).

Approval card. Refund $24.10 to order 55812 (Dana K.), item arrived damaged. Source: ticket message 3 and policy section 4.2. Safe choice: decline and ask for photos. Undo: not available.

Bad days tested. All five scenarios were run in a sandbox. Two failed: the tool error was read as success, and cost per ticket was not capped. Both were fixed before launch.

Capstone step 4. If your feature acts, write the agent spec, the risk register and approval cards. If it does not act yet, write them for the first action you may add. Run your five bad days.

Practitioner checklist

• Is every tool classified, with level, limit and undo?
• Are limits enforced in code, not only in the spec?
• Does the risk register have owners and tests?
• Do approval cards show who, what and how much?
• Are approval and edit rates tracked?
• Were the five bad days run in a sandbox?
• Is the rule for raising autonomy written as data conditions?
• Is the kill switch owned and tested?

Related lessons

• Designing agentic flows — the judgement behind this spec.
• Running an agent in production — operating the agent.
• Choosing and acceptance-testing an agent — acceptance tests.
• Red-teaming and proving your defenses — testing the injection risk.
• Tool calling — idempotency keys and safe retries for actions.

Sources

• Anthropic, "Building effective agents", anthropic.com/engineering/building-effective-agents, 2024-12-19, read 2026-10: human checkpoints, stopping conditions and sandboxed testing with guardrails.
• Relay, its tools, thresholds and every example are invented for teaching. This track's fact-check is pending.