Reliable outputs & tool use

TL;DR

When a downstream system expects JSON, or any schema, "usually valid" is a production incident waiting to happen. Reliable structured output is a pipeline, not a prompt: constrain generation where you can, validate every output against a schema, repair failures by feeding the error back, and fall back through progressively safer options so a malformed response never reaches your business logic. The goal is a component that returns valid data or a clean, typed error, never garbage.

🎯 For the AI-native PM

Why it matters — Most "the AI broke the workflow" incidents are a malformed output that a downstream system trusted. This is reliability your users — and your integrations — feel immediately.

What it changes in your decisions — Your integration/API commitments, your error budget, and what "done" means for an AI feature that feeds other systems.

Ask your eng team — "When the model returns invalid output, does it crash, retry, or degrade gracefully?"

Product risk if ignored — One parse error takes down an entire automated workflow and erodes trust in the whole feature.

Mental model

Model output is untrusted input. You wouldn't JSON.parse() a request body and use it without validation. Don't do it with a model either. The pipeline:

generate ──▶ parse ──▶ validate(schema) ──▶ ✅ typed object
   ▲           │            │
   │           └─ fail ─────┤
   │                        ▼
   └───── repair loop ◀── error fed back (bounded retries)
                            │ still failing
                            ▼
                      fallback chain ──▶ safer model / stricter mode / default / typed error

Layer 1 — Constrain generation (prevent, don't just detect)

The cheapest invalid output is the one that can't be produced.

• Constrained / structured decoding — grammar- or schema-guided, such as JSON-schema modes, GBNF grammars, "JSON mode," or tool/function schemas — masks the token distribution so the model can emit only schema-valid tokens. This makes syntactic validity near-certain.
• Caveat: constrained decoding guarantees the output parses and fits the schema, not that it's semantically correct. A grammar can force {"age": 200} to be valid JSON, but it can't make 200 a sensible age. You still need validation.
• Constrained decoding can interact with quality — over-tight grammars can push the model into awkward continuations. Pair it with clear schema descriptions.

Layer 2 — Validate (always, even with constrained decoding)

Validate every output against a real schema (Pydantic, JSON Schema, zod, protobuf):

• Syntactic: does it parse and match types and required fields?
• Semantic: are values in range, are enums legal, do references resolve, do invariants hold? (end_date > start_date, total == sum(items), ids that exist.)
• Validation produces a precise, machine-readable error, which is the fuel for the repair loop.

Layer 3 — Repair loop (bounded)

On validation failure, send the model the invalid output plus the specific error and ask it to fix only what's wrong:

for attempt in 1..MAX_REPAIRS:        # MAX_REPAIRS is small (1–2)
    out = model(prompt + last_output + validation_error)
    if validate(out): return out
return fallback(...)                   # don't loop forever

Discipline:

• Bound it. Allow one or two repairs, then fall back. Unbounded repair is an agent runaway and a cost leak.
• Be specific. "Field priority must be one of [low, med, high]; got urgent" repairs far better than "invalid JSON."
• Track repair rate in observability — a rising rate signals prompt, model, or schema drift.

Layer 4 — Fallback chain

When repair fails, degrade deliberately instead of throwing:

1. Stricter generation — re-run with constrained decoding or lower temperature.
2. A more capable model — escalate via model routing for the hard case.
3. A safe default or partial result — for example, return the fields you could validate and flag the rest.
4. A clean typed error — surface a structured failure the caller can handle and that degraded-mode UX can render. Never a stack trace or raw model text.

Tradeoffs

Failure modes

• Trusting JSON.parse with no schema — a stray markdown fence or trailing comma crashes the workflow. Strip fences defensively, and still validate.
• Unbounded repair loops — a persistently malformed case retries forever, burning cost. See budgets.
• Valid-but-wrong — the schema passes but values are nonsense. Only semantic validation and evals catch this.
• Quantization-induced malformation — aggressive quantization degrades strict-format adherence, so watch JSON validity after model swaps.
• Silent schema drift — you tightened the schema and repair rate spiked, but nobody noticed because it wasn't monitored.

Practitioner checklist

• Is every model output validated against an explicit schema before use?
• Do you validate semantics (ranges, enums, invariants), not just syntax?
• Is the repair loop bounded (1–2 tries) with specific error feedback?
• Is there a fallback chain ending in a clean typed error, never a crash?
• Do you track parse-failure and repair rates as metrics?
• Did structured-output validity hold after your last model/quantization change?

Related lessons

• Function calling reliability
• Agent guardrails
• Model routing & degraded-mode UX
• Quantization formats
• Production failure modes

TL;DR

Function calling is where a model stops generating text and starts causing effects — writing to databases, sending messages, moving money. That raises the stakes from "wrong answer" to "wrong action." Reliability comes from treating tool calls like an API boundary with an untrusted client: tight contracts, strict argument validation, idempotency so retries are safe, and authorization so the model can't do what the user couldn't. The model proposes; your harness disposes.

🎯 For the AI-native PM

Why it matters — This is where the AI stops talking and starts doing — moving money, sending email, editing records. The blast radius of a wrong action is far larger than a wrong sentence.

What it changes in your decisions — Which actions you let the AI take autonomously vs. behind a confirmation, and your audit/compliance posture.

Ask your eng team — "If the model calls a tool with a hallucinated argument, what stops it from actually acting on it?"

Product risk if ignored — A double-charge, a wrong-customer email, or an unauthorized change becomes a real-world incident, not a chat mistake.

Mental model

The model is an untrusted client of your tools. Its function call is a request, not a command you execute blindly:

model proposes call ──▶ validate args (schema + semantics)
                    ──▶ authorize (can THIS user/tenant do this?)
                    ──▶ execute idempotently (safe to retry)
                    ──▶ return structured result (incl. errors the model can act on)

Every arrow can reject. A hallucinated or malformed call should fail validation, not hit your database.

Tool contracts

A tool contract is the schema, semantics, and guarantees of a callable:

• Crisp schema. Typed, required versus optional, enums, formats, ranges. This is the same structured-output discipline applied to arguments.
• Descriptions are part of the contract. The model picks and fills tools from their names and descriptions. Vague descriptions cause wrong tool selection and bad arguments. Write them like API docs, with units and examples.
• Narrow surface area. Fewer, well-scoped tools beat many overlapping ones, because the model confuses similar tools. Don't expose a raw run_sql when you mean get_orders_by_customer.
• Errors are contractual. Return structured, actionable errors, such as "customer_id not found," so the model can recover — not opaque 500s.

Argument validation

Never pass model-provided arguments straight through:

• Syntactic and semantic validation against the schema — types, enums, ranges, referential existence.
• Hallucinated arguments are common — plausible-looking IDs, dates, and enum values that don't exist. Validate against the real system of record before acting.
• On invalid args, feed the error back for a bounded repair rather than executing or crashing.

Idempotency — the keystone for safe retries

Models retry, networks fail, agents re-issue calls. If a tool isn't idempotent, a retry can double-charge a card or send two emails.

• Idempotency keys. Have the caller supply, or the harness derive, a stable key per logical operation. The tool dedupes on it, so repeats are no-ops that return the prior result.
• Prefer idempotent designs. set_status(order, shipped) is naturally idempotent; increment_balance(+10) is not. Model it instead as apply_transaction(txn_id, +10).
• Separate read from write. Reads are freely retryable; writes need keys. Mark which tools have side effects so the harness knows what's safe to repeat.
• This is what makes agent guardrails and retry/fallback logic safe: you can re-run a step without fear.

Authorization & boundaries

The model must never be able to do something the user couldn't:

• Enforce permissions in the tool, keyed on the real session or tenant — not on arguments the model supplies. A model told "you are admin" by a prompt injection must still be blocked by the tool's own authz.
• Scope every call to the tenant or user context to prevent cross-tenant access.
• Least privilege. Give tools the minimum scope they need. Dangerous actions get confirmation or human approval.

Reliability patterns

• Parallel versus sequential calls — validate them, and order them where they have side effects. Watch for the model issuing conflicting parallel writes.
• Timeouts and retries with backoff — but only retry idempotent operations automatically.
• Circuit breakers — if a tool is failing, stop calling it and degrade rather than hammering it.
• Result shaping — return concise, structured results. Dumping a 10k-token API response back into context recreates a context-engineering problem.

Failure modes

• Hallucinated tool call — the model invents a tool or an argument. Catch this by validating tool names and args against the real contract or system.
• Non-idempotent retry — a double-send or double-charge after a timeout retry.
• Confused-deputy / injection — retrieved or tool-returned content tells the model to call a dangerous tool. Block this with tool-side authz, not by trusting the model.
• Wrong tool selected — overlapping or vague tools cause this. Fix descriptions and narrow the surface.
• Context blowup — verbose tool results crowd out the task. Shape and truncate them.

Practitioner checklist

• Does every tool have a typed schema and clear, example-rich descriptions?
• Are all model-supplied arguments validated against the real system before use?
• Is every side-effecting tool idempotent (keys) or modeled to be?
• Is authorization enforced in the tool on the real session — never on model claims?
• Are only idempotent operations auto-retried?
• Are tool errors structured and actionable for recovery?
• Is the tool surface narrow and least-privilege?

Related lessons

• Structured output
• Agent guardrails
• Safety engineering
• Multi-tenant isolation
• Production failure modes
• Tool calling: the product surface — this contract, one level up, at product-decision altitude.

TL;DR

An agent is a loop: think, act, observe, repeat. Loops need brakes. Without explicit budgets — max iterations, max tool calls, max tokens, max wall-clock, max cost — and clear termination conditions, an agent can spin forever, thrash a tool, or quietly run up a large bill. Each step looks locally reasonable, so nobody notices until the invoice or the incident. Guardrails make the worst case bounded and observable instead of open-ended.

🎯 For the AI-native PM

Why it matters — Agents can loop, thrash, and run up cost with no upper bound. Budgets are how you make worst-case behavior — and worst-case spend — predictable enough to ship.

What it changes in your decisions — Whether you ship autonomous agents at all, your per-task cost ceilings, and the UX for "I couldn't finish that."

Ask your eng team — "What's the most this agent can cost — or do — on a single request?"

Product risk if ignored — A runaway agent burns budget or takes too many actions, causing bill shock or a trust-destroying incident.

Mental model

Treat the agent loop like any unbounded recursion in production: it needs a base case and a depth limit. The model decides what to do next. Your harness decides whether it's still allowed to.

while not done:
    if over_any_budget(): terminate_with_partial_result_or_escalate()
    action = model.decide(state)
    if action == STOP or goal_met(state): return result
    observe = execute(action)        # validated, idempotent tools
    state = update(state, observe)

The budgets

Set defaults conservatively, and raise them per-task only with justification. The harness should enforce budgets, not request them of the model in the prompt — a model under injection or confusion won't respect a polite "please stop after 5 steps."

Termination conditions

An agent should stop on the first of these:

• Goal met — a checkable success condition (output validates, task verified), not the model's self-assessment alone.
• Explicit stop — the model emits a terminal action or answer.
• Budget exceeded — any cap above.
• No progress — repeated identical actions, oscillation, or repeated errors. Detect loops by breaking when the same tool and args repeat N times.
• Unrecoverable error — a tool hard-fails in a way retries won't fix.

Critically, define what happens at termination: return the best partial result, escalate to a human, fall back to a simpler path, or return a clean typed error. Never just hang or dump raw state.

Detecting "no progress"

Runaway agents often aren't infinite — they're circular. Here are some cheap detectors:

• Hash (action, args). If the same hash repeats K times, stop.
• Track whether state or goal-distance is changing. If not, stop.
• Cap consecutive tool errors.

These catch the common "agent keeps calling search with the same query" failure that a raw iteration cap would let run to the limit.

Observability for agents

Budgets are only safe if you can see them. Every agent run should emit a trace with one span per step: action, args, tool latency, tokens, cost, and which budget, if any, terminated it. Aggregate:

• the distribution of steps-per-task — a rising tail means a degrading agent,
• budget-hit rate — how often you terminate on a cap versus success,
• cost-per-task by feature/tenant.

Tradeoffs

• Tight budgets are safe and cheap, but they may cut off legitimately hard tasks — premature termination. Loose budgets solve more tasks but risk runaways and cost. Tune with evals: measure success rate and cost as you vary caps.
• Budgets interact with model routing: a cheap model may need more steps. Escalate hard tasks rather than letting a weak model loop.

Failure modes

• Runaway agent — no cap, circular reasoning, large bill. This is the canonical production failure.
• Tool thrash — the same call repeats; non-idempotent versions cause repeated side effects (see idempotency).
• Premature termination — the budget is too tight, so hard tasks fail. This looks like "the agent is dumb" when it's actually "the agent was cut off."
• Silent budget hits — terminating on a cap without surfacing it hides a quality problem as a "completed" run.

Practitioner checklist

• Are iteration, tool, token, cost, and time budgets all enforced in the harness?
• Is there a no-progress / loop detector beyond the raw iteration cap?
• Are termination conditions explicit, including what happens at termination?
• Are tool calls idempotent so a re-issued step is safe?
• Do agent traces record per-step action, tokens, cost, and the stop reason?
• Do you monitor budget-hit rate and steps-per-task over time?

Related lessons

• Function calling reliability & idempotency
• Structured output
• Model routing
• Observability
• Cost attribution
• Production failure modes

TL;DR

Don't hard-wire one model to every request. Routing sends each request to the right model for its difficulty, latency budget, and cost: a small, cheap model for easy cases, a large, expensive one for hard ones. Fallback keeps the system up when a model times out, errors, rate-limits, or returns low-confidence output, by trying another path. Degraded-mode UX makes the fallback honest and usable for the user, instead of a silent quality drop or a hard failure. Together they decouple your product's reliability from any single model's reliability.

🎯 For the AI-native PM

Why it matters — Routing decouples your product's reliability and cost from any single model. It's how you survive a provider outage and how you hit margin without tanking quality.

What it changes in your decisions — Your vendor strategy, the SLA you can credibly offer, and your degraded-mode UX.

Ask your eng team — "If our main provider has a bad hour, what do our users experience?"

Product risk if ignored — Single-vendor dependence means their outage is your outage. Silent quality drops on fallback quietly erode trust.

Mental model

Your provider or model is a dependency with its own latency, error rate, and outages. Production systems don't bet availability on a single dependency — they route and fall back. The same applies here:

request ──▶ classify (difficulty / cost ceiling / latency SLO / privacy)
        ──▶ pick model (cheap → capable)
        ──▶ call with timeout
        ──▶ on error / timeout / rate-limit / low-confidence:
              ├─ retry (backoff, idempotent only)
              ├─ failover to alt provider / model
              ├─ degrade (smaller model, cached, simpler answer)
              └─ clean typed error + degraded-mode UX

Routing strategies

• Difficulty-based (cascade). Try a cheap, fast model first, and escalate to a stronger one only when needed — low confidence, validation failure, an explicit "I'm not sure." Most traffic is easy, so cascades cut cost dramatically, but they add latency on escalated requests. This pairs naturally with structured-output fallback.
• Classifier / router model. A small upfront classifier predicts which model can handle the request, routing in one hop with no escalation latency, at the cost of router accuracy.
• Capability-based. Route by need: vision goes to a multimodal model, code goes to a code model, long context goes to a long-context model, cheap bulk work goes to a small model.
• Constraint-based. Honor latency SLOs (a fast model for interactive use), cost budgets, and data-residency and privacy rules — sensitive data goes only to an on-prem or approved model, a safety boundary.

A market note that settles the "is routing worth it?" debate: in August 2025, the biggest product in the industry shipped it as the default architecture. GPT-5's headline design is a real-time router deciding per request whether a fast model or a deliberate reasoning model answers. When routing is how the frontier lab spends its own margin, treating it as an optional optimization in your stack leaves the same economics on the table.

Graceful fallback logic

Fail over on the right signals, in order of cost:

1. Transient errors (timeout, 5xx, rate-limit) — retry with backoff, only for idempotent operations, then fail over to an alternate provider or model.
2. Quality signals (validation failure, low confidence, refusal) — escalate to a stronger model or stricter generation.
3. Hard outage — serve from a semantic cache, a smaller local model, or a non-LLM fallback such as a templated or rule-based response.

Principles:

• Multi-provider removes single-vendor outage risk. But you must keep prompts and evals portable, since models behave differently.
• Circuit breakers: when a model is failing, stop routing to it for a cooldown rather than piling on.
• Idempotency and budgets: fallback chains can multiply calls, so bound them like any agent loop.

Degraded-mode UX

A fallback that silently lowers quality is a trust bug. Make degradation legible:

• Be honest. "We're experiencing high load — here's a quick answer; ask for more detail to retry" beats a confidently worse answer presented as normal.
• Preserve the core job. Drop nice-to-haves — citations, rich formatting, long reasoning — before dropping the actual answer.
• Offer a path back. Let the user retry the full-quality path when it recovers.
• Never expose internals. No stack traces, raw model text, or "provider X 503." Surface a clean, typed, user-appropriate message instead — the end of the structured-output fallback chain.

Tradeoffs

Failure modes

• No fallback — the provider has a bad hour, and your whole feature is down.
• Silent degradation — quality quietly drops on fallback. Users lose trust, and evals don't catch it because they only test the happy path.
• Routing misclassification — an easy task goes to the expensive model (a cost problem), or a hard task goes to the weak model (a quality problem). Monitor per-route success and cost.
• Fallback amplification — chained retries and failovers explode cost and latency without budgets.
• Portability gaps — the failover model gives differently-shaped output that breaks downstream parsing. Validate every route against the same schema and evals.

Practitioner checklist

• Does every model call have a timeout and a defined fallback path?
• Do you route by difficulty/capability/cost/latency, not one-model-fits-all?
• Is there a second provider/model for outage failover?
• Are retries idempotent and the whole fallback chain budget-bounded?
• Is degraded mode honest, job-preserving, and free of raw internals?
• Do you monitor per-route success rate, latency, and cost?
• Are all routes validated against the same output schema and evals?

Related lessons

• Structured output & fallback chains
• Agent guardrails
• Cost attribution
• Evals
• Inference-stack tradeoffs