Foundations

TL;DR

Prompt engineering optimizes the string you send the model. Harness engineering optimizes the system that sends it: retries, validators, tool wiring, budgets, fallbacks, caching, and observability. In production, the harness is where most of your reliability — and most of your bugs — actually live. A great prompt inside a naive harness is fragile. An average prompt inside a strong harness ships.

🎯 For the AI-native PM

Why it matters — The reliability your users feel comes from the system around the model, not the prompt inside it. "Improve the prompt" is rarely your highest-leverage roadmap item. The work that actually moves retention is harness work — validation, fallbacks, budgets, routing.

What it changes in your decisions — How you scope "AI quality" epics, where you spend engineering cycles, and what you can credibly promise in an SLA.

Ask your eng team — "If the model returns garbage on a single call, what does the user actually see?"

Product risk if ignored — You ship a dazzling demo, then burn quarters firefighting reliability you never put on the roadmap.

Mental model

Think of the model as a single, fast, unreliable, stateless function call:

output = model(context)   // non-deterministic, occasionally wrong, no memory

Everything that makes that call useful and safe is the harness:

result = harness(input):
    context   = assemble_context(input)        # context engineering
    raw       = call_model(context)            # the easy part
    parsed    = validate_and_repair(raw)       # structured output
    effects   = execute_tools(parsed)          # function calling, idempotency
    guarded   = enforce_budgets(...)           # loop/tool budgets
    observed  = trace_everything(...)          # spans, tokens, cost
    return    fallback_if_needed(guarded)      # routing, degraded mode

Each of those lines is a lesson elsewhere in this repo. The point of this lesson is that they are one discipline, and that discipline — not prompt wording — is what separates a demo from a service.

Why the harness dominates in production

• Models are stochastic. The same input can produce a valid result once and a malformed one the next time. Only the harness can make the system deterministic enough to depend on (via validation, retries, and fallbacks).
• Models are stateless. Memory, history, scratchpads, and tool results all live in the harness. What the model "knows" on any given call is whatever your harness decided to put in the context.
• Models don't have side effects — your harness does. The moment a tool call writes to a database, sends an email, or moves money, correctness becomes an engineering property (see idempotency), not a prompting property.
• Models don't have SLAs — your service does. Latency, availability, and cost targets are met by routing, caching, and batching — all harness.

What lives in a serious harness

Failure modes (when teams over-invest in prompts and under-invest in harness)

• "It worked yesterday." No regression evals, so a prompt tweak or model update silently breaks 5% of cases. Fix: golden sets + CI gates.
• The JSON sometimes doesn't parse. No validation/repair, so one malformed output takes down a workflow. Fix: structured output pipeline.
• The agent ran for 40 steps and spent $12. No budgets. Fix: loop & tool budgets.
• Latency spikes at peak. No routing/caching. Fix: routing + caching.
• A prompt injection from a retrieved doc exfiltrated data. The harness trusted model output as control flow. Fix: permission boundaries.

Practitioner checklist

• Can your system survive the model returning garbage on any single call?
• Is every side-effecting tool call idempotent and validated?
• Is there a hard ceiling on iterations, tokens, and cost per request?
• Is every model call traced with tokens, latency, and cost?
• Can you change models without rewriting business logic?
• Do you have a regression eval that runs before prompt/model changes ship?

If the answer to any of these is "no," your reliability gap is in the harness, not the prompt.

Related lessons

• Context engineering, not just long prompts
• Shipping LLM systems as infrastructure, not demos
• Agent guardrails
• Production failure modes

TL;DR

The context window is a scarce, expensive, and quality-sensitive resource. Context engineering is the discipline of deciding what goes into it, in what order, in what form, and what gets left out — on every single call. "Just stuff in more text" is the anti-pattern. Longer context costs more, runs slower, and often makes answers worse, not better.

🎯 For the AI-native PM

Why it matters — Context is a budget you spend on every request. It shows up directly in latency, cost, and answer quality. "Just give the model more info" is a product decision with a bill and a quality cost attached.

What it changes in your decisions — What data you invest in making retrievable, how you scope memory/personalization features, and your cost-per-interaction.

Ask your eng team — "What's actually in the context window on a typical request, and what does each part cost us?"

Product risk if ignored — Prompt creep quietly triples unit cost and lowers quality. It stays invisible until the invoice or the churn shows up.

Mental model

Treat the context window like a working set in a memory hierarchy, not a junk drawer. The model can only attend to what is in front of it. Its effective attention is non-uniform: information at the start and end of the window is used more reliably than information buried in the middle ("lost in the middle"). So context engineering optimizes three things at once:

1. Relevance — is the right information present at all? (a retrieval problem → RAG architecture)
2. Signal-to-noise — is it crowded out by irrelevant tokens?
3. Position & format — is it placed and shaped so the model actually uses it?

From data-first to context-first

Two era-framings from the PM side of this discipline are worth knowing. First, the data-first playbook ("collect, clean, analyze; more data wins") is giving way to a context-first one. The question is no longer "what data do we have?" It is "what context does this decision need to be meaningful?" Data is inert until it's framed. The same model with the same data behaves entirely differently, depending on what makes it into the window and in what form.

Second: prompt engineering alone is stagecraft, not architecture. Clever wording dazzles in a demo and fails as a system, for three structural reasons:

• Fragility — a prompt tuned for "how do I request a refund?" falters on "can I get my money back if I cancel mid-cycle?" One phrasing shift collapses the trick.
• No memory — prompts operate in isolation. Without engineered context, the system contradicts itself across sessions: the assistant recommends the SMB market on Monday and enterprise-only on Tuesday, because nothing carried the strategy forward.
• Operational debt — every new use case demands fresh tinkering. Over time the team ends up maintaining a library of brittle bespoke prompts instead of one adaptive system.

The enterprise bar is higher than any single output. The context pipeline has to carry what the user is entitled to, what policy dictates, and when to escalate — CRM data, legal guardrails, strategic goals — assembled systematically on every request. That pipeline is this lesson's subject. The prompt is just its last mile.

Why more tokens is not more quality

• Distraction. Irrelevant retrieved chunks pull attention away from the answer. Precision often matters more than recall once you have enough signal — see retrieval evals.
• Lost in the middle. Long contexts degrade for facts placed mid-window. Put the most decision-critical material near the top or bottom.
• Cost & latency scale with input length. Every input token is paid for in the prefill phase and in dollars. Doubling context roughly doubles prefill compute.
• Contamination risk. Every extra source (a retrieved doc, a tool result, prior turns) is a potential prompt-injection vector. More context = larger attack surface.

The components of a well-engineered context

A typical context is assembled from layered sources, each with its own budget:

[ system / role + policy ]      ← stable, cache-friendly prefix
[ tool definitions / contracts ] ← stable
[ task instructions ]
[ retrieved knowledge ]          ← ranked, reranked, trimmed, cited
[ relevant memory / history ]    ← summarized, not raw
[ the user's actual request ]    ← often best placed last

Engineering decisions for each layer:

• Stable prefix first. Keep system prompt, policies, and tool contracts at the front and unchanging so they can be reused by prompt caching. Putting a timestamp or user name at the very top silently breaks prefix caching.
• Retrieve, then rerank, then trim. Don't paste your top-50 chunks. Rerank and keep the few that matter — see RAG architecture.
• Summarize history, don't replay it. Long agent transcripts should be compressed into running summaries plus the last few turns.
• Make provenance explicit. Tag each retrieved chunk with a source id so the model can produce citations and so you can defend against injection.

Tradeoffs & decisions

The recurring tension is recall vs. precision vs. budget. Context engineering is the art of spending your token budget where it moves the answer.

Failure modes

• Prompt bloat creep. Every incident adds "also, never do X" to the system prompt until it is 4,000 tokens of contradictory rules. Treat the system prompt like code: review it, test it, and prune it with evals.
• Cache-busting prefixes. Dynamic content near the top destroys prefix cache hits and quietly triples cost.
• Raw history replay. Feeding the entire conversation every turn → quadratic cost growth and lost-in-the-middle degradation.
• Unbounded retrieval. "Top-k = 20" with no reranking buries the answer in noise.

Practitioner checklist

• Is your context assembled from explicit, budgeted layers — not string concat?
• Is the stable prefix actually stable (cache-friendly)?
• Do you rerank and trim retrieved context instead of dumping it?
• Is conversation history summarized rather than replayed?
• Is every external chunk tagged with provenance and treated as untrusted?
• Have you measured whether adding context improves your evals, or just cost?

Related lessons

• Harness engineering
• RAG architecture
• Prompt caching vs. semantic caching
• Prefill vs. decode
• Safety engineering

TL;DR

A demo proves the model can do the task once. Infrastructure makes it do the task correctly, observably, affordably, and safely. It must work for every user, every tenant, at p99, indefinitely. The gap between the two is almost entirely engineering: evaluation, observability, cost control, safety, and graceful degradation. Demos are judged by their best case. Infrastructure is judged by its worst case.

🎯 For the AI-native PM

Why it matters — The gap between an impressive demo and a dependable feature is work that doesn't demo well. It is most of the project timeline. PMs who can't see that gap chronically under-scope AI projects.

What it changes in your decisions — Your launch-readiness criteria, your definition of done for AI features, and the expectations you set with execs.

Ask your eng team — "What would it take to operate this for a year without it surprising us?"

Product risk if ignored — Friday-demo, Monday-outage: you green-light a launch that was never operationally real.

Mental model

Ask of every LLM feature the same questions you'd ask of any production service:

• How do I know it's working? → observability
• How do I know it's correct? → evals
• What happens when a dependency fails? → routing & fallback
• What's the worst a malicious input can do? → safety engineering
• What does it cost, and who's paying? → cost attribution
• Can it leak one customer's data to another? → multi-tenant isolation

If you can't answer these, you have a demo.

The demo-to-infrastructure gap

Why LLM systems are harder to operationalize than typical services

• Non-determinism breaks "write a test that asserts equality." You need distribution-level and rubric-based evaluation, not exact-match unit tests.
• Silent failure. A wrong answer looks exactly like a right one. Errors don't throw; they read plausibly. Detecting them requires evals and drift monitoring, not exception handlers.
• Upstream drift. The model provider can update weights, your data can change, and your retrieval index can age. The system can regress with no code change at all.
• Unbounded cost and latency. Token usage — and therefore dollars and seconds — depends on inputs and on how many times an agent loops.

The operational baseline (minimum bar to call it "shipped")

1. A golden eval set that runs in CI and blocks regressions — Evals.
2. End-to-end tracing with tokens, latency, cost, and errors per request — Observability.
3. Output validation so malformed responses never reach downstream systems — Structured output.
4. Budgets on iterations, tokens, and tool calls — Agent guardrails.
5. Fallback paths for provider outage, timeout, or low confidence — Model routing.
6. Cost attribution per feature and tenant — Cost attribution.
7. Safety boundaries for injection and data leakage — Safety engineering.

Failure modes

• The "Friday demo, Monday outage" pattern — impressive launch, then a long tail of edge cases nobody measured.
• Cost surprise — a feature that was "basically free" in testing costs five figures a month at scale because nobody attributed tokens.
• Silent regression — quality drops after a model upgrade and no one notices for three weeks because there were no evals.

Practitioner checklist

• Could you operate this feature for a year without it surprising you?
• Do you measure correctness with numbers, not vibes?
• Does every external dependency have a fallback?
• Can you attribute every dollar of spend to a feature and tenant?
• Would a malicious document in your retrieval corpus be contained?

Related lessons

• Harness engineering
• Evals
• Observability
• Production failure modes