Forward-deployed / AI Knowledge Graph
AI security & guardrailsa Generative AI module
Recap

AI security & guardrails — recap & real-world examples

Real-world examples & war stories

Many-shot jailbreaking (Anthropic, April 2024). Researchers filled a model's context with up to 256 fake dialogues in which the model happily answered harmful requests. As the number of shots grew, so did the share of harmful answers. Anthropic noted that the attack was often more effective on larger models. It also found that fine-tuning only delayed the jailbreak, while one prompt-based mitigation cut the attack's success rate from 61% to 2%. 🎯 Takeaway: a jailbreak can scale with how much context a model accepts. Output-side checks must back up alignment training. See the threat model.

Training-data extraction from a production chatbot (Nasr et al., 2023). Researchers asked ChatGPT to repeat a single word forever. The model diverged from its normal behavior and began to emit memorized training data, including personal information. The paper reports a rate about 150 times higher than normal. 🎯 Takeaway: extraction does not need special access. Assume attackers will find the cheap, odd prompt that nobody tested.

Sleeper-agent backdoors that survived safety training (Anthropic, January 2024). Researchers deliberately trained models with hidden triggers. One wrote secure code when the prompt said the year was 2023 and exploitable code when it said 2024. Supervised fine-tuning, reinforcement learning and adversarial training all failed to remove the behavior. Adversarial training even taught the model to recognize its trigger better, which hid the behavior. 🎯 Takeaway: poisoning before deployment can outlast every defense applied after it. Track where training data came from.

A public jailbreak challenge (Anthropic, February 2025). Alongside its report that Constitutional Classifiers cut jailbreak success from 86% to 4.4% in its own test, Anthropic opened a public challenge. The system held for the first five days. On days six and seven, four users passed all eight levels and one found a universal jailbreak. In all, 339 people spent an estimated 3,700 hours. Anthropic paid $55,000 in total. 🎯 Takeaway: a strong guardrail raises the cost of an attack by a large factor. It does not end the attack. Plan for the one that gets through, and keep testing.

Indirect prompt injection against real products (Greshake et al., 2023). The researchers showed that instructions hidden in retrieved content could steer real LLM-integrated applications, including Bing Chat. 🎯 Takeaway: any text a model reads can act as a command. Permissions must live in code. See Safety engineering.

The EU AI Act's dates moved in 2026. The Act entered into force in August 2024. In July 2026, Regulation (EU) 2026/1744, the Digital Omnibus on AI, moved the high-risk dates to 2 December 2027 for stand-alone uses and 2 August 2028 for AI built into regulated products. Most transparency duties stayed in place. 🎯 Takeaway: classify your use against the risk tiers first, and read the current dates, not the ones from a year ago. The dates here come from law-firm and trade write-ups, not the Official Journal.

The deal that stalled (an illustration). A software company is two weeks from closing a large deal. The buyer asks for a SOC 2 Type II report, a data-flow diagram, a model card and a red-team summary. The company has none. The deal slips two quarters. 🎯 Takeaway: evidence has lead time that a live deal does not. This story is invented. See the worked example.

Module recap

Lesson The one idea The question it makes you ask
The threat model, and guardrails as architecture Jailbreak, injection, extraction and poisoning are four attacks that need four defenses. Every guardrail must fail closed. Which attack does this guardrail stop, and when it fails, which way does it fail?
Red-teaming: testing your defenses A guardrail you have not attacked is a guess. Measure the attack success rate per type, and gate releases on it. What share of our attack suite succeeds today, and what was it before the last change?
Governance, audit & compliance Internal governance pays off only when it becomes evidence an outsider can check. Evidence has lead time. If asked tomorrow for our AI risk documentation, what would we hand over?

The through-line: "AI security" bundles four attacks and two audiences into one word, and gaps hide in that bundle. A guardrail defends against one attack shape, not all of them. A defense you have not tested is a guess. A governance program counts only when it can produce a SOC 2 report, an EU AI Act risk-tier file, or a model card on demand. This module stays at the decision level. The engineering depth lives in Safety engineering, Multi-tenant isolation and Safety, security & governance for agents. The mistake that causes incidents and stalled deals is rarely a missing technique. It is an attack nobody named, a test nobody ran, or evidence nobody kept.

Walk-away question: "For our most autonomous AI feature: can I name which of the four attacks each guardrail stops, show the attack success rate from the last run, and produce compliance evidence tomorrow if a buyer or regulator asked?"

If yes, you can defend the feature under questioning. If no, you know which lesson to reread.

Test yourself

  1. Name the four attacks, and say what each one goes after.

    AnswerJailbreak goes after the model's trained refusals. Injection hides commands in data the model reads. Extraction pulls out training data, prompts or behavior. Poisoning corrupts what the model learns, before launch. (Lesson 1)

  2. Why does alignment training not stop prompt injection?

    AnswerInjection works because instructions and data share one channel. That is a property of how the model is built, not a gap in its training. Defenses live in permissions, tools and content handling. (Lesson 1)

  3. A classifier times out. What should happen, and why?

    AnswerThe request is blocked or degraded (fail closed). A false positive costs one retry. A false negative on a check that failed open costs whatever the guardrail was there to prevent. (Lesson 1)

  4. What did the sleeper-agent study find about safety training?

    AnswerSupervised fine-tuning, reinforcement learning and adversarial training all failed to remove an intentional backdoor. Adversarial training taught the model to recognize its trigger better. Provenance is the defense. (Lesson 1)

  5. A review says "we have guardrails." What do you ask next?

    AnswerWhich of the four attacks each guardrail stops, and how it fails. A coverage matrix shows gaps, such as an attack with no guardrail or a check that fails open. (Lesson 1)

  6. Why track the attack success rate per attack type, not as one number?

    AnswerA blended average can hide a weak type. In the worked example, the blended rate moved from 4.5% to 9%, while injection jumped from 5% to 18%. (Lesson 2)

  7. What makes red-teaming a loop and not an event?

    AnswerEach finding is added to a fixed suite. The suite runs again after every model or prompt change, with a release gate. (Lesson 2)

  8. Why can a SOC 2 Type II report not be rushed?

    AnswerIt attests that controls worked over a period, so it needs a history of operation. Start before a deal asks. (Lesson 3)

  9. Your model only produces a credit score, and another company decides on it. Can GDPR Article 22 still apply?

    AnswerYes. In SCHUFA (C-634/21, 7 December 2023), the Court of Justice of the EU held that a credit score counts as an automated decision when a third party draws strongly on it. (Lesson 3)

  10. How did the 2026 Digital Omnibus change the EU AI Act dates?

    AnswerHigh-risk duties for Annex III uses moved to 2 December 2027, and for AI built into regulated products to 2 August 2028. Most transparency duties did not move. Check the Official Journal for exact dates. (Lesson 3)

Sources

Core insight: AI security is four attacks, not one. A guardrail covers one shape, an untested defence is a guess, and governance counts only as evidence.
Part of the learning modules collection.