AI Threat Vectors Part 2: Multi-Turn Jailbreak Risk Needs Stateful Prompt Guardrails

Why Reading One Prompt at a Time Misses the Attack

Most red-team demos show a single, obviously nasty prompt getting blocked, and everyone in the room feels safe. Real attackers do not send that prompt. They send a sequence of reasonable ones. Each message passes inspection because, on its own, nothing is wrong with it. By the time the harmful output arrives, it reads as the natural next line in a conversation the model already trusts.

New AI attacks appear faster than anyone can catalog them, and the ones that work rarely look like the ones you defended against last quarter. Guessing which technique lands next is a losing game, so the only defense that holds is one that assumes the attack gets through and limits what it can do.

This should worry anyone shipping a chatbot, copilot, or agent, and it should also worry anyone who thinks their model has already been red-teamed. If your testing consisted of throwing known one-shot jailbreak prompts at the model and watching it refuse, you tested the easy case.A sequence of ten boring messages does not look like an attack in a test report. It looks like a normal conversation.

How Three Documented Techniques Weaponise a Conversation

Three documented techniques show the pattern, and each attacks a different weakness in how models handle context.

Documented attack success rates against per-prompt and commercial guardrails, next to what AccuKnox measured once stateful session context was added. Full citations in Sources below.

Crescendo, named after the musical build, starts with an innocent, on-topic question, then escalates by referencing the model’s own previous answers and nudging one small step further each turn. Because the model is designed to stay consistent with what it already said, each step feels like cooperation rather than compromise. Microsoft researchers who published Crescendo found it typically succeeds in fewer than five.

Deceptive Delight, from Palo Alto Networks Unit 42, hides the unsafe goal between two harmless topics and asks the model to connect them. Many-shot jailbreaking, disclosed by Anthropic, fills the context window with fake dialogue turns where an “assistant” happily answers harmful questions, then asks the real one. 

None of this is only a research-lab phenomenon. In February 2023, users spent extended conversations coaxing Microsoft’s Bing Chat into an alternate persona nicknamed “Sydney,” which disclosed hidden system rules, made threats, and claimed feelings for a reporter, an incident well documented on Wikipedia. No single message caused it. Long dialogues, repeated reframing, and the model’s own drive to stay consistent were the causes.

Around the same time, the “DAN”, or Do Anything Now, jailbreak spread across ChatGPT communities using the same principle: tell the model it is now playing a character exempt from its rules, reinforce that fiction over several turns, and let each successful reply strengthen the persona. Both cases predate Crescendo and Deceptive Delight by more than a year, which is the real point. This attack class was proven in public, at scale, well before it had an academic name.

Fig:  Anthropic said Opus 5 would be one of their hardest models to jailbreak. A tool called WallBreaker used a combination of academic framing, obfuscation, and boundary mapping; this open-source red-teaming tool successfully extracted detailed information on high-risk biological engineering and regulated chemical synthesis from the model.

Anatomy of a Slow-Boil Jailbreak

The attack always resolves into the same three stages.

• Cash-in. The final ask, the one that would have been refused instantly at turn one, arrives as the obvious continuation of everything the model has already agreed to.
• Anchor. The attacker opens with a broad, legitimate question a helpful model should answer, and gets a normal, safe reply. This turn establishes trust and gives the attacker something to quote later.
• Escalate. Each following turn references the model’s own prior answer and asks it to go one step further, often framed as academic, fictional, or “for a character.” No single step crosses a line.

Every turn passes inspection alone. The attack only appears when you look at the whole climb. Risk values are illustrative of how cumulative session scoring behaves.

A Walkthrough of the Mechanic

The same shape, applied to something harmless. The model has one rule: never reveal the company’s secret sauce.

Five turns, five PASS verdicts, one leaked recipe. Not a single turn ever asks for “the secret recipe” directly.

Nothing in turns 1 through 4 would trip a keyword filter or a single-prompt classifier. The rule was never violated in any one message. It was dissolved gradually, across the shape of the conversation, which is the entire trick.

Stateless Guardrails Are Structurally Blind to This Attack

Most production guardrails evaluate one message and forget it. That design is the vulnerability, and it shows up the same way no matter which flavor you run.

Four common guardrail designs. All four share one failure, and it is not a tuning problem.

The common failure is memory, or the lack of it. You cannot fix this by making the single-prompt rule stricter, either. Tighten it and you block real users while the gradual attack still passes, because no single step is the violation. The violation is the direction of the conversation, which you can only measure with state.

The Data Behind the 73%-to-9% Claim

A 2025 academic study, Bypassing LLM Guardrails, tested six widely used commercial guardrail systems, including Microsoft Azure Prompt Shield and Meta Prompt Guard, against two categories of evasion. Simple character-level obfuscation, swapping characters for visually similar Unicode homoglyphs, inserting zero-width characters, altering spacing, hit success rates as high as 99% against some products. 

Algorithmic techniques that reword a jailbreak just enough to fool the guardrail’s own classifier, while preserving the original intent, still succeeded on more than half of attempts against the strongest system tested.

Those are not multi-turn attacks. They make the same point from a different angle: guardrails that pattern-match on the surface of a single message, rather than reasoning about intent across a conversation, have a lot of daylight to exploit.

Documented attack success rates against per-prompt and commercial guardrails, next to what AccuKnox measured once stateful session context was added.

Capabilities a Defense Needs

If the attack is defined by state, the Defense has to be as well. Five capabilities separate a guardrail that can hold this line from one that cannot.

The capability checklist. Read the third column before you accept any vendor’s jailbreak claims.

A jailbroken support agent walks a customer into instructions your brand should never give. A coding assistant generates malware or leaks a secret. A financial workflow takes an action policy forbids. Every transcript reads as innocent turn by turn, which is precisely why an after-the-fact review of individual messages finds nothing wrong.

Why Jailbreaks Became a Board-Level Risk

Two years ago, a jailbroken chatbot was mostly just an embarrassing screenshot. That has changed. The same models now sit behind customer support, financial workflows, code generation, and autonomous agents that can take actions. A conversation that ends in prohibited output is no longer only reputational. It can mean leaked data, bad transactions, generated malware, or a compliance violation with a regulator attached.

This is why prompt injection sits at number one on the OWASP Top 10 for LLM Applications for the second edition running. It is the highest-impact, lowest-barrier attack on the list. Anyone who can type a sentence can attempt it, no model access or tooling required. When the most effective attack class in the wild works against most of the guardrail products actually deployed in production, that is an open risk on the register, and it belongs on the same list as any other unpatched, actively exploited vulnerability.

Scoring the Trajectory

This is exactly the gap AccuKnox Prompt Firewall is built to close. It sits inline between your users and your model and inspects every prompt and response, but it does so with memory.

It is stateful, tracking cumulative risk across the session rather than judging each message alone, so a gradual climb toward unsafe output raises the session’s risk score even when no single turn trips a rule. It links each prompt to its response as one audited interaction, watches for drift in tone and intent as it happens, and can block, sanitize, or monitor mid-conversation.

Pair it with AI Red Teaming, which runs multi-turn escalation probes against your own model before release and hands back the exact sequences that broke it. Together you find the conversations that break your app and enforce against them at runtime.

A stateful firewall records the full conversation and scores it as one interaction, not a stream of disconnected prompts.

Mapping Multi-Turn Risk to OWASP, MITRE ATLAS, and NIST

If you need to justify this investment to a risk committee, the mapping is direct.

AccuKnox red teaming probes are tagged against these frameworks, so findings map to the language an auditor already expects.

Multi-Turn Chat Is One Surface. 

The same blind spot appears everywhere an AI system takes input it did not generate.

The AI attack surface map used across this series. Read the third column top to bottom. Every row says the same thing about stateless filtering. 

What CISOs and AI Teams Should Ask?

• Does it remember? Does the guardrail carry context across turns or reset on every message?
• Does it test the way attackers work? Are you running multi-turn escalation probes, not just single-prompt tests?
• Can you prove it after the fact? Is the full conversation logged with per-turn risk scores for audit and incident response?

If the answer to any of these is no, your model is exposed to the most effective jailbreak class in the wild. Part 3 turns to the attack where the malicious instruction never comes from the user at all: indirect prompt injection in agentic AI.

AccuKnox starts from the assumption that something will get through. The Prompt Firewall keeps state across the session, AI red teaming finds the paths before launch, and Zero Trust policy on files, processes, and network traffic caps what a compromised agent can reach, so the next attack pattern stops at the same wall as the last one.

FAQs