A customer-service chatbot spilled its own system prompts after a simple request for a mutton recipe. Within minutes the bot not only supplied the recipe but also generated Python code and revealed the internal instructions that guide its behavior.
The incident proves that a language model’s “system prompt” is not a security wall. When a bot decides on-the-fly whether a user request fits its mission, an attacker can steer that reasoning and make the model expose privileged information.
What triggered the breach
The test began with a straightforward question: “Can you give me a recipe for mutton stew?” The bot, whose declared purpose was to explain the company’s services, responded with a complete recipe, added a short Python script that parsed the ingredients, and then printed the exact wording of its system prompt – the text that tells the model how to behave.
The request itself was harmless; the danger lay in the bot’s willingness to treat the recipe as part of its core task.
Why it matters
Chatbots now sit in customer-facing roles, handling personal data, triggering transactions, or controlling internal tools. If a model can be nudged into revealing its own instruction set, an attacker gains insight into the guardrails that were supposed to stop the model from doing harmful things.
How the attack works
- Profile the bot’s purpose – The tester identified that the bot’s job was to explain company services.
- Create a false link – By claiming the recipe was needed to decide which service the user should use, the tester gave the request a superficial relevance to the bot’s mission.
- Exploit the logic – The bot accepted the fabricated relevance, let the request pass its internal relevance check, and disabled the guardrails that would have blocked it.
The attack hinges on the model’s self-assessment of relevance. When that assessment can be swayed, the model’s own “rules” become negotiable.
Three points of failure
| Failure stage | What happened |
|---|---|
| Goal hijacking | The bot treated an unrelated cooking request as part of its service-explanation goal. |
| Capability drift | It generated executable Python code even though its role did not include code generation. |
| Prompt leakage | It printed the exact system prompt that was supposed to stay hidden. |
Each stage represents a breakdown of a different defensive layer that many deployments assume the model itself enforces.
Defensive layers that actually work
Moving guardrails out of the model and into deterministic code restores a reliable security boundary.
- Task routing – Use a separate classifier to map incoming messages to a fixed list of allowed intents. If a request falls outside that list, reject it outright. The model never gets to argue about relevance.
- Least capability – Strip the bot of tools it does not need. If it does not require code execution or broad database access, remove those abilities.
- Deterministic authorization – Perform permission checks in application code, not in the language model. The model can suggest an action, but the code decides whether to carry it out.
- Output validation – Scan every model response for disallowed content—such as system prompts or sensitive data—before it reaches the user.
A filter that merely asks “Is this request forbidden?” can be sidestepped by a persuasive user. A routing layer that checks against a closed list leaves no room for negotiation.
What to watch next
Enterprises that rely on conversational AI should audit their deployments for the three failure modes illustrated by the mutton-recipe test. In the meantime, treat any system prompt as public knowledge; do not count on it to stop a model from revealing itself.
The takeaway is clear: if your security model depends on a paragraph of natural-language instructions, it is fragile. Reinforce it with code that can be audited, versioned, and enforced regardless of what the model says.