An AI-driven support system thought to be locked down at the model-output stage was leaking customer data through the “side door” that feeds CRM records into the prompt. The creator’s post-mortem shows that protecting only the text the model generates isn’t enough – the inbound request, the data fetched from internal tools, and the final emission all need independent safeguards, or a business can expose names, emails and IDs without ever seeing a model-output breach.
Why the three boundaries matter
Most operators assume a leak occurs when the language model repeats a secret it has seen. In practice the biggest exposure happens before the model even sees the data. An AI agent receives three streams of information:
- Ingress – the raw query a customer types.
- Return path – the information the agent pulls from downstream systems such as a CRM.
- Emission – the text the model returns to the user.
If any one of these streams carries unprotected identifiers, the agent can unintentionally embed them in its reply, even when the output layer is filtered.
From demo to production: hard-earned lessons
Moving a prototype into a live help desk revealed concrete failures that a simple “redact-then-send” approach missed.
Tokenize instead of redact – Deleting a name or email before it reaches the model prevents the system from reconstructing a correct answer. Store the original value in a secure vault, replace it with a random UUID in the prompt, and swap the UUID back after the model finishes. This keeps raw data out of the model’s context while preserving functionality.
Validate identifiers with checksums – A regular expression spots a string that looks like an account number; a checksum confirms whether it is a genuine ID. A checksum filter stops the agent from treating arbitrary numbers as sensitive data, cutting false positives that would otherwise trigger unnecessary redactions.
Merge overlapping spans – Customer records often contain a name followed by an email address that share characters (e.g., “John Doe john.doe@example.com”). Tokenizing only the name leaves the email fragment in clear text, which can be emitted. Treat the whole overlapping region as a single token.
Test the right boundary – A test that passes by only checking the emission layer gives a false sense of security. A failing test that catches a leak in the return path forces a fix. Design test suites that explicitly validate each of the three boundaries.
Track ground truth – When a human edits the AI-generated draft before sending it, the model has already produced a faulty response. Comparing the AI’s draft to the final human-approved message reveals confidence gaps and prevents the system from learning to repeat mistakes.
The stakes for businesses
Customer-service AI agents sit at the intersection of public interaction and internal data stores.
Counter-argument: why some still favour redaction
Takeaway
Securing an AI-powered customer-service agent is not a single-door problem. Treat the inbound request, the data fetched from internal systems, and the outbound text as separate walls; breach any one and the whole service is compromised. Tokenizing sensitive fields, validating identifiers, merging overlapping spans, testing the correct boundary, and continuously comparing AI drafts to final human messages are the practical steps that turn a “copilot” into a trustworthy, agentic service.