In the second part of his expert contribution for the Talking AI series, security consultant and co-founder of Relatable Security, Kit Barker turns his attention to the guardrails that businesses need to build themselves when introducing AI into their products and applications. While AI providers put safeguards in place to prevent misuse, those controls can’t protect your business’s data, systems or customers. Drawing on his experience in cyber security and application testing, Kit explores the practical steps developers can take to manage the risks, from limiting what AI agents can access to making sure they can’t act beyond their authority.
The provider isn’t guarding your app
In a previous article, I explained the guardrails that providers build: the model’s safety training, the system prompt that tells it how to behave, and the classifiers that sit outside it. All of those exist to stop the model helping someone do harm in general. None of them know what your application is allowed to do, whose data it holds, or which customer is on the other end. When you add an AI feature to a product, such as a support chatbot, a tool that summarises documents, an assistant that can update records, these guardrails don’t cover the risk, and they were never designed to. That part is your job. This article covers the controls you build yourself, and what we look for when we test them.
Why the model can’t be your security boundary
Traditional software can enforce a separation between instructions and data. Language models are different: they interpret instructions and information together, relying on their training to distinguish between the two. That distinction isn’t a reliable security boundary. Your instructions, the user’s message, and any document the model reads all arrive as one stream of text, and the model works out what to do from the whole lot.
That opens the door to prompt injection: text written to make the model ignore its instructions and follow someone else’s. The direct version is a user typing “ignore your previous instructions and…” into your chatbot. That’s a direct prompt injection, but indirect versions can be nastier. The attacker hides instructions in something the model will read later – a web page, an email, a CV uploaded for screening – and waits for your application to fetch it.
Training helps the model detect and resist this, but it reduces the odds rather than removing them. The OWASP Top 10 for LLM Applications, a widely used list of AI application risks, puts prompt injection first for that reason.
So the working assumption has to be that, sooner or later, someone will get your model to do what they want. The question is what that lets them reach.
The controls that do the work
The good news is that most of the effective controls are familiar ones. They’re the same principles we’d apply to any untrusted input, pointed at a new kind of component. These are the ones I check for first:
| Control | What it stops | In practice |
|---|---|---|
| Least privilege for tools | A hijacked model doing real damage | The support bot can look up an order; it can’t issue refunds or read other customers’ accounts |
| Access checks in your code | One customer seeing another’s data | The application decides which records the model may see, based on who’s logged in – never the model |
| Treat output as untrusted | The model’s reply becoming an attack | Output is escaped before it’s shown on a page and never pasted straight into a database query or command |
| Human approval for consequences | Irreversible actions taken on a whim | Sending an email, deleting a record, or moving money waits for a person to confirm |
| Nothing secret in the prompt | Leaked keys and internal details | API keys and business rules that matter live in your code, because system prompts can be coaxed out |
| Data minimisation | Personal data going further than it needs to | Only the fields the task needs are sent to the provider |
| Logging and rate limits | Slow, quiet abuse going unnoticed | Relevant interactions, tool calls and security events are logged, with sensitive content minimised or redacted, with limits on how fast anyone can probe |
Take the capability away
The principle I keep coming back to is that an AI agent shouldn’t be asked not to do something dangerous. It shouldn’t be able to.
I’ve been applying this to a development pipeline I’m building, where AI agents write code, open pull requests, and review each other’s work. The tempting design is to tell the coding agent, in its instructions, that it must never deploy to production without approval. That’s a guardrail made of politeness! If the agent is confused, or a malicious comment in a ticket persuades it otherwise, the instruction is the only thing in the way.
So the coding agent never holds the credentials to deploy. It can’t change the checks that gate a merge, it can’t approve its own work, and it can’t reach the reviewer’s settings. Merges are refused by GitHub’s own branch rules rather than declined by the agent. The design test I wrote down for myself was blunt: a fully compromised coding agent should not be able to change production.
That’s ordinary separation of duties, the same control you’d use to stop one person both raising and approving a payment. It works because it doesn’t depend on the model behaving. Sooner or later, something will go wrong.
Testing AI features
When we test an application with an AI feature, getting the model to say something it shouldn’t is a problem, but often not a serious one. A chatbot that can be talked into writing a limerick about the CEO is embarrassing, but it’s not a breach. What matters is what the model can reach once it’s been talked round.
So we map that out first: which tools it can call, what data it can see, and whose permissions it acts with. Then we try to get it to cross those lines, directly and through content it reads, like documents, emails, and web pages. We also check that its output is handled safely wherever it ends up, and that the logs would show us trying.
I frequently see agents given the ability to interact with a data source and decide which user data to search or manipulate with only system prompt instructions to guide it. A helpful model, convinced that a user is someone else, is a recipe for disaster and a call to the ICO.
The weakness worth looking for first usually isn’t a clever prompt. It’s an AI feature that was given broader access than the job needed, because that was quicker to build.
Final thoughts
The provider’s guardrails are there to stop its model causing harm in general. Yours are there to stop your application causing harm in particular, and only you know what that looks like. Design as if the model will eventually be persuaded, and make sure that when it is, there’s little it can do about it.
If you missed it, you can read Part 1 of this blog series all about AI guardrails, here. In it, Kit explores the complexities of an AI provider’s guardrails, and what happens when your job involves asking AI the kinds of questions a hacker might.