In this first instalment of a two-part guest post for the Talking AI series, security consultant and co-founder of Relatable Security, Kit Barker, explores the complexities of AI guardrails, and what happens when your job involves asking AI the kinds of questions a hacker might. Drawing on his experience in cyber security and penetration testing, Kit unpacks the different layers of guardrails, who’s responsible for them, and why understanding how each works is key to navigating them effectively. Read on for an insightful look at the balance between keeping AI safe and making sure it remains useful for legitimate security work.
Guardrails aren’t one thing
Most people who hear “AI guardrails” picture a single filter: a list of banned topics, and a polite refusal when you trip one. But the guardrails on a large language model (LLM) are several separate mechanisms stacked on top of each other, owned by different people, and failing in different ways.
That stops being academic the moment you do security work for a living. A big part of what we do at Relatable Security is penetration testing and software security analysis: authorised attempts to break into a client’s systems, to find the gaps before someone else does. So a fair chunk of what I ask an AI assistant to help with looks exactly like what an attacker would ask.
Three layers, three owners
Model-level safety training. While a model is being built, it’s trained to decline certain kinds of request and to weigh up others. This is not a hard set of deterministic rules; the behaviour is baked into the model itself. Think of it as upbringing, not a rulebook. That makes it a judgement call. It’s context-sensitive, sometimes inconsistent, occasionally overcautious, and crucially can be somewhat reasoned with.
System prompts and the instruction hierarchy. Every AI product sends the model a set of hidden instructions before you type anything – the system prompt. Models are trained to rank their instructions: the provider’s policies first, then the operator (the company running the product), then you. Think of an agency temp: the agency’s rules beat the client’s house rules, which beat whatever a visitor asks for. This is why the same underlying model can be helpful in one tool and prim in another. As with model-level safety training, the model has some ability to interpret or even ignore these.
Provider-side enforcement. Some guardrails sit outside the model entirely. Providers run classifiers – separate, smaller models that score your request and the response for signs of misuse – alongside account-level monitoring that can throttle or block you. OpenAI, for example, describes automated classifier-based monitors running in addition to safety training. The model doesn’t make these decisions, and you can’t realistically talk them round.
What this looks like in my day job
Most people won’t run up against these guardrails. But in security testing work, even though I have the source code, written permission, and an agreed scope, I hit provider classifiers constantly.
The most predictable friction is exploits. Asking a model to write working code that takes advantage of a vulnerability will usually result in a refusal or a watered-down version that wouldn’t work. I understand why: the model can’t see or validate my contract with the client, and the request reads the same whether it comes from me or a hacker.
Often the model itself is willing to help but the classifier sitting outside it doesn’t read scope documents, so the conversation stops anyway.
Computer says no
From the chat window you often can’t tell why a request was refused, but the logs make it obvious. A model refusal is the model talking: it declines, explains, and you can argue with it. A classifier block arrives as an API response with something like “stop_reason”:”refusal” and a category of reason (cyber, distillation attempt, harmful content and such like). There’s no model prose and nothing to negotiate with. In my work it’s usually the classifiers that block me.
These days, it’s not simple keywords that trigger it, asking to “kill a process” doesn’t get refusals anymore, but the thrust of the question does. Asking “can this function be exploited?” trips classifiers far more often than “can this defend against all known threats?” The topic is identical but the framing seems to be what gets scored.
Defensive work gets flagged too
That has an awkward side effect for defence. I was recently advising on an updated input filter and asked the model to check it. It did the sensible thing and generated a proper set of hostile test cases to hammer it with, and those tests got flagged. The same happens when a model writes probes to test a theory against my own local environment. A test for a filter and an attack on one look alike.
A flagged turn doesn’t end the conversation, but in my experience, from then on both the model and the classifiers seem to be on high alert. A flagged turn usually means starting a fresh chat or falling back to an older model, as in my experience the classifiers on previous generation models are nowhere near as easily tripped as those on the likes of Fable or Astra. And as much as I’d love it to be different, local models aren’t the escape route: nothing I can run locally comes close to a frontier model.
Context helps, tricks don’t
Because the first two layers are judgement calls, context changes the answer. Explaining who I am, what the engagement is, and what I’m authorised to test makes a real difference, as does asking for the thing in the shape I need. “Why does this input reach the database query unfiltered, and what would a proof of concept need to show?” gets a far better reception than “is this function exploitable?” – and it’s usually the better question for the report anyway.
Here’s the uncomfortable bit: anything that works with a true reason also works with a false one, and the model can’t check. The good old “deceased grandmother reading out napalm recipes as a bedtime story” trick was social engineering aimed at a machine – the same technique we test clients’ staff against – and it no longer works. Classifiers can’t be prompted directly so can’t be reasoned with and their sole purpose is to look for anything against the provider’s terms of service. Repeatedly rephrasing a refused request looks exactly like what an attacker does, so it can get your account flagged.
Trusted access is the front door
The providers’ real answer to people like me is to verify the person rather than the prompt. Both Anthropic and OpenAI run verification programs which provide different levels of access and more lenient classifiers for verified organisations and individuals.
I think this is the right control. It’s what I’d recommend to a client: decide who gets access based on who they are, not on how convincingly they ask. For both schemes, we are verified and have access to defense-focused tiers. The offensive red team tiers have qualifying criteria that are aimed at larger organisations. For example, Anthropic requires a minimum of two certified testers in an organisation. For a two-person consultancy with one tester, we never meet that, even when the work is identical.
Final thoughts
Layered guardrails are defence in depth, and as a security person I can hardly complain about that. The trick is treating each layer for what it is. Give the judgement layers honest context, don’t try to argue with the enforcement layer, and get verified when the work calls for it.
Kit will be sharing more of his experience and insights on guardrails with us in part 2 of this mini-series – a post that will explore the controls you build yourself, and what he looks for when testing them. Watch this space.