You can patch a bug in code. You cannot patch a model that learned the wrong thing

When a program has a security flaw, somebody wrote a line that was wrong, and somebody else can rewrite it. The fix is verifiable: the bad input no longer produces the bad outcome, and the same code path is closed for everyone.

Machine-learning systems do not offer this. Their behaviour lives in millions of learned parameters that no person wrote, chose or can read. There is no line to correct. A model that responds badly to some input responds badly because of a shape in a function derived from data — and you can retrain, filter or fine-tune around it, but you cannot open the file and fix the mistake, because the mistake is not written anywhere. This is the structural reason AI security looks so different from ordinary software security, and it is why the most consequential vulnerability in current systems has no accepted solution.

Adversarial examples: the tiny nudge that flips the answer

The classic demonstration is now over a decade old and still not solved. Take an image a classifier labels correctly with high confidence. Compute a small perturbation — calculated, not random — and add it. To a person the picture is unchanged; the pixel shifts are below what the eye registers. The model now confidently reports something else entirely.

This is not a bug in a particular network. It is a consequence of how these models carve up their input space. A trained classifier draws decision boundaries through a space of enormous dimensionality, and in such a space the distance from a typical point to the nearest boundary is small in a direction that no human perceptual system attends to. The attacker is not fooling the model's "eyes"; they are walking a short distance in a direction the model happens to care about and people do not.

Two properties make it a practical problem rather than a curiosity. First, these perturbations often transfer: an attack computed against one model frequently works on a different model trained by a different team on different data, which means an attacker does not need access to the target system. Second, they survive the physical world — printed patterns, stickers placed on a road sign, patterns on clothing — so this is not confined to files being fed directly into an API.

Defences exist and are partial. Training on adversarial examples improves robustness against the attacks you anticipated, at some cost to ordinary accuracy, and generalises poorly to attacks you did not. Methods that provide mathematical guarantees exist but apply to limited perturbation types and scale badly. A decade of work has produced improvement, not closure.

Prompt injection: the one with no clean fix

The more urgent problem arrived with language models, and it is worth stating precisely because it is constantly confused with something else.

Jailbreaking is a user persuading a model to ignore its own usage rules. It is a content-policy problem, the user is the adversary, and the damage is mostly to the operator's reputation.

Prompt injection is a third party planting instructions in content the model will read, so that the model acts against the interests of the user it is working for. Here the user is the victim, not the attacker, and the damage is whatever the system was trusted to do.

The reason it is hard is architectural. A language model receives one stream of tokens. Your instructions, the system's instructions, the web page it fetched, the email it summarised and the document it was given all arrive in the same channel, and the model has no privileged mechanism for distinguishing "this is a command from my principal" from "this is text I was asked to read". Every operating system solved the equivalent problem decades ago with privilege separation; a transformer has no equivalent primitive, because instruction-following was learned from data rather than implemented.

The consequence is that any agent which reads untrusted content and can also take actions is executing text from strangers. A calendar invitation, a product review, a PDF, a code comment, white text on a white background in a web page — any of these can carry instructions, and the model has no reliable way to refuse them merely on the basis of where they came from.

In ordinary software, data cannot become instructions unless a programmer made a mistake. In a language model, data and instructions are the same kind of thing by construction. That is not an oversight to be patched; it is what the model is.

Mitigations are real but they are containment, not cure. Give the agent the least authority that lets it work, so a successful injection reaches little. Require human confirmation before anything consequential or irreversible — sending, paying, deleting, publishing. Separate the component that reads untrusted content from the component that holds credentials. Constrain outputs to a fixed set of allowed actions rather than free-form commands. Treat every model output derived from external content as untrusted input to the next stage. None of this makes the model resistant to injection; it makes injection less useful when it succeeds, which is the correct thing to engineer for.

Poisoning the training set

The third attack moves further upstream. If a model's behaviour comes from its data, then influencing the data influences the model.

Data poisoning means inserting crafted examples into a training set to degrade performance or, more interestingly, to implant a backdoor: the model behaves normally on everything except inputs carrying a specific trigger, on which it does what the attacker chose. A backdoored model passes ordinary evaluation, because evaluation samples the normal distribution and the trigger is not in it.

This sounds like it requires access to a training pipeline. For models trained on scraped web data it does not. Researchers have demonstrated that web-scale datasets are poisonable in practice — the contents of a listed URL can change after the dataset was compiled, and domains that appear in published datasets can expire and be re-registered. Modifying a very small fraction of a corpus can be enough to implant a specific behaviour.

The same reasoning applies to fine-tuning data, to retrieval corpora, and to anything a model is given at inference time. And a related family of attacks runs the other direction: repeated querying can be used to approximate a proprietary model's behaviour, or to establish whether a particular record was in its training data — a privacy question with real consequences for models trained on medical or personal records.

What defending these systems actually looks like

The practical posture that follows from all of this is not "make the model robust". It is to design systems on the assumption that the model will sometimes be wrong in a way an adversary chose.

That means the model is treated as an untrusted component inside a trusted system, rather than as the system. It means the security boundary sits at the actions — what can this thing actually do, with whose credentials, and what requires a person to approve. It means monitoring behaviour rather than only inputs, because an input designed to look benign will look benign. It means keeping a deterministic check on anything checkable: if the model produces a database query, validate it; if it produces a payment, verify the payee against a list; if it produces a citation, resolve it.

This is a familiar discipline. It is the same reasoning that produced sandboxing, privilege separation and input validation in conventional computing — applied to a component whose failure modes cannot be enumerated in advance.

Why it matters for students and researchers

Security is where the difference between a demonstration and a deployed system becomes obvious, and it is currently the least-taught part of applied machine learning. A student can build an impressive model without ever asking what happens when someone actively wants it to fail — and every one of these systems, the moment it is exposed to the public or given the ability to act, acquires exactly that adversary.

It is also intellectually the most interesting frontier in the field right now, because the standard answers do not work. You cannot verify a learned function the way you verify code. You cannot enumerate its inputs. You cannot patch it. What you can do is bound its authority, verify its outputs against something deterministic, and build the surrounding system so that being fooled is survivable. For anyone going into AI engineering, that habit of thinking — assume the component fails, design so it does not matter — is worth more than any particular defence technique, most of which will be obsolete before the graduates are.

Frequently asked questions

What is an adversarial example?

It is an input modified by a small, deliberately computed change that causes a model to produce a wrong output while looking unchanged to a person. It exploits the geometry of the model's decision boundaries rather than any coding error.

What is the difference between jailbreaking and prompt injection?

Jailbreaking is a user persuading a model to break its own usage rules. Prompt injection is a third party hiding instructions inside content the model reads, so the model acts against the user it is supposed to be helping — the user is the victim rather than the attacker.

Why is prompt injection so hard to fix?

Because a language model receives instructions and data in the same token stream and has no built-in privilege separation between them. Unlike conventional software, where data becomes executable only through a programmer's mistake, the model treats all text as potentially instructive by design.

What is data poisoning?

It is the insertion of crafted examples into training data to alter a model's behaviour, often implanting a backdoor that activates only on a specific trigger. Models trained on scraped web data are exposed to this because the content behind a listed address can change after collection.

How should AI systems be secured in practice?

By treating the model as an untrusted component: granting it minimal authority, requiring human approval for irreversible actions, separating the part that reads external content from the part holding credentials, and validating outputs against deterministic checks wherever possible.