Epistemic Noise
All notes

Note · 2 min

Prompt injection and guardrails

Prompt injection becomes much more consequential once a model can do something.

Traditional software has a useful separation between instructions and data. LLM applications weaken that boundary because system instructions, user messages, retrieved documents, webpages, emails and tool results can eventually become tokens inside the same context.

Direct injection comes from the user. Indirect injection arrives inside content the system retrieved on the user's behalf.

That second case is particularly uncomfortable for agents. A webpage is supposed to be data, yet it can contain text instructing the model to ignore the original task, reveal information or invoke another tool. To the model, both legitimate instructions and malicious content are ultimately language.

There is no sufficiently clever system prompt that turns this into a solved security problem.

The useful controls therefore exist around the model: least-privilege credentials, narrow tool permissions, deterministic authorization, parameter validation, isolation of untrusted content, rate limits, audit logs and confirmation before consequential actions.

Model-based classifiers and guardrails can add another layer, but they should not become the security boundary. They are probabilistic components too.

The model can interpret information and propose actions. The application should retain authority over what actually happens.