Putting a language model behind a product decision is the easy part. The difficult part is that the moment a model can generate text, retrieve documents or take an action, your application stops being a pure function of its input. Text you did not write now influences what your software does, and that text may itself have been influenced by someone you do not know. This is an ordinary engineering problem with an unusual input, and the defences are mostly ordinary.
What follows is the architecture we reach for. It is not a claim that these controls make an AI feature safe in any absolute sense. It is a description of how to shrink the blast radius, detect what slips through, and stay able to change the model without rebuilding the feature.
Threat-model the feature before the prompt
Security reviews of AI features tend to start with the prompt, which is the least productive place to start. Start from the capability instead: enumerate what the feature can cause to happen, then rank those consequences.
A useful set of questions to ask of any AI feature: can it read data the current user should not see, write or change records, spend money, contact a third party, disclose data it was given, or be persuaded to do several of those at once? A feature that drafts a summary for a human to read sits at one end of that list. A feature that approves refunds, sends email or modifies permissions sits at the other, and the two do not justify the same engineering budget.
This ranking is what makes the rest of the design decisions tractable. It also gives you an honest answer for a stakeholder who asks whether the feature is secure: here is what it can reach, here is what stands between it and that outcome, and here is what we would need to add to change the answer.
Decide what the model is allowed to affect
The most useful structural decision is to separate model output into two categories: content shown to a person, and content that changes the behaviour of the system. The first needs escaping, validation and review. The second needs a stricter gate, because it can read data, call tools or spend money.
A rule of thumb that holds up well: if a piece of model output would be dangerous if an attacker wrote it instead, it needs the same defences as though an attacker had written it. Retrieved documents, tool results and earlier conversation turns are all untrusted input by that definition, because the model can be induced to repeat them.
Prompt injection is an application problem
Prompt injection describes a case where instructions embedded in retrieved content override the instructions you gave the model. There is no known way to make a prompt immune to it, and treating it as a wording problem to be solved by better phrasing will eventually disappoint you. The published threat taxonomies for LLM applications treat it as an architectural risk for exactly that reason.
The defences that hold up sit in your code rather than in the prompt.
- Keep authority outside the model. The model proposes; deterministic code decides. Validate the returned action against an allow-list before executing it, never against a natural-language description of what it was asked to do.
- Give tools narrow, typed parameters rather than free-form instructions. A tool taking a validated identifier and an enum is far harder to misuse than one accepting a paragraph of text.
- Scope every tool call to the permissions of the current user, resolved server-side. Never let the model assert an identity, a role or a tenant.
- Treat retrieved text as data rather than instruction, delimit it clearly, and never let retrieved content occupy a position in the context that grants it authority.
- Require an explicit human decision for consequential actions. Sending, publishing, spending, deleting and permission changes belong behind a confirmation step.
The human-in-the-loop step is not a compromise while the rest of the design settles. For actions that are hard to reverse, it is frequently the correct design permanently.
Least privilege, applied twice
Tool execution is where an AI feature acquires real authority, so it deserves the same treatment as any other privileged integration.
Give each tool the narrowest scope that makes it work. A tool that reads a customer's invoices does not need the ability to write them, and separating those into two tools means an injected instruction can at worst reach the weaker one. Resolve the acting user on the server, from the session, on every call. Add per-tenant scoping to the query itself rather than filtering results afterwards, because a filter applied to returned rows is a control that a future code change can forget.
Isolation has a second axis that is easy to miss: retrieval. An index shared across tenants will return another tenant's document to a model that has no way to know it should not, and the model will treat that document as legitimate. Scope the index, or scope the query, to the tenant before generation rather than after.
Keep credentials and sensitive data out of context
A model will sometimes quote its context back. If an API key, a connection string or personal data is present in the prompt, treat the model as a component that can disclose it, because that is what it is.
- Keep credentials in the server environment, never in prompts, conversation history or retrieved documents. A key the model can read is a key the model can leak.
- Scope each key to the narrowest permission that works, so disclosure is not equivalent to full access.
- Send the minimum data the task needs. Data minimisation is the only control that removes the exposure rather than reducing it.
- Redact before logging, and test the redaction rather than assuming it. Request and response bodies are routinely written to application logs verbatim, and logs travel further than databases.
- Establish and record what your provider's terms and account settings mean for retention and training, and re-check them when the contract changes rather than at launch.
Constrain cost and output explicitly
Model-backed features have two failure modes that are easy to overlook: unbounded spend and unbounded output.
Enforce limits server-side. Cap tokens per request, cap requests per user or tenant, cap total daily usage, and decide what the user sees when a cap is reached. A rate limit enforced in the client is a suggestion. When a feature is genuinely expensive to run, gate it behind a plan or a quota rather than relying on usage staying low.
On the output side, assume a caller will eventually display whatever comes back. Sanitise any rendered markup, and never pass model output into an HTML injection sink without sanitising it first. Keep structured output constrained: request a defined shape, validate it against a schema before use, and reject the response rather than attempting to parse prose and hope.
Retrieval leaks more than teams expect
Retrieval systems disclose more than most teams anticipate, and the fix is usually a smaller index rather than a better ranking model.
- Filter by the current user's permissions before retrieval, not after generation.
- Scope retrieval to the tenant, the account and the feature in use.
- Re-derive authorisation for any record the model decides to surface. Generation is not authorisation.
- Version the index alongside the application, so a retrieval change is a reviewable deployment rather than a silent behaviour shift.
Keep an audit trail that survives the feature
AI features are audited later than their authors expect, usually after something went wrong and usually by someone who was not involved in building it. The record has to exist at the time it is needed.
Log the request in terms that let someone reconstruct a decision later: which user triggered it, which model and version answered, what tools were available on that call, which actions were actually executed, whether a human approved them, and what the resulting identifiers were. Record the data classes involved rather than the data itself, so the log is useful without becoming a second copy of the sensitive material you were careful about.
Decide retention deliberately and apply it to the log, the stored conversation history and any vector index as one decision. A feature that stores prompts indefinitely tends to accumulate the most sensitive text in the system, and the retention question is far harder to answer once the data is already there.
Provider boundaries and the abstraction that earns its keep
Providers change. Models are updated, safety defaults shift, deprecations arrive on their schedule rather than yours, and pricing moves. Anything your application depends on should therefore sit behind a small interface you own: one call that takes a request and returns a typed result, with provider-specific configuration, error mapping and redaction applied in one place.
The value of that boundary is not the possibility of switching providers, which is rarer than people assume. It is that a model change becomes a configuration change with a defined blast radius, and that the redaction, logging and rate-limit logic exists once instead of at every call site. Keeping the trust boundary narrow then means a provider-side update produces a degradation rather than an incident.
A reference shape, end to end
Pulling the above together, a defensible AI feature looks roughly like this. A request arrives at your server, where the session is resolved and the acting user's permissions are established before any model is involved. Your code decides what may be retrieved and assembles the minimum context, with retrieved text marked as data. The model is called through your provider interface, which applies the configured model version, the token budget and the redaction policy.
What comes back is treated as untrusted. Display-only output is escaped and rendered. Anything proposing an action is validated against an allow-list of permitted tools with typed parameters, re-authorised against the acting user, and scoped to the tenant. Consequential actions stop for explicit human approval. The execution, the approval and the model version are written to the audit log. Every layer of this is flaggable and reversible, and the feature has a defined behaviour when the provider is unavailable, the budget is exhausted, or the validation fails.
None of that is exotic. It is a permissioned integration with an untrusted input source, which is a pattern most backend work already follows.
Make the AI path observable and reversible
Model behaviour is probabilistic, which makes regressions hard to spot and easy to misattribute. The mitigations are unglamorous and effective. Note that latency belongs in this list too: a feature that waits several seconds for a response needs streaming, a progress state and a fallback, and those are the same measurement problems described in modern website performance.
- Log model, version and parameters for every request, alongside the identifiers of what was retrieved. Without this, a bad output is not debuggable.
- Build a small evaluation set of real prompts with expected behaviour and run it on every change to the prompt, the model or the retrieval code. A dozen representative cases catch more than monitoring will.
- Put AI features behind flags so they can be disabled without a deployment, and so behaviour can be compared before and after a change.
- Design a fallback for every AI-backed surface. A page that is useless without the model is a page with a single point of failure.
- Tell users when they are reading model-generated content, and give them a way to correct it.
A pre-launch checklist
- Every model output that changes behaviour passes a server-side allow-list check.
- No credential or connection string is reachable from the model context.
- Authorisation is re-derived after generation, never taken from model output.
- Tool calls are scoped to the acting user and to the tenant at query time.
- Retrieval is permission-filtered and tenant-scoped before it reaches the model.
- Token, request and spend limits are enforced server-side and fail closed.
- Logs are redacted, and the redaction is tested rather than assumed.
- Request-level tracing records model, version, parameters, tools and executed actions.
- Every AI-backed surface has a working fallback and can be disabled by flag.
- Retention is defined for logs, conversation history and any index.
Match the controls to the consequence
The single most useful habit is being explicit about what a given feature is optimising for. One that drafts content for a human to review has a very different risk profile from one that approves refunds, and applying one standard of rigour to every AI feature either overspends engineering effort on the harmless ones or under-protects the dangerous one.
Keep authority in deterministic code, keep credentials out of the context, scope every tool to the least privilege it needs, and make each surface reversible. With those in place, model-backed features become ordinary software to operate, which is the realistic bar.
Our AI engineering work covers this pattern end to end, and product engineering covers the application around it. If you already have an AI feature in production, a threat-model session on that specific feature is usually the most useful first step.
