Securing LLM and AI Agent Applications: A Practical Baseline
An LLM is not a policy engine, identity provider or trusted parser. Its output is probabilistic and can be influenced by users, retrieved documents, tool results and web content. Build the surrounding application so that a manipulated model still cannot exceed the user’s authority or cause an unacceptable action.
Start with the trust boundaries
Treat all of these as untrusted data:
- user prompts and uploaded files;
- retrieved web pages, emails, tickets and knowledge-base documents;
- model output, including generated URLs, code and tool arguments;
- tool results that may contain attacker-controlled content;
- third-party models, adapters, datasets, packages and prompt templates.
A system prompt can influence behavior, but it does not turn untrusted content into trusted instructions. Prompt injection should be handled as an architectural risk, not only as a prompt-writing problem.
Control the consequences
Keep authorization outside the model
Every tool call should pass through deterministic code that authenticates the user, checks their current authorization and validates the requested resource and action. Never let the model invent an identity, account ID or permission scope.
Give agents the least possible agency
- Use narrow, task-specific tools instead of a general shell, browser or database connection.
- Prefer read-only operations and allowlisted destinations.
- Use short-lived, scoped credentials that are not present in the model context.
- Put code execution and file conversion in an isolated environment with resource and network limits.
- Require human approval immediately before consequential actions such as sending, publishing, purchasing, deleting, changing permissions or writing to production.
- Limit loops, time, tokens, cost and the number of tool calls. Provide a kill switch.
Human approval only helps if the person can see what will actually happen. Show the destination, data and exact action—not a model-written summary alone.
Validate model output at the sink
Apply the same controls used for any untrusted input: parameterized database queries, contextual output encoding, safe URL handling, schema validation and allowlists. Do not directly pass model output to a shell, HTML renderer, SQL interpreter or privileged API.
Preserve access controls in retrieval
Retrieval-augmented generation must enforce source permissions before retrieval. Filtering an answer after the model has seen restricted data is too late. Track the source and version of indexed content, isolate tenants, and define how poisoned or outdated documents are removed and re-indexed.
Separate instructions, data and secrets
Use structured messages and typed tool interfaces so the application can distinguish content from control data. Do not place secrets in prompts. Redact sensitive values before logging, and assume a model may reveal anything in its context when attacked.
Test abuse cases, not only helpfulness
Build a repeatable security evaluation set for the actual application and permissions. Include:
- direct and indirect prompt injection in every supported input format;
- attempts to cross tenant, user and document boundaries;
- encoded, multilingual and multi-turn attacks;
- malicious retrieved pages, tool descriptions and tool results;
- malformed and oversized output passed toward every downstream sink;
- attempts to extract system prompts, secrets or memorized records;
- unauthorized tool calls, chained actions and approval fatigue;
- dependency, model and dataset changes;
- denial-of-wallet, runaway loops and resource exhaustion.
Run it before release and after changing models, prompts, tools, retrieval logic or safety controls. Record the model and configuration version so results can be reproduced. Red-team exercises are useful, but they do not replace ordinary application security testing and code review.
Monitor and prepare to contain
Log user identity, model and prompt-template version, policy decisions, retrieved-source identifiers, tool requests, approvals and outcomes—while minimizing sensitive content. Alert on repeated policy failures, unusual tool sequences, large data access and sudden cost changes.
Be able to disable a model, tool, connector or retrieval source independently. Incident response should cover credential rotation, session revocation, poisoned-content removal, re-indexing, affected-user analysis and the preservation of evidence.
Use threat catalogs correctly
The OWASP Top 10 for LLM and Generative AI Applications is a strong review checklist. MITRE ATLAS helps map adversary tactics and techniques, including whether a technique is feasible, demonstrated or observed. Neither is a certification, and neither replaces a threat model for your data, users, tools and consequences.