AI Security Checklist: What to Verify Before Deploying LLM Applications
The Detailed Checklist
Input classification is the first line of defense, catching the majority of known attack patterns before they reach the model.
Prompt injection classifier deployed. A dedicated classifier model (separate from the main application model) screens every user input for injection patterns. The classifier is trained on known injection attacks and evaluates inputs before they reach the main model. Open-source options include Meta's Prompt Guard and ProtectAI's rebuff. Cloud options include moderation endpoints from major LLM providers.
Heuristic rules for known patterns. Regex and keyword-based rules catch common injection phrases ("ignore previous instructions," "system prompt," "you are now") as a fast, cheap first-pass filter. These complement the classifier by catching patterns that are too simple to need ML-based detection.
Input length limits enforced. Maximum input lengths prevent abuse through excessively long inputs that could exploit the context window, run up token costs, or contain hidden instructions buried in lengthy text.
Encoding attack detection. Input validation checks for and handles encoded injection attempts: base64, rot13, unicode tricks, and other encoding schemes that bypass plain-text filters.
Multilingual injection coverage. Input classification works across languages, not just the application's primary language, because attackers routinely submit injections in different languages to bypass monolingual filters.
Output filtering catches attacks that bypass input classification by inspecting the model's response before it reaches the user.
Output safety classifier deployed. A safety classifier (such as Llama Guard) evaluates every model response for harmful content across safety categories including violence, sexual content, criminal instructions, and self-harm. Responses that exceed configured thresholds are blocked.
System prompt leakage detection. Output is scanned for content matching the system prompt using fuzzy matching. Responses that contain significant portions of the system prompt text are blocked and logged.
PII redaction on output. Responses are scanned for personally identifiable information (social security numbers, credit card numbers, email addresses, phone numbers) before reaching the user. Detected PII is redacted or the response is blocked.
Scope enforcement. Responses are evaluated for topic relevance, ensuring the model stays within its intended domain. Off-topic responses are caught and replaced with appropriate redirections.
Response format validation. For applications with structured output requirements, responses are validated against the expected schema. Malformed or unexpected formats are rejected.
Tool access controls determine the maximum possible damage from a successful attack. Detailed guidance is in the secure agent tool access guide.
Explicit tool whitelists per agent role. Each agent role has a defined list of tools it can access, enforced at the tool dispatch layer in code, not in the system prompt.
Permission scoping per tool. Each tool has defined permission boundaries: allowed operations (read/write/delete), data scope (current user only, specific tables), parameter ranges, and temporal constraints.
Confirmation gates for high-risk actions. Financial transactions, data deletion, external communications, and permission changes require human approval before execution.
Tool call parameter validation. Type checking, range validation, and semantic validation run on every tool call parameter before execution. SQL injection and command injection patterns in model-generated parameters are caught.
Per-tool rate limits. Tool call frequency is limited per session and per time window, calibrated against normal usage patterns, to prevent rapid exploitation.
Audit logging for all tool calls. Every tool call is logged with parameters, context, outcome, and timestamp in a write-only log store.
Data pipeline controls protect the knowledge base and training data from poisoning. See the data poisoning prevention guide for full details.
Source authentication for ingested content. Every document entering the knowledge base has a verified source. Anonymous or unattributed content is flagged for manual review.
Content validation on ingestion. Incoming documents are scanned for embedded injection instructions, anomalous formatting, and factual inconsistencies before entering the knowledge base.
Version control and change tracking. All knowledge base modifications are tracked with who, when, and what changed. Rollback capability is available for rapid response to discovered poisoning.
Retrieval quality monitoring. Regular automated checks compare retrieval results against a golden test set. Quality degradation triggers investigation of recent knowledge base changes.
Training data curation. Fine-tuning datasets are reviewed for adversarial examples, backdoor patterns, and data quality issues before use. Adversarial evaluation runs on every fine-tuned model before deployment.
Memory controls prevent stored information from becoming a persistent attack vector. See the memory integrity attacks guide for the full threat model.
Memory write validation. Information stored in persistent memory is validated for expected data types, formats, and content patterns. Instruction-like content, policy claims, and system configuration assertions in user-provided memories are flagged.
Source tagging with trust levels. Each stored memory is tagged with its source and a trust level (user-provided, system-inferred, admin-set, verified-external). Trust levels influence how retrieved memories are used in context.
Per-user memory isolation. In multi-tenant systems, each user's memories are stored in a separate partition with storage-level isolation, not just query-level filtering.
Memory auditing and expiration. Periodic reviews check stored memories for adversarial patterns. Unverified memories expire after a configurable period.
Conversation history management. History summarization preserves source attribution so that user-provided claims are not elevated to system-level facts through the summarization process.
API-level controls protect the external interface of the AI application. See the API security best practices guide for implementation details.
Per-user authentication and authorization. Every API request is authenticated, and the AI's capabilities are scoped to the authenticated user's permissions, not the application's service account.
Token-aware rate limiting. Rate limits operate on three dimensions: request count, token consumption, and estimated compute cost. Different endpoints have different limits based on cost and sensitivity.
No credentials in system prompts. API keys, database connection strings, internal URLs, and other sensitive configuration are stored in secrets management, not in the system prompt.
Model extraction detection. Monitoring identifies systematic query patterns that suggest automated model cloning attempts. Flagged accounts are rate-limited or blocked.
CORS, CSP, and transport security. Standard web security headers are configured. All traffic uses TLS. CORS policies restrict which origins can access the AI API.
Supply chain controls protect against compromised models, libraries, and dependencies. See the AI supply chain risks guide for the full threat landscape.
Model format safety. Models are loaded only from Safetensors or GGUF format. Pickle-serialized models are rejected unless audited by a trusted party.
Model provenance verification. Downloaded models come from verified publishers with checked cryptographic hashes. An internal model registry tracks approved models.
Dependency pinning with hash verification. All dependencies are pinned to exact versions with cryptographic hash verification. Lockfiles are committed to version control.
Automated vulnerability scanning. Dependency scanning (Dependabot, Snyk, pip-audit) runs in the CI/CD pipeline. High-severity vulnerabilities are treated as deployment blockers.
Isolated training and inference environments. Training runs in containers with restricted network access. Inference environments have minimal permissions, limited to the specific APIs and data stores the application requires.
Embedding model integrity checks. Embedding models are verified by hash at load time. Embedding quality benchmarks run regularly to detect substitution or degradation.
Monitoring ties all security layers together by providing visibility into what is happening across the system and enabling rapid response to detected attacks.
Real-time security alerting. Alerts fire on high-confidence prompt injection detection, system prompt content in output, unauthorized tool call attempts, rate limit breaches, and anomalous query patterns.
Dashboards for security metrics. Track injection detection rates, false positive rates, tool call distributions, retrieval accuracy trends, memory write patterns, and per-user activity profiles.
Comprehensive interaction logging. Every request, response, tool call, memory operation, and guardrail trigger is logged with context in a tamper-resistant log store.
Adversarial test suite in CI/CD. Automated adversarial tests run before every deployment, catching security regressions from prompt changes, model updates, or configuration modifications. See the adversarial testing guide.
Incident response playbook. Documented procedures for responding to detected attacks: prompt injection incidents, data poisoning discovery, memory corruption, system prompt extraction, and tool exploitation. The playbook includes containment steps, investigation procedures, and recovery actions for each scenario.
Regular manual red teaming. Quarterly (at minimum) manual red team assessments by testers with both security expertise and AI domain knowledge, supplementing continuous automated testing with creative adversarial thinking.
Using This Checklist
Not every item applies to every application. A read-only chatbot with no tool access does not need tool permission scoping. An application using a hosted model API does not need supply chain controls for model files. Start with the items that address your application's specific threat model: if your application has tool access, the tool access section is critical; if it uses a knowledge base, the data pipeline section is critical; if it stores persistent memory, the memory integrity section is critical. The input, output, API, and monitoring sections apply to virtually every AI application.
This checklist represents the current state of AI security best practices as of mid-2026. The field evolves rapidly as new attack techniques emerge and new defenses are developed. Review and update your security controls at least quarterly, and subscribe to security advisories from OWASP (LLM Top 10), your LLM provider, and the open-source tools you use for detection and monitoring.
AI security is defense in depth. No single checklist item protects the application on its own. The value is in the combination: input classification catches most attacks, output filtering catches what input classification misses, tool access controls limit damage when both filters fail, memory validation prevents persistence, and monitoring detects everything else. Implement all layers that apply to your application's architecture, test them adversarially, and monitor them continuously.