AI Incidents Are Different
Traditional security incidents have a discrete structure: a breach happens at a moment in time, a threshold is crossed, a payload executes. The before-state and after-state are distinguishable. The incident timeline has a beginning and an end. Detection is often about finding the signature of the intrusion — the malicious binary, the anomalous login, the exfiltrated file.
AI security incidents don't work this way. There is rarely a clear "breach moment." Instead, there is behavioral drift — outputs that gradually shift toward unexpected territory, tool calls that occur slightly outside normal patterns, responses that comply with requests they should decline. The compromise is often a spectrum rather than a binary, and the cause is often invisible in any single interaction. It requires looking at patterns across many interactions over time.
This structural difference has significant operational implications. Your incident response runbook, built for traditional security events, will not map cleanly onto an AI compromise. The concepts carry over — detection, containment, investigation, recovery, post-incident review — but the specific actions in each phase are fundamentally different. Teams that haven't thought through these differences before an incident will discover them under pressure, at the worst possible time.
Detection Signals
The goal of detection is not to find the attack — it's to notice that something is wrong before the damage becomes unrecoverable. For AI systems, that means instrumenting for behavioral deviations rather than signature matches.
Outputs that don't match the application's expected behavior envelope: a customer service bot that starts giving financial advice, a summarization model that starts including extra instructions in its summaries, a coding assistant that starts recommending insecure patterns it previously avoided. Automated output quality checks with LLM-based evaluation can catch distributional shift before humans notice it.
For agentic systems, unexpected tool usage is a primary signal. A model that starts calling the email tool during a read-only summarization task, or that begins querying unusual database tables, or that makes external HTTP requests to domains not in its expected call pattern — these are signs that the model's behavior is being influenced by something outside normal operation. Log every tool call with its triggering context.
Users are often the first to notice behavioral anomalies. "The bot told me to do something weird," "it asked me for information it never needed before," "it seems to be talking about something unrelated to my question." Build a lightweight feedback mechanism that routes these reports directly to your security team. User reports are noisy but high-value — they represent real behavioral deviations that made it to production.
Run your model against a fixed set of behavioral benchmark prompts on a regular schedule — ideally after every deployment and at least daily in production. Include prompts specifically designed to test refusal behaviors, safety constraints, and task-relevant competency. Significant degradation on refusal benchmarks is an early warning sign of model compromise or prompt configuration drift.
Containment Steps
Once you suspect an AI system is behaving abnormally, the immediate priority is containing the blast radius — preventing the compromised behavior from affecting more users or taking more actions. Do not wait for full investigation before containing. Contain first, investigate second.
Immediately reduce the number of users exposed to the potentially compromised model. Route new traffic to a known-good backup model or model version. If you don't have a backup, apply aggressive rate limiting to slow down exposure while you investigate. For agentic systems with tool access, consider disabling the highest-risk tools first — those that take irreversible external actions (email, file writes, API calls).
For agentic deployments, the tools available to the model determine the blast radius of any compromise. When you suspect compromise, disable tools in order of potential impact: external communication tools first (email, messaging), then write tools (database writes, file system), then read tools (unless the reads are the attack vector). Leave only the minimum toolset needed to confirm the anomaly you're investigating.
Before you make any changes to the system prompt, model version, or configuration — preserve the current state. Take a snapshot of the active system prompt, the deployed model version, the tool configuration, and a sample of recent inputs and outputs. You will need this for forensic investigation. Modifying the system without preserving state first is a common mistake that destroys investigation evidence.
Investigation Framework
Investigation means answering one central question: where did the unexpected behavior originate? For AI systems, there are five primary vectors to examine. Work through them systematically, as more than one may be involved simultaneously.
Vector 1 — Prompt injection via user input: Did a user provide input that successfully overrode the system prompt? Review conversations leading up to the anomalous behavior. Look for attempts to use role-play framing, authority claims, or instruction override language. Check whether the anomalous behavior started after a specific user interaction.
Vector 2 — Indirect injection via tool outputs: Did the model process data from an external source that contained adversarial instructions? If the model reads from any external data source — web pages, databases, files, API responses — audit recent tool outputs for instruction-like patterns. Cross-reference the timing of anomalous behavior with specific tool calls.
Vector 3 — System prompt misconfiguration: Was the system prompt modified, corrupted, or incorrectly deployed? Compare the active system prompt against your version-controlled baseline. Even small unintended changes can have large behavioral effects. Check your deployment pipeline for unauthorized modifications.
Vector 4 — Model version issues: Was a new model version deployed that behaves differently on your use case? Model providers update models frequently, and behavioral changes can be subtle. If anomalous behavior started after a model update, compare behavior across versions using your behavioral benchmarks.
Vector 5 — Fine-tuning or training data compromise: For teams using fine-tuned models, was the training dataset poisoned? This is the hardest vector to detect and the slowest to manifest — the effects of training data poisoning may only become apparent under specific input conditions that trigger the backdoor.
Recovery and Hardening
Recovery is not complete when the anomalous behavior stops. Recovery is complete when you have established that the root cause has been addressed, verified that behavior has returned to the expected envelope, and hardened the system against the recurrence of the identified attack vector.
Before restoring full service, run a targeted red team exercise specifically against the attack vector you identified. If the incident was caused by prompt injection, attempt prompt injection against the remediated system. If it was caused by indirect injection, attempt to inject malicious instructions through the same data channel. Do not re-deploy without explicit confirmation that the identified vector has been closed.
Have pre-staged rollback procedures for your model version, system prompt, and tool configuration. A rollback should be executable in under five minutes by a single on-call engineer without requiring coordination with external parties. If your current deployment process requires more than this, the gap between "incident detected" and "anomalous model contained" will be measured in tens of minutes — during which the compromised behavior continues.
Post-Incident Documentation
The post-incident review is not bureaucratic process — it is how your organization learns to detect and respond faster to the next incident. For AI incidents specifically, the review should produce four artifacts: a timeline of the incident from first detection to full recovery; a root cause analysis that goes beyond "the model was compromised" to identify the specific technical mechanism; a list of detection improvements that would have caught the anomaly sooner; and an architecture review that identifies structural changes that would reduce the impact of similar incidents.
Consider sharing sanitized incident details with the broader AI security community. The community is still building its collective knowledge base for AI security incidents. Every published incident, even anonymized, helps other teams build better detection and response capabilities. The adversarial landscape for AI systems is evolving rapidly — the community benefits from shared intelligence in ways that traditional security incident sharing already recognizes but AI security is still developing the culture and channels for.