AI incident response: building the runbook
By Sam Rivera, Founder, SentinelPanda · June 3, 2026 · 3 min read · AI Governance
Your existing IR runbook does not cover the failure modes that matter for AI systems. Build the AI-specific one before you need it.
Why AI incident response is different
A classical security incident has a defined attack surface — exploited vulnerability, compromised asset, lateral movement, exfiltration. An AI incident may be a misclassified loan applicant, a generated piece of false medical guidance, a chatbot that walked a user into giving away credentials, or an agentic system that booked a flight nobody approved. The failure modes do not fit the same taxonomy.
They also do not surface the same way. SIEM correlation does not detect bias drift. Endpoint protection does not catch prompt injection. AI incident detection relies on evaluation suites running continuously, monitoring metrics on output quality, and reports from human reviewers who notice something off. The detection capability is what most teams under-invest in.
What counts as an AI incident
- Model behaviour failure: hallucination causing material harm, severe bias incident, accuracy below contracted threshold, drift outside acceptable bounds.
- Security incident: prompt injection that exfiltrated data or triggered unauthorised action; training-data exposure; model weight theft; supply-chain compromise.
- Policy violation: model emitting prohibited content, system used for an excluded purpose, model deployed without required impact assessment.
- Third-party failure: vendor model change degrading output, sub-processor breach, upstream model deprecation breaking your service.
- Affected-person harm: complaint received, regulator inquiry, media incident involving your AI system.
The runbook structure
Treat the AI runbook the same way you treat the security runbook: roles, triggers, classification, containment, eradication, recovery, post-mortem. Specific roles needed: AI incident commander (typically the AI programme owner), model owner (engineering lead for the affected system), legal counsel (especially for affected-person incidents), communications, data scientist or evaluator (for technical analysis), and human reviewer pool (for re-checking decisions).
Classification needs an additional dimension beyond severity: which framework's notification obligations does this trigger? EU AI Act Article 73 requires providers of high-risk AI systems to report serious incidents to market surveillance authorities — define what counts as "serious" in advance (death or serious damage to health, significant infrastructure disruption, infringement of fundamental rights, serious property damage). ISO 42001 incidents may need to be reported as nonconformities. NIST AI RMF includes ongoing incident management in the Manage function but no external notification.
Detection capability
You cannot respond to incidents you cannot see. AI incident detection requires three things: an evaluation suite that runs continuously against the production model and alerts on regression; monitoring metrics on outputs (quality, bias, refusal rate, latency, token spend, anomalous prompt patterns); and a reporting channel humans actually use (a button in the product UI for end users to flag bad outputs, internal review processes for human-in-the-loop cases).
The detection signals you collect feed both the runbook and your post-incident learning. A pattern of similar incidents indicates a model-level problem that requires retraining, fine-tuning, or prompt engineering — not a one-off response.
What to do this quarter
- Stand up an AI incident classification taxonomy that maps to your framework notification obligations (Article 73, ISO 42001 nonconformity, contractual customer notification).
- Write or extend your IR runbook with AI-specific roles, triggers, and containment procedures.
- Define "serious incident" for each in-scope high-risk system in plain language, agreed with legal and the model owner.
- Stand up the detection capability: at least one evaluation suite running continuously, one user-facing report channel, one production metric on output quality.
- Run a tabletop on a synthetic AI incident — prompt injection, hallucination harm, biased decision — to test the runbook.