Incident Response for AI Systems

Even with good design, incidents happen. A prompt injection leaks a document. A key is exposed. An agent takes an action it should not. What matters is how fast and how well you respond. This lesson gives you an incident playbook for AI systems, from the first alert to the lessons afterwards.

What counts as an AI incident

• Data exposure: private data shown to the wrong person or sent out.
• Harmful action: an agent sent, changed or deleted something wrongly.
• Safety failure: the system produced harmful or illegal content.
• Abuse: someone used the system to attack others or to run up costs.
• Compromise: a key, model, package or tool server was taken over.
• Serious quality failure: wrong answers that caused real harm, such as wrong medical or financial guidance.

Prepare before it happens

• Write the playbook and keep it where people can find it fast.
• Name roles: incident lead, technical lead, communications, legal and privacy contacts.
• Know your kill switches: how to turn off a tool, an agent or a whole feature in minutes.
• Keep logs that can answer who, what and when.
• Practice with a drill, at least once a year.

The playbook

1. DETECT AND REPORT   Anyone can raise it. Open an incident record with time and facts.
2. ASSESS              What happened? How bad? Who is affected? Is it still happening?
3. CONTAIN             Stop the harm first: switch off the tool, revoke keys, block the user,
                       disable the feature, roll back the prompt or model.
4. PRESERVE EVIDENCE   Save logs, prompts, tool calls and versions before they rotate away.
5. INVESTIGATE         Find the cause. Was it injection, a leak, a bug, a stolen key?
6. COMMUNICATE         Tell leaders at once. Tell affected people and regulators when required.
                       Ask legal and privacy about duties and deadlines.
7. FIX                 Remove the cause. Add a test that would have caught it.
8. RECOVER             Restore service carefully. Watch closely for a repeat.
9. REVIEW              Hold a blame-free review within a week. Write the lessons and owners.

Containment first

In the first minutes, stop the harm. Do not search for the root cause while data is still leaking. Use your kill switch. Revoke exposed keys. Block the offending account. You can investigate once the situation is stable.

Communicate clearly

• Use plain facts: what happened, who is affected, what you have done, what happens next.
• Do not guess or blame in public messages.
• Update on a fixed rhythm, even when there is little new.
• Legal and privacy rules may require you to tell regulators or users within a set time. Know them in advance.

The blame-free review

After the incident, write down the timeline, the cause, what worked, what did not and what you will change. Focus on systems, not people. Each action gets an owner and a date. Add the attack or failure to your test set so it cannot return silently.

Special points for AI

• Behavior may differ on replay. Save the exact prompt, model version and settings.
• Check whether the same weakness exists in other features that share the prompt, tool or data.
• Consider whether the provider's model update played a part, and tell the provider if needed.
• If a model produced harmful content, record the exact input and output for the review.