Governance, Compliance & Incident Response

Governance is what lets an organization say yes to agents with its eyes open. It answers questions that will arrive sooner than you expect. Who owns this agent? What data does it touch? What can it do? Who approved it? What happens when it goes wrong? This lesson covers light structures that work for small teams and that grow. It also covers how to run an incident when, not if, something breaks. This is general guidance, not legal advice. Involve a lawyer for your country and your sector.

The agent register

Start with a list. For every agent in production, keep a one-page record. Include its name and purpose. Include its business owner and its technical owner. List its users and the people it affects. List the data it can read and write, with sensitivity classes. List its tools and permissions, its autonomy level and its approval rules. Note the models and vendors it uses, and its risk class. Add links to the design doc, the eval results, the runbook and the date of the last review. You cannot govern what you have not listed. And shadow agents, built by keen employees, are the biggest gap in most organizations.

name: support-triage
purpose: Classify inbound tickets and draft replies for human send
owners: { business: head-of-support, technical: platform-ai-team }
users: [support-agents]            # internal
affected_parties: [customers]
data:
  reads:  [ticket_text (contains personal data), kb_articles]
  writes: [ticket_tags, reply_drafts]
tools: [kb.search, ticket.add_tag, email.save_draft]
autonomy: "propose-only; no external send"
risk_tier: medium
vendors: [model-provider-A (DPA signed 2026-03), observability-vendor-B]
disclosure: "Drafts are labelled 'AI-assisted' to staff; customers always receive human-sent replies"
last_review: 2026-09-12
links: { design_doc: ..., evals: ..., runbook: ..., audit_log: ... }

Classify the risk and scale the controls

• Low: internal, read-only, no sensitive data and easy to undo. Light review and normal monitoring.
• Medium: touches personal or confidential data, or drafts messages to outsiders. A design review, an eval gate, a person sends the message, and an audit log.
• High: takes important actions, affects people's rights, money, safety or jobs, or works in a regulated area. A formal risk assessment, approval before launch, human oversight built in, ongoing monitoring and a regular independent review.

Data governance

• Use as little data as you can: send the model only what the task needs, and remove or replace identifiers where possible.
• Know where data goes: the vendor's terms on keeping data and on training, sub-processors, and the regions where it is processed and stored. Have data processing agreements in place.
• Retention and deletion: how long prompts, traces and memory are kept, and how a deletion request reaches traces, vector stores and caches.
• Access control: who can see traces, prompts and audit logs, and log that access.
• Cross-border and sector rules: where data may live, and rules for health, finance and education, can limit which vendors and regions you may use.

Frameworks worth knowing

• NIST AI Risk Management Framework: it organizes AI risk work into Govern, Map, Measure and Manage. It is voluntary and widely used.
• ISO/IEC 42001: a standard for a management system that organizations can be certified against, when they build or use AI.
• EU AI Act: a rule based on risk, with duties that start in phases over several years. It includes transparency duties, so that people know when they are dealing with AI, and stricter rules for high-risk uses. The dates and details have been changing, so check the current position with a lawyer.
• OWASP guidance for LLM and agent applications: practical security checklists. See the guardrails lesson.

Be open and honest

Tell users when they are dealing with an AI, what it can and cannot do, and how to reach a person. Do not let an agent claim to be human, or claim abilities it does not have. Say when content is AI-made, where the situation or the law requires it. Clear, honest communication prevents more trust incidents than any technical control.

Incident response for agents

Decide your severity levels and playbooks now. A reasonable ladder looks like this. Sev 1: an agent took harmful or unauthorized actions, leaked data or is spending out of control. Sev 2: quality dropped a lot, or there is a widespread outage. Sev 3: problems that are contained and have workarounds.

# Agent incident runbook

## 1. Detect & declare
Alert / user report / audit finding -> on-call declares severity, opens incident channel, names an incident lead.

## 2. Contain  (minutes, not hours)
- Flip the kill switch for the affected agent or tool (feature flag, no deploy).
- Revoke or rotate credentials the agent used if data or actions may be compromised.
- Pause queues; stop runaway spend with the circuit breaker.
- Switch to the human fallback path and tell support.

## 3. Assess
- Use traces + audit log: which runs, users, tenants, actions and data were affected? Time window?
- Identify the first wrong span: bad input? prompt injection? model change? tool bug? policy gap?

## 4. Remediate
- Reverse harmful actions where possible (undo/compensation list), notify affected parties as required by policy/law.
- Ship the fix through the normal release pipeline (fast-track, but still eval-gated).

## 5. Communicate
- Internal updates on a cadence; customer/regulator notices per policy (involve legal/privacy early).

## 6. Learn (blameless postmortem within 5 days)
- Timeline, root cause, contributing factors, what detection missed.
- Action items with owners and dates.
- ADD THE FAILING CASES TO THE EVAL SET and the attack suite.