19 min

Threat Modeling AI Systems

Draw the system, mark trust boundaries, ask the right questions and rank the threats.

Lesson 2 of 10

0 done

In plain English

Draw the whole system with its data flows and mark every trust boundary. Ask what could be spoofed, changed, leaked, overloaded or over-powered, and add the AI questions about injection and over-agency. Rank threats, give each an owner and turn each into a test.

Data flow: How information moves between parts of a system.
Threat register: A list of threats with likelihood, harm, control and owner.
Over-agency: An agent that can do more than its task needs.

Picture it

A threat model in four moves

Draw the system

Parts and data flows

Mark trust boundaries

Where trust changes

Ask what could go wrong

Spoof, change, leak, overload

Rank and assign

Owner, date, test

A threat model is a plan for what can go wrong. You draw the system, mark where trust changes, and list the attacks that could happen at each point. It sounds formal, but it can be done in an hour with a whiteboard. This lesson shows a simple method for AI systems, and how to turn the results into actions.

Draw the system

Start with a picture. Show every part: the user, the web app, the model, the search index, the tools, the databases, the logs and the third-party services. Draw arrows for the data flow. Mark where each piece of data comes from and who controls it.

Mark the trust boundaries

A trust boundary is a line where the level of trust changes. Data crossing it needs a check. In an AI system, the key boundaries are often these.

Between the user and the system: users can be hostile or simply careless.
Between the system and the documents it reads: a web page or a shared file may hold hidden instructions.
Between the model and your tools: model output is untrusted input to your code.
Between customers in a shared system: one customer must never reach another's data.
Between your system and outside services: vendors, plugins and tool servers.

Ask the questions at each point

threat-questions.md

For every arrow and every box, ask:

SPOOFING       Can someone pretend to be a user, a tool or a service?
TAMPERING      Can someone change the input, the documents, the prompt or the output?
LEAKING        Can data reach someone who should not see it?
DENIAL / COST  Can someone overload the system or run up the bill?
ELEVATION      Can someone get more power than they should have, through the agent?
HIDING         Could an attack happen without leaving a trace in the logs?

AI-SPECIFIC
INJECTION      Can text the model reads (user, document, web, tool result) steer it?
OUTPUT TRUST   Does other code act on the model's output without checking it?
OVER-AGENCY    Can the agent do more than the task needs?
SECRET LEAK    Can the model reveal the system prompt, keys or private data?

Rate and rank

You will find more threats than you can fix. Rank them. For each, estimate how likely it is and how bad the harm would be. Fix the likely and harmful ones first. A simple high, medium or low rating is enough. Write down who owns each fix.

threat-register.md

ID | THREAT                                   | LIKELY | HARM | CONTROL                              | OWNER | DATE
T1 | Hidden text in a PDF makes the agent     | High   | High | Treat doc text as data; no tools     | Sam   | 12 Nov
   email private files to an outside address|        |      | that send email; human approval      |       |
T2 | One customer's search shows another's    | Medium | High | Filter by customer id in the query;  | Priya | 20 Nov
   documents                                |        |      | test with two accounts               |       |
T3 | Long prompts run up the model bill       | Medium | Low  | Token limits and per-user budgets    | Lee   | 01 Dec

Make it part of the build

Do a quick threat model for every new feature that adds data sources or tools.
Update the model when the design changes. An old picture misleads.
Turn each threat into a test, so the control is checked again on every release.
Keep the register where engineers will see it, and review it on a schedule.

⚠Threat modeling fails when only the security team does it. Include the engineers who built the system. They know where the shortcuts are.

Key takeaways

✓Draw the whole system with data flows, then mark every trust boundary.
✓Ask spoofing, tampering, leaking, cost, elevation and hiding questions, plus the AI-specific ones about injection, output trust and over-agency.
✓Rank threats by likelihood and harm, assign owners and dates, and turn each into a test.
✓Update the model with each design change, and include the engineers who built the system.

Hands-on exercise

Threat-model a small system

Draw an assistant that reads a shared drive and can send email on a user's behalf.
Mark five trust boundaries on your drawing.
Write at least six threats using the questions list. Rate each by likelihood and harm.
Write a control and an owner for the top three, and one test for each.

Check yourself

Q1What is a trust boundary?show answer

A line where the level of trust changes, such as between a user and the system or between model output and your tools. Data crossing it needs a check.

Q2Why treat model output as untrusted?show answer

The model can be steered by hostile text. If other code acts on its output without checks, an attacker can reach that code.

Q3How do you turn a threat model into action?show answer

Rank threats, assign owners and dates, add controls and write a test for each so they are checked on every release.

← What an AI Security Engineer Does Prompt Injection & Jailbreaks →