Red Teaming & Security Testing

You find weaknesses before an attacker does by attacking your own system. This is red teaming. It is part test and part creative thinking. This lesson shows how to plan a red team exercise, which attacks to try, how to record findings and how to turn them into permanent tests.

What red teaming is

A red team plays the attacker. It tries to make the system break its rules, leak data or misuse its tools. A blue team defends. In smaller organizations the same people may play both roles at different times. The goal is to learn, not to blame.

Set the scope and rules

• Agree what is in scope: which systems, environments and attack types.
• Use a test environment with fake data and accounts. Never attack real customers.
• Get written permission from the owners.
• Agree how to report serious findings at once.
• Set a time box, so the exercise has an end.

A plan of attacks

AREA                 ATTACKS TO TRY
Instructions         Override, role-play, reveal the system prompt, long and confusing input
Hidden content       Instructions in documents, web pages, emails, images, tool results
Data                 Ask for other customers' data; ask for secrets; search for private terms
Tools                Wrong arguments, huge amounts, other users' ids, chained calls
Output handling      Make the output carry code, links or markup that other systems run
Safety rules         Ask for harmful or off-limits content in disguised ways
Cost and abuse       Very long prompts, loops, repeated calls, many accounts
Languages and formats  The same attack in other languages, encodings and file types

For each: goal, steps, expected safe result, actual result, severity, fix.

Use tools carefully

Open-source and commercial tools can run many known attacks against a model or an app. They save time and give coverage. They do not replace human creativity. The best findings often come from a person who understands the product and thinks of a misuse the tools never imagined.

Record findings clearly

TITLE: Hidden text in a shared PDF makes the assistant email private notes
SEVERITY: High        STATUS: Open        OWNER: {{name}}

WHAT HAPPENED (plain words)
A PDF with white text told the assistant to send the user's last five notes to an outside
address. The assistant did it without asking.

HOW TO REPRODUCE
1. Upload the test PDF named {{file}}  2. Ask: "Summarize my files"  3. Watch the tool log

WHY IT MATTERS
A stranger could steal private notes by sharing a file.

FIX
Treat file text as data only; block outbound email without approval; add this PDF to the test set.

RETEST: {{date}}    RESULT: {{...}}

Rate severity

• Critical: leaks other customers' data or lets an attacker act with high power.
• High: leaks data about one user, or lets an attacker misuse a tool.
• Medium: breaks a safety rule without a direct path to harm.
• Low: unusual behavior with little impact.

Turn findings into tests

A fix without a test will break again. For every finding, add the attack as a permanent test. Run the attack set on every release. Over time you build a regression suite of real attacks. New models and new prompts must pass it before shipping.

Report with care

Write for two readers. Leaders need the risk and the cost of the fix. Engineers need the steps to reproduce. Report problems without blame, and share lessons across teams.