{}Route To AI
Courses
← AI Security Engineer
18 min

What an AI Security Engineer Does

What is new about AI security, what stays the same and what the role covers.

Lesson 1 of 10
0 done
In plain English

AI systems read text they cannot trust, and instructions and data mix in one stream. That is the root of many attacks. Classic security rules still apply. The role covers threat models, design reviews, limits on agents, testing, guardrails, monitoring and incident response. Real controls belong in code and permissions, not in the prompt.

Trust boundary
A line where the level of trust changes.
Least privilege
Giving only the smallest permissions needed.
Threat model
A plan of what can go wrong and where.
Picture it
What is new, and what is the same

Put real controls in code and permissions, not in the prompt.

New with AI
  • Instructions and data mix in one stream
  • Hidden text can steer the model
  • An agent can act with its permissions
Still true
  • Least privilege
  • Strong login and encryption
  • Logging, patching and review

An AI security engineer protects systems that use language models and agents. The job mixes normal security with new problems. These systems read text they cannot fully trust. They can be tricked by words. And they can take actions. This lesson explains what is new, what is not, and what the role does day to day.

What is new about AI security

Classic software has a clear line between code and data. A database does not treat a customer's name as a command. A language model has no such line. Instructions and data arrive in the same stream of text. A web page, an email or a document can contain words that the model treats as instructions. This is the root of many AI attacks.

  • Inputs can carry hidden instructions, not only from users but from documents, web pages and tool results.
  • Outputs can cause harm when other software trusts them, such as running a command or showing raw code.
  • Models can leak data they were shown, in the prompt, in search results or in training.
  • Agents add power: an attacker who steers an agent may act with the agent's permissions.
  • The model's behavior is not fixed. A defense that worked yesterday may fail after a model update.

What stays the same

The old rules still apply. Least privilege. Strong login. Encryption. Logging. Patching. Reviewing code and dependencies. Many AI incidents come from plain mistakes, such as a secret key in a prompt or an admin tool open to everyone. An AI security engineer must be a good security engineer first.

What the role does

  • Maps the system and its trust boundaries, and writes the threat model.
  • Reviews designs before they are built, and asks what could go wrong.
  • Sets limits on tools, data and permissions for agents.
  • Tests the system with attacks before launch and on a schedule after.
  • Builds or chooses guardrails, filters and monitoring.
  • Plans and leads the response when something goes wrong.
  • Teaches engineers and product managers how to build safely.

The assets you protect

assets.md
DATA        Customer records, company documents, secrets, the system prompt, logs
ACTIONS     What the system can do: send email, move money, change records, run code
IDENTITY    Accounts, keys and tokens the system uses
TRUST       The reputation of the product and the safety of its users
MONEY       Model costs, which an attacker can run up
AVAILABILITY  The service must keep working

Know the main reference lists

  • OWASP Top 10 for LLM Applications: a widely used list of the main risks, such as prompt injection, sensitive information disclosure, supply chain issues, improper output handling and excessive agency.
  • MITRE ATLAS: a catalog of attacker tactics and techniques against AI systems, in the style of the ATT&CK framework.
  • NIST AI Risk Management Framework: a general guide to managing AI risk.
⚠These lists change. Check the current versions before you quote them in a design review or an interview.

A healthy mindset

Assume that any text the model reads can be hostile. Assume that the model will sometimes ignore its instructions. Design so that a mistake or an attack cannot do serious harm. Do not rely on the prompt to keep secrets or enforce rules. Put the real controls in code and in permissions.

→Try attacking a small AI app yourself this week. Ask it to ignore its instructions. Hide a command in a document it reads. You will learn the problem faster by doing it.
Key takeaways
  • ✓AI systems read text they cannot trust, and instructions and data mix in one stream. That is the root of many attacks.
  • ✓Classic security rules still apply, and many incidents come from plain mistakes such as exposed secrets.
  • ✓The role covers threat models, design reviews, limits on agents, testing, guardrails, monitoring and incident response.
  • ✓Do not rely on prompts to enforce rules. Put real controls in code and permissions, and check current reference lists.
Hands-on exercise

List the assets of an AI assistant

  1. Pick an AI assistant, such as one that answers customer questions and can look up orders.
  2. Fill in the assets list: data, actions, identity, trust, money and availability.
  3. For each asset, write one way an attacker could target it through text the model reads.
  4. Write one control in code or permissions for the two worst cases.
Check yourself
Q1Why is AI security different from classic security?show answer

Instructions and data share one stream of text, so hidden words in documents or web pages can steer the model. The model's behavior is also not fixed.

Q2What does least privilege mean for an agent?show answer

Give it only the smallest permissions and tools it needs, so a tricked agent can do limited harm.

Q3Why not rely on the prompt to enforce rules?show answer

A model can be tricked or can make mistakes. Controls in code and permissions cannot be argued away.

Threat Modeling AI Systems →
0 / 10 finished0%