Security, Safety & Human Oversight

An agent that reads untrusted text and can act is a new kind of attack surface. This lesson covers prompt injection, the OWASP risk words, secrets and authorization, and human-in-the-loop design. One rule runs through all of it: do not rely on the model to enforce security.

Prompt injection

Prompt injection happens when untrusted content tries to influence the model beyond its intended role. It can be direct, such as ignore previous instructions. Or it can be indirect, with bad instructions hidden in a web page, document, email or retrieved record. The core defense is in the design. Do not rely on prompts as your only security boundary. Keep data apart from instructions. Limit the tools. Enforce authorization outside the model. Clean and label untrusted content. And ask for confirmation before sensitive actions. An agent that can read any web page and send email is especially exposed.

• Assume that retrieved and web content is hostile.
• Use tools with least privilege.
• Require policy checks and human approval for high-impact actions.

OWASP LLM risks as a checklist for threat modeling

The OWASP Top 10 for LLM Applications gives a shared set of risk words. They are prompt injection, leaking sensitive information, supply-chain risks, mishandled output, too much agency and weak system design. The architect's job is to turn each risk into controls.

Excessive agency          -> narrow tools, authorization, approval workflows, budgets, audit logs
Sensitive data exposure   -> data classification, filtering, redaction, access control, provider configuration
Prompt injection          -> isolate untrusted content, least privilege, output validation, confirmation gates
Improper output handling  -> never execute/render model output unescaped; validate before use
Supply chain              -> vet and pin tools, models and MCP servers; scan dependencies

For each risk list PREVENTIVE, DETECTIVE and CORRECTIVE controls. Test agent permissions as aggressively as API permissions.

Secrets, identity and authorization

AI applications connect to sensitive systems. Never put long-lived secrets into prompts or model context. Give tools short-lived credentials, or identities that live on the server. Enforce authorization at the boundary of each tool and service. The model may suggest delete customer. But only an authorized backend can decide whether this person may do it. In systems with many customers, carry the customer identity through every layer. That means the request, the retrieval, the tool call, the database query, the logs and the background jobs. A missing customer filter is a critical security bug.

• Use IAM and service identities, and short-lived credentials.
• Enforce authorization on the server, at the boundary of every tool and every data access.
• Carry the customer and user identity through asynchronous workflows.

Human-in-the-loop

Human approval is right when the cost of a wrong action is high, or when policy demands human judgment. Examples are money transactions, destructive operations, legal commitments, production deployments and sensitive messages. An approval request must hold enough context to decide. That means the proposed action, the target, the evidence, the expected effect, the risk and the alternatives. Agent wants to proceed is not an approval screen. Review can also be sampled instead of applied to everything. A system based on risk approves low-risk actions automatically, and sends high-risk ones to reviewers.

PROPOSED -> (policy: low risk) -> AUTO_APPROVED -> EXECUTED
PROPOSED -> (policy: high risk) -> NEEDS_HUMAN -> APPROVED | REJECTED | EDITED -> EXECUTED | CANCELLED
Every transition is logged: who/what decided, when, on what evidence, under which policy version.