AI Security Questions

In plain words: It is a trick hidden inside content the AI reads, like a web page or email. There is no bad user to block, so you must limit what the AI can do.

Full answer · Instructions hidden in content the agent reads — a web page, email, document, retrieved record or tool output — that attempt to steer its behavior. The attacker never talks to the agent; the content does. It's dangerous because there is no 'bad user' to block, and the model can't reliably distinguish your instructions from instructions embedded in data. Defenses are architectural: isolate untrusted content, restrict tools, authorize outside the model and require approval for sensitive actions.

In plain words: Through anything it can write to: email, links, web requests or shared files. Keep private data, outside content and sending apart.

Full answer · Through any channel it can write to: sending email, making HTTP requests, creating links or rendered markdown images whose URLs carry data, writing to shared documents or logs. Combine private data access, untrusted content and an outbound channel in one agent — the 'lethal trifecta' — and an injected instruction can leak data. Break one leg: split the agent, remove the channel, allow-list destinations, or require approval for outbound actions.

In plain words: Because the AI can be fooled, but code cannot. Permission checks in code work every time.

Full answer · The model only proposes actions and can be influenced by what it reads; a prompt rule can be ignored or argued around. Authorization in the tool and service layer is deterministic: it checks the authenticated principal, the resource and the operation, regardless of what the model says. 'The model decided' is never an authorization decision.

In plain words: Giving an agent more tools or freedom than the job needs. A mistake or trick then causes bigger damage. Give only what is needed.

Full answer · Giving an agent more tools, permissions or autonomy than its task needs, so any mistake or manipulation has a large blast radius. Controls: narrow tools around business capabilities, least-privilege short-lived credentials, per-tool limits, budgets, approval workflows for high-impact actions and audit logs. Ask of every tool: what's the worst thing a manipulated agent could do with it?

In plain words: Attack the whole system with hidden instructions, bad tools, other customers' data and cost attacks. Turn each failure into a test.

Full answer · Attack the whole system, not just the model: direct and indirect injection through every input channel, tool abuse and malicious tool responses, data exfiltration attempts, privilege escalation, cross-tenant access, loop and cost attacks, and memory poisoning. Automate the scenarios, track attack success rate, turn every finding into a regression test and a design control, and re-run on every release. The key question is what a successful trick can cause, not whether the model can be tricked.

In plain words: Never put secrets in prompts. Tools use short-lived keys kept on the server, and logs hide secrets.

Full answer · Never put long-lived secrets in prompts, context, memory or logs. Tools receive short-lived credentials or run under server-side service identities scoped to least privilege, retrieved from a secrets manager at call time; the model only sees tool names and results. Redact secrets in telemetry, rotate regularly and scan repositories and traces for leaks.

In plain words: Send only what is needed. Hide private details. Use business terms that do not train on your data. Choose the right region.

Full answer · Minimize and redact what's sent, classify data and filter by sensitivity, use enterprise or API terms with no training on your data and appropriate retention, choose regions that meet residency rules, have data-processing agreements in place, and keep a model gateway where policy and logging are centralized. For regulated data, confirm contractual and regulatory coverage before sending anything.

In plain words: It is trusting the AI's output blindly, such as running it as code. Treat it like any outside input and check it first.

Full answer · Using model output as trusted input to another system — rendering it as HTML, executing it as code or SQL, passing it to a shell or following links — which turns a manipulated output into XSS, injection or command execution. Treat output as untrusted: encode on render, parameterize queries, validate against schemas and allow-lists, sandbox any code execution and never let free-form text trigger privileged operations.

In plain words: Tools, models and plugins are outside code you trust. Check them, pin versions, limit their access and watch what they do.

Full answer · Models, tools, MCP servers, plugins, prompts and dependencies are all third-party code or text the agent trusts. A compromised tool description can steer behavior; an updated server can change what a tool does. Vet sources, pin versions and review changes, scope each integration's credentials, scan dependencies and monitor tool behavior — and don't join a server that reads untrusted content to one holding sensitive access without gates.