Data Leakage & Privacy

AI systems handle a lot of sensitive text: customer messages, contracts, medical notes, source code. Every step is a chance for that text to leak. It can leak into prompts, logs, search results, answers and outside services. This lesson maps the main leak paths and the controls that close them.

Where data can leak

• The prompt: private data is sent to a model provider you may not control.
• Logs and traces: prompts and answers are stored, often for long periods and for many people to read.
• Search and memory: one user's documents appear in another user's answers.
• The model's answer: it reveals data it was shown, such as the system prompt or hidden context.
• Training and fine-tuning: private data ends up in a model that others can query.
• Third-party tools: plugins, tool servers and analytics see data you send them.
• Outputs: links, images or code that carry data to an attacker's address.

Know your data

Start by classifying data. Which items are public, internal, confidential or regulated? Mark personal data, health data, payment data and secrets. You cannot protect what you have not named. Write down which classes may go to which model or service.

Provider and contract questions

• Is our data used to train the provider's models? Is there an opt-out in writing?
• Where is data processed and stored? For how long?
• Who at the provider can read it, and under what rules?
• Can we delete data on request, and get proof?
• What happens to data if the provider is breached?

Customer isolation

When many customers share one system, a single missing filter can show one customer's documents to another. This is among the most serious AI bugs. Carry a customer identifier through every step: the request, the search, the tool call, the cache and the logs.

[ ] Every search query filters by customer id, in the database or index, not after the fact
[ ] Every tool call carries the user's identity and checks permission in code
[ ] Caches and memory are keyed by customer, never shared across customers
[ ] Logs are separated by customer, and access to them is limited and recorded
[ ] Test with two customers: ask for the other's data, expect nothing back
[ ] Background jobs and batch tasks also carry the customer id
[ ] Shared prompts and examples contain no real customer data

Controls that work

• Data minimization: send only what the task needs. Remove names and numbers where the task does not need them.
• Masking and tokenization: replace personal data with placeholders before it reaches the model, and restore it after.
• Access control in code: check permissions at search and tool time. Never rely on the model to hide things.
• Log hygiene: avoid storing full prompts with personal data, or mask them, and set short retention.
• Encryption in transit and at rest, and good key management.
• Output checks: scan answers for secrets and private patterns before showing them.
• Egress limits: block the system from sending data to unknown addresses.

Privacy law basics

Rules such as GDPR give people rights over their personal data, such as access and deletion. They apply to AI systems as they do to any system. Work with your privacy officer. Record what data you use, why, where it goes and how long you keep it. Rules differ by region, so ask a lawyer about yours.