Supply Chain & Model Risks
Models, packages, data, tool servers and prompts as entry points, and how to reduce the risk.
Your AI system depends on models, packages, data, tool servers and prompts that you did not write. Each is a possible entry point. Use trusted sources and pinned versions, treat open data sources as untrusted, and keep a bill of materials so you can answer quickly when a problem is announced.
- Supply chain
- All the outside parts your system depends on.
- Data poisoning
- Planting text in a source so answers are steered or wrong.
- Bill of materials
- A list of every part, with version and owner.
Every AI system stands on things you did not write. A model from a provider. Software packages. Datasets. Tool servers and plugins. Prompt templates copied from the internet. Each is a place where an attacker can slip in. This lesson covers the supply chain of an AI system, and how to reduce the risk from borrowed parts.
What is in the chain
- Models: hosted from a provider, or downloaded files from a public hub.
- Libraries and frameworks: the code packages your system imports.
- Data: training sets, fine-tuning data and the documents you index.
- Tool servers and plugins: the code that gives your agent new abilities.
- Prompts and templates: instructions you copied from elsewhere.
- Infrastructure: cloud services, build systems and container images.
Risks in each link
Downloaded model files can hide harmful code. Some older file formats can run code when loaded. A model may also have been changed to behave badly under a secret trigger. A package can be replaced with a lookalike name. A popular tool server can be taken over and updated with hostile code. Data can be poisoned: someone adds crafted text to a source you trust, so the system learns or retrieves the wrong thing.
Model risks
- Use models from sources you trust, and check their origin and publisher.
- Prefer safe file formats that cannot run code on loading.
- Pin the exact version, and record its checksum.
- Test behavior after any model change, including safety and must-never-happen cases.
- Remember that a hosted provider can change a model. Re-run your tests when it does.
Data poisoning
If your system searches documents that outsiders can edit, they can plant text that steers answers. Examples are a public wiki, a shared folder, product reviews or a support inbox. Treat open sources as untrusted. Limit who can add to your index, track where each document came from, and check new content before it joins the index.
Packages and tools
PACKAGES AND IMAGES [ ] Pin versions; use a lock file [ ] Scan dependencies for known problems on every build [ ] Watch for lookalike package names [ ] Build from trusted base images, and rebuild often TOOL SERVERS AND PLUGINS [ ] Install only from trusted publishers; read what the code does [ ] Pin the version; review each update before accepting it [ ] Give each one the least access; no secrets within its reach [ ] Treat its descriptions and results as untrusted text MODELS AND DATA [ ] Record the source, version and checksum of every model [ ] Keep a list of data sources and who can change them [ ] Review new documents before indexing when the source is open [ ] Keep a rollback for models, prompts and indexes
A bill of materials
Keep a list of every part: models, packages, tool servers, datasets and prompt templates, with versions and owners. When a problem is announced, you can answer in minutes whether you are affected. Some teams call this an AI bill of materials. A simple spreadsheet is a fine start.
Vendor review
- Ask for security reports or certifications relevant to the service.
- Read the data terms: training use, retention and sub-processors.
- Check how they handle incidents and how fast they tell you.
- Plan for the vendor failing: have a fallback model or a manual process.
- ✓Your AI system depends on models, packages, data, tool servers, prompts and infrastructure. Each is a possible entry point.
- ✓Use trusted sources, safe file formats and pinned versions. Re-test behavior after every model change.
- ✓Treat open data sources as untrusted, since outsiders can plant text that steers answers.
- ✓Keep a bill of materials, review vendors and keep rollbacks for models, prompts and indexes.
Make a bill of materials
- Take an AI app you know, or invent one with a hosted model, a vector store, two packages and a tool server.
- List each part with its source, version and owner.
- For each part, write what an attacker could do if it were compromised.
- Write the three controls you would add first and why.
Q1What is data poisoning in a search system?show answer
Someone plants crafted text in a source the system indexes, such as a public wiki, so answers are steered or wrong.
Q2Why pin versions of models and packages?show answer
An unpinned part can change without notice, including through an attack. A pinned version stays known and can be reviewed.
Q3What is the use of a bill of materials?show answer
When a problem is announced, you can quickly check whether any part you use is affected.