Quality, Cost & Speed Trade-offs
Move the triangle with the right levers, and do the unit economics before you scale.
Quality, cost and speed pull against each other, so choose the point for each feature on purpose. Cost comes from model size, length, number of calls and review. Move the triangle with smaller models, less text, caching and better search. Do the unit economics and plan for price and model changes.
- Unit economics
- Cost and value of one task.
- Caching
- Saving an answer to reuse it.
- Latency
- How long the user waits for a response.
Every AI feature sits inside a triangle. Quality, cost and speed pull against each other. A bigger model may answer better, cost more and respond slower. A shorter prompt is cheaper and faster and may lose accuracy. The product manager's job is to choose the right point in the triangle for each feature. This lesson shows how to think about it and how to explain it to others.
The three corners
- Quality: how often the output is right and useful, measured on your test set.
- Cost: what each task costs to run, including model use, search, review and support.
- Speed: how long the user waits, and how fast the system responds under load.
What drives cost
- Model size and type: larger models cost more per token.
- Length: more text in and out costs more. Long documents and long chat histories add up.
- Number of calls: an agent or a multi-step workflow may call the model many times for one task.
- Retries and checks: a second model that checks the first adds cost.
- Support and review: people checking results are part of the cost.
Prices and model options change often. Check current prices, and build in the ability to switch models.
Ways to move the triangle
TO CUT COST - Use a smaller model for easy steps; keep the big one for hard steps - Send less text: shorter prompts, only the relevant passages, trimmed history - Cache answers to common questions and repeated work - Limit steps and retries; set a budget per task - Batch work that does not need to be instant TO CUT DELAY - Stream the answer as it is written - Run independent steps at the same time - Use a faster model for the first response - Do heavy work in the background and notify the user TO RAISE QUALITY - Better instructions and examples - Better search and fresher documents - A stronger model for the hard part - A checking step, or a person for risky cases - Narrow the task so the AI does one clear job
Unit economics
A feature must pay for itself. Work out cost per task and value per task. If a support answer costs a few cents to produce and saves several minutes of an agent's time, the sums are likely good. If a casual feature costs more than users will pay, you have a problem. Look at it for typical users and for heavy users, because a few heavy users can drive most of the cost.
COST PER TASK model calls x average tokens x price + search/storage + review time + support VALUE PER TASK time saved x hourly cost | errors avoided x cost per error | revenue gained MARGIN value (or price) minus cost per task HEAVY USERS What does the top 10 percent of users cost? Do limits or tiers protect margin? SENSITIVITY What if usage doubles? If prices change? If quality needs a bigger model?
Choose the good-enough point
Do not chase the highest quality at any price. Ask what level of quality makes the feature useful and safe, and what the user will pay or wait for. A draft that is right nine times out of ten with a quick edit may beat a slow, costly one that is right ninety-nine times out of a hundred. Set the point on purpose, using data.
Plan for change
- Keep the model behind a thin layer so you can switch.
- Re-run your test set when you change model or prompt, and compare quality, cost and speed.
- Watch provider announcements for new models and prices.
- Set alerts on cost per task and total spend.
- ✓Quality, cost and speed pull against each other. Choose the point for each feature on purpose.
- ✓Cost comes from model size, length, number of calls, retries and checks, and human review.
- ✓Move the triangle with smaller models, less text, caching, streaming, parallel steps, better search and narrower tasks.
- ✓Do unit economics, including heavy users and sensitivity. Plan for model and price changes, and set cost alerts.
Do the sums for one feature
- Pick a feature, such as an AI that summarizes support tickets.
- Estimate tokens per task, the price per token from a current price list, and review time. Work out cost per task.
- Estimate value per task with a clear method. Compute margin and a heavy-user case.
- List three levers you would try first, and what each might do to quality.
Q1Name two ways to cut cost without a big drop in quality.show answer
Use a smaller model for easy steps, send less text, cache repeated answers, or limit steps and retries.
Q2Why look at heavy users separately?show answer
A small group of heavy users can drive most of the cost, so limits or tiers may be needed to protect margin.
Q3Why keep the model behind a thin layer?show answer
Models and prices change. A thin layer lets you switch with little rework, and compare options with your test set.