Cost, Latency & Evaluation

A multi-agent system that works but costs ten dollars and takes six minutes for each request is a demo, not a product. This lesson is about money and measuring. It shows how to work out cost and delay before you build, and how to bring both down. Most of all, it shows how to test whether the whole machine beats the simple baseline.

A cost model you can do on a napkin

The cost of a run is roughly the sum, over every model call, of input tokens times the input price, plus output tokens times the output price. The part people forget is that in an agent loop every turn sends the growing context again. So input tokens grow very fast with the number of turns, unless you cache or shrink the context. Add the number of agents and workers, and the costs compound quickly.

def run_cost(agents):
    """agents: list of dicts with turns, avg_in, avg_out, in_price, out_price (per 1M tokens)"""
    total = 0.0
    for a in agents:
        # context grows each turn; approximate re-sent input as avg_in * turns * (turns+1)/2 / turns
        resent_in = a["avg_in"] * (a["turns"] + 1) / 2 * a["turns"]
        total += resent_in * a["in_price"] / 1e6 + a["turns"] * a["avg_out"] * a["out_price"] / 1e6
    return total

lead    = dict(turns=6,  avg_in=6_000, avg_out=900, in_price=15, out_price=75)   # strong model
workers = [dict(turns=10, avg_in=4_000, avg_out=600, in_price=3,  out_price=15)] * 5
print(f"~${run_cost([lead, *workers]):.2f} per run")   # plug in YOUR model prices

What really moves the bill

• Tiers of models: a strong model to plan and combine, and small, fast models for extracting, routing and the grunt work.
• Prompt caching: keep the long, stable beginning of the prompt exactly the same, such as instructions, tool descriptions and shared documents. Then repeated input is billed at the cheaper cached rate.
• Clean context: cut tool outputs short, summarize at the boundaries, and do not carry dead ends forward.
• Fewer, better workers: use effort rules so that trivial questions do not start a research team.
• Early exit: stop as soon as the done-rules are met, instead of running a fixed number of rounds.
• Batch or slower tiers for work that is not urgent, where providers offer a cheaper asynchronous option.

Delay: think about the critical path

Total delay is the longest chain of steps that depend on each other. It is not the number of agents. Draw your plan as a graph and find the critical path. Only shortening that path helps. Running things in parallel shortens it. Adding a review stage that must wait for the others makes it longer. Also stream partial progress to the user, such as searched 12 sources, now cross-checking. How fast it feels matters as much as how fast it is. For long jobs, switch to an asynchronous we will notify you model.

Evaluating a multi-agent system

Test in layers, from the outside in:

• The end result: did the final result achieve the user's goal? This is the number that matters, so start here. For tasks that change things, check the end state, such as was the ticket really resolved, and not only the text.
• A baseline comparison: run the same test set through your best single-agent version. If the team of agents does not win on quality per dollar or per minute, it has not earned its complexity.
• Component checks: routing accuracy (was the right specialist picked?), plan quality (did the split cover the question without overlap?) and hand-off integrity (did the limit survive the hop?).
• Behavior numbers from traces: the number of steps, tool calls, replans, budget hits and retries. A drift in these is an early warning, even when quality looks fine.

Start small. Anthropic's research team described starting with about twenty typical queries, because early changes have big effects that show up even in tiny samples. Then grow the set from real failures. Use AI judges with a written rubric for factual accuracy, citation accuracy, completeness and source quality. Check the judges against human review, because judges drift and have biases. Spot-check them often.

{
  "case_id": "q_017",
  "input": "Compare EU and US data-residency rules for health SaaS",
  "variants": {
    "single_agent":  { "score": 0.71, "cost_usd": 0.42, "seconds": 48,  "steps": 14 },
    "orchestrator":  { "score": 0.86, "cost_usd": 3.10, "seconds": 95,  "steps": 61 }
  },
  "judge_rubric": ["factual_accuracy", "citation_accuracy", "completeness", "source_quality"],
  "human_reviewed": true
}
// Decision rule you might write down: ship the orchestrator only for query classes
// where score gain >= 0.10 AND the task's value justifies ~7x cost.