Capstone: Design a Multi-Agent System

This capstone ties the whole path together. You design one complete system on paper before you write code. That is how experienced teams work. We will design a market-research report maker. Given a company and a question, it makes a decision-ready brief with sources. The work is wide, it is valuable on each run, and it has natural points to check. That makes it a good fit for orchestration.

Step 1: goal and success measure

Write the goal as a sentence you can check. For example: given a company name and a decision to support, deliver a brief of two pages or less where every factual claim has a source address. It covers competitors, pricing and recent news. It takes under 5 minutes and costs under $3. Then say how you will score it: factual accuracy, citation accuracy, completeness against a checklist, and how useful the reader finds it, on a scale of 1 to 5.

Step 2: the baseline

Start with one agent that has web search and page-fetch tools and a good prompt. Run it on 20 sample requests and keep the numbers. The likely problems you will see are these. The context fills with raw page text, so later sections come out thin. And claims with no source creep in. Those two problems are your reason to split: separate contexts and independent checking.

Step 3: roles, tools and permissions

• Lead researcher (strong model): plans, writes the briefs for the workers and combines the results. Its only tools are spawn_worker and read_artifact. It never browses.
• Search workers, as many as needed (small, fast model): each takes one angle, such as competitors, pricing or news. Tools: web_search and fetch_page. They can write only to the file store.
• Fact-checker (independent, mid-size model): it receives only the claims and source references, not the lead's reasoning. Tool: fetch_page. For each claim it returns verified, contradicted or unsupported.
• Writer (mid-size model): receives the verified claims and writes the formatted brief. It has no tools.

Notice how the permissions are split. Only the workers touch untrusted web content, and none of them can do anything except save notes. The lead and the writer see extracted facts with sources, and never raw pages.

Step 4: hand-off contracts and state

Define the schemas once: WorkerBrief, VendorFinding (claim, evidence quote, source address, time fetched, confidence), ClaimVerdict and FinalBrief. The state holds the goal (fixed), the plan, the findings for each task, the verdicts, the budget used and the status. The reducers are: findings merge by key, verdicts merge by claim ID, and errors are append-only. Anything large goes to the file store by reference.

Step 5: the reliability budget

run:
  max_seconds: 300
  max_tokens: 600000
  max_cost_usd: 3.00
lead:
  max_workers: 6          # effort scaling: 1-2 for simple lookups, 4-6 for full brief
  max_replans: 1
worker:
  max_tool_calls: 8
  timeout_seconds: 60
fact_checker:
  max_claims_checked: 25  # prioritize numbers, dates, comparisons
  rounds: 1
on_budget_exceeded: return_partial_with_gaps
on_worker_failure: retry_once_then_mark_gap

Step 6: testing and rollout

• Before launch: run the 20-case set and score the outcomes, with the single-agent baseline as the control. Ship only if the gain in quality justifies the extra cost.
• By component: check the split (no overlap, no gaps), the citation accuracy, and how well the fact-checker finds planted false claims.
• Live: run it in shadow mode on real requests. Compare with what people produced, and review the differences.
• Rollout: start with internal users, a cost cap on each run, a human review step before anything leaves the building, and traces on every run.

The design-doc template

# <System name> — design doc

## 1. Goal & success metric
(one checkable sentence + how it is scored)

## 2. Baseline
(single-agent version, results on N cases, observed failure modes)

## 3. Why multi-agent
(which of: context isolation / parallelism / specialization / privilege split / verification)

## 4. Agents
| Agent | Model tier | Tools | Permissions | Sees raw untrusted data? |
|-------|-----------|-------|-------------|--------------------------|

## 5. Pattern(s) and flow
(diagram: router? orchestrator-workers? critic loop?)

## 6. Hand-off contracts & shared state
(schemas, reducers, artifact store, what is pinned)

## 7. Reliability
(budgets, timeouts, retries, termination criteria, partial-result policy)

## 8. Cost & latency target
(napkin math, critical path)

## 9. Evaluation
(offline set, baseline comparison, component checks, online plan)

## 10. Risks & rollout
(security, abuse, human-in-the-loop points, kill switch)