Model Selection & Coding Agents

Two decisions quietly decide how well an AI system ages. One is which models you depend on. The other is how much freedom you give agents that change code. This lesson covers choosing a model as an engineering test, and it designs a coding agent platform.

Choosing a model is a testing problem

Define typical tasks and compare the candidate models on correctness, tool use, structured output, speed, cost, context needs and safety behavior. Use a model router when tasks differ. A small model can do classifying or extracting. A stronger one can do difficult reasoning. A fallback protects availability. Do not hard-code assumptions about one model across the codebase. Put the model settings behind a provider layer, and keep prompts and tests versioned.

• Create an evaluation set before you choose a model.
• Route by ability, not by brand.
• Track the model version. Behavior changes over time.
• Test fallback models against the same eval set. An untested fallback is a surprise.

routes:
  classify_intent:
    model: <small-fast-model-snapshot>
    max_output_tokens: 50
    fallback: <alt-small-model-snapshot>
  draft_reply:
    model: <mid-model-snapshot>
    fallback: <alt-mid-model-snapshot>
  hard_reasoning:
    model: <strong-model-snapshot>
    budget_usd_per_task: 0.40
eval_gate:
  suite: evals/support@v31
  min_task_success: 0.88
  max_cost_per_success_usd: 0.12
# Pin snapshots, version this file, and run the eval gate whenever any value changes.

Coding agents

Coding agents combine knowledge of the repository, code search, file editing, running commands and tools, testing and step-by-step reasoning. Their biggest advantage is closing the loop. They look at the code, make a change, run the tests, see the failure and refine. Their biggest risk is uncontrolled change. Treat a coding agent like an untrusted junior engineer who iterates unusually fast. Do not treat it like a trusted production operator.

• Run it in isolated workspaces or containers. Give it only the repository and tools it needs, with limited network access where possible.
• Limit command permissions. Record every command and every file change.
• Require tests and review gates before anything reaches production.
• For company use, combine static analysis, dependency checks, secret scanning and human approval.

Designing a coding agent platform

A platform needs repository access, search, editing, running commands, running tests and managing results. Use isolated sandboxes, record the file changes and commands, and run the tests before it proposes that it is done. For company use, connect branch creation, code review, CI, static analysis, dependency scanning and approval gates. And make sure the agent leaves a work trail that can be reproduced.

request: "Fix failing test test_refund_limit"
  1. create sandbox (fresh container, repo @ commit abc123, no secrets, egress allow-list)
  2. agent loop: search -> read -> edit -> run tests -> read failure -> edit ... (budget: 40 steps / 15 min / $2)
  3. every command + file diff appended to an immutable audit trail
  4. gates: unit tests pass, linter clean, dependency + secret scan clean, diff size under limit
  5. open a branch + pull request with the trail attached; a human reviews and merges
  6. sandbox destroyed; artifacts retained per policy

Evaluating a coding agent

Build the evaluation set from your own repositories. Use real past issues and bug fixes. Each one has the commit before the fix and the tests that prove it. Run the agent in a fresh sandbox for each task, several tries each. Score it on whether the hidden tests pass without breaking the rest of the suite. Then look beyond the pass rate. Look at the size and clarity of the diff and how many review comments it attracts. Check whether it changed tests to make them pass. And check its cost for each merged change. A high pass rate with large, hard-to-review diffs may cost more reviewer time than it saves.