Change Management & Releases

In an agent, the things that decide its behavior are not just code. They are prompts, the choice of model, tool definitions, search settings and policy rules. Change any one of them and the product changes, sometimes by a lot. Change management means shipping those changes with the same care as code: with versions, review, tests, stages and a way back. This is what separates teams that move fast and safely from teams that are afraid to touch their prompts.

Treat the whole behavior bundle as one release

Define a release as a fixed bundle with an ID. It holds the prompts and their template variables, the model snapshot and settings, the tool schemas and descriptions, the version of the search index, and the guardrail and policy settings. Log the release ID on every trace and usage event. Then what changed, and which version made this, always have answers.

release_id: support-agent@2026-10-05.1
git_sha: 4be9c21
model:
  name: <pinned-model-snapshot-id>
  temperature: 0.2
  max_output_tokens: 900
prompts:
  system: prompts/support_system.v14.md
  triage: prompts/triage.v6.md
tools:
  - { name: kb.search,   schema: tools/kb_search.v3.json }
  - { name: refund.issue, schema: tools/refund_issue.v2.json }
retrieval: { index: kb-2026-10-01, top_k: 6, reranker: v2 }
policy: policy/action_policy.v9.yaml
evals:
  suite: evals/support@v31
  required: { task_success: ">= 0.88", policy_violations: 0, cost_per_success: "<= 0.12" }

The release pipeline

• 1. Make the change in version control, with review. Review prompt diffs like code, and put the reason in the description.
• 2. An automatic eval gate. The regression suite must meet the targets for quality, safety and cost.
• 3. A staging setup with realistic data and injected faults.
• 4. Shadow mode. Run the new release on live inputs without showing its output, and compare it with production.
• 5. A canary. Send 1 to 5 percent of traffic to it, with automatic guardrail checks and a waiting period.
• 6. A step-by-step ramp: 5, then 25, then 50, then 100 percent, pausing by itself if quality drops.
• 7. Watch it after release. Increase monitoring and name an owner for the first day.

def canary_verdict(canary, control, min_samples=300):
    if canary.n < min_samples:
        return "keep_waiting"
    if canary.policy_violations > 0:
        return "rollback"                                  # zero-tolerance metrics first
    if canary.task_success < control.task_success - 0.03:
        return "rollback"
    if canary.escalation_rate > control.escalation_rate * 1.5:
        return "rollback"
    if canary.cost_per_success > control.cost_per_success * 1.3:
        return "hold_and_investigate"
    if canary.task_success >= control.task_success - 0.005 and canary.p95_latency <= control.p95_latency * 1.2:
        return "ramp_up"
    return "hold_and_investigate"

Rollback must be boring

A release is a fixed bundle. So rollback means pointing traffic at the previous release ID. That is a change of settings, not a new deploy. Practice it. Keep the previous release's resources alive, such as the old index and the old tool schemas, until the new one has proved itself. And make sure that state written by the new release can be read by the old one, or that the migration can be reversed.

Feature flags for behavior

• Put new tools, new levels of autonomy and new models behind flags that you can turn off at once, without a deploy.
• Roll out by customer group: internal users first, then friendly customers, then everyone.
• Use overrides for each customer who has special compliance needs.
• Keep one global kill switch for each agent.

The hidden changes

• Tool schema changes: renaming a field or tightening a check can break prompts that mention the old shape. Version tools, and keep the old form working for a while.
• Updates to the knowledge base and index: search quality changes when documents change. Version and test the index too.
• Changes by the provider: quiet updates or retirements. Pin snapshots, and run scheduled evals that would catch drift.
• Changes to memory and state formats: changing a stored format can strand runs in progress. Migrate or version them.
• Policy changes: rules like refund limits change behavior. Treat them as releases.

Notes that save you at 2 a.m.

Keep a changelog. For each release, record what changed, why, the eval results and the known regressions you accepted. Keep the design intent of each part of the prompt next to it. A future maintainer who does not know why a strange instruction exists will delete it, and bring back an old bug.