Reliability Engineering for Agents

Production is where every dependency you rely on will, one day, fail. The model API returns errors in an outage. A tool times out. A rate limit trips. A provider quietly updates a model. Reliability engineering for agents uses decades of distributed-systems practice on a new kind of part. This part is also unpredictable, which makes some of the old rules harder. The goal is not zero failures. The goal is that failures are contained, visible and recoverable.

Know how things fail

• Model API: timeouts, server errors, rate limits (429), overload errors, cut-off or broken replies and regional outages.
• Tools: slow or failing APIs, schema changes, partial results and expired logins.
• Agent logic: loops, made-up tool calls that call a tool or input that does not exist, and structured output that cannot be parsed.
• Infrastructure: a worker crashes in the middle of a run, queues back up, and releases kill runs in progress.
• Silent change: a model snapshot is updated or retired, and quality shifts.

A call wrapper that copes

import random, time

RETRYABLE = {408, 409, 429, 500, 502, 503, 504, 529}   # 529: provider overloaded (varies by vendor)

def call_with_resilience(make_request, *, attempts=4, base=0.5, cap=20, timeout=60):
    for attempt in range(1, attempts + 1):
        try:
            return make_request(timeout=timeout)
        except ApiError as e:
            if e.status not in RETRYABLE or attempt == attempts:
                raise
            # honour Retry-After when the provider sends it; otherwise exp backoff + full jitter
            delay = e.retry_after or random.uniform(0, min(cap, base * 2 ** attempt))
            time.sleep(delay)
        except TimeoutError:
            if attempt == attempts: raise
            time.sleep(random.uniform(0, min(cap, base * 2 ** attempt)))

# Rules baked in: bounded attempts, jitter (no thundering herd), only retry what is retryable,
# an explicit timeout on every call. Never retry non-idempotent side effects blindly.

Idempotency: retry without acting twice

Retrying a read is safe. Retrying send the invoice is not. Give every tool that changes things an idempotency key built from the run and the step, for example run_id plus step_id. Have the system downstream ignore repeats of the same key. Then retries, resumes after crashes and even repeats started by the model cannot charge twice or email twice.

Fallbacks and graceful reduction

• A chain of fallback models: the main model, then a second model or another provider, if errors continue. Different models behave differently. So test every fallback against your eval set in advance. An untested fallback is a surprise.
• Reduced features: drop to a simpler path, such as no re-ranking of search results, shorter output or a saved answer, instead of failing.
• Circuit breakers: after N failures in a row from a dependency, stop calling it for a cool-down time. Serve the reduced path, and test now and then whether it has recovered.
• A smooth hand-off to a person: if the agent cannot go on, send the case to a person with its context, instead of showing an error.
• Honest partial results: return what worked and say what did not.

Long-running work: queues, state and recovery

• Run agent jobs on durable queues or workflow engines, not inside web requests that time out.
• Save checkpoints, so a crashed worker's job can resume somewhere else.
• Apply backpressure. Limit the queue length and how many jobs run at once, so bursts do not overwhelm provider limits.
• Drain gracefully on releases. Let runs in progress finish, or save a checkpoint, before a worker stops.
• Dead-letter queues for jobs that keep failing, with an owner who reviews them.

Model versions and retirement

In production, use pinned, dated model snapshots, not floating aliases that can change under you. Record the exact snapshot in every trace. Subscribe to your providers' retirement notices and keep a calendar of retirement dates. Treat a model move as a release. Run the full eval suite on the new model. Compare cost and delay. Adjust prompts. Try it as a canary. And keep a way back until the old model is retired. Running an unplanned move in the week a model is switched off is a classic incident that you can avoid.

Test failure on purpose

• Inject faults in a test setup: tool timeouts, 429 errors, broken model output, empty search results, and a dependency that returns believable but wrong data.
• Check that each one leads to the behavior you intended, such as retry, fallback, reduced service or escalation, and that the trace shows it.
• Run game days, where the team practices an outage using the runbooks.
• Keep a kill switch, a feature flag that turns off an agent or certain tools at once, without a release.