Production AI Engineering

A demo proves that the happy path works. Production engineering proves that the unsafe, expensive, unavailable, incorrect and hostile paths are under control. This lesson covers observability, reliability, cost, speed, API design, managing settings and the production-readiness checklist.

Observability and tracing

Ordinary logs are not enough. You need traces that show model calls, prompts or safe prompt details, search results, tool calls, delay, tokens, errors, retries and final outcomes. Use correlation IDs all the way along the chain: user request, agent run, model call, search, tool call and outside service. Store enough to debug without logging sensitive data you do not need. Dashboards should show task success, delay percentiles, model error rate, tool error rate, token use, cost and escalation. For agents they should also show steps per task and the reasons runs ended.

• Trace the whole execution graph of the agent.
• Remove secrets and sensitive content from telemetry.
• Make cost and reliability visible for each workflow.

Reliability, retries and recovery

AI systems fail in particular ways. The provider times out. Rate limits hit. Outputs are malformed. Tools fail. Search fails. The context overflows. Model quality gets worse. Retry selectively. Retry short-lived infrastructure errors with longer waits and jitter. Do not blindly retry invalid tool inputs forever. Use circuit breakers and fallbacks for dependencies that are struggling. Long-running agents need checkpoints and the ability to resume. If a process fails after five good steps, resume from a saved checkpoint. Do not start from zero.

TRANSIENT   timeout, 429, 503            -> retry with backoff + jitter (bounded), then fall back or fail clearly
PERMANENT   400, auth failure, bad schema -> do NOT retry; surface to caller / fix the prompt or schema
POLICY      denied by authorization       -> never retry; explain, or escalate to a human
QUALITY     valid but wrong output        -> repair once with the validator's errors, then escalate; fix the root cause

Cost and token economics

AI cost comes from model prices, input and output tokens, search, running tools, infrastructure and retries. Agent loops multiply cost, because one user request may trigger many model calls. Measure the cost for each successful business task. A cheap model that often fails can cost more than a stronger one that succeeds the first time. Common savings are caching, shorter context, better search, smaller models for simple tasks, batching, parallel work where it suits, and limiting agent rounds. Cost controls belong in the architecture. They do not belong in a finance report after launch.

• Set budgets for each request and each workflow.
• Track tokens by customer and by feature.
• Optimize the cost of a successful task, not token count alone.

Delay and performance

Delay has many parts: network, model queueing, inference, search, tool calls, serialization and orchestration. Streaming improves how fast it feels, but it does not reduce the total work. Running search and tool calls in parallel cuts waiting time when the operations are independent. Caching removes repeated work. Set a delay budget for each workflow. A real-time chat assistant needs a different design from a background report agent. Measure p50, p95 and p99, and move long-running work to background jobs.

API design, Git and CI/CD for AI

AI APIs should offer clear operations, not one endpoint that accepts any instruction. Useful fields include a request ID, customer ID, user context, task type, input, settings and response details. Long tasks return a run ID and a status endpoint. Version your contracts. And remember that a change of model or prompt can change behavior even when the API schema stays the same. So record settings and version details internally, for tracing.

Prompts, tool schemas, evaluation datasets, model settings and policies are code. They belong in version control. CI should run fixed tests, schema checks, security checks and chosen AI evaluations. A model upgrade should not ship just because the application builds. Use staged rollouts and canaries. Compare quality, cost, delay and failure rates before the full rollout.

Production readiness checklist

• Identity, authorization and customer isolation are checked.
• Defenses against prompt injection and limits on tools are in place. Schemas are checked.
• Rate limits, budgets, timeouts, retries and cancellation are built.
• Audit logs, tracing and dashboards with alerts exist.
• An evaluation baseline and regression tests exist. Model and version tracking is in place.
• Incident response, data retention and a tested rollback plan are written down.