AWS AI Architecture
Reference architecture, the model gateway, serverless agents and multi-tenant AI SaaS.
This shows how the ideas map onto Amazon's cloud: a front door, short functions, queues for work waiting its turn, a workflow engine for long jobs and a gateway for models. The same building blocks exist on every cloud. Never keep a short function waiting on a long task.
- Serverless
- Code that runs only when needed, without managing servers.
- Queue
- A waiting line for jobs.
- Model gateway
- One doorway through which all model calls pass.
Every cloud has the same building blocks. Never keep a short function waiting on a long task.
Cloud architecture turns AI ideas into systems you can deploy. This lesson uses AWS as the worked example. The same building blocks exist on any cloud. It covers a reference architecture, model access through a gateway, agents that run serverless and multi-customer SaaS. Check exact service limits and features in the current documentation.
AWS AI reference architecture
A typical application uses API Gateway as the entry point. It uses Lambda or ECS for application logic and Amazon Bedrock for model access. S3 holds documents. A vector or semantic store handles search. DynamoDB or RDS holds application state. SQS and EventBridge handle asynchronous workflows. Step Functions handle orchestration. CloudWatch, X-Ray or OpenTelemetry give observability. The right service depends on the workload. Lambda is great for short, event-driven tool functions. ECS or EKS suits long-running processes, custom runtimes or longer execution. Step Functions suits clear workflows and durable orchestration.
Client -> API Gateway (auth, throttling)
-> Lambda "api" (validate, create run, return run_id)
-> SQS queue ----------------------------+
v
Step Functions state machine (durable orchestration, retries, human approval wait)
|-> Lambda tool functions (short, idempotent)
|-> Model gateway -> Bedrock (policy, quotas, logging)
|-> Retrieval service -> vector store / SQL (tenant filters)
|-> DynamoDB (run state, checkpoints, audit) / S3 (artifacts)
<- EventBridge events -> status API / webhooks
Observability: CloudWatch + X-Ray/OpenTelemetry traces with run_id on every span- Map AI components to the distributed-systems building blocks you already know.
- Use serverless for event-driven, bounded work. Use containers for long-running or custom workloads.
- Keep model-provider details behind a stable application boundary.
Model access and the gateway pattern
A model gateway centralizes access, policy, logging, routing and provider settings. In a company this beats every microservice calling outside model APIs directly. A gateway can apply quotas, customer budgets, safety filters, fallback routing, request details and audit logging. It also gives you one place to change providers without rewriting every application. Do not let it become a huge synchronous bottleneck. Keep it light, cache where it is safe, and keep trace IDs across calls.
- Centralize policy and model settings.
- Avoid hard-coding provider credentials into application services.
- Use fallback models on purpose, and test how their behavior differs.
Serverless agent architecture
Serverless suits agents when execution is event-driven and tasks break into bounded steps. API Gateway receives the request. A workflow service starts an execution. Functions run the tools. Queues handle asynchronous work, and state is saved. The key challenge is that a simple agent loop does not fit a short function call when tasks run long. Use durable orchestration and checkpoints. Do not have one function wait a long time on an LLM. And do not hold compute while waiting for slow outside operations.
Multi-customer AI SaaS
A platform with many customers needs isolation across data, search, tools, budgets, settings and observability. Carry a tenant_id through every request, and enforce it at the data-access boundaries. Vector retrieval must filter by customer. Tool calls must check the customer's permissions. Logs must not leak another customer's prompts or outputs. Allocating cost is its own challenge. Track tokens, model calls, storage, search and tool use for each customer. Provide quotas and rate limits. Enterprise customers may need separate encryption keys, model policies or data residency.
- Customer isolation is a rule that cuts across everything. It is never a feature of the interface.
- Design metering and quotas before customers arrive.
- Plan overrides for each customer, such as keys, models and residency, from the start.
- ✓Match the workload to the compute: Lambda for short event-driven tools, containers for long-running work, and Step Functions for durable workflows.
- ✓A model gateway centralizes policy, quotas, logging, routing and provider changes.
- ✓Never hold a short-lived function open while it waits on a long LLM task. Checkpoint and orchestrate in a durable way.
- ✓Multi-customer systems mean tenant_id everywhere, metering and quotas for each customer, and logs that cannot leak.
Design a multi-customer RAG platform on AWS, on paper
- Draw the architecture: the ingestion path, the query path, the points of customer isolation, the state stores and the observability.
- Say where you use Lambda, ECS, SQS and Step Functions, and justify each choice in one sentence.
- Define metering: which events you count for each customer, and how quotas degrade gently.
- List five failure modes, with a guard for each: a queue backlog, a poison document, a leak across customers, a provider outage and runaway cost.
Q1When would you choose Step Functions over a design with Lambda only?show answer
When the workflow is long-running or has many steps, retries and waits, such as human approval. Also when it needs durable state and visibility, instead of one function held open while it waits.
Q2What is the benefit of a model gateway?show answer
One place for quotas, budgets, safety filters, fallback routing, audit logging and provider changes, instead of every service calling model APIs directly.
Q3How do you control cost for each customer?show answer
Meter tokens, model calls, storage, search and tool use by customer. Enforce quotas and rate limits at the gateway, with graceful degradation.