Hands-On Labs
Five build labs with deliverables and acceptance criteria, from a validated LLM service to a production agent on AWS.
Reading is not enough, so build five projects in order: a small AI service, a RAG system, an agent with tools, a tool server and a real deployment. Each has clear 'done' checks. Write a short story of each one: the choices you made and what failed.
- Lab
- A hands-on project to practise a skill.
- Acceptance criteria
- The checks that show the work is finished.
- Case study
- A short write-up of what you built and learned.
Write a short story of each one: the choices you made and what failed.
Reading turns into skill only when you build. These five labs form a progression. Each has a deliverable, acceptance criteria and a story you can tell in an interview. Build them in order. Keep the code in a repository. Write a short case study for each one.
Do not skip the early labs. Each lab depends on the one before it. If Lab 1 is weak, every later lab inherits its problems.
Lab 1: build a minimal LLM application
The goal is a backend endpoint. It accepts a task and calls a model. It checks that the output has the right structure. It records delay, token use and cost. Then it returns a typed response.
Work through these milestones in order: a wrapper around the provider, a prompt template, a schema, validation, error handling, logging and tests. The aim is to understand the model boundary before you add search or agents.
Deliverable: API + tests + request trace Accept when: [ ] typed request/response schemas; invalid input returns a clear 4xx [ ] model output validated by schema AND one business rule; invalid output triggers one repair then a clean error [ ] every call logs request_id, model + version, latency, input/output tokens, estimated cost [ ] failure cases tested: timeout, malformed output, provider error Interview story: "why validation belongs outside the model".
Lab 2: build production RAG
Build document upload and asynchronous ingestion. Then add chunking, embeddings, vector or hybrid search, metadata filtering and reranking. Finish with answer generation and citations.
Create an evaluation set of 50 to 100 questions. Measure retrieval recall and answer groundedness. Whenever you find a retrieval bug, add a regression test so it cannot come back.
Deliverable: RAG API + ingestion worker + evaluation report Accept when: [ ] ingestion is async, idempotent, observable; a poison document does not block the queue [ ] hybrid retrieval with tenant + ACL filters applied BEFORE ranking; tenant-isolation tests pass [ ] answers cite sources and say "I don't have enough information" when unsupported [ ] eval report: recall@5 for vector vs hybrid vs hybrid+rerank; groundedness rate; failures categorized [ ] deleting a document removes it from retrieval; re-indexing a changed document is demonstrated
Lab 3: build a tool-using agent
Create an agent with four to six narrow tools. Good examples are search customer, search order, calculate amount, create ticket, update ticket and escalate.
Add schema validation and authorization. Make side effects idempotent. Set timeouts and handle tool errors. Set an explicit maximum number of steps. Add human approval for one sensitive action. Then test with malicious tool arguments.
Deliverable: agent + trace viewer Accept when: [ ] max steps, tool budget and deadline enforced; a deliberate loop is stopped [ ] malicious arguments (other customer's id, huge amount, SQL-like strings) are rejected by validation/authorization, not by the prompt [ ] side-effecting tools take idempotency keys; a replayed run does not double-act [ ] the sensitive action pauses for approval and resumes after approve/reject [ ] a trace shows every model call, tool call, result and termination reason
Lab 4: build an MCP server
Expose a small business capability through MCP-compatible tools and resources. Then build a client that finds it and calls it.
Do not stop at proving the connection works. Focus on the protocol boundaries instead. That means authentication, authorization, input validation and audit logs. Document the tool schemas. Then show what happens when the server is unavailable.
Lab 5: a production agent on AWS
Deploy an asynchronous agent workflow. Use API Gateway and Lambda or containers. Add a durable workflow layer. Use SQS or EventBridge where they help. Add stored state, model access, search and tracing.
Then add the production features. Build retries, checkpoints and cancellation. Set a cost budget. Require human approval for one high-impact action.
Deliverable: architecture diagram + deployed service + runbook Accept when: [ ] request returns a run_id immediately; status endpoint and events work [ ] a worker killed mid-run resumes from the last checkpoint (demonstrate it) [ ] cancelling a run prevents any later side effect [ ] per-run and per-tenant cost budgets enforced; p95 latency and cost per task measured and reported [ ] a failure-recovery demonstration is written up (what broke, how it was detected, how it recovered)
- ✓The labs build on each other: the model boundary, then RAG, then tools and agents, then a protocol, then production deployment.
- ✓Every lab has a deliverable, acceptance criteria and failure cases. Treat tests and traces as part of the deliverable.
- ✓Each lab gives you an interview story about a decision and a failure.
- ✓Document trade-offs and results as case studies.
Complete Lab 1 and plan the rest
- Implement Lab 1 to its acceptance criteria. Tick every box with evidence, such as test output or a sample trace.
- Write the Lab 1 case study in under 400 words. Include one failure you hit.
- For Labs 2 to 5, write the milestone plan and the riskiest assumption in each.
- Put the five labs on your calendar with dates. Share the plan with someone who will hold you to it.
Q1Why does Lab 1 come before RAG and agents?show answer
Understanding the model boundary is the foundation. That means validation, errors, logging and cost. Adding search or agents first only hides model-layer problems inside more complexity.
Q2What separates a Lab 3 agent from a toy?show answer
Budgets, authorization and validation that are enforced outside the prompt. Side effects that are idempotent. Human approval for a sensitive action. And a trace you can read.
Q3What belongs in a lab case study?show answer
The problem, the architecture diagram, the key decisions and their trade-offs, the failures and fixes, and measured evaluation results.