18 min

Prototyping on the Customer's Real Data

Build a first version fast, test it honestly and show progress every week.

Lesson 4 of 10

0 done

In plain English

A working prototype on the customer's own data beats a slide deck. Get real examples in week one. Build the smallest thing that works. Make a test set and run it after every change. Show progress every week, including failures, and add a path for when the AI is unsure.

Prototype: A first working version, built to learn.
Test set: Real cases with known good answers, used to check every change.
Unsure path: A rule that sends doubtful cases to a person.

Picture it

The weekly prototype loop

Get real examples

Hard cases too

Build a thin slice

Input to output

Run the test set

After every change

Demo to the customer

Show failures too

Customers believe what they can see. A working prototype on their own data beats a slide deck. This lesson shows how to build a first version quickly, test it honestly, and keep the customer involved each week.

Get real data early

Ask for real examples in the first week. Demos on made-up data impress nobody and hide the hard parts. Real data is messy. It has odd formats, missing fields and slang. You need to meet that mess early.

Agree privacy rules first. Hide names and numbers where possible.
Ask for a small set: 20 to 50 cases is enough to start.
Include hard and strange cases, not only the easy ones.
Keep the data in the place the customer approves. Do not copy it to your laptop without permission.

Build the smallest thing that works

Aim for a thin slice from start to finish. It should take a real input and produce a real output. Use the simplest tools that work. A prompt, a short script and a spreadsheet can be enough. Add search, agents and fancy parts only when the simple version fails.

Start with a fixed workflow. A fixed series of steps is easier to test, cheaper and safer than a free-roaming agent. Move to an agent only when the path really changes from case to case.

Build your test set on day one

Turn the real cases into a test set. For each case, write what a good answer looks like. Run the prototype on all cases after every change. This is how you know whether a change helped or hurt.

prototype-test-sheet.md

CASE ID | INPUT (short)        | GOOD ANSWER                | PROTOTYPE ANSWER | PASS? | NOTE
001     | Invoice from Acme    | Vendor=Acme, total=1,240   | Vendor=Acme, ... | yes   |
002     | Blurry scan          | Needs human review         | Guessed a total  | NO    | add "unsure" rule
003     | Email in Spanish     | Route to billing           | Route to billing | yes   |

Rules
- Run every case after every change.
- Track the pass rate in one place.
- Add every new failure as a new case.

Demo every week

Show progress weekly, even if it is small. A weekly demo catches wrong turns early. It also keeps the sponsor calm. Use the customer's own examples. Show a failure too. Say, here is a case that fails, and here is what we will do about it. Honesty builds trust.

Handle failures well

Add an unsure path: when the AI is not confident, it asks a person.
Show sources or reasons, so reviewers can check fast.
Log inputs and outputs, so you can find the cause when something goes wrong.
Fix the biggest group of failures first. Do not chase rare oddities.

→Keep a short list titled what we learned this week. It becomes your project diary, your customer update and, later, your notes for your own product team.

Know when to stop prototyping

Stop when the prototype passes the agreed mark on the test set, and users say it helps. Then move to integration. A prototype is for learning. Do not let it quietly become production without tests, access controls and monitoring.

Key takeaways

✓Use real data from week one. Demos on made-up data hide the hard parts.
✓Build a thin slice from input to output with simple tools. Prefer a fixed workflow before an agent.
✓Make a test set on day one and run it after every change.
✓Demo weekly, show failures honestly, and add an unsure path. A prototype is for learning, not for quiet production use.

Hands-on exercise

Prototype on a small set

Collect 20 real or realistic examples of a task, with private details hidden.
Write the good answer for each. Put them in a test sheet.
Build a prototype with one prompt in the AI tool of your choice. Run all 20 cases and record the pass rate.
Improve the prompt once, run again, and compare. Write down what changed and why.

Check yourself

Q1Why use real data early?show answer

Real data is messy, with odd formats and missing fields. Meeting that mess early avoids surprises, and the customer believes what they see on their own cases.

Q2Why start with a fixed workflow instead of an agent?show answer

A fixed series of steps is easier to test, cheaper and safer. Move to an agent only when the path truly changes case by case.

Q3What is an unsure path?show answer

A rule that sends a case to a person when the AI is not confident. It keeps mistakes from reaching the customer's customers.

← Scoping, Success Numbers & the One-Page Spec Integrating With the Customer's Systems, Safely →