AI Technology Literacy for Leaders

A strategist does not write code. A strategist must still understand what the technology can do, what it costs and where it breaks. Without that, you cannot judge a proposal or spot a weak claim. An interviewer will test this early. This lesson gives you the working vocabulary and the questions to ask engineers and vendors.

What a model is

A large language model is software trained on a huge amount of text. It predicts a likely next piece of text. From this one skill it can write, summarize, translate, sort and answer questions. It does not look things up unless you connect it to a source. It does not know your company's private facts unless you give them to it.

• Token: a small piece of text, a word or part of a word. Prices and limits are counted in tokens.
• Context window: how much text the model can read at once, including your question and the documents you give it.
• Hallucination: a confident answer that is wrong or invented.
• Prompt: the instructions and information you send to the model.

Four ways to make a model useful for your business

1. BETTER PROMPTS      Clear instructions and examples.
                       Cheap, fast, try this first.

2. RETRIEVAL (RAG)     The system finds your documents and gives them to the model.
                       Good for company facts that change. Answers can cite sources.

3. TOOLS AND ACTIONS   The model can call software: look up an order, create a ticket.
                       Needs strict permission checks and logs.

4. FINE-TUNING         Train a model further on your examples.
                       Costly and slower to change. Use it when the first three are not enough,
                       for a style or a narrow task that must be very consistent.

Many leaders hear fine-tuning first, because it sounds serious. Often the first three solve the problem at a lower cost. A strategist should ask, have we tried the simpler options?

Workflows and agents

A workflow is a fixed series of steps, some of which use AI. An agent lets the model decide the next step in a loop. Workflows are easier to test and control. Agents are more flexible and more risky. Most business value today comes from well-designed workflows with people checking the risky steps.

Where the cost comes from

• Model use: more text in and out costs more, and bigger models cost more per token.
• Search and storage for your documents.
• People: building, testing, reviewing and supporting.
• Change over time: models and prices change, so tests and settings need upkeep.

Ask for a cost per completed task, not only a price per token. A cheap model that fails often can cost more in rework than a stronger model that gets it right.

Questions to ask engineers

• What is the simplest design that could work? Why not simpler?
• How will we know it works? What is the test set and the pass mark?
• What happens when it is wrong? Who catches it?
• What data does it see? Who may see the answers?
• What does one task cost, and what changes that cost?

Questions to ask vendors

• Is our data used to train your models? Can we opt out in writing?
• Where is data stored and processed? For how long?
• Can we export our prompts, settings and data if we leave?
• What happens when you change the underlying model? Do you tell us first?
• Can you show results on cases like ours, not a generic demo?