How LLMs Actually Work

A chat app hides a few details from you. The moment you build an agent, those details become your job. An agent is a program that calls the model in a loop and lets it use tools. Four things matter most: tokens, the context window, messages and sampling.

Tokens: the pieces a model reads

A model does not read letters or whole words. It reads tokens. A token is a small chunk of text. Common words are often one token. Rare words are split into pieces.

"unbelievable"      -> ["un", "believ", "able"]      (3 tokens)
"the"                -> ["the"]                        (1 token)
"gpt-5-mini"         -> ["gpt", "-", "5", "-", "mini"]  (5 tokens)
"🚀"                  -> ["🚀"] or multiple bytes         (varies by tokenizer)

Why this matters: everything you send is counted in tokens. That includes the tool descriptions, every earlier message and every tool result. As a rough rule for English, one token is about four characters, or three quarters of a word. Code, JSON and other languages usually take more tokens than plain English.

Context window: one shared budget

The context window is how much text the model can look at in one request. Your input and the model's answer both count. Today's big models can handle from about a hundred thousand tokens to over a million. That sounds like a lot. An agent fills it faster than you expect.

Here is what shares the space:

• The system prompt, which holds your agent's instructions
• The name, description and fields of every tool
• The whole conversation so far, including every earlier tool call and result
• Any documents you add for RAG
• The model's own answer for this turn

This is the trap that catches most beginners. A simple agent adds every message to the history and sends all of it again on each turn. So turn 10 sends the results of turns 1 to 9 again. The total grows fast. Big tool results, like a full web page, can use up the window in just a few turns.

Real agents manage this. They shrink old turns into short notes. They drop big tool results once they are used. They fetch only the history they need. You will build this later in the path. For now, remember one thing: the context is a budget. You manage it. You do not just fill it.

Messages and roles: how a chat becomes one prompt

You do not send plain text to a chat model. You send a list of messages. Each message has a role, such as system, user, assistant or tool. The provider joins them into one long sequence, using special marker tokens, in the form the model was trained on.

[
  { "role": "system", "content": "You are a support agent. Use tools to look up orders." },
  { "role": "user", "content": "Where's my order #4821?" },
  { "role": "assistant", "content": null, "tool_calls": [
      { "id": "call_1", "name": "get_order", "arguments": { "order_id": "4821" } }
  ]},
  { "role": "tool", "tool_call_id": "call_1", "content": "{ \"status\": \"shipped\", \"eta\": \"Thu\" }" },
  { "role": "assistant", "content": "Order #4821 shipped and should arrive Thursday." }
]

Look at the tool message. This is how nearly every agent works. The model does not run anything itself. It writes a request: please call this tool. Your code runs the tool. Then you send the result back as a new message with the same id. If the roles or ids are wrong, most APIs reject the request. This is a common first bug.

Sampling: how the model picks the next token

At each step the model scores every possible next token. Sampling settings decide how one gets picked.

• temperature: A low value (0) means it nearly always picks the top choice. A higher value (0.7 to 1) allows more variety.
• top_p: It picks only from the most likely tokens and ignores the unlikely ones.
• max_tokens: This is a hard limit on the answer length. If it is reached, the answer stops half way. That can cut a tool call or a JSON block in half.

Put it together. Each turn, your instructions, tool descriptions and message history are turned into tokens. They count against the window. They are arranged into the exact format the model expects. The model then picks its answer using your sampling settings. Every later lesson adds structure to this same loop.