23 min

Tool Calling Fundamentals

Defining tools, handling the request/response loop, and common failure modes.

Lesson 3 of 9

0 done

In plain English

AI cannot do things by itself. It asks your code to run a tool, like looking up an order, and your code does it and gives back the result. Always set a limit on how many steps it can take, check every tool result, and retry only the errors that can fix themselves.

Tool: A function the AI can ask your program to run.
Agent loop: AI thinks, uses a tool, reads the result, and repeats.
Backoff: Waiting a little longer between each retry.

Picture it

The tool-calling loop

AI asks for a tool

For example: look up an order

Your code runs the tool

The AI cannot do this itself

The result goes back

Check it first

AI decides the next step

Stop at the step limit

Then it goes back to step 1 and repeats

A model cannot do anything by itself. It can only ask for things. Your code does the work. This lesson turns that idea into a loop that survives real life: bad answers, slow tools and retries that make things worse. Most agent failures come from how tools are used, not from the model being dumb.

The loop

agent-loop.ts

let messages = [systemMsg, userMsg];

for (let step = 0; step < MAX_STEPS; step++) {
  const res = await model.call({ messages, tools });
  if (!res.tool_calls) return res.content;           // done

  for (const call of res.tool_calls) {
    const result = await runToolSafely(call);         // never throws — see below
    messages.push(res, { role: "tool", tool_call_id: call.id, content: result });
  }
}
throw new Error("hit MAX_STEPS without finishing");   // hard ceiling, not optional

⚠MAX_STEPS is not optional. Without a hard limit, an agent can repeat the same failing call until it runs out of money or time.

Four ways tool use fails

Tool skipping: the model answers from its own knowledge, which may be old or wrong, instead of calling the tool that has the real answer.
Result ignoring: the tool returns the right result, and the model's next message contradicts it anyway.
Making things up: the model writes a believable result without ever calling the tool.
Needless tool use: the model calls a tool when it already had the answer. That wastes time, money and context.

The sneakiest failure is a silent one. A tool says OK but returns an empty or cut-off answer. If your code does not check, the agent thinks it worked and reasons from missing data. One bad step then spoils every step after it. Errors in a loop spread. They do not stay where they started.

runToolSafely — validate before you trust

async function runToolSafely(call: ToolCall): Promise<string> {
  try {
    const raw = await tools[call.name].run(call.arguments);
    if (!raw || (typeof raw === "object" && Object.keys(raw).length === 0)) {
      return JSON.stringify({ error: "empty_result", tool: call.name });
    }
    return JSON.stringify(raw);
  } catch (err) {
    return JSON.stringify({ error: classify(err), tool: call.name });
    // return the error AS a tool result — let the model see and react to it,
    // don't throw and silently drop the step
  }
}

Retries that do not make things worse

Retrying everything straight away is dangerous. Say a service answers: too many requests. If every agent retries at the same moment, they all hit the service together, and it gets worse. This is called a retry storm. The fix is to wait longer after each failure, and to add a small random delay, called jitter, so the agents do not all retry at once.

First sort the error. Short-lived errors, like a timeout or a busy server, can be retried. Permanent errors, like a bad request or a failed login, cannot.
Retry short-lived errors with longer and longer waits, plus jitter. Stop after a few tries.
Never retry a permanent error. Show it to the calling code or to the user.
Limit the total tool calls in a turn, not just per tool. Five tools called ten times each cost as much as one tool called fifty times.

→Try it: take the loop above and make one tool return an empty or cut-off answer on purpose. Check that your code catches it before the model sees it as a success.

Going deeper: designing tools the model can use

When a model uses a tool badly, the tool's design is usually the problem. The model knows only the tool's name, its description and its fields. That text is its whole manual. Write it as you would for a smart colleague who cannot ask questions.

Name tools with a verb and a noun, and make them clearly different. search_orders and find_orders will be confused.
In the description, say what the tool does, when to use it, when NOT to use it and what it returns.
Use flat, typed fields. Use a fixed list of choices where possible. Avoid free-form nested objects.
Return small, clear results. Cut big results short and add a hint for the next page. Do not dump 50 KB into the context.
Make errors helpful. For example: order_not_found, check the order id format ORD-12345. Then the model can fix its own mistake.
Make tools that change things safe to repeat, or give them an idempotency key, so a retry cannot do it twice.
Prefer a few strong tools to dozens of near copies. A long menu confuses the model and uses up context.

tool-description-before-after.txt

# BEFORE (vague - the model will guess)
name: lookup
description: Looks things up.
parameters: { q: string }

# AFTER (clear contract)
name: search_orders
description: >
  Find a customer's orders by email or order number. Use this FIRST whenever the customer
  mentions an order, delivery or refund. Do NOT use it for product questions (use search_docs).
  Returns up to 5 orders: id, status, total_cents, placed_at. Use page=2 for more.
parameters:
  email:        { type: string, description: "Customer email. Optional if order_id given." }
  order_id:     { type: string, pattern: "^ORD-[0-9]{5}$", description: "Exact order id, e.g. ORD-12345" }
  page:         { type: integer, default: 1 }

A complete small agent loop

agent_loop.py (provider-agnostic shape)

TOOLS = {}                                   # name -> (schema, function)

def tool(schema):
    def wrap(fn):
        TOOLS[schema["name"]] = (schema, fn)
        return fn
    return wrap

def run_agent(user_message: str, max_steps: int = 8, max_tool_calls: int = 12) -> str:
    messages = [{"role": "system", "content": SYSTEM}, {"role": "user", "content": user_message}]
    tool_calls_used = 0
    for step in range(max_steps):
        reply = llm.generate(messages, tools=[s for s, _ in TOOLS.values()], temperature=0.1)
        messages.append(reply.as_message())
        if not reply.tool_calls:
            return reply.text                                    # finished
        # models may request several independent calls at once - run them concurrently
        results = run_parallel(reply.tool_calls)
        for call, result in zip(reply.tool_calls, results):
            tool_calls_used += 1
            if tool_calls_used > max_tool_calls:
                return "I couldn't finish within my tool budget. Here's what I found so far: ..."
            messages.append({"role": "tool", "tool_call_id": call.id, "content": result})
    return "I wasn't able to finish in the allowed number of steps."

def run_safe(call):
    schema, fn = TOOLS.get(call.name, (None, None))
    if fn is None:
        return json.dumps({"error": "unknown_tool", "available": list(TOOLS)})
    try:
        args = validate(schema, call.arguments)               # reject bad args BEFORE running
        out = fn(**args)
        return json.dumps(out)[:8000]                         # bound result size
    except ValidationError as e:
        return json.dumps({"error": "invalid_arguments", "detail": str(e)})
    except Exception as e:
        return json.dumps({"error": type(e).__name__, "detail": str(e)[:300]})

Key takeaways

✓Every agent loop needs a hard limit on steps and a limit on total tool calls per turn.
✓Four failures to expect: skipping a tool, ignoring a result, making up a result, and needless tool use.
✓Check tool results for empty or cut-off answers before the model sees them as successes. Return errors as tool results.
✓Retry only short-lived errors, with longer waits and jitter. Never retry permanent ones.

Hands-on exercise

Make a tool fail on purpose

Build the loop from this lesson with two tools. Add a switch that makes one tool sometimes return nothing, time out or return cut-off JSON.
Run 20 tasks with the switch on. Count how often the agent confidently answers from missing data.
Add checks like runToolSafely, plus a way to tell short-lived errors from permanent ones, with growing waits and jitter.
Run the same 20 tasks again and compare how many silent failures remain.

Check yourself

Q1A tool returns OK with an empty body. Why is that dangerous?show answer

Without a check, the agent treats it as a success and reasons from missing data. The mistake then spreads through every step that depends on it.

Q2Why add jitter to the waiting time between retries?show answer

Without randomness, every client retries at the same moment and hits the service together, which makes the problem worse.

Q3Why limit tool calls per turn and not only per tool?show answer

Five tools called ten times each cost as much as one tool called fifty times. The limit has to cover the total.

← Prompting & Structured Outputs for Engineers RAG: Retrieval, Embeddings & Vector Databases →