LLM Application Engineering

With the foundations in place, this lesson covers building applications that call models. It shows how to think about what an LLM really is, how to design instructions and context, and how to get structured output that you can trust. The theme that keeps coming back is this: the model proposes, and your code decides.

What a large language model actually does

An LLM estimates the likely continuation of a sequence of tokens, based on patterns it learned in training. It predicts the next token is technically right, but it is not a full picture for engineers. Scale, data, design, instruction tuning, preference tuning and the context you give at run time all combine. Together they let the system do sophisticated language and reasoning tasks.

• It is probabilistic. The same input can give different outputs. Even at a low temperature, do not assume the answer is always right. Treat model output as the result of a computation you do not trust. Check it whenever correctness matters.
• Its abilities are uneven. It is excellent at summarizing. It is unreliable at arithmetic, at interpreting policy, and at your own company's workflow. Tools, search, limited outputs and checking move work from uncertain generating to dependable software.
• Choose models on quality, speed, cost, context size, how reliably they use tools, support for structured output, privacy needs, regional availability and operating limits. Do not choose on leaderboard scores alone.

Prompt engineering and context engineering

Prompt engineering is the deliberate design of instructions, examples, limits and output requirements. Context engineering is wider. It covers everything the model receives when it runs: system instructions, user input, retrieved documents, tool definitions, conversation state, memory and intermediate results. A strong prompt defines the job, the limits, the information available, the output contract and what to do on failure.

INSTRUCTION   You extract invoice data for the accounts team. Be literal; never guess.
CONTEXT       <invoice_text> ...untrusted document content... </invoice_text>
              <vendor_list> ...the approved vendors... </vendor_list>
TASK          Extract the fields below from the invoice.
OUTPUT        JSON matching the schema; use null for anything not stated.
FAILURE       If the document is not an invoice, return {"error": "not_an_invoice"}.

Pattern: instruction -> context -> task -> output contract -> validation (in code, after the model).
Keep secrets, authorization decisions and business-critical invariants OUT of the prompt.

• Few-shot examples show the behavior you want. But they use up context and can add accidental bias. Pick examples that are representative and varied.
• Long instructions can clash with user content or retrieved text. Give priority by relevance, authority, how recent it is, and clear boundaries.
• Treat retrieved content as data, not as trusted instructions.

Structured outputs and validation

You can ask a model for JSON. But valid syntax is not correct meaning. A production service checks the output against a schema, and then checks business rules in application code. For an invoice, a JSON schema checks the types and required fields. Business checks confirm that the total is not negative, that the tax is plausible, that the dates are valid and that the vendor is allowed.

from pydantic import BaseModel, ValidationError

class Invoice(BaseModel):
    invoice_number: str
    vendor: str
    total_cents: int
    tax_cents: int

def accept(raw_model_output: str, approved_vendors: set[str]) -> Invoice:
    inv = Invoice.model_validate_json(raw_model_output)       # 1) shape
    if inv.total_cents < 0 or inv.tax_cents > inv.total_cents:  # 2) business invariants
        raise ValueError("implausible amounts")
    if inv.vendor not in approved_vendors:                     # 3) policy
        raise ValueError("vendor not permitted")
    return inv

# The model PROPOSES a result. Your validator DECIDES whether it is acceptable.
# Invalid output can trigger repair, a retry, a fallback model, or human review.

This separation becomes critical for agents, because tool inputs are also made by the model. A tool schema reduces accidental misuse. It does not replace authorization and policy checks. Use fixed lists, ranges, required fields and bounded strings. And never treat inputs that pass the schema as automatically authorized.