RAG & Knowledge Systems

Retrieval-augmented generation (RAG) is the most widely used AI design, and also the most widely misunderstood. This lesson treats RAG as what it really is: a search system plus a generation system. Each must be built and tested on its own. And security must be enforced when the search happens.

The pipeline

INGEST: documents -> normalize -> chunk -> embed -> index
QUERY:  question -> (rewrite) -> retrieve -> rerank/filter -> build context -> generate -> cite

Retrieve  = hybrid (keyword + vector) search with metadata/ACL filters
Rerank    = cheap first stage returns ~50, a better model scores them, keep the top 5-10
Generate  = answer ONLY from supplied context, with citations, and say "I don't know" when unsupported

RAG is useful when knowledge is outside the model, private, changing or too large for a prompt, and when you need answers grounded in sources. It does not cure made-up answers. If the search returns irrelevant or contradictory content, the model can still give a poor answer. The key insight is that RAG is a search system plus a generation system. So test them independently. Search measures such as recall@k and precision@k show whether the right evidence was found. Testing the generation shows whether the answer used it correctly.

Ingestion and chunking

Chunking splits documents into units that can be retrieved. Chunks that are too small separate important context. Chunks that are too large make search less precise and use up context. There is no universal size. A technical manual, a contract, a source file and a product catalog each need a different approach. Chunking that follows the structure usually beats cutting at arbitrary character counts. It keeps headings, section names, page numbers, web addresses, timestamps, customer IDs, document IDs and access-control details. Tables and code can lose their meaning as plain text. So keep table boundaries and code symbols intact, or extract structured forms.

• Compare fixed-size, recursive, semantic, structure-aware and parent-child chunking on your own questions.
• Store the metadata you need for filtering and citations with every chunk.
• Make ingestion asynchronous. Uploading a file must not block a request. Use a queue or workflow engine, record the ingestion state, make jobs safe to repeat and show failures so they can be reprocessed.
• Keep the original documents, which never change, apart from the chunks and embeddings made from them. Then a changed document can be re-indexed without losing history.

Vector search, hybrid search and reranking

Vector search finds content with a similar meaning. Keyword search is strong for exact identifiers, product codes, error messages and names. Hybrid search combines both. Reranking adds a second stage. A cheap retriever returns around 50 candidates. Then a more careful model scores them, so only the best 5 to 10 reach the model. So a production search stack is not just putting text in a vector database. It is an information-retrieval system. It has indexing, filtering, ranking, freshness, deletion, access control, monitoring and testing.

def recall_at_k(golden, retrieve, k=5):
    hits = 0
    for case in golden:   # {"question": ..., "relevant_ids": {...}}
        got = {c.id for c in retrieve(case["question"], k=k, tenant=case["tenant"])}
        hits += bool(got & case["relevant_ids"])
    return hits / len(golden)

# Build 50-100 labelled questions, include tenant-isolation cases, and re-run on every retrieval change.

Data modeling, and choosing search by data type

AI systems still need ordinary data modeling. Conversations, runs, tools, documents, chunks, embeddings, permissions, evaluations and audit events deserve clear identifiers and relationships. Prompts, tools, models and search settings should be versioned. An agent run gets a lasting run ID, a customer, a user, a status, timestamps, a model configuration, a parent run, a current state and a reason for ending. Tool calls get their own IDs, linked to the run.

And vector search is not the only way to retrieve. Knowledge graphs show entities and relationships directly. That suits questions about ownership, dependencies and multi-step links. A structured question such as what were sales last month should go to a database with SQL. It should not go to a semantic search over sales documents. A mature system picks the retrieval by data type. Use SQL for structured facts, keyword search for exact terms, vector search for meaning in text, graph traversal for relationships, and APIs for live operational state.

• A transactional fact: use structured SQL.
• Exact terms dominate: use keyword search.
• Matching by meaning: use vector search.
• Both exact terms and meaning matter: use hybrid retrieval.
• Relationships and multi-step links are central: use a graph.
• Do not start by choosing a vector database. Start by defining the types of question and what correct means.