RAG: from documents to grounded answers

I recently studied retrieval-augmented generation, or RAG. The basic idea is easy to describe: retrieve relevant information before asking a language model to answer. What made it interesting to me was everything that happens between those two steps.

How do we split a document without losing its meaning? How does a question find the right piece of text? And once the model produces an answer, how do we know which source supports it?

This is my current understanding of that pipeline, from the original document to an answer with traceable sources.

What RAG is, and why it is used

A language model does not automatically know what is inside a private document collection. Its training also does not guarantee knowledge of recent changes, and it can produce a confident answer even when it lacks the information needed.

RAG gives the model relevant external information at answer time. Instead of relying only on what the model learned during training, the application retrieves material from a knowledge base and includes it in the model's context alongside the question.

That makes RAG useful for things like internal documentation, product manuals, support articles, and research collections. When a document changes, we can update the retrieval index without retraining the language model just to make that information available.

The distinction I found useful is this: RAG changes the context given to the model, not the model's weights. The model still generates the answer; retrieval supplies evidence it can use.

This does not eliminate hallucinations. Retrieval can miss the right passage, the source itself can be wrong, and the model can misread what it receives. RAG provides a way to ground an answer, not a guarantee that the answer is correct.

The pipeline has two parts

I find it easier to reason about RAG as two connected workflows.

Preparing the knowledge base:

1. Load and extract text from the source documents.
2. Split the text into chunks.
3. Create an embedding for each chunk.
4. Store the embeddings, text, and source metadata in a vector database.

Answering a question:

1. Create an embedding for the user's query.
2. Retrieve relevant chunks from the database.
3. Give the question and retrieved text to the language model.
4. Generate an answer and connect its claims to the original sources.

A hypothetical example is a question about rotating an API key. The system should find the relevant section of a service handbook, supply those instructions to the model, and let the reader open the exact section behind the answer.

The details of each stage determine whether that actually works.

Pipeline diagram

Expand image

Open the diagram at full size

Chunking: deciding what gets retrieved

A chunk is the unit of text we embed and retrieve. Sending an entire document is not always practical: embedding models have input limits, language models have context limits, and a large document may contain mostly irrelevant material for a particular question.

But splitting text too aggressively creates a different problem. A sentence such as "This requires administrator access" is not very useful if the instruction it refers to is in another chunk.

The goal is to make chunks focused enough to retrieve precisely, while keeping enough context to understand them.

Fixed-size chunking

This approach splits text into units of a chosen size, measured in characters or tokens. For example, a token-based splitter might target 400 tokens per chunk. The unit matters: 400 characters and 400 tokens are not equivalent.

Fixed-size chunking is simple and predictable. It is a reasonable baseline, but it does not necessarily respect the structure of the document. A boundary can fall in the middle of an explanation, separate a heading from its content, or divide a numbered procedure.

Fixed-size chunking with overlap

Overlap repeats part of one chunk in the next. With a target size of 400 tokens and an overlap of 80 tokens, consecutive chunks share some context around their boundary.

This reduces the chance that a useful connection disappears just because it falls across a split. It does not solve every boundary problem, though, and it introduces duplication: more text is embedded and stored, and near-identical chunks may appear together in retrieval results.

I see overlap as a tradeoff, not a setting to maximize. Too much of it can fill the model's context with repeated evidence instead of additional useful information.

Paragraph or content-based chunking

Rather than cutting only by length, this approach follows natural structure: paragraphs, headings, sections, or other content boundaries.

For a handbook, keeping a heading with the paragraphs beneath it may preserve meaning better than a strict character boundary. Lists and tables also need care because individual fragments can lose their relationship to the surrounding text.

The challenge is that natural sections vary in size. A short paragraph may not contain enough context, while a long section may exceed the model's input limit. In practice, structure-aware splitting can still use size limits, merging small sections or splitting large ones where necessary.

Agentic chunking

The term "agentic chunking" is used for different implementations. The version I studied uses a language model or agent to decide which statements belong together, rather than relying only on fixed boundaries.

For example, it might group related statements about API-key rotation into a coherent chunk even when the document's formatting is inconsistent. Some approaches also generate a title or summary to describe each group.

That flexibility comes with additional cost, latency, and less predictable decisions. If generated summaries are used, they also need checking: a generated description should not silently replace the original evidence. The original text and its source mapping still matter.

There is no universally best strategy. I would compare them using representative questions and check whether the retrieved chunks contain complete, useful evidence—not just whether the chunks look tidy.

Embeddings: making text searchable by similarity

Once the chunks are ready, an embedding model converts each one into a vector: a list of numbers representing aspects of its meaning in a learned space.

At query time, the question is embedded using a compatible model and the appropriate query settings. The database then compares that query vector with the stored vectors using a distance or similarity metric.

This is why a question such as "How do I replace a leaked API key?" can retrieve a passage titled "Credential rotation," even when the wording differs.

An embedding is not a reversible encoding of the original text. We cannot reliably reconstruct the source document from its vector. We need to keep the chunk text and source information separately.

The embedding model is also part of the index's design. Vectors from unrelated models are generally not comparable, even if they have the same number of dimensions. Changing the model usually means re-embedding the documents.

Storing chunks in ChromaDB

ChromaDB is one vector database I studied for this workflow. A stored record can associate a chunk ID with an embedding, the original chunk text, and metadata.

Useful metadata includes:

• A stable document ID and chunk ID.
• The source title and URL or file path.
• A page number, section heading, or text offset.
• The document version or update timestamp.

For the API-key example, storing only the embedding might help identify a similar vector, but it would not provide the actual instructions or tell us where they came from.

That is why I now think of indexing as more than storing vectors. It also establishes the connection between searchable representations and readable evidence. When documents are updated or deleted, their indexed chunks need to be updated or removed too.

Retrieval with ANN and HNSW

A straightforward nearest-neighbor search compares the query vector with every stored vector. That can work for small collections, but its cost grows with the number of vectors.

Approximate nearest-neighbor search, or ANN, trades some retrieval accuracy for speed. It tries to find close matches without exhaustively comparing every vector. "Approximate" matters: the results are not guaranteed to be the exact nearest neighbors.

HNSW stands for Hierarchical Navigable Small World. It is a graph-based ANN algorithm that Chroma can use; the exact index and configuration depend on the deployment.

The mental model that helped me is a layered map. The upper layers provide sparse connections for moving quickly toward a promising region. The lower layers provide more detailed connections for exploring nearby candidates. A query navigates those layers instead of scanning the full collection.

The amount of searching affects the tradeoff between latency and recall. Exploring more candidates generally improves the chance of finding close neighbors, but takes more work. The graph itself also uses memory.

There are two different questions here: did the index find close vectors, and do those chunks actually answer the user's question? Strong ANN recall does not fix poorly chosen chunk boundaries or an unsuitable embedding model.

After retrieval, an application may filter, deduplicate, or rerank candidates before selecting the final context. Increasing the number of retrieved chunks is not automatically better if it adds irrelevant or repetitive text.

From retrieved chunks to an answer

The retrieved text is placed in the prompt with the question. It should be clearly separated from the application's instructions and labelled with identifiers that can be used for citations.

The model can then be asked to answer using the supplied evidence, cite supporting passages, and say when the available information is insufficient. Those instructions help, but they do not enforce correctness by themselves.

Retrieved documents also need to be treated as data, not trusted instructions. A passage that tells the model to ignore its rules should not gain authority just because it came back from a vector search.

Tracing the answer back to its source

Source tracing starts during ingestion, not after the answer is generated. If the source location is discarded when creating chunks, adding a trustworthy citation later becomes much harder.

The chain I want to preserve is:

Answer claim → retrieved chunk ID → document ID and version → original passage.

For the handbook example, a claim about administrator access should point to the specific passage that states that requirement—not simply to the handbook's home page.

The application should resolve citation identifiers against the chunks it actually retrieved. A citation to a nonexistent chunk should be rejected. Even a valid citation still needs a second check: does that passage support the claim being made?

This is a distinction I do not want to lose: traceability tells us where evidence came from; it does not prove that the answer interpreted that evidence correctly.

Keeping the query, retrieved chunk IDs, source versions, and answer together can also help debug failures. If an answer is wrong, we can investigate whether the problem was retrieval, outdated content, missing context, or generation. Any such logs need appropriate privacy and access controls.

What I took away

My main takeaway is that RAG is not just a language model connected to a vector database. It is a sequence of decisions about what evidence to preserve, how to find it, and how to make its use inspectable.

Chunking decides what can be retrieved. Embeddings make similarity search possible. The vector index makes that search practical at scale. Metadata keeps the retrieved text connected to its source. The language model turns the selected context into an answer.

As I explore this further, I want to test those stages separately: whether the right passage is retrieved, whether it contains enough context, whether the answer stays within that evidence, and whether its citations really support its claims.

A fluent response is the visible result. The evidence behind it is what makes the system worth trusting.