How RAG Works

A language model knows what was in its training data. It does not know your policies, your product catalogue, last week's incident report, or anything else that lives inside your organisation. Ask it about those and you get a confident answer assembled from nothing.

Retrieval augmented generation, usually shortened to RAG, is the standard fix. The idea is simple enough to state in a sentence: before answering, go and find the relevant documents, put them in front of the model, and tell it to answer from those. The complexity is entirely in the going and finding.

The shape of the system

Two pipelines, running at different times.

Ingestion happens ahead of time. Documents are loaded, split into chunks, converted into vectors, and stored in an index. This is a batch job that runs when documents change.

Query happens per request. The question comes in, gets converted into a vector, the index returns the closest chunks, and those chunks go into the prompt alongside the question. The model answers from what it was given.

Ingestion:  documents → chunks → embeddings → vector store

Query:      question → embedding → search → top chunks
                                              ↓
                              prompt = chunks + question → model → answer

Nearly every problem people have with RAG lives in the retrieval half. If the right chunk does not make it into the prompt, no amount of prompt engineering saves the answer. Debugging a bad RAG system almost always means looking at what was retrieved, not at what the model said.

Chunking

You cannot embed a hundred page document as a single unit and expect useful results. It has to be split. How you split it turns out to matter more than most teams expect.

The naive approach is a fixed character count with some overlap:

def chunk(text, size=1000, overlap=200):
    chunks = []
    start = 0
    while start < len(text):
        chunks.append(text[start:start + size])
        start += size - overlap
    return chunks

The overlap exists so a sentence spanning a boundary appears intact in at least one chunk. This works, and it is a reasonable starting point, but it cuts through headings, tables and code blocks without noticing.

Better is to split on the document's own structure. Markdown by heading. HTML by section. Code by function. A chunk that corresponds to one section of one document is far more likely to be a good retrieval result than an arbitrary thousand characters.

Two rules of thumb that hold up in practice:

• Keep related things together. A table split from its caption is worthless to both the search and the model.

• Attach context to every chunk. Store the document title, the section heading and the source alongside the text. A chunk reading "This must be completed within 30 days" means nothing on its own and means something specific under the heading it came from. Prepending that heading to the chunk text before embedding measurably improves retrieval.

Embeddings and vector search

An embedding model turns a piece of text into a list of numbers, typically several hundred or a couple of thousand of them. Texts with similar meanings end up with similar vectors. That is the whole trick, and it is what lets you search by meaning rather than by keyword.

"How do I reset my password" and "I forgot my login details" share almost no words. Their embeddings sit close together. Keyword search misses that; vector search does not.

The store holds the vectors and answers the question "which of these are nearest to this one". Options range from a Postgres extension to a dedicated database:

• pgvector if you already run Postgres. Under a few million chunks this is usually the right answer, and it keeps your metadata and your vectors in one place where you can join across them.

• Qdrant, Weaviate, Milvus as dedicated stores when scale or filtering complexity justifies the extra moving part.

• Pinecone or similar if you would rather not run it yourself.

Two practical constraints. First, the embedding model used at query time must be the same one used at ingestion, otherwise the vectors are not comparable and results become nonsense. Second, changing the embedding model means re embedding everything, so treat that choice as semi permanent.

Why pure vector search is not enough

This is the part most introductory explanations skip, and it is the difference between a demo and a system people trust.

Vector search is excellent at meaning and poor at specifics. Search for an error code, a part number, a person's surname or an exact phrase from a policy, and semantic similarity will happily return things that are about the right topic but do not contain the term you asked for.

The fix is hybrid search: run a keyword search such as BM25 alongside the vector search and combine the results. Reciprocal rank fusion is the usual method and is simple enough to write yourself:

def fuse(vector_hits, keyword_hits, k=60):
    scores = {}
    for ranking in (vector_hits, keyword_hits):
        for rank, doc_id in enumerate(ranking):
            scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank + 1)
    return sorted(scores, key=scores.get, reverse=True)

A document ranked well by either method rises. A document ranked well by both rises further. In practice hybrid search is one of the largest single improvements available to a struggling RAG system, and it costs very little to add.

Reranking

The second large improvement. Retrieval is tuned for speed across a large corpus, so it is approximate. A reranker is a slower, more accurate model that scores each candidate against the query properly.

The pattern is to retrieve broadly and rerank narrowly: fetch fifty candidates cheaply, rerank them, keep the top five for the prompt. You get the recall of a wide search with the precision of an expensive comparison, paid only on fifty items rather than the whole index.

Metadata filtering

Not every search should cover everything. Filter before or during the vector search when you can:

results = store.search(
    query_vector=embed(question),
    filter={"department": "finance", "status": "current"},
    limit=20,
)

This solves a problem that pure semantic search cannot: superseded documents. If the 2019 policy and the 2026 policy say different things, they are semantically almost identical and the search has no way to prefer the current one. A status field does what similarity cannot.

Permissions belong here too. If a user should not see a document, it must be excluded at the search, not filtered out of the answer afterwards. By the time the text reaches the model it is already too late.

Generation

Once retrieval is right, the generation step is comparatively straightforward. Assemble the chunks, state the constraint plainly, and ask for citations.

Answer the question using only the sources below.
Cite the source number for each claim, like [2].
If the sources do not contain the answer, say so. Do not use
outside knowledge.

Sources:
[1] {chunk_1}
[2] {chunk_2}
[3] {chunk_3}

Question: {question}

Two things are load bearing here. The instruction to say when the answer is absent is what prevents confident invention, and it works far better than people expect. The citation requirement is what makes the answer auditable, which is usually the difference between a system people use and one they do not trust.

Return the citations through to the interface as links back to the source documents. Users check them, and the ability to check is a large part of why they trust the tool.

Evaluation

Without evaluation you are guessing, and every change you make is a coin flip. This is the single most skipped step and the most reliable predictor of whether a RAG project succeeds.

Build a set of question and answer pairs. Thirty to fifty real questions with known correct answers and the documents those answers come from is enough to start. Then measure the two halves separately.

Retrieval quality. For each question, did the correct chunk appear in the results, and how far down? This is the metric to fix first, because generation cannot recover from a retrieval failure.

Answer quality. Given the correct chunks, was the answer right? Is every claim supported by a cited source? Did it correctly decline when the answer was not in the documents?

Track both over time. When someone suggests a change to chunking, embeddings or the prompt, you will have a number rather than an opinion.

The failure modes, and what causes them

SymptomUsual causeAnswers are vague and genericRetrieved chunks are topically related but do not contain the specific answer. Look at chunk size and try reranking.Exact terms and codes are not foundNo keyword search. Add hybrid retrieval.Confidently wrong answersThe prompt does not forbid outside knowledge, or the retrieval returned nothing and the model filled the gap anyway.Contradictory answers on the same questionSuperseded documents in the index. Filter by status or date.Correct but incomplete answersThe answer spans multiple documents and only one was retrieved. Increase the number of chunks retrieved and rerank.Slow responsesUsually the reranker or an oversized prompt. Measure each stage before optimising.

Keeping the index current

Documents change. An index that reflects last quarter is a liability, because the answers look just as confident while being wrong.

Store a content hash with each document and re embed only what changed. Delete chunks belonging to removed documents rather than leaving them orphaned. Run this on a schedule, or trigger it from whatever system owns the documents, and monitor it the way you would monitor any other job. A silently failing sync is worse than no sync, because nobody notices.

What to build first

Start simple and add complexity only where the evaluation tells you to. Fixed chunking, a single vector store, a plain prompt with citations, and an evaluation set. Get that working end to end and measure it.

Then add hybrid search, then reranking, then structural chunking. Each of those is a real improvement, and each is easier to justify when you can see what it did to the numbers.

If you are building a RAG system over your own documents, or you have one that returns disappointing answers and you want to know why, talk to Eight Mile. AI assistants, RAG systems, backend APIs and system architecture are the work we take on.