RAG Pipeline Architecture: Components and Build Order

A retrieval-augmented generation pipeline is seven components in a fixed order, and the order matters more than the choice of any single component. Build it back to front and you will spend weeks tuning a reranker that sits on top of an extractor that destroyed the document headings. The short version: get ingestion and extraction right, freeze a golden question set before you optimise anything, then add sophistication one layer at a time and re-measure after each.

The seven components

Stage	Decides	Cheapest to fix
Ingestion and extraction	Whether structure survives	At the start
Chunking	The retrieval unit	At the start
Embedding	What “similar” means	Before indexing
Vector index	Recall and latency at scale	Before scale
Retrieval and filtering	Which candidates exist	Any time
Reranking	Which candidates reach the model	Late
Generation and evaluation	What the user sees	Continuously

Each stage inherits the mistakes of the stage above it. That is the whole argument for a build order.

1. Ingestion and extraction

Extraction is the least glamorous stage and the one that caps everything else. A PDF flattened to raw text loses heading levels, table boundaries, list nesting, and reading order in multi-column layouts. Once that information is gone, no splitter can reconstruct it and no embedding model can infer it.

Target a structured intermediate format: Markdown or HTML with the heading hierarchy intact. Assess an extractor on three things before adopting it:

• Heading fidelity. Do H2 and H3 levels come out as headings, or as bold paragraphs indistinguishable from body text?
• Table handling. Are tables emitted as tables, or as a stream of cells with the header row orphaned somewhere above?
• Reading order. Two-column PDFs and sidebars are where extractors interleave text most visibly.

Also settle the boring operational questions here, because retrofitting them later means a full re-index: a stable document ID, a content hash for change detection, a source URL, and a timestamp. Incremental re-indexing is impossible without them.

2. Chunking

Chunking converts documents into retrieval units. Retrieval returns whole chunks, so the chunk boundary is the resolution limit of the entire system.

The working defaults are 300 to 500 tokens with 10 to 15 percent overlap, split on document structure rather than character counts, with the heading path prepended into the embedded text. The reasoning behind those numbers, the per-content-type variations, and the metadata every chunk should carry are covered in detail in the RAG chunking strategy guide.

The build-order point is narrower: do not tune chunk size yet. You cannot tell whether 400 tokens beats 600 without the golden set from stage seven. Pick a defensible default, move on, and come back with a measurement.

3. Embedding

The embedding model defines the geometry the retriever searches. Three decisions matter and one of them is irreversible.

Dimensionality. OpenAI’s text-embedding-3-small returns 1,536 dimensions and text-embedding-3-large returns 3,072, both accepting up to 8,192 input tokens. The text-embedding-3 family also supports a dimensions parameter that truncates output, trading a little retrieval quality for a large reduction in index size. Check that the store can actually index the width you choose: pgvector accepts vectors up to 16,000 dimensions but will not build an HNSW index above 2,000.

Distance metric. Cosine, dot product, and Euclidean are not interchangeable. Cosine and dot product are equivalent only when vectors are normalised to unit length, which some providers do for you and some do not. Set the metric the model was trained for when you create the collection, because on most stores it cannot be changed afterwards.

Model family. Vectors from different models are not comparable, so switching models means re-embedding and re-indexing the entire corpus. That is the irreversible one. Shortlist candidates from a public leaderboard, then evaluate two or three on your own questions, because domain jargon, non-English content, and source code all reshuffle public rankings.

Estimate the cost and index footprint before you commit. The chunk calculator converts a corpus size, chunk size, overlap ratio, and embedding width into a vector count, index RAM, and one-time embedding spend.

4. Vector index

Two choices here: which database, and which index inside it.

Below roughly a million vectors, a flat brute-force index over an exact scan is fast enough and gives perfect recall, which makes it a useful reference point for measuring what an approximate index costs you. Above that, HNSW is the default approximate structure in most stores, with m and ef_construction set at build time and ef at query time. Raising query-time ef buys recall at the cost of latency without rebuilding the index, so it is the first knob to reach for.

Size the index before you build it, and do not size it as raw vector bytes. Qdrant’s capacity planning guide budgets an in-memory collection at vectors x dimensions x 4 bytes x 1.5, where the extra 50 percent covers index structures, point versions and temporary segments created during optimisation; payload storage and replication sit on top of that again. At normal embedding widths the vectors themselves still dominate that total, which is why quantisation is the lever that actually moves it: scalar quantisation to 8-bit cuts vector storage roughly fourfold with a small recall penalty, and binary quantisation goes much further at a much larger one, usually paired with a rescoring pass over full-precision vectors.

Choosing between the major stores turns on deployment model and filtering behaviour more than on raw speed. That comparison is in Qdrant vs Milvus vs Pinecone.

5. Retrieval and filtering

Retrieval is where two features earn their keep.

Metadata filtering is a correctness feature, not a convenience. A system serving several product versions that cannot filter by version will answer v2 questions from v1 documentation. Access control is the same mechanism with worse consequences. The critical detail is where the filter runs: filtering after the vector search means restricted or irrelevant chunks consume top-k slots and push out the answer, so the filter has to be applied inside the search, not over its output.

Hybrid search runs a lexical index alongside the dense one. Dense retrieval is weak on exact tokens such as error codes, part numbers, and function names, which are precisely the queries where users expect an exact hit. Reciprocal rank fusion is the usual way to merge the two result lists because it needs no score normalisation between systems, scoring each document by summing 1 / (k + rank) across lists with k = 60 as the conventional constant.

Retrieve wider than you intend to use. Fifty to a hundred candidates at this stage costs little and gives the reranker something to work with.

6. Reranking

A cross-encoder reranker scores the query and each candidate document together rather than comparing pre-computed vectors. That is far more accurate and far slower per pair, so it runs over a shortlist of tens of candidates and never over the corpus. Retrieve 50 to 100, rerank, pass the top 5 to 10 to the generator.

Reranking fixes ordering. It cannot fix a candidate set that never contained the answer, which is why it belongs at stage six and not stage two. If the correct chunk is absent from the top 100, the problem is upstream in chunking, embedding, or filtering.

7. Generation and evaluation

Two things belong in the generation stage beyond prompt assembly.

Context ordering. Long-context models attend unevenly across the window, and relevant material placed in the middle of a long prompt is used less reliably than the same material at either end. With a small reranked set this is easy: put the strongest evidence first and last rather than dumping the list in rank order.

Grounding instructions. Tell the model to answer only from the provided context and to say when the context does not contain the answer. Without that, a generator improvises over a failed retrieval and masks the failure.

Evaluation is listed last but has to be built first. Before any tuning, assemble a golden set of 50 to 200 real questions, each labelled with the chunk that answers it, sourced from support logs rather than invented by the team. Then track the retrieval stage in isolation with recall@k, nDCG@10, and MRR, and change one variable at a time.

Generation-side evaluation is a separate layer with its own frameworks. Ragas and similar tools score faithfulness, answer relevancy, and context precision and recall, which tells you whether the model used the context it was given. Keep the two layers separate: an end-to-end score cannot distinguish a retrieval failure from a prompting failure, and the fixes are unrelated.

The build order in practice

1. Fix extraction until headings and tables survive.
2. Chunk on structure with a defensible default size.
3. Embed with one model, correct metric, and a width the index supports.
4. Index flat if the corpus is small, HNSW if it is not.
5. Build the golden set and measure recall@k. Stop here until the number exists.
6. Add filtering and hybrid search, re-measure.
7. Add reranking, re-measure.
8. Only now tune chunk size, overlap, and embedding model, one at a time.

Steps 6 through 8 are cheap and reversible. Steps 1 through 4 are expensive to redo. That asymmetry is the whole reason to build in this order.

When recall stays flat after all of it, the failure is diagnostic rather than architectural, and the next step is a systematic walk down the pipeline: see RAG retrieval debugging.

See also

• RAG chunking strategy: picking chunk size and overlap
• Qdrant vs Milvus vs Pinecone: vector database comparison
• RAG retrieval debugging: why results come back wrong
• Chunk and index sizing calculator
• Production LLM telemetry and ops
• LLM observability and tracing