RAG Pipeline Diagram: Indexing and Query Paths Explained

A RAG pipeline diagram has two lanes that share one store. The offline indexing path extracts, chunks, embeds, and writes to a vector index. The online query path embeds the question, retrieves top-k with filters, reranks, assembles the prompt, and generates. Draw them as separate lanes: they fail, scale, and deploy on different schedules.

The single-arrow “documents to answer” picture works on a slide and fails during an incident, because it hides the edge that silently breaks recall: the query-side embedding model must be the exact model and version that built the index.

What does a RAG pipeline diagram show?

It shows two data flows joined at the index: a batch flow that turns documents into searchable chunks, and a per-request flow that turns a question into a grounded answer. A useful diagram adds one more edge, from answer logs back to an evaluation golden set, because that edge is how you know whether any box works.

Lewis et al. (NeurIPS 2020) drew the original as a pretrained neural retriever over a dense vector index of Wikipedia feeding a seq2seq generator. The Gao et al. survey calls the baseline “Naive RAG”: indexing, retrieval, generation. LangChain’s RAG tutorial splits indexing into Load, Split, Embed, Store and runs it separately from Retrieve and Generate.

Here is the reference diagram as a Mermaid flowchart, so it lives in the repo and changes in the same PR as the pipeline:

flowchart LR
    subgraph offline [Indexing path, batch]
        S[Source documents] --> X[Extract and parse]
        X --> C[Chunk and attach metadata]
        C --> E1[Embedding model v1]
        E1 --> V[(Vector index)]
        C --> L[(Lexical index)]
    end
    subgraph online [Query path, per request]
        Q[User query] --> RW[Rewrite or route]
        RW --> E2[Embedding model v1]
        E2 -->|query vector| R[Retrieve top-k with filters]
        RW -->|query text| R
        R --> RR[Rerank]
        RR --> P[Assemble prompt]
        P --> G[LLM generate]
        G --> A[Answer with citations]
    end
    V --> R
    L --> R
    A -.-> F[Logs and user feedback]
    F -.-> GS[(Golden eval set)]
    GS -.->|recall at k| R

Two details are worth copying. The embedding model appears twice with the same version, because it is one artifact deployed in two places. And the lexical index receives raw query text, not the vector, which is how hybrid search catches exact tokens such as error codes.

The indexing path, box by box

Extract and parse. Turns PDF, HTML, and Office files into text with headings and tables intact. Structure lost here caps every box downstream; nothing later recovers it.
Chunk and attach metadata. Sets the retrieval unit. Each chunk carries a document ID, heading path, content hash, source URL, timestamp, and any ACL or product-version fields you will filter on. Size and overlap defaults are in the RAG chunking strategy guide.
Embedding model. Batch work, bound by GPU or API throughput. Write the model name and version into the index metadata, not just the job config.
Vector and lexical indexes. HNSW or IVF for dense, BM25 for lexical. Picking the store is its own decision; see Qdrant vs Milvus vs Pinecone.

This path runs on a schedule or on change events. Its service-level objective is freshness, the time from a source edit to a searchable chunk, not request latency.

The query path, box by box

Rewrite or route. Optional. Query expansion, HyDE, or a router that sends summarisation requests away from top-k search. Gao et al. classify this as pre-retrieval work in “Advanced RAG”.
Embed query. Same model, version, and normalisation as the indexing side.
Retrieve. Dense plus lexical, merged with reciprocal rank fusion, with metadata filters applied inside the search, not over its output. Retrieve wide so the reranker has material.
Rerank. A cross-encoder over the shortlist; post-retrieval work in the survey’s terms.
Assemble prompt. Liu et al. found performance is often highest when relevant information sits at the start or end of the input and drops in the middle. Put the strongest evidence at the edges.
Generate. Grounding instructions, a citation format, and an explicit “not in context” path.

Why each component exists, and the order to build them, is in RAG pipeline architecture: components and build order.

The metric that matters: recall@k at the Retrieve box

If the diagram carries one number, put recall@k on the arrow out of Retrieve. Over a golden set Q, where each question q has a labelled answer chunk g(q):

recall@k = (1 / |Q|) * sum over q in Q of [ g(q) in top_k(q) ]

It beats the obvious alternative, an end-to-end answer quality score, because the end-to-end score cannot tell a retrieval miss from a generator that ignored good context, and those failures have unrelated fixes. Measure it twice: at the retrieve width, and again after rerank at the width you actually pass to the model. The gap between the two numbers is the reranker’s job. When recall at the retrieve width is low, the fault is upstream of rerank, and the stage-by-stage walk in RAG retrieval debugging applies.

Wiring it up: the diagram as a trace tree

Every box on the query path should be one span, named per the OpenTelemetry GenAI conventions so your tracing backend groups them without custom mapping. The GenAI spans spec lists retrieval and embeddings as well-known values of gen_ai.operation.name, with span names {gen_ai.operation.name} {gen_ai.data_source.id} and {gen_ai.operation.name} {gen_ai.request.model} respectively. The client inference spec uses the model-name pattern for chat spans and records gen_ai.usage.input_tokens and gen_ai.usage.output_tokens. Span calls follow the OpenTelemetry Python instrumentation docs.

import os

from opentelemetry import trace

# store, embedder, reranker, llm and build_prompt are your own clients.
tracer = trace.get_tracer("rag.query_path")

EMBED_MODEL = os.environ["RAG_EMBED_MODEL"]
GEN_MODEL = os.environ["RAG_GEN_MODEL"]
DATA_SOURCE = "docs-prod"
TOP_K = 50
CONTEXT_K = 8


def check_index_matches(store) -> None:
    """Run at startup: fail the deploy if the query side drifts from the index."""
    built_with = store.metadata()["embed_model"]
    if built_with != EMBED_MODEL:
        raise RuntimeError(f"index built with {built_with}, query uses {EMBED_MODEL}")


def answer(question: str, tenant_id: str) -> str:
    with tracer.start_as_current_span("rag.query") as root:
        root.set_attribute("rag.index_version", store.metadata()["index_version"])

        with tracer.start_as_current_span(f"embeddings {EMBED_MODEL}") as span:
            span.set_attribute("gen_ai.operation.name", "embeddings")
            span.set_attribute("gen_ai.request.model", EMBED_MODEL)
            qvec = embedder.embed(question)

        with tracer.start_as_current_span(f"retrieval {DATA_SOURCE}") as span:
            span.set_attribute("gen_ai.operation.name", "retrieval")
            span.set_attribute("gen_ai.data_source.id", DATA_SOURCE)
            span.set_attribute("gen_ai.retrieval.top_k", TOP_K)
            candidates = store.hybrid_search(
                vector=qvec, text=question, top_k=TOP_K,
                filter={"tenant_id": tenant_id},
            )
            span.set_attribute("rag.candidates.returned", len(candidates))

        with tracer.start_as_current_span("rag.rerank") as span:
            span.set_attribute("rag.rerank.input_count", len(candidates))
            context = reranker.rank(question, candidates)[:CONTEXT_K]

        with tracer.start_as_current_span(f"chat {GEN_MODEL}") as span:
            span.set_attribute("gen_ai.operation.name", "chat")
            span.set_attribute("gen_ai.request.model", GEN_MODEL)
            resp = llm.generate(build_prompt(question, context))
            span.set_attribute("gen_ai.usage.input_tokens", resp.input_tokens)
            span.set_attribute("gen_ai.usage.output_tokens", resp.output_tokens)

        return resp.text

Rerank has no well-known operation name, so it gets a custom rag. namespace. The startup check is the cheapest guardrail in the file: it turns the diagram’s most dangerous edge into a failed deploy instead of a slow recall regression.

What you’ll see

A healthy trace waterfall is boring: the chat span usually takes most of the wall-clock time, embeddings and retrieval are thin, and rag.candidates.returned sits at or near top_k. The bad patterns map back to specific boxes:

Retrieval returns far fewer candidates than top_k. A filter is too narrow, or new chunks are missing tenant or ACL metadata. The fix is on the indexing path.
The rerank span widens after a config change. Someone raised top_k; cross-encoder cost is per query-document pair.
Chat input tokens creep up with flat traffic. Chunks got bigger or the assembler passes more of them. Cost and time-to-first-token follow.
Recall@k drops while every span looks green. A partial reindex, with replicas serving different index versions. Group traces by rag.index_version and the split shows.

Caveats

The conventions are not stable. The GenAI spans document is marked Development. Pin the semantic conventions version you target.
Opt-in attributes carry user data. gen_ai.retrieval.query.text and gen_ai.retrieval.documents are Opt-In for good reason: they put raw queries and document content in your trace store, a PII and retention problem that also inflates span size. Redact or sample them.
Cardinality. Document IDs, query text, and tenant IDs belong in trace attributes, never in Prometheus label sets.
Modular pipelines are not a line. Routing, iterative retrieve-read loops, and adaptive retrieval (Gao et al. cite FLARE and Self-RAG) add cycles. Draw the loop and give each iteration its own retrieval span, or the trace hides how many retrievals one answer cost.

FAQ

how do i draw a rag pipeline diagram

Draw it as a Mermaid flowchart with two subgraphs, one for the batch indexing path and one for the per-request query path, joined at cylinder-shaped index nodes. Keeping it as text in the repository means it is diffed and reviewed alongside pipeline code, so it stays accurate when someone adds a reranker or router.

what is the difference between naive rag and advanced rag

Naive RAG is a straight indexing, retrieval, generation chain; advanced RAG adds boxes before and after retrieval. In the Gao et al. survey, pre-retrieval work covers index improvements and query rewriting or expansion, and post-retrieval work covers reranking and context compression. On a diagram, advanced RAG means extra nodes around the Retrieve box.

does a rag pipeline need a reranker

No, a reranker is optional, and it should be added after you can measure recall@k at the retrieve step. A cross-encoder reranker reorders a shortlist of candidates; it cannot recover an answer chunk that retrieval never returned. If the right chunk is missing from the candidates, fix chunking, embeddings, or filters first.

should indexing and querying run in the same service

No, run them as separate deployments that share only the index and the pinned embedding model version. Indexing is batch work measured by freshness and throughput; querying is per-request work measured by p95 and p99 latency. Coupling them means a large reindex competes with live traffic for the same embedding capacity and memory.