RAG vs Fine-Tuning Explained: A Production Decision Guide

If you searched for rag vs fine tuning explained, the production question is which failure mode you can afford. RAG can turn a healthy vLLM deployment into a p99 time-to-first-token (TTFT) incident when retrieval and extra context lengthen the request path. Fine-tuning can ship a fast model that regresses the golden set, or end a PyTorch training job with a GPU OOM. Average accuracy will hide both failures.

The short answer: use RAG for current, private, or sourceable knowledge; use fine-tuning for stable behavior, terminology, output format, or task specialization. Combine them when the application needs both. Start with neither if a prompt plus ordinary context already passes the eval set.

RAG and fine-tuning change different things

RAG changes what the model can see at inference. A retriever searches a corpus and inserts selected chunks into the prompt. The original RAG paper describes this as combining parametric model memory with a non-parametric dense index. Updating documents and embeddings can change available knowledge without replacing the generator weights.

Fine-tuning changes learned parameters by continuing training on a smaller task or domain dataset, as the Hugging Face training documentation explains. Full fine-tuning updates the model weights. LoRA instead freezes the base and learns low-rank updates, following the LoRA paper; Hugging Face PEFT supplies the practical LoraConfig, adapter loading, and switching path in its official quicktour.

Operating concern	RAG	Fine-tuning
Best fit	Changing facts, private documents, citations	Stable behavior, style, schema, classification
Update path	Re-chunk, embed, and update the index	Train, evaluate, register, and deploy weights or an adapter
Online path	Embed, retrieve, possibly rerank, then generate	Generate with the base or adapted model
Primary failure	Low recall@k, stale chunks, irrelevant context	Overfit, label leakage, general-capability regression
Main cost	Vector-index QPS and longer prompt prefill	Curated examples, GPU training, model or adapter lifecycle

These are not competing religions. The original RAG system fine-tuned its retriever and generator. A common production shape is RAG for knowledge plus LoRA for response format or domain behavior.

The metric that matters

Use SLO-qualified golden-set pass rate:

qualified pass rate = count(golden pass AND end-to-end latency <= SLO)
                      / count(all evaluated requests)

Run the same golden set and offered QPS against base, rag, lora, and rag_lora shadow deployments. This beats task accuracy alone because a correct response after the latency SLO is still a failed request. It beats p99 latency alone because fast wrong answers are not useful.

Keep component metrics beside it. For RAG, track recall@k or MRR, groundedness, retrieval p95/p99, TTFT, and end-to-end p99. For fine-tuning, track task pass rate, format compliance, and regression slices unrelated to the training task. MLflow’s RAG evaluation documentation separates retrieval relevance, groundedness, and sufficiency for exactly this reason.

Wiring it up

vLLM already exposes serving data at /metrics, including token and scheduler measurements in its production metrics documentation. Add low-cardinality comparison metrics from the eval runner:

from prometheus_client import Counter, Histogram, start_http_server

VARIANTS = {"base", "rag", "lora", "rag_lora"}
BUCKETS = (0.1, 0.25, 0.5, 1.0, 2.0, 5.0, 10.0, 30.0)

E2E = Histogram("llm_eval_e2e_seconds", "Eval request latency", ["variant"], buckets=BUCKETS)
TTFT = Histogram("llm_eval_ttft_seconds", "Eval TTFT", ["variant"], buckets=BUCKETS)
RETRIEVAL = Histogram("rag_retrieval_seconds", "Retriever latency", ["variant"], buckets=BUCKETS)
RESULTS = Counter("llm_eval_requests_total", "SLO-qualified eval outcomes", ["variant", "outcome"])

def record_eval(*, variant: str, passed: bool, e2e_s: float,
                ttft_s: float, slo_s: float,
                retrieval_s: float | None = None) -> None:
    if variant not in VARIANTS:
        raise ValueError(f"unknown variant: {variant}")
    E2E.labels(variant).observe(e2e_s)
    TTFT.labels(variant).observe(ttft_s)
    if retrieval_s is not None:
        RETRIEVAL.labels(variant).observe(retrieval_s)
    outcome = "qualified" if passed and e2e_s <= slo_s else "unqualified"
    RESULTS.labels(variant, outcome).inc()

start_http_server(9108)

Graph the rate with:

sum by (variant) (rate(llm_eval_requests_total{outcome="qualified"}[15m]))
/
sum by (variant) (rate(llm_eval_requests_total[15m]))

Use histograms for p50/p95/p99 across replicas. Prometheus explains why aggregating precomputed summary quantiles is invalid in its histogram guidance. Store row-level prompts, document IDs, and judge rationales in MLflow artifacts or traces, not metric labels.

What you’ll see

A good RAG canary raises qualified pass rate while recall@k and MRR stay stable. Retrieval p99 consumes a bounded part of the SLO, and TTFT remains acceptable at target QPS. If recall drops while retrieval latency stays flat, suspect embedding, chunking, metadata-filter, or index drift. If retrieval p99, TTFT, queued requests, and KV-cache pressure rise together under load, suspect a capacity or context-length problem.

A good LoRA canary improves its task slice without moving the general golden set. Compare p99, tokens/sec, batch size, and GPU memory with the base deployment. A rising task score plus a falling retention slice usually means overfit or contaminated evaluation data, not a successful release. Promote from shadow to canary only after both quality and serving SLOs hold.

For a broader monitoring treatment of input drift, eval regressions, and production debugging, see SentryML’s MLOps coverage.

Caveats

• False alarms: A different canary query mix, cold vector index, changed corpus, or changed judge model can move the chart without concept drift. PSI, KL divergence, and KS tests flag input-distribution shifts; they do not prove concept drift. Label drift changes the outcome distribution, while concept drift changes the input-to-outcome relationship.
• Sampling cost: LLM judges add latency and spend. Run them asynchronously on a sampled stream, then anchor alerts to deterministic golden-set checks and reviewed labels.
• Cardinality: Never label Prometheus series with query_id, user, prompt, or document URL. Use an adapter label only from a bounded allowlist. Labels multiply; put high-cardinality evidence in OpenTelemetry traces.
• Label leakage: Do not train on the golden set. Keep a separate retention set and rotate adversarial cases after they become training examples.
• Security: Retrieved documents are untrusted input. RAG and fine-tuning do not remove prompt-injection risk, according to OWASP’s LLM01 guidance. Threat-model poisoned chunks and indirect instructions; AISec’s prompt-injection coverage is a useful next stop.