RAG vs Fine-Tuning

The cheap fix works more often than you'd guess (mostly

RAG and fine-tuning solve different problems. RAG pulls external text into the prompt at query time, so the model answers from documents you can edit or cite. Fine-tuning retrains the model’s weights until a skill, format, or tone is built in. Use RAG when the knowledge changes; use fine-tuning when the behavior is what you need to change.

🧭 Part 10 of the RAG & Search course

TL;DR

• The one-line rule: RAG changes what the model sees at answer time; fine-tuning changes how it behaves.

• Reach for RAG when the knowledge moves: facts that change, that are private, or that must be cited.

• Reach for fine-tuning when behavior is the problem: a fixed output format, your company’s voice, or a narrow domain skill.

• Prompting wins more often than people admit: a prompt costs one API call; a fine-tune costs days of data prep and a GPU bill. Test the cheap option first.

• The strongest systems do both: retrieve the facts, fine-tune the behavior.

Before Either, There Was the Prompt

Before either tool, there is just the prompt. You ask the model a question, and it answers from what training taught it. For a huge share of tasks, that is all you need, because training packed most of the public internet into its weights.

You can stretch prompting further by pasting context into the request. Put the relevant passage above your question and the model uses it on the spot. That is in-context learning: the model takes its cue from the prompt itself, with no training run and no change to its weights.

Prompting runs into two walls. The first is the context window, the fixed amount of text a model can read at once. You cannot paste a 500-page handbook into every question. Even when the text fits, padding the prompt with marginally relevant passages makes answers worse. The second wall is everything a prompt cannot fix: knowledge the model was never trained on, a format it keeps breaking, a tone it will not hold across a long answer. Hit either wall and you have two ways out, and they lead through different doors.

How Each One Works

Both techniques fix the same failure, a model answering from memory, and they differ only in where the knowledge comes from. One adds it at the moment of the question. The other builds it into the model ahead of time.

RAG Hands the Model an Open Book

RAG, retrieval-augmented generation, gives the model the open book. Before the model writes a word, a search step finds the passages the question needs and pastes them into the prompt. The model then answers from that text instead of from memory. We took the full mechanism apart in What is RAG?, and the one-line version is retrieve, then generate.

The payoff is that the knowledge lives outside the model, in an index you own. A price changed? Edit the document. A claim needs a source? The retrieved passage is the citation. None of the model’s weights move, so there is no training run.

Fine-Tuning Makes the Model Study

Fine-tuning trains the model for the closed-book exam. You take a base model and keep training it on a few hundred to a few thousand examples of the exact output you want. The weights shift until the model does this by default, with no prompt needed to trigger it. We covered the full process in What is Fine-Tuning?.

Fine-tuning can absorb knowledge too: a domain’s vocabulary, its phrasing, even facts that never change. That knowledge freezes into the checkpoint the moment training ends. The real payoff is stable behavior no prompt reliably delivers: valid JSON in your schema every time, your company’s voice, a niche domain’s phrasing.

When to Use Each

Your support bot keeps quoting last quarter’s prices. The obvious fix looks like fine-tuning: the pricing lives in a PDF, so you train on it, and a weekend of data prep and a GPU bill later the bot has the new numbers. Then sales launches a promo, three SKUs change overnight, and the bot is stale again, one more training run behind reality. Prices are a lookup. A lookup belongs in a retrieval layer you can edit in seconds; weights take a retraining run to change. That is the whole decision in miniature.

Your decision comes down to three things: whether the knowledge changes, whether you have data to train on, and what you can afford to serve.

1. Change rate: facts that move, that are private to your company, or that must be cited belong in RAG. OpenAI’s own guidance agrees: fine-tuning adjusts behavior, style, and format, and retrieval is the better tool once the model needs current or proprietary facts¹.

2. The data you have: fine-tuning needs a few hundred to a few thousand labeled examples of the output you want. Without them, RAG over your existing documents is what you can build today.

3. The runtime bill. RAG adds a retrieval hop and a search system to maintain. A fine-tuned small model skips that step entirely, so it can serve faster and cheaper once trained.

Sometimes you need both, and that’s fine. A knowledge problem and a behavior problem can show up in the same system: RAG fixes the first, fine-tuning fixes the second. RAFT, a 2024 Berkeley method, fine-tunes a model specifically to read retrieved documents well and ignore the irrelevant ones. The teams getting the most out of LLMs in production rarely pick a side.

Who’s Actually Building With This

Predibase fine-tuned more than 25 small models built on Mistral-7B, a compact open-weight model. They beat GPT-4 by 4 to 15 points on narrow tasks. Each fine-tune cost under $8 and serves from a single A100 GPU². When a task is narrow and stable, a small fine-tuned model can outrun a frontier model at a fraction of the serving cost: it only has to be good at one thing.

Microsoft researchers wrapped GPT-4 in a careful prompting strategy called Medprompt and scored 90.2% on MedQA, the US medical licensing question set, beating the fine-tuned specialist Med-PaLM 2 with no fine-tuning at all³. The team expecting to need a specialist found the prompted generalist got there first. Before you train anything, find out how far a good prompt and good retrieval already carry you.

DoorDash built its support system on retrieval, so its answers track an internal knowledge base that changes constantly. We pulled that architecture apart in How DoorDash Built Their RAG System.

What This Costs You

A sharper prompt costs one API call to test. Standing up retrieval over documents you already have costs an afternoon. A fine-tune costs real days and real money: data prep, a training run, a GPU bill, even at the cheap end. The real waste is training before confirming the cheap option would not have worked.

The tradeoff is real on both sides. RAG adds a retrieval system to build and maintain, and a weak retriever quietly caps your answer quality. Fine-tuning adds a training and evaluation loop, plus a model that drifts out of date. Neither is free, and the two costs land in different places, which is what makes the choice worth getting right.

The One Thing to Remember

RAG and fine-tuning were never rivals. Diagnosing whether your problem is about knowledge or about behavior decides the tool before you write a line of code. One keeps your system editable. The other locks a behaviour into the model.

---

💬 Ever fine-tuned a model for something retrieval should have handled, or the reverse? Tell me what the problem turned out to be in the comments. I read every one.

---

Where to Next?

• 📖 Go deeper: Unsloth vs Axolotl vs LLaMA-Factory, the hands-on fine-tuning stack once you have decided to train.

• 🔗 Prerequisite: What are Embeddings?, the representation retrieval relies on.

• 🔀 Related: What is Semantic Search?, how the retrieval step finds the passages RAG feeds the model.

🔜 Friday: How OpenAI Codex Works, the agent that writes, runs, and fixes code on its own.

FAQ

What is the difference between RAG and fine-tuning?
RAG (retrieval-augmented generation) pulls relevant text from an external index at query time and pastes it into the prompt, so the model answers from documents you control and can cite. Fine-tuning keeps training the model on examples until a behavior, format, or skill lives in its weights. In one line: RAG changes what the model sees, and fine-tuning changes how it responds.

When should I use RAG instead of fine-tuning?
Use RAG when the problem is knowledge, especially knowledge that changes often, is private to your company, or must be cited. You update them with an edit instead of a training run, since the facts live in an index outside the model. Support docs, product catalogs, and shifting policies are textbook RAG, since a single document fix corrects every future answer.

When is fine-tuning the better choice?
Fine-tuning wins when the problem is behavior rather than facts: a strict output format, a consistent tone, or a narrow domain whose phrasing the model fumbles. Predibase fine-tuned small open models that beat GPT-4 by 4 to 15 points on narrow tasks for under $8 each, so a tuned small model can outperform a frontier model on one stable, well-defined job.

Can I use RAG and fine-tuning together?
Yes, and strong production systems often do. Retrieval supplies the current facts while fine-tuning sharpens how the model uses them. A 2024 Berkeley method called RAFT fine-tunes a model specifically to read retrieved passages and ignore irrelevant ones, which improves answer quality in domain-specific RAG. The two are complementary, since one handles knowledge and the other handles behavior.

Is fine-tuning a good way to add new knowledge to a model?
Usually not. OpenAI’s guidance is that fine-tuning adjusts behavior, style, and format, while retrieval is the better tool for giving a model current or proprietary facts. Knowledge baked into weights freezes at training time and goes stale, and the only fix is another training run. For facts that change, RAG keeps the knowledge editable without retraining.

1. Optimizing LLM Accuracy, OpenAI (June 2024)

2. Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicine, arXiv (November 2023)

3. RAFT: Adapting Language Model to Domain Specific RAG, arXiv (March 2024)