What is Chunking?

You set your RAG quality ceiling at ingest, before the first query runs

Chunking splits documents into smaller passages before a RAG system embeds and indexes them, so a query retrieves only the passage that answers it. The split sets a ceiling on answer quality: chunk size, boundaries, and overlap decide what the model can ever see, and no later stage can recover what a bad cut threw away.

🧭 Part 11 of the RAG & Search course

TL;DR

• The definition: a chunk is a few hundred tokens of text, embedded and indexed as its own retrievable unit.

• Why it matters: retrieval ranks only the chunks that exist. A policy split across two chunks answers no question.

• Size and boundaries: too small strips the context a passage needs, and too big averages unrelated passages into one vector that matches no specific question. The good cuts land on paragraphs, headings, and sentence ends.

• The overlap: the tail of each chunk gets copied onto the head of the next, so a sentence that straddles a cut survives whole in at least one chunk. The repeats cost extra vectors, so you hold overlap to 10 to 20 percent of the chunk.

• The catch: fancy cutting strategies barely beat the simple ones, and in one benchmark the simple one won outright. Tune the basics before reaching for anything exotic.

Before Chunking, One Vector Per Document

The simplest version of the pipeline has no splitter in it. You have a folder of company PDFs, an embedding model, and a vector database. You embed each document as one vector, store one row per document, and match queries against them. This is document-level indexing. On small single-topic files it works. We took the full pipeline apart in What is RAG?.

An embedding compresses a passage into one fixed-length vector that sits near texts with similar meaning. What are Embeddings? takes the mechanics apart. One vector per document means one vector for the whole 80-page employee handbook.

No embedding model reads 80 pages in a single pass, so the pipeline embeds the handbook in pieces and averages those into one document vector. Onboarding, expenses, security training, and vacation policy all land in the same point.

Why One Vector Per Document Breaks Down

A new hire asks the company bot how many vacation days they get after year three. The answer sits on page 41 of the handbook. Retrieval compares the question against one vector per document and returns the closest.

The handbook’s vector is an average of every subject in the book, and an average of concepts far apart scores badly against any specific question.

Page 12 of that handbook says:

Expense reports above $500 require director approval before submission.

Page 41 says:

Employees with three or more years of service accrue twenty days of paid vacation annually.

Both go into the embedding model together, along with the other seventy-eight pages, and come out as one list of numbers. That list is what the vacation question gets compared against.

The handbook ranks below documents narrow enough to match the question, so the bot answers the new hire without ever seeing page 41.

How Chunking Actually Works

A chunk has two jobs at once. It has to be small enough that its vector stays specific. It also has to be complete enough that the passage answers the question by itself.

Those two pull against each other, and three decisions settle where you land between them:

1. where to cut,

2. how big to cut, and

3. how much to repeat across cuts.

The pipeline runs in two halves. Ingest happens once:

1. A splitter cuts each document into chunks.

2. An embedding model turns each chunk into its own vector.

3. The vectors land in a vector store with the chunk text attached. We compared the stores themselves in Vector Database Showdown.

Query time runs the same steps from the other side:

4. The same model embeds the question.

5. The index returns the top-k closest chunks, 3 to 10 in most setups.

6. The pipeline pastes those chunks into the prompt as the context the model answers from.

Every stage downstream of step 1 handles chunks. The document as a whole never appears again.

(1) Where to cut is which splitter you run, and the options runs from dumb to expensive:

1. Fixed-size splitting cuts every N tokens no matter what the text is doing, fast and uniform. It has no notion of where a sentence or a rule ends, so a cut lands wherever the count runs out.

2. Recursive splitting cuts where the writing already pauses. It tries paragraph breaks first, then feeds any piece that is still too big back through itself on line breaks, then sentence ends. Running on its own output is what makes it recursive.

3. Structure-based splitting follows the document’s own skeleton: markdown headings, HTML sections, PDF pages. The author grouped related content when they wrote it, and the split inherits that grouping, so tables and code blocks survive whole.

4. Semantic chunking embeds every sentence, then compares each one against the sentence before it. A large jump between two neighbours means the subject changed, so the splitter cuts at that point. The cost is an embedding pass over every sentence at ingest.

5. Agentic chunking hands the document to an LLM and lets it choose the boundaries, the way a person marking up a manual would. It is the only strategy that can tell a heading from a caption from a footnote, at a cost of one model call per document.

6. Contextual enrichment does not move the cuts at all. After splitting, a model writes a one-line description per chunk, naming the document and section it came from, and that line goes in front of the chunk before embedding. A chunk that begins mid-rule then carries the policy name its own text left out.

(2) How big to cut is the size setting, and the working band is 250 to 500 tokens. Tune from there against your own queries. A token-level evaluation by Chroma ran nine chunker configurations over identical corpora, models, and queries and found an 8-point recall spread between best and worst. Recall here is the share of each needed passage’s tokens that lands in the retrieved set. Plain recursive splitting at 400 tokens scored 89.5 percent. The best semantic chunker cleared that by under two points. The worst one sat at the bottom of the table at 83.6. OpenAI’s popular 800-token default gave up a point to recursive. Between the best cut and the worst, the share of each needed passage that never reaches the model doubles, from about 8 percent missing to about 16¹. Changing a splitter setting costs one re-index, which makes this the cheapest 8 points of recall in the RAG pipeline.

(3) How much to repeat is the overlap setting. A fixed-size splitter on the handbook ends chunk 12 mid-rule:

...Employees with three or more years of service accrue

Chunk 13 opens with the payoff:

twenty days of paid vacation annually, subject to manager approval...

Retrieval pulls chunk 13, since that is where “twenty days” and “vacation” live, and the model quotes the number with no eligibility rule attached. Overlap is the fix. The splitter copies the last 40 to 60 tokens of each chunk onto the front of the next, so a sentence that crosses a cut survives whole in at least one piece.

The repetition has a price. Fifteen percent overlap means fifteen percent more vectors to embed, store, and search. The duplicated text can also return twice, so two of your ten retrieved slots hold the same sentences and the model sees less of the corpus than you think.

The working baseline fits in one call:

from langchain_text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter.from_tiktoken_encoder(
    encoding_name="cl100k_base",
    chunk_size=400,
    chunk_overlap=60,
)
chunks = splitter.split_text(handbook_text)

The plain RecursiveCharacterTextSplitter constructor measures chunk_size in characters rather than tokens, so asking for 400 gets you about 100 tokens of text. Building it through from_tiktoken_encoder measures in real tokens instead. This is the first thing I check when a pipeline is retrieving fragments.

Who’s Actually Building With This

Anthropic tested the ingest-side fix at scale. Their embeddings-only baseline failed to surface the needed chunk in the top 20 results 5.7 percent of the time². Contextual enrichment cut that to 3.7 percent, a 35 percent drop bought entirely at ingest, with no change to the model or the query. Two retrieval levers on top took it to 1.9 percent. The first is BM25, the classic keyword scorer we unpacked in What is Semantic Search?. The second is a reranker, which re-scores the shortlist before the top results ship.

NVIDIA benchmarked chunking strategies across five document sets, and the winner was page-level chunking: one chunk per PDF page, with no splitting logic at all. It averaged 0.648 accuracy, where 1.0 is a perfect score, beating every other strategy on average with the lowest variance across document types³. A page break rarely lands mid-table or mid-clause, which is most of what a splitter is trying to avoid.

A Hugging Face team benchmarking RAG over nuclear engineering documents got the opposite result: the purpose-built chunker lost to the default one⁴. Their baseline was LangChain’s recursive splitter at 2,000 characters with a 400-character overlap, one of several sizes they tried, and the size barely moved the result. Across 156 test queries it beat their section-aware chunker 70.5 percent to 63.8, scoring the share of queries whose source document landed in the ten retrieved results. Their reading of why: cutting strictly on section headers orphaned content that ran across a boundary. The overlap in the default setup kept that same content together.

What Can Go Wrong (and What’s Overhyped)

Five failure modes account for most chunking pain:

1. Boundary casualties. A cut that lands inside a table, a code block, or a numbered procedure breaks it. Split the handbook’s vacation-accrual table between its header row and its data rows, and the chunk holding the numbers arrives with no column names, so 15 and 20 mean nothing. Structure-based splitting exists for these documents.

2. Near-duplicate retrieval. Generous overlap plus repetitive sections fills the top-k list with copies of the same paragraph. Every duplicate takes a slot, so the passages sitting just below the cutoff get pushed out and the model sees the same fact three times instead. Cap overlap and deduplicate retrieved chunks by text similarity.

3. The precision trap. Shrinking chunks makes each vector more specific and strips away the context the passage needed to answer the question. Cut the handbook fine enough and the vacation number lands in a chunk of its own, with the years-of-service condition sitting in the chunk above it. Retrieval finds the number and the model answers without the rule.

4. Ingest-time model bills. Semantic chunking runs an embedding pass over every sentence in the corpus. Agentic chunking runs a model call per document. On a million documents those passes cost real compute before the first query ships. Contextual enrichment runs a call per chunk and is the exception, because prompt caching drops it to about a dollar per million document tokens.

5. Silent truncation. Every embedding model has a maximum input length, and it drops whatever runs past that point without raising an error. all-MiniLM-L6-v2, pulled about 254 million times a month, reads 256 tokens. Feed it the 400-token chunks the baseline above produces and it embeds the opening stretch, drops the rest, and returns a vector that looks exactly like any other. Its 256 limit counts MiniLM’s own word pieces, which are not the tokens your splitter counted⁵. Check your model’s limit before you set chunk size.

The loudest claim about chunking right now is that long-context models killed it. Anthropic’s own guidance puts the no-RAG threshold at 200,000 tokens, about 500 pages. Below it, keep the corpus in the prompt and let prompt caching cover the rereads. Above it, you are retrieving, and retrieval runs on chunks. The other hype magnet is the exotic chunker, and the measured gap between the fanciest one and tuned recursive splitting was two recall points.

🏗️ Engineering Lesson: No single chunking setup is right for every corpus, and you cannot reason your way to yours. It depends on how much structure your documents carry, how narrow your queries run, and how much ingest compute you can spend. Keep fifty real queries with their known source passages, re-run them after every change, and read the number. A two-point gain and random noise look identical until you do.

Which of the six you land on comes down to two questions:

1. The first is what your documents look like. Paginated files mean page-level splitting, which NVIDIA found beats every strategy on average. Headings are less reliable, since the Hugging Face team cut strictly on them and lost. Everything else starts recursive at 400 tokens with 60 of overlap.

2. The second is what your eval set says. Misses that trace to a cut landing mid-rule or mid-topic are what justify semantic or agentic chunking, or a contextual enrichment pass over the chunks you already have. Misses that trace anywhere else mean the splitter is not your bottleneck and the next fix sits further down the pipeline.

The One Thing to Remember

A RAG system assembles every answer from pieces it cut before the first question arrived. Tuning retrieval, swapping embedding models, and adding rerankers all operate downstream of the same ceiling: the retriever chooses among the chunks the splitter made, and the model reads only what the retriever returns. Ingestion is where an answer becomes findable, or stops existing.

---

💬 What did a bad chunk boundary cost you? A wrong number, a split table, a policy quoted without its condition: tell me in the comments. I read every one.

---

Where to Next?

• 📖 Go deeper: How Perplexity Built Their Search Engine, what retrieval looks like when the corpus is the whole web.

• 🔀 Related: Agentic RAG vs CUA vs A2A, what happens when the retrieval loop itself becomes an agent.

• 🔗 Prerequisite: How DoorDash Built Their RAG System, the guardrail side that catches bad answers after generation.

🔜 Friday: How Meta Trained Llama 3 on 16,000 GPUs, four ways to split one model across a cluster, and why coordination beat compute as the bottleneck.

FAQ

What is chunking in RAG?
Chunking cuts each document into passages of a few hundred tokens, and the RAG pipeline embeds and indexes those passages instead of the whole file. Retrieval then ranks passages, so the model receives the paragraph that answers the question rather than everything around it. Size, boundaries, and overlap decide whether that passage arrives focused and complete.

What chunk size should I use for RAG?
Start between 250 and 500 tokens with 10 to 20 percent overlap, then tune upward against your own queries. The starting band is deliberately below where benchmarks land, since small chunks fail loudly and big ones fail quietly. NVIDIA’s five-dataset benchmark found 512 to 1,024 tokens best for token-based splitting, with page-level chunking the most consistent overall and 128-token chunks worst. Cut too small and a chunk keeps the number while losing its condition. Cut too big and its vector stops matching any specific question.

What is chunk overlap and how much should I use?
Overlap repeats the tail of each chunk at the head of the next, so sentences that straddle a boundary survive whole in at least one piece. The working range is 10 to 20 percent of chunk size. More overlap means more vectors to store and search, plus near-duplicate passages competing for top-k slots, so raise it only when boundary failures show up in your evals.

Do long-context models make chunking obsolete?
No. Anthropic’s guidance puts the crossover near 200,000 tokens, about 500 pages. A corpus smaller than that can skip retrieval: it sits in the prompt, and prompt caching keeps rereading it cheap. Anything larger still needs retrieval, and retrieval operates on chunks.

Is semantic chunking better than fixed-size chunking?
Yes on some corpora, and by less than the name promises. A token-level benchmark measured the best semantic chunkers about two recall points above tuned recursive splitting, and the worst semantic chunker scored below plain recursive. A Hugging Face team found the stock recursive splitter beat section-aware chunking by almost seven points on nuclear engineering documents.

1. Evaluating Chunking Strategies for Retrieval, Chroma Research (July 2024)

2. Introducing Contextual Retrieval, Anthropic (September 2024)

3. Finding the Best Chunking Strategy for Accurate AI Responses, NVIDIA Developer Blog (June 2025)

4. Evaluate Your Own RAG: Why Best Practices Failed Us, Hugging Face Blog (November 2025)

5. all-MiniLM-L6-v2 model card, Hugging Face