AI

RAG chunking strategies — pick the right splitter for your docs

Five chunking strategies that cover 95% of real RAG use cases. Runnable Python. Rules for picking chunk size + overlap by document type.

The RAG pipeline from scratch walkthrough used a fixed-size character splitter — 1500 characters per chunk with 200 characters of overlap. That works for a tutorial on five Q&A documents. It fails quietly on anything real.

This article is the deeper look. Why the simple splitter breaks, what the five strategies that cover almost everything actually are, and how to pick chunk size and overlap without guessing.

Prerequisite reading: if the overall RAG loop (chunk → embed → store → retrieve + generate) is unfamiliar, the pillar walkthrough covers it in ~300 lines of Python with no framework.

Why fixed-size character splitting fails on real docs

Fixed-size character splitting cuts text every N characters, regardless of structure. The retrieval step then becomes unreliable in three ways:

1. Cuts mid-sentence or mid-word. A chunk ending in “the primary concern of the API gateway is” has lost the predicate — the sentence trails off before saying what the concern actually is. The embedding of this fragment doesn’t match what a user would ask about (“What does an API gateway do?”), so the correct chunk never ranks high in retrieval.

2. Splits related content into different chunks. A technical paragraph often introduces a concept in the first sentence and explains it in the next three. If the cut lands mid-paragraph, the “concept introduction” chunk and the “explanation” chunk live apart. Each on its own retrieves poorly — the first is too abstract, the second lacks context.

3. Different document types fail differently. 1500 characters is one short section in a Markdown doc, half a function in Python code, and three slide titles in a converted PowerPoint. Picking a single character count treats all of these the same way, which is wrong for all of them.

The fix is not a better magic number for chunk size. It’s choosing a splitter that understands the structure of the document being split.

The five strategies that cover 95% of use cases

Each strategy below is shown as pure Python — no framework. All of them are easy to drop into the pillar walkthrough’s chunk_text function.

Strategy 1 — Fixed-size character (baseline, use when nothing else fits)

The pillar used this. It’s the right starting point when documents have no exploitable structure and you need something working today.

def chunk_fixed_char(text: str, size: int = 1500, overlap: int = 200) -> list[str]:
    chunks = []
    start = 0
    while start < len(text):
        chunks.append(text[start:start + size])
        start += size - overlap
    return chunks

When to use it: OCR’d text with no reliable paragraph breaks. Scraped HTML stripped to plain text with no structure left. Chat transcripts with inconsistent formatting.

When it breaks: everything else. If paragraphs, headings, or sentences survive in the source, use Strategy 3 or 4 instead.

Strategy 2 — Token-aware splitting

Characters are a poor proxy for what the embedding model actually sees. Every embedding model has a hard token limit, and it varies more than people assume:

  • all-MiniLM-L6-v2 — 256 tokens (text past that is silently truncated)
  • all-mpnet-base-v2 — 384 tokens
  • OpenAI text-embedding-3-small and -large — 8191 tokens
  • Cohere embed-v3 — 512 tokens

A 1500-character chunk is roughly 300-400 tokens of English prose, but can be 500+ tokens of code with punctuation, or 1500+ tokens of CJK text where one character often maps to one or more tokens. For all-MiniLM-L6-v2 with its 256-token cap, that 1500-character English chunk already overshoots and gets truncated. The missing content is gone from the embedding.

Count tokens with the embedding model’s own tokenizer. For sentence-transformers (the embedding library used in the pillar):

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-MiniLM-L6-v2")
tokenizer = model.tokenizer

# max_tokens defaults to 256 because that's the model's hard limit.
# For a model with a higher ceiling (e.g. all-mpnet-base-v2 at 384), raise it.
def chunk_fixed_token(text: str, max_tokens: int = 256, overlap_tokens: int = 32) -> list[str]:
    ids = tokenizer.encode(text, add_special_tokens=False)
    chunks = []
    start = 0
    while start < len(ids):
        slice_ids = ids[start:start + max_tokens]
        chunks.append(tokenizer.decode(slice_ids, skip_special_tokens=True))
        start += max_tokens - overlap_tokens
    return chunks

When to use it: mixed-language content, code-heavy docs, or any time you want the chunk size to actually match what the model can process.

Gotcha 1: tokenizer.decode round-trips can lose exact whitespace. For retrieval that’s fine (the embedding is derived from the decoded form anyway), but if downstream code needs byte-perfect reconstruction, keep an index to the original character range alongside each chunk.

Gotcha 2: picking max_tokens higher than the model’s limit does not help — the model truncates internally. Check your embedding model’s max_seq_length attribute (model.get_max_seq_length() on a SentenceTransformer instance) before setting this.

Strategy 3 — Recursive character splitting (the default for prose)

Instead of cutting every N characters blindly, try a sequence of separators from biggest to smallest: paragraph break, then sentence break, then word break, then character. Each level cuts only if the previous level produced a chunk that’s still too big.

import re

SEPARATORS = ["\n\n", "\n", ". ", " "]

def chunk_recursive(text: str, size: int = 1500, overlap: int = 200) -> list[str]:
    # Base case: text fits in one chunk
    if len(text) <= size:
        return [text]

    # Try each separator in turn
    for sep in SEPARATORS:
        if sep in text:
            parts = text.split(sep)
            chunks = []
            current = ""
            for part in parts:
                piece = (current + sep + part) if current else part
                if len(piece) <= size:
                    current = piece
                else:
                    if current:
                        chunks.append(current)
                    # Part itself may still be too big — recurse
                    if len(part) > size:
                        chunks.extend(chunk_recursive(part, size, overlap))
                        current = ""
                    else:
                        current = part
            if current:
                chunks.append(current)
            # Apply overlap between adjacent chunks
            return _apply_overlap(chunks, overlap)

    # No separator present — fall back to fixed character
    return chunk_fixed_char(text, size, overlap)


def _apply_overlap(chunks: list[str], overlap: int) -> list[str]:
    if overlap <= 0 or len(chunks) <= 1:
        return chunks
    with_overlap = [chunks[0]]
    for i in range(1, len(chunks)):
        tail = chunks[i - 1][-overlap:]
        with_overlap.append(tail + chunks[i])
    return with_overlap

The result: chunks almost always end at a paragraph or sentence boundary. The content is still coherent after the split.

When to use it: 80% of real-world cases. Blog posts, documentation, articles, emails, chat transcripts with proper formatting.

When to swap for Strategy 4: when documents have explicit structure (Markdown headings, HTML sections) that already carries meaning.

Strategy 4 — Markdown / heading-aware splitting

For Markdown or any document with explicit section headers, split at the headings. Each section becomes its own chunk, and the heading text travels with the body — which the embedding model uses to understand what the section is about.

import re

HEADING_RE = re.compile(r"^(#{1,6})\s+(.+)$", re.MULTILINE)

def chunk_markdown(text: str, max_size: int = 2000) -> list[str]:
    chunks = []
    headings = list(HEADING_RE.finditer(text))

    if not headings:
        # No headings at all — fall back to recursive
        return chunk_recursive(text, max_size, overlap=200)

    # Preamble: any text before the first heading (title, intro paragraph, etc.)
    preamble = text[:headings[0].start()].strip()
    if preamble:
        if len(preamble) <= max_size:
            chunks.append(preamble)
        else:
            chunks.extend(chunk_recursive(preamble, max_size, overlap=200))

    for i, match in enumerate(headings):
        start = match.start()
        end = headings[i + 1].start() if i + 1 < len(headings) else len(text)
        section = text[start:end].strip()

        if len(section) <= max_size:
            chunks.append(section)
        else:
            # Section too big — recursively split, but prepend the heading
            # to each sub-chunk so embeddings still know what the section is about
            heading_line = match.group(0)
            body = section[len(heading_line):].strip()
            sub_chunks = chunk_recursive(body, max_size, overlap=200)
            chunks.extend(f"{heading_line}\n\n{sub}" for sub in sub_chunks)

    return chunks

The payoff is practical: a user asking “How do I configure logging?” retrieves chunks whose headings mention logging. The heading itself is strong semantic signal, cheaper and more reliable than hoping it leaks into prose.

When to use it: technical docs, markdown wikis, Confluence exports, any content with clear section headers.

Gotcha: if headings are extremely short or decorative (e.g., ### 📊), prepending them doesn’t help. In that case strip emoji-only headings before chunking.

Strategy 5 — Semantic chunking (slower, use for high-value content)

The strategies above split by structure. Semantic chunking splits where the meaning actually shifts — detected by comparing embeddings of successive sentences and cutting where similarity drops.

import numpy as np
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-MiniLM-L6-v2")

def chunk_semantic(text: str, similarity_threshold: float = 0.6, buffer: int = 1) -> list[str]:
    # Split into sentences first
    sentences = [s.strip() for s in re.split(r"(?<=[.!?])\s+", text) if s.strip()]
    if len(sentences) <= 1:
        return [text]

    # Combine each sentence with its neighbours for stable embeddings
    windows = []
    for i in range(len(sentences)):
        lo = max(0, i - buffer)
        hi = min(len(sentences), i + buffer + 1)
        windows.append(" ".join(sentences[lo:hi]))

    embeddings = model.encode(windows, convert_to_numpy=True)

    # Cosine similarity between adjacent windows
    def cosine(a, b):
        return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))

    similarities = [
        cosine(embeddings[i], embeddings[i + 1])
        for i in range(len(embeddings) - 1)
    ]

    # Cut wherever similarity falls below the threshold
    chunks = []
    current = [sentences[0]]
    for i, sim in enumerate(similarities):
        if sim < similarity_threshold:
            chunks.append(" ".join(current))
            current = [sentences[i + 1]]
        else:
            current.append(sentences[i + 1])
    if current:
        chunks.append(" ".join(current))

    return chunks

When to use it: a small set of high-value documents where retrieval quality matters more than indexing speed. Legal contracts, research papers, long-form editorial content.

Why not everywhere: semantic chunking runs the embedding model once per sentence during indexing — 10-100× slower than Strategy 3. For a 10,000-document corpus, it’s measurable extra time and compute. For 50 contracts, it’s cheap and the quality lift is real.

Threshold tuning: 0.6 is a reasonable starting point for all-MiniLM-L6-v2. Different embedding models produce different similarity distributions — the right threshold is one that produces chunks of about the size you’d pick by hand on a sample document.

Choosing chunk size

Chunk size trades off two things:

  • Smaller chunks make retrieval more precise (the chunk is “about” one tight idea), but risk losing surrounding context the LLM needs to answer.
  • Larger chunks carry more context, but dilute the embedding — one big chunk about three loosely related topics ranks poorly for any single-topic query.

Three practical rules:

Rule 1: Chunk in tokens, not characters, and respect the embedding model’s cap. For the SBERT family (all-MiniLM-L6-v2 at 256 tokens, all-mpnet-base-v2 at 384 tokens), set chunks AT or below that cap — overshooting means silent truncation. For OpenAI embeddings (8191-token input), chunks of 500-1000 tokens are safe and leave room for context-rich passages.

Rule 2: Multiple chunks must fit the LLM’s context window at retrieval time. If retrieval returns the top 5 chunks and each is 500 tokens, that’s 2500 tokens of context before prompt and answer. Keep headroom. Current frontier model windows: Claude Sonnet 5.5 and Opus 5.5 support 1M tokens, GPT-5 supports 400K tokens — generous for most RAG. For cheaper or older models with 32K-128K windows, it bites.

Rule 3: Match chunk size to the question type. Factual lookups (“what port does Prometheus listen on?”) work with 100-200 token chunks. Analytical questions (“compare the deployment strategies of X and Y”) need 500-1000 token chunks so the comparison target lives in one chunk. If you’re using an SBERT model capped at 256-384 tokens, analytical queries push you toward retrieving top-K=3-5 small chunks and letting the LLM synthesize rather than a few big chunks.

If uncertain, start at 256 tokens with Strategy 3 and iterate once there’s traffic to measure against.

Overlap: when it helps, when it doesn’t

Overlap repeats the last few sentences of chunk N at the start of chunk N+1. The point is to prevent an idea from being cut exactly at the boundary.

Helps: flowing prose where sentences depend on preceding context — editorial content, legal reasoning, step-by-step explanations.

Doesn’t help, or hurts:

  • Structured content that already has clean breaks (Markdown headings, bullet lists, code blocks). The structure IS the overlap — adding more creates duplicate retrieval hits.
  • Tabular data flattened to text. Overlap smears cells across chunks.
  • Chat logs where each message is self-contained.

Rule of thumb: 10-20% overlap on prose; 0% on structured content.

Document-type specific advice

Markdown and HTML

Strategy 4. Headings are the single strongest semantic signal in these docs — use them. Keep the heading inline with the chunk body so the embedding sees it. Strip boilerplate (<nav>, <footer>) before chunking, otherwise every chunk gets contaminated with navigation terms.

PDFs

PDFs are a special pain. Text extraction tools (pdfplumber, pypdf, unstructured) produce output with:

  • Headers and footers repeating on every page
  • Line breaks mid-sentence from the two-column layout
  • Tables flattened into unreadable text

Pre-process before chunking: strip repeated header/footer lines, join lines that end without punctuation, extract tables separately. Then apply Strategy 3 or 4 depending on whether the document has detectable section structure.

For PDFs with heavy tables or figures, consider Vision Language Models (Claude Sonnet 5.5 with image input, GPT-5 with vision) to extract structured representations before chunking — pure text extraction loses too much.

Code

Chunk at function or class boundaries, not character count. A function with its docstring, signature, and body belongs together — retrieval returns a code unit that’s genuinely useful.

import ast

def chunk_python_code(source: str) -> list[str]:
    tree = ast.parse(source)
    chunks = []
    for node in ast.walk(tree):
        if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef)):
            chunk = ast.get_source_segment(source, node)
            if chunk:
                chunks.append(chunk)
    return chunks

ast.walk traverses the whole tree, so methods inside classes become their own chunks as well as the class as a whole. For very large files that leads to overlap — both the class and each method appear as chunks. If duplication in retrieval is a problem, switch ast.walk to ast.iter_child_nodes(tree) to get only module-level definitions (classes stay whole; methods ride with their class).

For other languages, Tree-sitter parsers are the standard tool. The principle is the same: respect the language’s structural units.

Chat and conversation logs

Each turn (user message or assistant reply) is already a natural chunk. The complication is context — one turn alone often lacks meaning. Attach the preceding 2-4 turns as context in the chunk’s text, but store the primary turn separately in metadata so retrieval can return just the direct answer when needed.

How to tell if your chunking is actually helping

Short answer: measure retrieval quality on a held-out set of query → correct-chunk pairs. Build 20-50 of these by hand from your corpus. For each query, check whether the “correct” chunk shows up in the top-K retrieved results.

A detailed walkthrough of this process is in the next article in this series: how to evaluate RAG retrieval + generation.

For now, the quick qualitative test: pick five chunks at random from your output. Can each one, read on its own, answer at least one plausible user question? If a chunk is a fragment that doesn’t make sense in isolation, chunking is too aggressive. If a chunk covers three unrelated topics, chunking is too loose.

Common mistakes

  • Splitting by newlines only. A single paragraph in a well-formatted document can be 2000+ words. Line count isn’t token count.
  • Treating “all documents the same”. Running Strategy 1 on Markdown wastes the structure that’s right there. Running Strategy 4 on OCR’d PDFs crashes because there are no headings.
  • Chunking before cleaning. Extracting text from HTML and then chunking carries over every [Edit] link and sidebar fragment. Clean first; chunk second.
  • Over-tuning overlap. 50% overlap doesn’t make retrieval twice as good; it doubles the index size and makes every chunk look partially similar to its neighbours, hurting ranking.
  • Fixing the splitter and never measuring. A good first chunker gets to 70% of ideal retrieval. The last 30% comes from the retrieval + evaluation loop, not from a smarter splitter.

Bottom line

Fixed-size character splitting works for a demo. For anything real:

  • Use Strategy 3 (recursive) as the default for prose.
  • Use Strategy 4 (heading-aware) for Markdown, HTML, and structured docs.
  • Use Strategy 2 (token-aware) when the character-to-token ratio varies a lot.
  • Use Strategy 5 (semantic) only on small, high-value document sets.
  • Fall back to Strategy 1 (fixed character) only when nothing exploitable survives in the source.

Chunk sizes: 200-400 tokens for most prose, up to 1000 for comparison or synthesis tasks. Overlap: 10-20% on prose, zero on structured content.

The retrieval quality floor is set by chunking. No reranker, hybrid search, or query rewrite recovers from chunks that split the right answer across three fragments. Get this layer right once, then everything downstream has something to work with.

What’s next in this series

  • RAG evaluation metrics and tooling — how to measure whether chunking + retrieval are actually working, and what to log so you can debug a bad answer six weeks later.
  • When to skip RAG entirely — the problems a longer context window, fine-tuning, or a straight SQL query solves better than any RAG pipeline.