Lesson 1: Chunking Strategies That Actually Work

Chunking is the step most teams get wrong first. Too large and the retrieved chunk floods the context with irrelevant text. Too small and the model loses the surrounding context needed to answer. This lesson runs controlled experiments so you can see the difference yourself.

Why Chunking Strategy Dominates Retrieval Quality

Your embedding model encodes the average meaning of a chunk. A 2,000-word chunk that mixes three different topics gets an embedding that is mediocre at representing any of them. A well-bounded chunk — one paragraph, one concept — gets an embedding that precisely represents exactly one idea.

Studies on production RAG systems consistently find that chunking strategy accounts for 20–40% of end-to-end retrieval quality variance — more than the choice of embedding model.

The Four Strategies

Strategy: Recursive CharacterSize: ~200 charsOverlap: 30 chars

[1]Introduction to Large Language Models Large Language Models (LLMs) are neural networks trained on vast amounts of text data. They can generate coherent text, answer questions, and perform complex reasoning tasks.

[2]Training Process LLMs are trained using self-supervised learning. The model predicts the next token given the preceding context. This process, repeated billions of times, teaches the model grammar, facts, and reasoning patterns.

[3]The training data matters enormously. Models trained on higher-quality data tend to perform better, even with fewer parameters.

[4]Applications LLMs power modern AI assistants, code completion tools, and document summarization systems. They are increasingly used in enterprise workflows to automate knowledge work.

✅ Respects paragraph boundaries. Each chunk is a complete thought. Significantly better retrieval quality than fixed-size.

Implementation with LangChain

from langchain.text_splitter import (
    RecursiveCharacterTextSplitter,
    CharacterTextSplitter,
    SentenceTransformersTokenTextSplitter,
)

# ── Strategy 1: Fixed-Size (baseline — usually worst) ──
fixed = CharacterTextSplitter(chunk_size=200, chunk_overlap=20)

# ── Strategy 2: Recursive Character (default for most use cases) ──
recursive = RecursiveCharacterTextSplitter(
    chunk_size=512,
    chunk_overlap=64,
    separators=["

", "
", ". ", " ", ""],  # Tries in order
)

# ── Strategy 3: Token-based (respects LLM token limits) ──
token_splitter = SentenceTransformersTokenTextSplitter(
    model_name="sentence-transformers/all-MiniLM-L6-v2",
    chunk_overlap=25,
    tokens_per_chunk=256,
)

# ── Strategy 4: Document-aware (for Markdown/HTML) ──
from langchain.text_splitter import MarkdownHeaderTextSplitter

md_splitter = MarkdownHeaderTextSplitter(
    headers_to_split_on=[
        ("#", "H1"),
        ("##", "H2"),
        ("###", "H3"),
    ]
)

# Each chunk inherits the header context as metadata
md_chunks = md_splitter.split_text(markdown_document)
for chunk in md_chunks[:2]:
    print(chunk.metadata)  # → {'H1': 'Introduction', 'H2': 'Training Process'}
    print(chunk.page_content[:80])

The Parent-Child Retrieval Pattern

A powerful hybrid: index small chunks for precise retrieval, but return the parent chunkto the LLM for richer context. Embed 128-token sentences; when a sentence is retrieved, return its 512-token parent paragraph to the LLM.

from langchain.retrievers import ParentDocumentRetriever
from langchain.storage import InMemoryStore
from langchain_community.vectorstores import Chroma

child_splitter = RecursiveCharacterTextSplitter(chunk_size=128)
parent_splitter = RecursiveCharacterTextSplitter(chunk_size=512)

store = InMemoryStore()  # Or a persistent store
vectorstore = Chroma(embedding_function=embedder)

retriever = ParentDocumentRetriever(
    vectorstore=vectorstore,
    docstore=store,
    child_splitter=child_splitter,
    parent_splitter=parent_splitter,
)

# Add documents — small chunks go into vector store, parents into docstore
retriever.add_documents(documents)

# Retrieve: finds small chunks, returns full parents
results = retriever.get_relevant_documents("How does training work?")

Chunking Decision Guide

Document Type	Recommended Strategy
Markdown / wikis / documentation	MarkdownHeaderTextSplitter (section-aware)
Plain prose (reports, books)	RecursiveCharacterTextSplitter (512 chars, 64 overlap)
FAQs / Q&A pairs	SentenceTransformers (sentence-level)
Legal / financial documents	Parent-child (precise retrieval, rich context)
Code files	Language-specific splitter (by function/class)
PDFs with mixed content	Page-aware + recursive within pages

In the next lesson we move from better chunks to better search: combining semantic vectors with keyword BM25 in a hybrid retrieval system that outperforms either approach alone.

from langchain.text_splitter import ( RecursiveCharacterTextSplitter, CharacterTextSplitter, SentenceTransformersTokenTextSplitter, ) # ── Strategy 1: Fixed-Size (baseline — usually worst) ── fixed = CharacterTextSplitter(chunk_size=200, chunk_overlap=20) # ── Strategy 2: Recursive Character (default for most use cases) ── recursive = RecursiveCharacterTextSplitter( chunk_size=512, chunk_overlap=64, separators=[" ", " ", ". ", " ", ""], # Tries in order ) # ── Strategy 3: Token-based (respects LLM token limits) ── token_splitter = SentenceTransformersTokenTextSplitter( model_name="sentence-transformers/all-MiniLM-L6-v2", chunk_overlap=25, tokens_per_chunk=256, ) # ── Strategy 4: Document-aware (for Markdown/HTML) ── from langchain.text_splitter import MarkdownHeaderTextSplitter md_splitter = MarkdownHeaderTextSplitter( headers_to_split_on=[ ("#", "H1"), ("##", "H2"), ("###", "H3"), ] ) # Each chunk inherits the header context as metadata md_chunks = md_splitter.split_text(markdown_document) for chunk in md_chunks[:2]: print(chunk.metadata) # → {'H1': 'Introduction', 'H2': 'Training Process'} print(chunk.page_content[:80])

from langchain.retrievers import ParentDocumentRetriever from langchain.storage import InMemoryStore from langchain_community.vectorstores import Chroma child_splitter = RecursiveCharacterTextSplitter(chunk_size=128) parent_splitter = RecursiveCharacterTextSplitter(chunk_size=512) store = InMemoryStore() # Or a persistent store vectorstore = Chroma(embedding_function=embedder) retriever = ParentDocumentRetriever( vectorstore=vectorstore, docstore=store, child_splitter=child_splitter, parent_splitter=parent_splitter, ) # Add documents — small chunks go into vector store, parents into docstore retriever.add_documents(documents) # Retrieve: finds small chunks, returns full parents results = retriever.get_relevant_documents("How does training work?")

Document Type

Recommended Strategy

Markdown / wikis / documentation

MarkdownHeaderTextSplitter (section-aware)

Plain prose (reports, books)

RecursiveCharacterTextSplitter (512 chars, 64 overlap)

FAQs / Q&A pairs

SentenceTransformers (sentence-level)

Legal / financial documents

Parent-child (precise retrieval, rich context)

Code files

Language-specific splitter (by function/class)

PDFs with mixed content

Page-aware + recursive within pages