What is document chunking? Preserving conditions and provenance in RAG
Understand character, token, and structure-based chunking through a returns manual, including headings, tables, exceptions, and retrieval checks.
Understand the concept
Chunking divides documents into independently processed passages for embeddings and RAG. Length- or structure-based splitting can meet limits and focus retrieval. Smaller chunks can separate conditions from conclusions; larger ones can add irrelevant content. Preserve source positions and test retrieval with real questions.
Different units produce different effective lengths
Character and token splitting differ, especially for Chinese and code. LangChain’s recursive splitter defaults to characters and separators, while Microsoft discusses several approaches. Check the length function before treating equal numeric settings as equal size.
Heading or paragraph splits preserve some structure, depending on parsing quality. Fine splitting cannot repair scrambled multi-column OCR. Compare extracted text with original pages before blaming retrieval models.
Keep exceptions and table headers with their content
A manual allowing seven-day returns may immediately exclude customized products. Separating the sentences can produce a false promise. Keep related passages or necessary scope information rather than deleting qualifications to shorten chunks.
Amounts without row and column labels are hard to interpret. Ensure chunks trace to the original table and retain headers or identifiers where needed. This is processing design without a real manual import or measured accuracy.
Evaluate chunks against the evidence needed to answer
Label required passages for questions involving exceptions or multiple sections and inspect retrieval coverage. Missing indexed evidence is not necessarily fixed by generation prompts. Distinguish parsing, chunk coherence, and ranking failures first.
Compare complete answers and duplication across settings rather than chunk count alone. Overlap helps continuity but adds storage and duplicate retrieval. There is no universally best size; use document form and task evidence and retain configuration versions.
Checked sources
These sources support the concepts and methods described here. Examples and test scope are labeled separately; check the version of the tools you use.