Concepts
The practical model behind chunk sizes, overlap, tokenizers, and offsets.
What is a chunk?
A chunk is a retrievable piece of text with its original text, byte offsets, and optional metadata. Chunkers preserve boundaries where possible instead of cutting blindly at a character count.
chunk = chunks[0]
print(chunk.text)
print(chunk.start, chunk.end)Chunk size and tokenizer units
chunk_size is measured using the chunker's tokenizer. Depending on the chunker, that unit is bytes, characters, tokens, sentences, or table rows. TokenChunker uses its configured tokenizer; Chunker measures UTF-8 bytes.
| Unit | Used by | Meaning |
|---|---|---|
| bytes | Chunker | UTF-8 byte length |
| characters | Recursive, Sentence, Code | Unicode characters |
| tokens | TokenChunker | Configured tokenizer tokens |
| rows | TableChunker | Table data rows |
Overlap
Overlap repeats a boundary region between adjacent chunks. It improves retrieval when an answer crosses a split, but increases indexing volume. Keep it smaller than the chunk size and tune it in the same unit as that chunker.
from blazechunk import RecursiveChunker
chunker = RecursiveChunker(chunk_size=1200, overlap=120)Offsets and safety
Text chunk offsets are UTF-8 byte offsets into the original input. This makes exact slicing safe for multibyte text. DocumentChunker offsets instead point into its converted Markdown output; see the Documents guide.