The Chunk object

Understand the return type that every chunker produces.

Every chunker returns Chunk objects:

class Chunk:
    text: str          # the chunk text
    start_index: int    # byte offset where the chunk starts in the original input
    end_index: int      # byte offset where the chunk ends
    token_count: int    # number of tokens (per the chosen token counter)

Fields

ParameterTypeDefaultDescription
textstr—The chunk's text.
start_indexint—Byte offset into the original input where the chunk begins.
end_indexint—Byte offset where the chunk ends.
token_countint—Number of tokens in the chunk (depends on the token counter).
Note
The slice invariant: For every chunker except TableChunker, chunk.text == original_text[start_index:end_index]. Chunk text is never rebuilt from decoded lengths, which is what makes multi-byte / CJK text safe by construction. TableChunker is the one documented exception (it repeats the header in every chunk).