Reference

Core objects and low-level primitives for building custom chunking pipelines.

Core objects

Chunk stores text, byte offsets, and metadata. LateChunk extends a chunk with contextual embedding data. The tokenizer argument lets token-aware chunkers measure the unit used for their limit.

from blazechunk import TokenChunker

chunker = TokenChunker(chunk_size=512, tokenizer="cl100k_base")
chunks = chunker.chunk(text)

Low-level primitives

chunk() returns chunks, while chunk_offsets() returns offset pairs. Use split_offsets() to split a range, merge_splits() to combine compatible ranges, and PatternSplitter to define custom boundaries.

from blazechunk import Chunker

primitive = Chunker(chunk_size=4096)
chunks = primitive.chunk(text)
offsets = primitive.chunk_offsets(text)

Custom chunker interface

Subclass Chunker when you need custom splitting. Keep synchronous and asynchronous methods consistent, preserve ordered offsets, and return the standard chunk objects so downstream metadata handling remains predictable.