TokenChunker

Fixed-size token windows with optional overlap, ideal for token-limited models.

Tip

When to use: Splitting for token-limited LLM APIs where you need exact token budgets and deterministic splits.

Initialization

from blazechunk import TokenChunker

chunker = TokenChunker(
    tokenizer="character",     # size unit
    chunk_size=2048,           # fixed token window size
    overlap=0,                 # tokens to repeat for context
)

Parameters

Parameter	Type	Default	Description
tokenizer	str	"character"	Token counter name, or path to a tokenizer.json.
chunk_size	int	2048	Fixed token window size.
overlap	int	0	Number of tokens to repeat in adjacent chunks.

Usage

from blazechunk import TokenChunker

chunker = TokenChunker(chunk_size=10, overlap=2)
chunks = chunker.chunk("0123456789abcdefghij")
for c in chunks:
    print(c.text)

Output

0123456789
89abcdefgh
ghij

Batch processing

chunks = chunker.chunk_batch(texts)            # sync
chunks = await chunker.chunk_batch_async(texts) # async

Note

Deterministic: TokenChunker produces the same chunk boundaries every time, making it ideal for reproducible pipelines and token budgeting.