RecursiveChunker

Recursively chunk documents by descending a hierarchy of delimiters.

Tip

When to use: Long but well-structured documents — books, papers, docs — where you want chunks near the target size while respecting natural structure.

Initialization

from blazechunk import RecursiveChunker

chunker = RecursiveChunker(
    tokenizer="character",          # size unit
    chunk_size=2048,                # target max tokens per chunk
    min_characters_per_chunk=24,    # merge fragments smaller than this
    rules=None,                     # optional custom delimiter hierarchy
)

Parameters

Parameter	Type	Default	Description
tokenizer	str	"character"	Token counter name, or path to a tokenizer.json.
chunk_size	int	2048	Target maximum tokens per chunk. Must be > 0.
min_characters_per_chunk	int	24	Fragments shorter than this are merged into a neighbour.
rules	list[dict] \| None	None	Custom delimiter hierarchy. None = built-in 5-level hierarchy.

Usage

from blazechunk import RecursiveChunker

rules = [{"delimiters": ["|"]}, {"type": "token"}]
chunker = RecursiveChunker(chunk_size=3, min_characters_per_chunk=1, rules=rules)

for c in chunker.chunk("a|bb|ccc"):
    print(c.text, c.start_index, c.end_index, c.token_count)

Output

a| 0 2 2
bb| 2 5 3
ccc 5 8 3

Batch processing

chunker.chunk_batch(["a|bb", "ccc|d"])            # sync
await chunker.chunk_batch_async(["a|bb", "ccc|d"]) # async

Note

Custom rules: The rules parameter is a list of level dicts applied top-to-bottom:

• {"delimiters": ["\n\n", "\n"], "include_delim": "prev"} — split on any of these
• {"type": "whitespace"} — split on ASCII spaces
• {"type": "token"} — terminal level: hard-split oversized text

Warning

Unlike other chunkers, whitespace-only input is not treated as empty — it still produces a chunk. Only fully empty input returns [].