CodeChunker

Structure-aware source-code chunking that respects syntax boundaries.

Tip

When to use: Large source files where you want chunks to respect function/class boundaries and syntax structure.

Initialization

from blazechunk import CodeChunker

chunker = CodeChunker(
    tokenizer="character",     # size unit
    chunk_size=2048,           # target max tokens per chunk
    language="python",         # code language
    min_characters_per_chunk=24,  # merge tiny fragments
)

Parameters

Parameter	Type	Default	Description
tokenizer	str	"character"	Token counter name, or path to a tokenizer.json.
chunk_size	int	2048	Target maximum tokens per chunk.
language	str	"python"	Code language (e.g. "python", "rust", "javascript").
min_characters_per_chunk	int	24	Fragments shorter than this are merged.

Usage

from blazechunk import CodeChunker

code = """def hello():
    print("world")

def goodbye():
    print("see you")"""

chunker = CodeChunker(language="python", chunk_size=50)
chunks = chunker.chunk(code)
for c in chunks:
    print(c.text)
    print("---")

Batch processing

chunks = chunker.chunk_batch(code_files)            # sync
chunks = await chunker.chunk_batch_async(code_files) # async

Note

Supported languages: python, rust, javascript/typescript, go, java, c++, and more. The chunker respects syntax boundaries like function and class definitions.