Low-level primitives

Direct access to blazechunk's core algorithms for advanced use cases.

Chunker methods

All high-level chunkers expose the same four methods:

from blazechunk import RecursiveChunker

chunker = RecursiveChunker(chunk_size=2048)

# Synchronous
chunks: list[Chunk] = chunker.chunk(text: str)

# Asynchronous (runs off the event loop)
chunks: list[Chunk] = await chunker.chunk_async(text: str)

# Batch synchronous
batches: list[list[Chunk]] = chunker.chunk_batch(texts: Sequence[str])

# Batch asynchronous (with optional concurrency limit)
batches: list[list[Chunk]] = await chunker.chunk_batch_async(
    texts: Sequence[str],
    max_concurrency: int | None = None
)

Chunk object

Every chunk has four read-only attributes:

chunk.text: str                # The chunk text
chunk.start_index: int         # Byte offset into the original text
chunk.end_index: int           # Byte offset into the original text
chunk.token_count: int         # Token count per the configured tokenizer

# Invariant: chunk.text == original_text[start_index:end_index]
# (except TableChunker, which re-includes the table header in each chunk)

Tokenizer options

The tokenizer parameter accepts:

# Built-in tokenizers:
tokenizer="character"   # Unicode grapheme clusters
tokenizer="word"        # Word boundaries
tokenizer="byte"        # Raw UTF-8 bytes
tokenizer="row"         # Row-based (for TableChunker)

# Advanced: path to HuggingFace tokenizer.json (requires hf-tokenizer feature)
tokenizer="/path/to/tokenizer.json"

Note

All tokenizers are optimized for performance and used internally by every chunker. Specify the tokenizer when initializing any chunker.

Low-level zero-copy fast path

For maximum throughput, use the functional API directly:

from blazechunk import chunk, chunk_async

# Sync generator of zero-copy memoryview slices
for view in chunk(b"Hello. World. Test.", size=10, delimiters=b"."):
    print(bytes(view))

# Async variant returns owned bytes
chunks = await chunk_async(b"Hello. World.", size=10, delimiters=b".")

Output

b'Hello.'
b' World.'
b' Test.'

Chunker constructors

All chunkers use keyword-only arguments with sensible defaults:

from blazechunk import (
    RecursiveChunker, SentenceChunker, TokenChunker,
    TableChunker, CodeChunker
)

RecursiveChunker(*, tokenizer="character", chunk_size=2048, 
                 min_characters_per_chunk=24, rules=None)

SentenceChunker(*, tokenizer="character", chunk_size=2048, 
                chunk_overlap=0, min_sentences_per_chunk=1,
                min_characters_per_sentence=12, delim=None, 
                include_delim="prev")

TokenChunker(*, tokenizer="character", chunk_size=2048, 
             chunk_overlap=None)  # int | float | None

TableChunker(*, tokenizer="row", chunk_size=3)

CodeChunker(*, tokenizer="character", chunk_size=2048, 
            language="auto")

Note

include_delim accepts "prev", "next", or "none". For TokenChunker, chunk_overlap can be an int (token count) or float (fraction, e.g., 0.1).