SentenceChunker

Pack whole sentences into chunks up to chunk_size, never splitting mid-sentence, with optional overlap.

Tip

When to use: Retrieval/RAG where sentence integrity matters and you want fast semantic chunk boundaries.

Initialization

from blazechunk import SentenceChunker

chunker = SentenceChunker(
    tokenizer="character",     # size unit
    chunk_size=2048,           # target max tokens per chunk
    overlap=0,                 # sentences to repeat for context
    min_characters_per_chunk=24,  # merge tiny fragments
)

Parameters

Parameter	Type	Default	Description
tokenizer	str	"character"	Token counter name, or path to a tokenizer.json.
chunk_size	int	2048	Target maximum tokens per chunk.
overlap	int	0	Number of sentences to repeat in adjacent chunks.
min_characters_per_chunk	int	24	Fragments shorter than this are merged.

Usage

from blazechunk import SentenceChunker

chunker = SentenceChunker(chunk_size=50, overlap=1)
chunks = chunker.chunk("First sentence. Second sentence. Third sentence.")
for c in chunks:
    print(c.text)

Output

First sentence. Second sentence.
Second sentence. Third sentence.

Batch processing

chunker.chunk_batch(texts)            # sync
await chunker.chunk_batch_async(texts) # async

Note

Overlap: Set overlap=1 or higher to repeat the last N sentences from the previous chunk at the start of the next chunk, providing context for retrieval.