blazechunk
The fastest semantic text chunking library — up to 1 TB/s throughput.
blazechunk splits text at semantic boundaries and does it stupid fast. It ships nine chunkers: a zero-copy byte Chunker, six structural chunkers ( RecursiveChunker, SentenceChunker, TokenChunker, TableChunker, CodeChunker), and three embedding-based chunkers ( SemanticChunker, SDPMChunker, LateChunker) — all with matching synchronous and asynchronous methods.
Why blazechunk
Blazing fast
SIMD memchr/lookup-table/Aho-Corasick delimiter search; up to 1 TB/s on the raw primitive.
Correct by construction
Chunk offsets are real byte offsets into your input; multi-byte / CJK text is never split mid-character.
Sync and async everywhere
Every chunker has chunk/chunk_async and chunk_batch/chunk_batch_async, safe to use inside FastAPI/asyncio without blocking the event loop.
Pick a chunker
| Chunker | Best for | Unit | Default |
|---|---|---|---|
| Chunker | max-throughput byte splitting | bytes | 4096 |
| RecursiveChunker | long, well-structured docs | characters | 2048 |
| SentenceChunker | retrieval where sentence integrity matters | characters | 2048 |
| TokenChunker | fixed-size windows for token-limited models | characters | 2048 |
| TableChunker | large Markdown/HTML tables | rows | 3 |
| CodeChunker | source code, structure-aware | characters | 2048 |
| SemanticChunker | semantic boundaries via embeddings | characters | 2048 |
| SDPMChunker | semantic + bridging gaps (skip-window merge) | characters | 2048 |
| LateChunker | late-interaction embeddings per chunk | characters | 2048 |
Framework integrations
Use blazechunk directly in your favorite frameworks with official integrations:
Availability: Python (sync+async)