Chunker
Zero-copy, SIMD byte chunking — split raw bytes at delimiter boundaries under a size budget, at up to 1 TB/s.
Tip
When to use:High-throughput pipelines where a byte-size limit is acceptable and you don't need sentence/token awareness.
Low-level byte chunking
Import the low-level chunking functions for zero-copy byte-level chunking:
from blazechunk import chunk, chunk_async, Chunker
# Signatures:
chunk(text: str | bytes, *, size=4096, delimiters=None, patterns=None) -> Iterator[memoryview]
chunk_async(text: str | bytes, *, size=4096, delimiters=None, patterns=None) -> list[bytes]Parameters
| Parameter | Type | Default | Description |
|---|---|---|---|
| text | str | bytes | — | Input; str is UTF-8 encoded. |
| size | int | 4096 | Target chunk size in bytes. |
| delimiters | str | bytes | None | b"\n.?" | Single-byte delimiters to break on. |
| patterns | Sequence[str | bytes] | None | None | Multi-byte patterns (e.g. CJK ["。","，"]), composable with delimiters. |
Usage
from blazechunk import chunk
for view in chunk("Hello. World. Test.", size=10, delimiters=b"."):
print(bytes(view))Output
b'Hello.'
b' World.'
b' Test.'Parameters
| Parameter | Type | Default | Description |
|---|---|---|---|
| text | str | bytes | — | Input; str is UTF-8 encoded. |
| size | int | 4096 | Target chunk size in bytes. |
| delimiters | bytes | None | None | Single-byte delimiters to break on. |
| patterns | Sequence[bytes] | None | None | Multi-byte patterns (e.g. CJK ["。","，"]). |
Note
chunk() yields zero-copy memoryviews into the original buffer — materialize with bytes(view) when you need to own the data. chunk_async() returns owned bytes because the chunks outlive the worker thread.