Sync vs Async
Understand the uniform four-method API and when to use each.
Methods
| Method | Async equivalent | Description |
|---|---|---|
| chunk(text) | chunk_async(text) | Chunk a single string. |
| chunk_batch(texts) | chunk_batch_async(texts) | Chunk many strings. |
Async methods offload the CPU-bound chunking to a worker thread via asyncio.to_thread, so awaiting them never blocks the event loop — ideal inside FastAPI/Starlette/aiohttp handlers. chunk_batch_async accepts an optional max_concurrency for back-pressure on large batches.
from blazechunk import TokenChunker
chunker = TokenChunker(chunk_size=3)
batches = chunker.chunk_batch(["abcdef", "xy"])
print([[c.text for c in b] for b in batches])Output
[['abc', 'def'], ['xy']]