Multibyte safety

Built-in guarantees for correct text handling across all character encodings.

blazechunk bakes in two guarantees by construction:

•Offsets from byte-spans: Offsets exposed to callers are always derived from the original text / byte-spans, never from accumulating decoded (char) lengths — so token chunking never corrupts multi-byte characters.
•Multi-byte delimiters: Non-ASCII / multi-byte delimiters (e.g. CJK punctuation 。, ，) go through pattern matching, never a per-byte split — so CJK text is not mis-split.

Tip

This is why every chunk's .text is an exact slice of your input (except TableChunker), even for emoji, CJK, and other multi-byte text.