Token counters
Understand how chunk sizes are measured and customize the tokenizer.
Chunk size is measured by a token counter, injected into every chunker via the tokenizer= argument. Here are the built-ins:
| Name | Counts | Default for |
|---|---|---|
| "character" | Unicode code points | most chunkers |
| "word" | whitespace-separated words | — |
| "byte" | raw UTF-8 bytes | — |
| "row" | newline-separated rows | TableChunker |
Note
You can also pass a filesystem path to a HuggingFace
tokenizer.json for real subword counts (requires the hf-tokenizer build).