The toolkit

Chunkers

Nine strategies for turning long context into useful, retrievable pieces. Start with structure, then reach for embeddings when meaning matters.

SIMD-accelerated byte-based chunking at up to 1 TB/s.

Recursively chunk long, well-structured docs.

Split at sentence boundaries with optional overlap.

Fixed-size token windows with overlap.

Split large Markdown/HTML tables, header repeated per chunk.

Structure-aware source-code chunking.

Splits at semantic-similarity troughs using embeddings.

Semantic + double-pass merge for bridging gaps.

Whole-document embeddings, chunk-level contextual vectors.

Compatibility

One interface, every strategy

These all chunk text. To chunk a file — PDF, Word, PowerPoint, or Excel — see Documents.