SemanticChunker

Splits at semantic-similarity troughs between sentence windows using Savitzky–Golay minima detection.

Tip

When to use: When you have an embedding model and want chunks that respect semantic boundaries. Detects topic shifts and natural breakpoints in the text using sentence-level embeddings.

Initialization

from sentence_transformers import SentenceTransformer
from blazechunk import SemanticChunker

model = SentenceTransformer("all-MiniLM-L6-v2")
chunker = SemanticChunker(
    embed_batch=model.encode,        # embedding function
    threshold=0.8,                   # percentile for minima detection
    chunk_size=2048,                 # target max tokens per chunk
    tokenizer="character",           # size unit
)

Parameters

Parameter	Type	Default	Description
embed_batch	Callable[[list[str]], NDArray]	—	Function that takes a list of strings and returns a 2D embedding array.
threshold	float	0.8	Percentile threshold for detecting similarity minima (0.0–1.0). Higher = stricter chunking.
chunk_size	int	2048	Target maximum tokens per chunk.
tokenizer	str	"character"	Token counter name, or path to a tokenizer.json.

Usage

from sentence_transformers import SentenceTransformer
from blazechunk import SemanticChunker

model = SentenceTransformer("all-MiniLM-L6-v2")
chunker = SemanticChunker(model.encode, threshold=0.8, chunk_size=2048)

chunks = chunker.chunk(prose)
for c in chunks:
    print(c.text, c.start_index, c.end_index, c.token_count)

How it works

1.Splits text into sentences using punctuation and space boundaries.
2.Embeds sliding windows of sentences (e.g., 2–3 sentence spans).
3.Computes cosine similarity between consecutive windows.
4.Applies Savitzky–Golay filter to smooth the similarity curve and detect minima (topic shifts).
5.Splits at the detected minima, then rebalances to respect chunk_size.

Note

Byte-exact offsets: Like all blazechunk chunkers, start_index and end_index are byte offsets into the original text, enabling precise reconstruction.

Tip

Embedding model choice: Works with any embedding function — sentence-transformers, OpenAI, custom models. The pure Rust core ships no model; you inject the embeddings.