SDPMChunker

Semantic + double-pass merge: splits on semantic boundaries, then re-merges related non-adjacent chunks.

Tip

When to use: When your text has short digressions or side comments that break semantic flow. SDPM bridges these gaps by merging non-adjacent chunks that are semantically similar, filling in short gaps between related content.

Initialization

from sentence_transformers import SentenceTransformer
from blazechunk import SDPMChunker

model = SentenceTransformer("all-MiniLM-L6-v2")
chunker = SDPMChunker(
    embed_batch=model.encode,        # embedding function
    skip_window=1,                   # how many chunks to skip when looking for merges
    threshold=0.8,                   # percentile for semantic boundaries
    chunk_size=2048,                 # target max tokens per chunk
    tokenizer="character",           # size unit
)

Parameters

Parameter	Type	Default	Description
embed_batch	Callable[[list[str]], NDArray]	—	Function that takes a list of strings and returns a 2D embedding array.
skip_window	int	1	Number of chunks to skip when searching for merge candidates. Default 1 merges adjacent chunks.
threshold	float	0.8	Percentile threshold for detecting similarity minima (0.0–1.0).
chunk_size	int	2048	Target maximum tokens per chunk after merging.
tokenizer	str	"character"	Token counter name, or path to a tokenizer.json.

Usage

from sentence_transformers import SentenceTransformer
from blazechunk import SDPMChunker

model = SentenceTransformer("all-MiniLM-L6-v2")
chunker = SDPMChunker(model.encode, skip_window=1, threshold=0.8, chunk_size=2048)

chunks = chunker.chunk(prose)
for c in chunks:
    print(c.text, c.start_index, c.end_index, c.token_count)

How it works

1.First pass: Run SemanticChunker to detect semantic boundaries.
2.Second pass: For each chunk, compute its mean embedding.
3.Compare each chunk to chunks skip_window positions away.
4.If similarity is above threshold and merged size fits chunk_size, bridge the gap and merge.
5.Repeat until no more merges are possible.

Note

What "SDPM" means: Semantic + Double-Pass Merge — two passes over the text: first to detect semantic chunks, second to merge related but separated groups.

Tip

Tuning skip_window: Increase skip_window to allow merging of chunks further apart. Default 1 only merges immediately adjacent chunks; 2 allows bridges over one intermediate chunk, etc.