Documents

Installation

pip install "blazechunk[anydoc]"

Conversion is handled by anydoc, a pure-Rust document converter from Firecrawl — no ML and no network calls. Markdown and plain text need no extra.

Quickstart

from blazechunk.loaders import DocumentChunker

result = DocumentChunker().chunk("report.pdf")

for c in result.chunks:
    print(c.heading_path, c.kind, c.text[:60])
    # ('Methods', 'Sample Preparation')  prose  'We sampled two hundred sites across …'

chunk() accepts a path, a Path, or raw bytes.

Supported formats

Format is detected from the file's content signature, with the extension as a fallback. Raw CSV bytes need format="csv".

Why not just convert and chunk?

Converting a file to Markdown and handing the string to a text chunker throws the structure away on the way in. Heading hierarchy, table boundaries and code-block extents are gone, so the text chunker can split tables mid-row and functions mid-body.

DocumentChunker segments the document first, then routes each piece to the chunker that suits it:

from blazechunk import RecursiveChunker, TableChunker, CodeChunker
from blazechunk.loaders import DocumentChunker

loader = DocumentChunker(
    chunker=RecursiveChunker(chunk_size=2048),
    table_chunker=TableChunker(chunk_size=3),
    code_chunker=CodeChunker(chunk_size=2048),
    respect_headings=True,
    min_chunk_size=256,
)

result = loader.chunk("handbook.docx")

heading_path

Every chunk carries the chain of headings above it, giving retrievers and rerankers useful document context.

for c in result.chunks:
    store.add(
        text=c.text,
        metadata={
            "section": " > ".join(c.heading_path),
            "kind": c.kind,
            "format": c.source_format,
        },
    )

Undersized segments merge forward and downward. Moving sideways between sibling sections never merges while respect_headings is on.

Async and batch

from blazechunk.loaders import DocumentChunker

loader = DocumentChunker()
result = loader.chunk("report.pdf")
results = loader.chunk_batch(["a.pdf", "b.docx"])
results = loader.chunk_batch(paths, on_error="skip")

Conversion is CPU-bound Rust and releases the GIL while it runs, so async methods genuinely overlap work.

Offsets and provenance

md_start / md_end are UTF-8 byte offsets into result.markdown, the converted Markdown returned alongside the chunks.

data = result.markdown_bytes
assert data[c.md_start:c.md_end].decode() == c.text

Exact chunks are byte-for-byte slices and chunk spans provide full ordered coverage; later table chunks may repeat a header row and set is_exact=False.

Scanned PDFs

anydoc reads a PDF's text layer; it does not do OCR. A scanned or image-only PDF raises a named error.

from blazechunk.loaders import DocumentChunker, ScannedDocumentError, DocumentError

try:
    result = DocumentChunker().chunk("scan.pdf")
except ScannedDocumentError:
    ...
except DocumentError:
    ...

Run OCR upstream or use Firecrawl Parse, then pass the extracted text through chunk_markdown().

result = DocumentChunker().chunk_markdown(text_from_ocr)

Warnings

Large tables may be split across chunks with a repeated header row. Inspect result.warnings when exactness matters.

result = DocumentChunker().chunk("report.pdf")

for note in result.warnings:
    log.info("%s", note)
    # e.g. "3 table(s) exceeded ... is_exact=False"

PDF to retrieval example

Keep the heading path and source format beside each embedded chunk so retrieval results remain explainable.

result = DocumentChunker().chunk("handbook.pdf")

for c in result.chunks:
    vector_store.upsert(
        id=f"handbook:{c.md_start}",
        text=c.text,
        embedding=embed(c.text),
        metadata={"heading_path": c.heading_path, "kind": c.kind, "format": c.source_format},
    )

hits = vector_store.search(embed("vacation policy"), limit=5)

Troubleshooting

• Scanned PDFs need OCR; DocumentChunker reads a text layer only.
• Encrypted or unsupported files raise a DocumentError; check the original file and format.
• If blazechunk[anydoc] is missing, install the extra before importing document loaders.
• For raw CSV bytes, pass format="csv" because bytes have no extension.

Semantic chunkers