Document API Reference

Every class, method and parameter in blazechunk.loaders.

DocumentChunker

Turns documents into structure-aware chunks. It is deliberately not a BaseChunker subclass: text chunkers map str to chunks, while this maps files to a ChunkedDocument.

DocumentChunker(
    chunker=None,
    *,
    table_chunker=TableChunker(chunk_size=3),
    code_chunker=CodeChunker(chunk_size=2048),
    respect_headings=True,
    merge_small_segments=True,
    min_chunk_size=256,
    max_concurrency=8,
)

Parameter	Type	Default	Description
chunker	BaseChunker \| None	RecursiveChunker(chunk_size=2048)	Chunker for prose; positional or keyword.
table_chunker	BaseChunker \| None	TableChunker(chunk_size=3)	Chunker for tables; None routes tables to chunker.
code_chunker	BaseChunker \| None	CodeChunker(chunk_size=2048)	Chunker for fenced code; None routes code to chunker.
respect_headings	bool	True	Prevent merges sideways across heading boundaries.
merge_small_segments	bool	True	Combine undersized neighbours.
min_chunk_size	int	256	Minimum segment size before looking for a merge partner.
max_concurrency	int \| None	8	Default async document conversion bound.

Note

Passing None explicitly for table or code routing disables that route; omitting the argument uses its default. min_chunk_size < 0 and max_concurrency < 1 raise ValueError.

Methods

loader.chunk(source, *, format=None) -> ChunkedDocument
loader.chunk_batch(sources, *, format=None, on_error="raise") -> list[ChunkedDocument]
loader.chunk_markdown(markdown, *, format="md") -> ChunkedDocument
loader(source, *, format=None) -> ChunkedDocument

source accepts str, os.PathLike[str], bytes, or bytearray.
format is required for raw CSV bytes and invalid names raise ValueError.
chunk_markdown needs no anydoc install and preserves the same structural routing.

on_error

Value	Behaviour
"raise"	Propagate the first DocumentError.
"skip"	Leave failing documents out; the list is shorter.
"collect"	Keep positions aligned with the DocumentError in the failing slot.

results = loader.chunk_batch(paths, on_error="collect")

for path, result in zip(paths, results):
    if isinstance(result, DocumentError):
        log.warning("skipped %s: %s", path, result)
        continue
    index(result.chunks)

ChunkedDocument

The result of chunking one document. Markdown returns alongside chunks because offsets index into it.

Attribute	Type	Description
chunks	list[DocumentChunk]	Chunks in document order.
markdown	str	Converted document; every offset indexes into this.
format	str	Format read from, such as pdf.
warnings	list[str]	Non-fatal notes.
markdown_bytes	bytes	Property containing Markdown as UTF-8.

Supports len(), iteration, and integer indexing.

DocumentChunk

Attribute	Type	Description
text	str	Chunk text.
md_start / md_end	int	UTF-8 byte offsets into ChunkedDocument.markdown.
kind	str	prose, table, code, list, quote, heading or rule.
heading_path	tuple[str, ...]	Enclosing headings, outermost first.
lang	str \| None	Fenced code info string, otherwise None.
source_format	str	Source document format.
token_count	int	Tokens counted by the producing chunker.
is_exact	bool	Whether text is a verbatim Markdown slice.

Tip

Offsets are bytes, not code points. Slice result.markdown_bytes or multibyte characters can misalign the result.

Errors

DocumentError
├── UnsupportedDocument
│   └── ScannedDocumentError
├── MalformedDocument      (.part)
├── EncryptedDocument
└── DocumentResourceLimit  (.limit)

Exception	Raised when
DocumentError	Base class for every load failure.
UnsupportedDocument	Unknown format or no extractable text.
ScannedDocumentError	PDF has no text layer; subclasses UnsupportedDocument.
MalformedDocument	Structurally unusable or missing a required part.
EncryptedDocument	Encrypted or password-protected.
DocumentResourceLimit	Parser safety limit crossed.
OSError	File could not be read; deliberately not wrapped.

Constants

Name	Contents
SUPPORTED_FORMATS	frozenset of csv, doc, docx, epub, odp, ods, odt, pdf, ppt, pptx, rtf, xlsx.
PASSTHROUGH_FORMATS	frozenset of md, markdown, txt, text.