API reference¶

Use these pages for constructor arguments, method signatures, and reader contracts. For a complete preprocessing and training workflow, start with the quick-start.

Module	Reference	Purpose
`guideloom.storage`	Storage	Store documents, streams, and streams with metadata; open readers and writers.
`guideloom.storage_format`	Built-in storage formats	Encode array mappings with `StreamNumpyFormat` or documents with `DocumentPickleFormat`.
`guideloom.io`	Built-in I/O backends	Choose seek, memory-mapped, or POSIX positional reads.
`guideloom.dataset`	Datasets	Turn stored data into indexed training samples.
`guideloom.pytorch`	PyTorch DataLoader	Batch, shuffle, prefetch, partition by rank, and checkpoint sample delivery.
`guideloom.types`	Format interfaces and extension interfaces	Implement custom codecs, storage, datasets, or I/O backends.

Storage classes, dataset classes, and built-in formats can also be imported directly from guideloom. StatefulDataLoader requires the torch extra; storage, formats, datasets, and I/O backends are available in the base package.

How the pieces fit¶

A storage format encodes your records and decodes read results. A storage class manages shards and reads documents or stream ranges. An I/O backend reads byte ranges from individual shard files. A dataset maps sample indices to storage reads, and the DataLoader turns those samples into training batches.

Readers and writers returned by storage and dataset factories are context managers: use with for synchronous access and async with for asynchronous access. The reference documents their public interfaces; private implementation classes are internal. See the compatibility policy for release guarantees.