GuideLoom¶
Flexible data. Reliable training. Built to scale.¶
GuideLoom is randomly-addressable dataset storage and a resumable PyTorch DataLoader for large-scale distributed training. Keep documents, token streams, and span annotations in one data pipeline, with global shuffling, distributed reads, exact checkpointing, and high throughput.
Flexible data formats¶
Choose a layout to match your training samples, with custom encoders and decoders for your data:
- Streams: Concatenate tokens or other fixed-size values into a continuous stream. Read fixed-length windows, with optional aligned fields such as loss masks.
- Documents: Keep variable-length records, such as conversations, separate. Each sample is a complete record; your collator handles padding.
- Streams with metadata: Attach variable-size annotations to spans in a stream, such as concept lists or document IDs. Each window includes the relevant metadata, even when it crosses document boundaries.
Switch formats below to compare how the same documents become training samples. Change the window size or select a token to explore.
The interactive comparison requires JavaScript.
See choose a storage format for a comparison table and examples.
Why GuideLoom?¶
-
Support complex data layouts
Train on complete documents, fixed-length streams, or streams with annotations on token spans. Carry document IDs for traceability or variable-size concept lists into each training window, even across document boundaries.
-
Choose what your data contains
Define your own encoders and decoders for documents and span metadata. Store multiple aligned fields in a stream, such as token IDs, loss masks, and per-token concepts. Change the context window at read time without rewriting the stream.
-
Instant global shuffling
Seeded shuffling samples without replacement across the entire dataset, including shard boundaries. Compute shuffled indices as you need them, without a dataset-sized permutation or a separate shuffled copy for each epoch.
-
Resume exactly, with prefetching
Prepare future batches in the background while checkpointing the position actually delivered to training. With the same dataset, seed, and batch partitioning, resume at the next batch without skipping prefetched data or repeating consumed batches.
-
Scale reads across GPUs and nodes
Each rank reads its assigned samples from shared storage, without communication between loaders or staging whole shards on every node. Use local disks or HPC parallel filesystems; extend the read backend for other storage systems.
-
High performance from the ground up
NumPy buffer views avoid extra decoding copies for compatible numeric formats. Compiled indexing and shuffling release the GIL, while configurable workers and prefetching overlap CPU data preparation with training.
Start with a complete example¶
The quick-start has two scripts: one writes a persistent dataset; the other opens it later and produces PyTorch batches. Preprocessing and training can run at different times and on different machines with access to the same files.
Beginner¶
Follow these in order. You only need basic Python and NumPy arrays; structured arrays are introduced when you need them.
- Choose a storage format: documents, streams, or streams with metadata.
- Write a dataset: prepare shards, including parallel writers with exclusive shard ownership.
- Read into PyTorch: create a PyTorch dataloader, collate samples, and partition loading by rank and world size.
Intermediate¶
- Save and resume: checkpoint state and requirements for preserving sample order.
- NumPy structured arrays: are central to the design of guideloom, so while it's a NumPy API we include a breif tutorial.
- Understand each storage type in detail with examples:
- Document storage - for variable-length records that should be kept seperate.
- streams for concatenating records, such as token-ids in pre-training.
- streams with metadata for attaching custom metadata, to spans of stream content.
Advanced¶
- Performance tips: pread, workers, prefetching, serialization, and device placement.
- Use the dataset API directly.
- Use asynchronous readers.
- Write a custom I/O backend, including HTTP range reads.
- Use another framework, such as JAX.
The API reference documents storage, built-in formats, I/O backends, datasets, and extension interfaces. Read the compatibility policy before upgrading an existing dataset or training job.