Datasets¶
Dataset views provide indexed samples. PyTorch training uses
StatefulDataLoader.from_storage, which creates these views for you. See
the dataset API to open them directly.
| View | Sample at index i |
|---|---|
StreamDataset(storage, sequence_length) |
A complete, non-overlapping window starting at i * sequence_length. Trailing elements that do not fill a window are omitted. |
DocumentDataset(storage) |
The complete document at global document index i. |
Open sync_reader() with with, then use len(reader), reader[i], or
reader[start:stop]. For async access, use async with dataset.async_reader()
and await the indexed result. Negative integer indices raise IndexError;
slices use normal slice semantics. The
dataset reader interfaces document these operations.
guideloom.dataset.StreamDataset
¶
Bases: DatasetAbstract[OutDataT]
__init__(storage: StreamStorageAbstract[Any, OutDataT], sequence_length: int) -> None
¶
Create a dataset view of stream storage, with one item per fixed-length window.
Each observation contains sequence_length stream elements. The opened view length is
total_length // sequence_length, so a trailing partial window is omitted. Open a reader
before indexing it.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
storage
|
StreamStorageAbstract[Any, OutDataT]
|
Storage that provides access to the database. |
required |
sequence_length
|
int
|
Number of stream elements, such as tokens, in each observation. |
required |
sync_reader() -> StreamSyncReaderDataset[OutDataT]
¶
async_reader() -> StreamAsyncReaderDataset[OutDataT]
¶
guideloom.dataset.DocumentDataset
¶
Bases: DatasetAbstract[OutDataT]
__init__(storage: DocumentStorageAbstract[Any, OutDataT]) -> None
¶
Create a dataset view of document storage, with one item per document.
The opened view length is the number of documents. Open a reader before indexing it.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
storage
|
DocumentStorageAbstract[Any, OutDataT]
|
Storage that provides access to the database. |
required |