Extension interfaces¶
Import these interfaces from guideloom.types. They support custom storage,
dataset readers, and I/O backends. For codecs, see the
format interfaces. See also
custom I/O backends and
asynchronous reads.
Storage factories¶
Implement one of these factories to provide storage to a dataset view or
StatefulDataLoader.from_storage. Stream offsets count stream elements;
document offsets count whole documents. Reader and writer methods are documented
under Storage.
guideloom.types.StreamStorageAbstract
¶
sync_writer() -> StreamStorageSyncWriterAbstract[InDataT]
abstractmethod
¶
Return a synchronous writer that appends to explicit shards.
Use the writer as a context manager.
Returns:
| Type | Description |
|---|---|
StreamStorageSyncWriterAbstract[InDataT]
|
StreamStorageSyncWriterAbstract[InDataT]: Sync writer instance. |
sync_reader() -> StreamStorageSyncReaderAbstract[OutDataT]
abstractmethod
¶
Return a synchronous reader over the entire dataset.
Use the reader as a context manager.
Returns:
| Type | Description |
|---|---|
StreamStorageSyncReaderAbstract[OutDataT]
|
StreamStorageSyncReaderAbstract[OutDataT]: Sync reader instance. |
async_reader() -> StreamStorageAsyncReaderAbstract[OutDataT]
abstractmethod
¶
Return an asynchronous reader over the entire dataset.
Use the reader as a context manager.
Returns:
| Type | Description |
|---|---|
StreamStorageAsyncReaderAbstract[OutDataT]
|
StreamStorageAsyncReaderAbstract[OutDataT]: Async reader instance. |
guideloom.types.DocumentStorageAbstract
¶
sync_writer() -> DocumentStorageSyncWriterAbstract[InDataT]
abstractmethod
¶
Return a synchronous writer that appends to explicit shards.
Use the writer as a context manager.
Returns:
| Type | Description |
|---|---|
DocumentStorageSyncWriterAbstract[InDataT]
|
DocumentStorageSyncWriterAbstract[InDataT]: Sync writer instance. |
sync_reader() -> DocumentStorageSyncReaderAbstract[OutDataT]
abstractmethod
¶
Return a synchronous reader over the entire dataset.
Use the reader as a context manager.
Returns:
| Type | Description |
|---|---|
DocumentStorageSyncReaderAbstract[OutDataT]
|
DocumentStorageSyncReaderAbstract[OutDataT]: Sync reader instance. |
async_reader() -> DocumentStorageAsyncReaderAbstract[OutDataT]
abstractmethod
¶
Return an asynchronous reader over the entire dataset.
Use the reader as a context manager.
Returns:
| Type | Description |
|---|---|
DocumentStorageAsyncReaderAbstract[OutDataT]
|
DocumentStorageAsyncReaderAbstract[OutDataT]: Async reader instance. |
Dataset interfaces¶
Dataset factories return context-managed readers. Synchronous readers provide
len(reader) and reader[index]; asynchronous readers provide len(reader)
and await reader[index]. Both accept integer indices and slices. See
Datasets for the built-in views.
guideloom.types.DatasetAbstract
¶
sync_reader() -> DatasetSyncReaderAbstract[OutT]
abstractmethod
¶
Return a synchronous reader over the entire dataset.
Use the reader as a context manager.
Returns:
| Type | Description |
|---|---|
DatasetSyncReaderAbstract[OutT]
|
DatasetSyncReaderAbstract[OutT]: Sync reader instance. |
async_reader() -> DatasetAsyncReaderAbstract[OutT]
abstractmethod
¶
Return an asynchronous reader over the entire dataset.
Use the reader as a context manager.
Returns:
| Type | Description |
|---|---|
DatasetAsyncReaderAbstract[OutT]
|
DatasetAsyncReaderAbstract[OutT]: Async reader instance. |
guideloom.types.DatasetSyncReaderAbstract
¶
Bases: Sequence[OutT], SyncContextManager, ABC
__len__() -> int
abstractmethod
¶
Return the number of observations in the opened dataset view.
Calling this before the reader is opened raises RuntimeError.
Returns:
| Name | Type | Description |
|---|---|---|
int |
int
|
Number of observations. |
__getitem__(index: int | slice) -> OutT | Sequence[OutT]
abstractmethod
¶
Return the observation at index.
An integer index must satisfy 0 <= index < len(self). A negative integer raises
IndexError. A slice uses normal slice semantics over the opened view.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
index
|
int | slice
|
Integer position or slice within the opened view. |
required |
Returns:
| Type | Description |
|---|---|
OutT | Sequence[OutT]
|
OutT | Sequence[OutT]: One observation, or a sequence of observations for a slice. |
guideloom.types.DatasetAsyncReaderAbstract
¶
Bases: AsyncContextManager, ABC
__len__() -> int
abstractmethod
¶
Return the number of observations in the opened dataset view.
Calling this before the reader is opened raises RuntimeError.
Returns:
| Name | Type | Description |
|---|---|---|
int |
int
|
Number of observations. |
__getitem__(index: int | slice) -> Awaitable[OutT] | Awaitable[Sequence[OutT]]
abstractmethod
¶
Return an awaitable observation at index.
An integer index must satisfy 0 <= index < len(self). A negative integer raises
IndexError. A slice uses normal slice semantics over the opened view.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
index
|
int | slice
|
Integer position or slice within the opened view. |
required |
Returns:
| Type | Description |
|---|---|
Awaitable[OutT] | Awaitable[Sequence[OutT]]
|
Awaitable[OutT] | Awaitable[Sequence[OutT]]: Awaitable observation, or an awaitable sequence of observations for a slice. |
I/O backend interfaces¶
An IOReaderAbstract returns constructors for readers of individual files.
Those readers take a file path, support explicit open/close operations, and
return the requested byte range. The asynchronous reader also provides
close_sync for cleanup. See built-in I/O backends for implementations.
guideloom.types.IOReaderAbstract
¶
Factory for readers of one file.
PathT is the path type passed to each file reader. It defaults to pathlib.PurePath. Built-in
backends use pathlib.Path because they open local files. A custom backend can keep the default
or require a PurePath subclass.
is_supported() -> bool
abstractmethod
staticmethod
¶
Detects if this I/O interface is supported.
Returns:
| Name | Type | Description |
|---|---|---|
bool |
bool
|
true if supported. |
sync_reader() -> Callable[[PathT], IOSyncReaderAbstract]
abstractmethod
¶
Get the constructor for the sync reader.
Returns:
| Type | Description |
|---|---|
Callable[[PathT], IOSyncReaderAbstract]
|
Callable[[PathT], IOSyncReaderAbstract]: Sync reader constructor. Called with a path of
type |
async_reader() -> Callable[[PathT], IOAsyncReaderAbstract]
abstractmethod
¶
Get the constructor for the async reader.
Returns:
| Type | Description |
|---|---|
Callable[[PathT], IOAsyncReaderAbstract]
|
Callable[[PathT], IOAsyncReaderAbstract]: Async reader constructor. Called with a path of
type |
guideloom.types.IOSyncReaderAbstract
¶
Synchronous reader for one file.
Implementations are constructed with the file path: Reader(path).
open() -> None
abstractmethod
¶
Open the file.
Raises:
| Type | Description |
|---|---|
FileNotFoundError
|
if the path cannot be found. |
close() -> None
abstractmethod
¶
Close the file.
filesize() -> int
abstractmethod
¶
Return the size of the file in bytes.
Returns:
| Name | Type | Description |
|---|---|---|
int |
int
|
Number of bytes. |
read(offset: int, size: int) -> StorageBufferTypes
abstractmethod
¶
Read size bytes starting at offset.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
offset
|
int
|
Byte index to start reading from. A negative offset raises |
required |
size
|
int
|
Number of bytes to read. A short read raises |
required |
Returns:
| Name | Type | Description |
|---|---|---|
StorageBufferTypes |
StorageBufferTypes
|
Bytes read, as a |
guideloom.types.IOAsyncReaderAbstract
¶
Asynchronous reader for one file.
Implementations are constructed with the file path: Reader(path).
open() -> None
abstractmethod
async
¶
Open the file.
Raises:
| Type | Description |
|---|---|
FileNotFoundError
|
if the path cannot be found. |
close() -> None
abstractmethod
async
¶
Close the file.
close_sync() -> None
abstractmethod
¶
Close the file synchronously.
__del__ calls this.
filesize() -> int
abstractmethod
async
¶
Return the size of the file in bytes.
Returns:
| Name | Type | Description |
|---|---|---|
int |
int
|
Number of bytes. |
read(offset: int, size: int) -> StorageBufferTypes
abstractmethod
async
¶
Read size bytes starting at offset.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
offset
|
int
|
Byte index to start reading from. A negative offset raises |
required |
size
|
int
|
Number of bytes to read. A short read raises |
required |
Returns:
| Name | Type | Description |
|---|---|---|
StorageBufferTypes |
StorageBufferTypes
|
Bytes read, as a |
Asynchronous storage readers¶
These readers have the same global offsets and shard metadata as their
synchronous counterparts. Open them
with async with and await read or read_many; num_shards, total_length,
and length(shard) remain synchronous.
guideloom.types.StreamStorageAsyncReaderAbstract
¶
Bases: _StreamStorageSharedReaderAbstract, AsyncContextManager, ABC
read(global_index: int, global_length: int) -> OutDataT
abstractmethod
async
¶
Read global_length stream elements starting at global_index.
global_index is an element offset across all shards and must lie in the dataset. An index
past the end raises IndexError. A negative global_length raises IndexError.
The read continues into the next shard when it passes the end of the current one. Past the
last element it continues at shard 0. A global_length longer than the dataset wraps more
than once.
The result is the value returned by storage_format.unpack. A plain stream format receives
the concatenated elements, the length of each shard segment, and the shard id of each
segment. A metadata format receives the stream plus the metadata payloads for that window.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
global_index
|
int
|
Element offset where the read starts, across all shards. |
required |
global_length
|
int
|
Number of stream elements to read. The span may cross shards. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
OutDataT |
OutDataT
|
Value returned by the storage format. |
guideloom.types.DocumentStorageAsyncReaderAbstract
¶
Bases: _DocumentStorageSharedReaderAbstract, AsyncContextManager, ABC
read(global_index: int) -> OutDataT
abstractmethod
async
¶
Read the document at global_index.
global_index is a document offset across all shards. An index outside the dataset raises
IndexError.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
global_index
|
int
|
Document offset across all shards. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
OutDataT |
OutDataT
|
Value returned by |
read_many(global_indexes: Vector[np.int64]) -> Sequence[OutDataT]
async
¶
Read the documents at global_indexes.
Overrides may read more efficiently than one read call per index. The result matches
await asyncio.gather(*(self.read(int(i)) for i in global_indexes)), including order.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
global_indexes
|
Vector[int64]
|
Document offsets across all shards. |
required |
Returns:
| Type | Description |
|---|---|
Sequence[OutDataT]
|
Sequence[OutDataT]: Values returned by |