Custom storage¶
Create filesystem storage with the storage constructor, then open reader or writer contexts. The abstract reader interfaces below document the methods returned by these factories; private implementation classes are not part of this reference.
| Storage | Records | Format |
|---|---|---|
StreamStorage |
Fixed-size elements read as windows across shards. | StreamNumpyFormat or a custom StreamStorageFormat. |
StreamWithMetaStorage |
Stream elements with variable-size annotations on spans. | Custom StreamWithMetaStorageFormat. |
DocumentStorage |
Separate variable-length documents. | DocumentPickleFormat or a custom DocumentStorageFormat. |
See storage formats for encoding contracts and
I/O backends for the read_interface option.
Storage constructors¶
guideloom.storage.StreamStorage
¶
Bases: StreamStorageAbstract[InDataT, OutDataT]
__init__(storage_dir: PathT, storage_format: StreamStorageFormat[InDataT, OutDataT, StreamDType], read_interface: IOReaderAbstract[PathT] | None = None) -> None
¶
Create stream storage for a directory of shard files.
storage_dir and read_interface share PathT. Built-in backends require a pathlib.Path.
A custom backend can require pathlib.PurePath or a subclass. When read_interface is
omitted, SeekIO is used and storage_dir must be a pathlib.Path.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
storage_dir
|
PathT
|
Directory that contains the shard files. |
required |
storage_format
|
StreamStorageFormat[InDataT, OutDataT, StreamDType]
|
Encodes and decodes each stream record. |
required |
read_interface
|
IOReaderAbstract[PathT] | None
|
Byte-range reader. Defaults to |
None
|
sync_writer() -> _StreamStorageSyncWriter[InDataT, StreamDType]
¶
Return a synchronous writer for this storage.
Use the writer as a context manager. See StreamStorageSyncWriterAbstract.
Returns:
| Type | Description |
|---|---|
_StreamStorageSyncWriter[InDataT, StreamDType]
|
StreamStorageSyncWriterAbstract[InDataT]: Writer that appends stream elements to shards. |
sync_reader() -> _StreamStorageSyncReader[OutDataT, StreamDType]
¶
Return a synchronous reader over every shard in this storage.
Use the reader as a context manager. See StreamStorageSyncReaderAbstract.
Returns:
| Type | Description |
|---|---|
_StreamStorageSyncReader[OutDataT, StreamDType]
|
StreamStorageSyncReaderAbstract[OutDataT]: Reader over the stored stream. |
async_reader() -> _StreamStorageAsyncReader[OutDataT, StreamDType]
¶
Return an asynchronous reader over every shard in this storage.
Use the reader as a context manager. See StreamStorageAsyncReaderAbstract.
Returns:
| Type | Description |
|---|---|
_StreamStorageAsyncReader[OutDataT, StreamDType]
|
StreamStorageAsyncReaderAbstract[OutDataT]: Reader over the stored stream. |
guideloom.storage.StreamWithMetaStorage
¶
Bases: StreamStorageAbstract[InDataT, OutDataT]
__init__(storage_dir: PathT, storage_format: StreamWithMetaStorageFormat[InDataT, OutDataT, StreamDType], read_interface: IOReaderAbstract[PathT] | None = None) -> None
¶
Create stream-with-metadata storage for a directory of shard files.
Stream elements and their metadata payloads are both stored under storage_dir.
storage_dir and read_interface share PathT. Built-in backends require a pathlib.Path.
A custom backend can require pathlib.PurePath or a subclass. When read_interface is
omitted, SeekIO is used and storage_dir must be a pathlib.Path.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
storage_dir
|
PathT
|
Directory that contains the shard files. |
required |
storage_format
|
StreamWithMetaStorageFormat[InDataT, OutDataT, StreamDType]
|
Encodes and decodes each record. |
required |
read_interface
|
IOReaderAbstract[PathT] | None
|
Byte-range reader. Defaults to |
None
|
sync_writer() -> _StreamWithMetaStorageSyncWriter[InDataT, StreamDType]
¶
Return a synchronous writer for this storage.
Use the writer as a context manager. See StreamStorageSyncWriterAbstract.
Returns:
| Type | Description |
|---|---|
_StreamWithMetaStorageSyncWriter[InDataT, StreamDType]
|
StreamStorageSyncWriterAbstract[InDataT]: Writer that appends stream elements and metadata. |
sync_reader() -> _StreamWithMetaStorageSyncReader[OutDataT, StreamDType]
¶
Return a synchronous reader over every shard in this storage.
Use the reader as a context manager. See StreamStorageSyncReaderAbstract.
Returns:
| Type | Description |
|---|---|
_StreamWithMetaStorageSyncReader[OutDataT, StreamDType]
|
StreamStorageSyncReaderAbstract[OutDataT]: Reader over the stored stream and metadata. |
async_reader() -> _StreamWithMetaStorageAsyncReader[OutDataT, StreamDType]
¶
Return an asynchronous reader over every shard in this storage.
Use the reader as a context manager. See StreamStorageAsyncReaderAbstract.
Returns:
| Type | Description |
|---|---|
_StreamWithMetaStorageAsyncReader[OutDataT, StreamDType]
|
StreamStorageAsyncReaderAbstract[OutDataT]: Reader over the stored stream and metadata. |
guideloom.storage.DocumentStorage
¶
Bases: DocumentStorageAbstract[InDataT, OutDataT]
__init__(storage_dir: PathT, storage_format: DocumentStorageFormat[InDataT, OutDataT], read_interface: IOReaderAbstract[PathT] | None = None) -> None
¶
Create document storage for a directory of shard files.
storage_dir and read_interface share PathT. Built-in backends require a pathlib.Path.
A custom backend can require pathlib.PurePath or a subclass. When read_interface is
omitted, SeekIO is used and storage_dir must be a pathlib.Path.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
storage_dir
|
PathT
|
Directory that contains the shard files. |
required |
storage_format
|
DocumentStorageFormat[InDataT, OutDataT]
|
Encodes and decodes each document. |
required |
read_interface
|
IOReaderAbstract[PathT] | None
|
Byte-range reader. Defaults to |
None
|
sync_writer() -> _DocumentStorageSyncWriter[InDataT]
¶
Return a synchronous writer for this storage.
Use the writer as a context manager. See DocumentStorageSyncWriterAbstract.
Returns:
| Type | Description |
|---|---|
_DocumentStorageSyncWriter[InDataT]
|
DocumentStorageSyncWriterAbstract[InDataT]: Writer that appends documents to shards. |
sync_reader() -> _DocumentStorageSyncReader[OutDataT]
¶
Return a synchronous reader over every shard in this storage.
Use the reader as a context manager. See DocumentStorageSyncReaderAbstract.
Returns:
| Type | Description |
|---|---|
_DocumentStorageSyncReader[OutDataT]
|
DocumentStorageSyncReaderAbstract[OutDataT]: Reader over the stored documents. |
async_reader() -> _DocumentStorageAsyncReader[OutDataT]
¶
Return an asynchronous reader over every shard in this storage.
Use the reader as a context manager. See DocumentStorageAsyncReaderAbstract.
Returns:
| Type | Description |
|---|---|
_DocumentStorageAsyncReader[OutDataT]
|
DocumentStorageAsyncReaderAbstract[OutDataT]: Reader over the stored documents. |
Reader and writer interfaces¶
Use the returned objects in with blocks, or call open() and close()
explicitly. Readers expose num_shards, total_length, and length(shard);
stream lengths count elements and document lengths count records. Writer
length(shard) counts what was appended since that writer was opened.
For awaitable reads, see asynchronous storage readers.
guideloom.types.StreamStorageSyncReaderAbstract
¶
Bases: _StreamStorageSharedReaderAbstract, SyncContextManager, ABC
num_shards: int
abstractmethod
property
¶
The total number of shards in the database.
Returns:
| Name | Type | Description |
|---|---|---|
int |
int
|
number of shards. |
total_length: int
abstractmethod
property
¶
The total number of stream elements (e.g. tokens) in the database.
Returns:
| Name | Type | Description |
|---|---|---|
int |
int
|
number of stream elements. |
open() -> None
abstractmethod
¶
Open this context.
close() -> None
abstractmethod
¶
Close this context.
length(shard: int) -> int
abstractmethod
¶
Return the number of stream elements stored in the shard.
shard must satisfy 0 <= shard < num_shards. An out-of-range shard raises IndexError.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
shard
|
int
|
Shard index. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
int |
int
|
Stream elements in the shard. |
read(global_index: int, global_length: int) -> OutDataT
abstractmethod
¶
Read global_length stream elements starting at global_index.
global_index is an element offset across all shards and must lie in the dataset. An index
past the end raises IndexError. A negative global_length raises IndexError.
The read continues into the next shard when it passes the end of the current one. Past the
last element it continues at shard 0. A global_length longer than the dataset wraps more
than once.
The result is the value returned by storage_format.unpack. A plain stream format receives
the concatenated elements, the length of each shard segment, and the shard id of each
segment. A metadata format receives the stream plus the metadata payloads for that window.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
global_index
|
int
|
Element offset where the read starts, across all shards. |
required |
global_length
|
int
|
Number of stream elements to read. The span may cross shards. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
OutDataT |
OutDataT
|
Value returned by the storage format. |
guideloom.types.StreamStorageSyncWriterAbstract
¶
Bases: SyncContextManager, ABC
open() -> None
abstractmethod
¶
Open this context.
close() -> None
abstractmethod
¶
Close this context.
append(shard: int, data: InDataT) -> None
abstractmethod
¶
Append one encoded record to shard.
The storage format's pack function encodes data. The written length is the number of
stream elements in that record.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
shard
|
int
|
Shard to write. |
required |
data
|
InDataT
|
User data to encode and append. |
required |
length(shard: int) -> int
abstractmethod
¶
Return the stream elements appended to shard since open().
This count is not the on-disk size of an existing shard. A shard that has not been appended
to in this session raises KeyError.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
shard
|
int
|
Shard written during this session. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
int |
int
|
Stream elements appended to |
guideloom.types.DocumentStorageSyncReaderAbstract
¶
Bases: _DocumentStorageSharedReaderAbstract, SyncContextManager, ABC
num_shards: int
abstractmethod
property
¶
The total number of shards in the database.
Returns:
| Name | Type | Description |
|---|---|---|
int |
int
|
number of shards. |
total_length: int
abstractmethod
property
¶
The total number of documents in the database.
Returns:
| Name | Type | Description |
|---|---|---|
int |
int
|
number of documents. |
open() -> None
abstractmethod
¶
Open this context.
close() -> None
abstractmethod
¶
Close this context.
length(shard: int) -> int
abstractmethod
¶
Return the number of documents stored in the shard.
shard must satisfy 0 <= shard < num_shards. An out-of-range shard raises IndexError.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
shard
|
int
|
Shard index. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
int |
int
|
Documents in the shard. |
read(global_index: int) -> OutDataT
abstractmethod
¶
Read the document at global_index.
global_index is a document offset across all shards. An index outside the dataset raises
IndexError.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
global_index
|
int
|
Document offset across all shards. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
OutDataT |
OutDataT
|
Value returned by |
read_many(global_indexes: Vector[np.int64]) -> Sequence[OutDataT]
¶
Read the documents at global_indexes.
Overrides may read more efficiently than one read call per index. The result matches
[self.read(int(i)) for i in global_indexes], including order.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
global_indexes
|
Vector[int64]
|
Document offsets across all shards. |
required |
Returns:
| Type | Description |
|---|---|
Sequence[OutDataT]
|
Sequence[OutDataT]: Values returned by |
guideloom.types.DocumentStorageSyncWriterAbstract
¶
Bases: SyncContextManager, ABC
open() -> None
abstractmethod
¶
Open this context.
close() -> None
abstractmethod
¶
Close this context.
append(shard: int, data: InDataT) -> None
abstractmethod
¶
Append one encoded document to shard.
The storage format's pack function encodes data. Built-in storage accepts bytes or
memoryview from pack.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
shard
|
int
|
Shard to write. |
required |
data
|
InDataT
|
User data to encode and append. |
required |
length(shard: int) -> int
abstractmethod
¶
Return the documents appended to shard since open().
This count is not the on-disk size of an existing shard. A shard that has not been appended
to in this session raises KeyError.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
shard
|
int
|
Shard written during this session. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
int |
int
|
Documents appended to |