Skip to content

Storage formats

Formats define what writer.append accepts and what a reader returns. Pass a format instance as storage_format to the matching storage constructor. The pack method encodes one appended record; unpack decodes a requested document or stream window.

Built-in formats

Import these classes from guideloom.storage_format or directly from guideloom.

Format Use with Encoding
StreamNumpyFormat StreamStorage A mapping of equal-length, one-dimensional arrays, stored as fields of a structured NumPy dtype.
DocumentPickleFormat DocumentStorage One pickled Python object per document.

StreamNumpyFormat requires a structured dtype whose field names match the input mapping. pack builds a structured array; unpack returns field views of the read array. Multiple fields can hold tokens, loss masks, or other aligned values. The dtype is part of your schema: reopen storage with the same format.

DocumentPickleFormat takes no constructor arguments and uses pickle.dumps and pickle.loads. Use it for trusted data; unpickling can execute code. For numeric workloads, a custom binary format can avoid Python object reconstruction. See encoding tradeoffs.

guideloom.storage_format.StreamNumpyFormat

Bases: StreamStorageFormat[DataT, DataT, DType]

Pack a mapping of equal-length arrays into one structured stream array.

unpack returns those fields and ignores shard segmentation.

__init__(stream_dtype: np.dtype[DType], data_type: type[DataT] | None = None) -> None

Create a format for a structured dtype.

Parameters:

Name Type Description Default
stream_dtype dtype[DType]

Structured dtype. Its field names are the mapping keys.

 required
data_type type[DataT] | None

Mapping type used only to set the DataT type parameter. Ignored at runtime. Defaults to None.

 None

Raises:

Type Description
ValueError

If stream_dtype is not a structured dtype.

pack(data: DataT) -> Vector[DType]

Stack mapping fields into one structured array.

Every field named by stream_dtype must be a one-dimensional array of the same length.

Parameters:

Name Type Description Default
data DataT

Mapping from field name to array.

 required

Returns:

Type Description
Vector[DType]

Vector[DType]: Structured array with stream_dtype.

unpack(stream: Vector[DType], shard_lengths: Vector[np.int64], shards: Vector[np.int64]) -> DataT

Return each structured field as a mapping entry.

Parameters:

Name Type Description Default
stream Vector[DType]

Structured array read from storage.

 required
shard_lengths Vector[int64]

Length of each shard segment. Ignored.

 required
shards Vector[int64]

Shard id of each segment. Ignored.

 required

Returns:

Name Type Description
DataT DataT

Mapping from field name to that field's array.

guideloom.storage_format.DocumentPickleFormat

Bases: DocumentStorageFormat[DocumentT, DocumentT]

Pickle each document with pickle.dumps and pickle.loads.

unpack ignores shard.

pack(data: DocumentT) -> bytes

Pickle a document.

Parameters:

Name Type Description Default
data DocumentT

Document to store.

 required

Returns:

Name Type Description
bytes bytes

Pickled document.

unpack(encoded_data: StorageBufferTypes, shard: int) -> DocumentT

Unpickle a document.

Parameters:

Name Type Description Default
encoded_data StorageBufferTypes

Bytes produced by pack.

 required
shard int

Shard the document was read from. Ignored.

 required

Returns:

Name Type Description
DocumentT DocumentT

Unpickled document.

Custom format interfaces

Implement the interface matching your storage layout. Input and output types can differ: a record appended during preprocessing need not have the same shape as a window read during training.

Interface Encoding contract Example
StreamStorageFormat stream_dtype, pack, and unpack define a fixed-size stream element and its decoded window. Custom stream format
DocumentStorageFormat pack serializes a document; unpack decodes its bytes. Custom document format
StreamWithMetaStorageFormat size counts stream elements; pack yields metadata pairs and returns the stream array; unpack aligns metadata with the read window. Streams with metadata

Streams with metadata require a custom format; there is no built-in metadata codec. StorageFormat[InT, OutT] in guideloom.types is the union of these three format interfaces.

guideloom.types.StreamStorageFormat

Bases: Protocol

__init__(stream_dtype: np.dtype[StreamDType]) -> None

Create the storage format.

Example

StreamStorageFormat(stream_dtype=np.dtype(np.int64))

Parameters:

Name Type Description Default
stream_dtype dtype[StreamDType]

the numpy.dtype of the stream data.

 required

pack(data: InDataT) -> Vector[StreamDType]

Pack the data for the database to store.

Example

return data['tokens']

Parameters:

Name Type Description Default
data InDataT

user provided data structure.

 required

Returns:

Type Description
Vector[StreamDType]

Vector[StreamDType]: The stream data, should be a 1d numpy vector. A structured dtype can
Vector[StreamDType]

be used to store more complex data, such as multiple columns.

unpack(stream: Vector[StreamDType], shard_lengths: Vector[np.int64], shards: Vector[np.int64]) -> OutDataT

Decode a window from concatenated stream segments.

stream is the concatenation of the arrays returned by pack for this read. Segment i is shard_lengths[i] elements long and comes from shards[i]. The segments are in order, so stream[:shard_lengths[0]] comes from shards[0] when the window is non-empty.

Example

shard_ids = np.concat([ np.full((length,), shard, dtype=np.int64) for length, shard in zip(shard_lengths.tolist(), shards.tolist(), strict=True) ]) return {"tokens": stream, "shard": shard_ids}

Parameters:

Name Type Description Default
stream Vector[StreamDType]

Concatenated stream elements for this read.

 required
shard_lengths Vector[int64]

Length of each contiguous shard segment, in read order.

 required
shards Vector[int64]

Shard id of each segment, parallel to shard_lengths.

 required

Returns:

Name Type Description
OutDataT OutDataT

Decoded window.

guideloom.types.StreamWithMetaStorageFormat

Bases: Protocol

__init__(stream_dtype: np.dtype[StreamDType]) -> None

Create the storage format.

Example

StreamWithMetaStorageFormat(stream_dtype=np.dtype(np.int64))

Parameters:

Name Type Description Default
stream_dtype dtype[StreamDType]

the numpy.dtype of the stream data.

 required

size(data: InDataT) -> int

Return the number of stream elements in data.

pack must yield this many pairs.

Parameters:

Name Type Description Default
data InDataT

Record passed to pack.

 required

Returns:

Name Type Description
int int

Number of stream elements.

pack(data: InDataT) -> Generator[tuple[Hashable, UserBufferTypes | None], None, Vector[StreamDType]]

Encode one record as one metadata pair per stream element and a stream array.

Yield exactly size(data) pairs, then return a one-dimensional array with dtype stream_dtype and that same length. Return the array from the generator. Do not yield it. A structured dtype can store several columns in that array.

A key may repeat only inside a contiguous run: [0, 0, 1, 1] is valid and [0, 1, 0] is not. The first pair in a run must include a bytes or memoryview payload. Later pairs in that run may pass None instead of repeating the payload. Keys are compared only within this call and are not stored. Encode document and chunk ids in the payload when readers need them. A key can be any hashable object.

Example 
for chunk_id in data["chunk_ids"]:
    payload = data["concepts"][int(chunk_id)].tobytes()
    yield ((data["document_id"], int(chunk_id)), payload)
return np.asarray(data["tokens"], dtype=self.stream_dtype)

data["chunk_ids"] has one id per stream element, in stream order.

Parameters:

Name Type Description Default
data InDataT

User-provided record.

 required

Returns:

Type Description
Vector[StreamDType]

Vector[StreamDType]: Stream array of length size(data).

Yields:

Type Description
tuple[Hashable, UserBufferTypes | None]

tuple[Hashable, bytes | memoryview | None]: Key and payload for one stream element.

unpack(stream_data: Vector[StreamDType], document_shard: Vector[np.int64], document_index: Vector[np.int32], document_data: Sequence[StorageBufferTypes]) -> OutDataT

Decode a window into stream elements and the metadata payloads they reference.

document_index has one entry per stream element and indexes document_data. document_shard[i] is the shard that stored document_data[i]. Both are remapped for this read. They are not original chunk ids. Decode ids from the payload itself when you need provenance. The shard of the element at token_idx is document_shard[document_index[token_idx]]. Each payload still belongs to a single shard: one append call writes its metadata into the shard it was given, while a read can span several shards.

Example

return { "tokens": stream_data, "annotation_index": document_index, "annotations": [np.frombuffer(payload, dtype=np.int32) for payload in document_data], }

Parameters:

Name Type Description Default
stream_data Vector[StreamDType]

Concatenated stream array for this read. It may span several records, so it is not aligned with one pack call.

 required
document_shard Vector[int64]

Shard of each payload in document_data.

 required
document_index Vector[int32]

Index into document_data for each stream element. document_index.shape == stream_data.shape. The payload for element token_idx is document_data[document_index[token_idx]].

 required
document_data Sequence[StorageBufferTypes]

Payloads for the distinct metadata objects referenced by this read. Decode them in unpack.

 required

Returns:

Name Type Description
OutDataT OutDataT

Decoded window.

guideloom.types.DocumentStorageFormat

Bases: Protocol

pack(data: InDataT) -> UserBufferTypes

Encode one document for storage.

Built-in document storage accepts bytes or memoryview from this method.

Example

return pickle.dumps(data)

Parameters:

Name Type Description Default
data InDataT

User-provided document.

 required

Returns:

Name Type Description
UserBufferTypes UserBufferTypes

Encoded document. Built-in storage accepts bytes or memoryview.

unpack(encoded_data: StorageBufferTypes, shard: int) -> OutDataT

Decode one document.

Storage does not keep a document index. Encode one in pack when you need it. Together with shard, that identifies the shard and the document within that shard.

Example

return pickle.loads(encoded_data)

Parameters:

Name Type Description Default
encoded_data StorageBufferTypes

Bytes produced by pack.

 required
shard int

Shard the document was read from.

 required

Returns:

Name Type Description
OutDataT OutDataT

Decoded document.