quantlab.backend.base.DataBackend

class quantlab.backend.base.DataBackend

Bases: ABC

Contract for a storage medium that holds one tabular or panel dataset.

A backend owns a single in-memory object, exposed as data, and knows how to move it to and from a path, narrow it in place, and convert it to the two exchange formats used across the pipeline: an xarray.Dataset and a polars.LazyFrame. Every mutating method returns self so calls can be chained.

Two details of the contract are easy to get wrong when implementing it. filter_by_date and filter_by_symbol narrow data in place, and every object sharing the backend instance sees the result. head must do the opposite: open the store at the given path, read at most n rows without materialising the whole store, and leave data untouched.

Examples

>>> backend = XrBackend().read("prices.zarr")
>>> panel = backend.filter_by_date("timestamp", "2022-01-01",
...                                "2022-12-31")
>>> ds = panel.get_xarray_dataset(["timestamp", "symbol"])

property data

The object this backend currently holds.

Raises:

AttributeError

If nothing has been loaded yet. Accessing an unpopulated backend fails immediately rather than returning an empty dataset that downstream code would mistake for “no data in this range”.

Examples

>>> XrBackend().to_internal(panel).data is panel
True
>>> XrBackend().data
Traceback (most recent call last):
AttributeError: Please call 'read' or 'to_internal' first.

abstractmethod filter_by_date(col, start_date, end_date)

Narrow data in place to start_date..end_date on col.

Parameters:

colstr

The date column or dimension to filter on.

start_datestr

First date to keep, inclusive.

end_datestr

Last date to keep, inclusive.

Examples

>>> backend.filter_by_date("timestamp", "2024-01-02", "2024-01-03")
XrBackend()
>>> backend.data["timestamp"].values.astype("datetime64[D]")
array(['2024-01-02', '2024-01-03'], dtype='datetime64[D]')

abstractmethod filter_by_symbol(col, symbols)

Narrow data in place to the rows whose col is in symbols.

Parameters:

colstr

The symbol column or dimension to filter on.

symbolstuple[str, …]

The symbols to keep.

Examples

>>> backend.filter_by_symbol("symbol", ("BBB",))
XrBackend()
>>> backend.data["symbol"].values.tolist()
['BBB']

abstractmethod get_lazyframe()

Return the held data as a polars.LazyFrame.

Examples

>>> backend.get_lazyframe().collect().columns
['timestamp', 'symbol', 'close']

abstractmethod get_xarray_dataset(indexes=None)

Return the held data as an xarray.Dataset.

Parameters:

indexeslist[str], optional

The dimensions the returned dataset must be indexed by, in order. None asks for no particular shape.

Examples

>>> backend = XrBackend().to_internal(panel)
>>> tuple(backend.get_xarray_dataset(["timestamp", "symbol"]).dims)
('timestamp', 'symbol')

abstractmethod head(path, n)

Return at most n rows from the store at path.

Implementations must not materialise the whole store, must not touch data, and must raise FileNotFoundError when path does not exist rather than deferring the failure to a later collect.

Parameters:

pathstr

Location of the store.

nint

Maximum number of rows to return.

Examples

>>> XrBackend().head("prices.zarr", 2).collect().shape
(2, 3)

abstractmethod read(path, **kwargs)

Load the store at path into data and return self.

Parameters:

pathstr

Location of the store.

**kwargs

Options for the underlying reader.

Examples

>>> dict(XrBackend().read("prices.zarr").data.sizes)
{'timestamp': 4, 'symbol': 2}

abstractmethod resample(labels, how)

Aggregate data onto the coarser time grid labels describes.

labels maps every source timestamp (its index) to the timestamp of the bar it belongs to (its value); the caller works those out, so the backend knows nothing about clocks or trading sessions. how names one ResampleMethod for every variable. Each variable is reduced over the timestamps sharing a label, NaN cells skipped, and the result replaces data in place, sorted by the new timestamps.

Parameters:

labelspd.Series

Source timestamp to target timestamp, covering every timestamp data holds.

howdict[str, str]

Variable name to aggregation method.

Examples

>>> labels = pd.Series(minutes.floor("D"), index=minutes)
>>> backend.resample(labels, {"close": "last", "volume": "sum"})
>>> dict(backend.data.sizes)
{'timestamp': 2, 'symbol': 3}

abstractmethod to_internal(data)

Adopt an in-memory object as data, bypassing disk.

Parameters:

dataobject

The object to hold, typically an xarray.Dataset or a polars.LazyFrame.

Examples

>>> XrBackend().to_internal(panel).data is panel
True

abstractmethod write(path, **kwargs)

Persist data to path and return self.

Parameters:

pathstr

Location of the store.

**kwargs

Options for the underlying writer.

Examples

>>> XrBackend().to_internal(panel).write("prices.zarr")
XrBackend()