quantlab.backend.parquet.PlBackend

class quantlab.backend.parquet.PlBackend

Bases: DataBackend

Parquet-backed storage for a long-format table held as a lazy frame.

data is a polars.LazyFrame produced by scan_parquet, so reads and filters stay lazy until write or get_xarray_dataset collects them. Used for reference tables that are tabular rather than panel shaped.

Examples

>>> table = PlBackend().read("universe.parquet")
>>> frame = table.filter_by_symbol("symbol", ("AAPL",)).get_lazyframe()
>>> frame.collect()

filter_by_date(col, start_date, end_date)

Narrow data in place to rows whose col lies in the range.

Parameters:

colstr

The date column to filter on.

start_datestr

First date to keep, inclusive.

end_datestr

Last date to keep, inclusive.

Examples

>>> table.filter_by_date("timestamp", "2024-01-02", "2024-01-03")
PlBackend()
>>> table.get_lazyframe().collect().height  # two days of two symbols
4

filter_by_symbol(col, symbols)

Narrow data in place to rows whose col is in symbols.

Parameters:

colstr

The column to filter on.

symbolstuple[str, …]

The values to keep.

Examples

>>> table = PlBackend().read("universe.parquet")
>>> table.filter_by_symbol("symbol", ("BBB",))
PlBackend()
>>> table.get_lazyframe().collect()["symbol"].unique().to_list()
['BBB']

get_lazyframe()

Return the held lazy frame.

Examples

>>> table.get_lazyframe().collect().shape
(8, 3)

get_xarray_dataset(indexes=None)

Collect data and convert it to a dataset indexed by indexes.

The named columns become the dataset’s dimensions and every other column becomes a data variable.

Parameters:

indexeslist[str]

The columns to index by. Required: a lazy frame has no dimensions to fall back on.

Raises:

ValueError

If indexes is None.

Examples

>>> ds = table.get_xarray_dataset(["timestamp", "symbol"])
>>> tuple(ds.dims), list(ds.data_vars)
(('timestamp', 'symbol'), ['close'])

head(path, n)

Return at most n rows scanned lazily from path.

Parameters:

pathstr

Path of the Parquet file.

nint

Maximum number of rows to return.

The limit is pushed down into the Parquet reader, and a fresh frame

is returned without touching ``data``. The existence check is done

here because ``scan_parquet`` on a missing file only fails at

collect time.

Raises:

FileNotFoundError

If path does not exist.

Examples

>>> PlBackend().head("universe.parquet", 3).collect().shape
(3, 3)

read(path, **kwargs)

Lazily scan the Parquet file at path into data.

Parameters:

pathstr

Path of the Parquet file.

**kwargs

Accepted for interface compatibility and ignored.

Raises:

FileNotFoundError

If path does not exist.

Examples

>>> table = PlBackend().read("universe.parquet")
>>> type(table.data).__name__
LazyFrame

resample(labels, how)

Aggregate the long-format frame onto the bars labels assigns.

Rows are grouped by their label and by every non-timestamp index column the frame has (symbol, in a panel). how names a method for every other column. The result replaces data lazily; nothing is collected here.

Parameters:

labelspd.Series

Source timestamp to target timestamp; see DataBackend.

howdict[str, str]

Column name to aggregation method, one entry per value column.

Examples

>>> table.resample(labels, {"close": "last"})
>>> table.get_lazyframe().collect().shape
(2, 3)

to_internal(data)

Adopt an in-memory polars.LazyFrame as data.

Parameters:

datapl.LazyFrame

The table to hold.

Examples

>>> PlBackend().to_internal(frame.lazy())
PlBackend()

write(path, **kwargs)

Collect data and write it to path as Parquet.

Parameters:

pathstr

Path of the Parquet file to write.

**kwargs

Passed through to polars.DataFrame.write_parquet.

Examples

>>> PlBackend().to_internal(frame.lazy()).write("universe.parquet")
PlBackend()