quantlab.dataset.memory.FrameDataset

class quantlab.dataset.memory.FrameDataset(data, *, columns=None)

Bases: MarketDataset

A market dataset whose panel is held in memory.

Built from a long pandas or polars frame (one row per timestamp and symbol) or from an xarray.Dataset panel. The input rules of quantlab.dataset._support.frame apply: a pandas (timestamp, symbol) MultiIndex is reset, columns renames first, a repeated (timestamp, symbol) raises, timezone-aware timestamps become naive UTC, symbols become str and missing cells become NaN. Every other column becomes a variable under its own name, and to_kunquant exports the variables as they are named, so name them as the factor reading them expects (adjClose for the stock factors, close for the crypto ones).

Built from a frame or panel, nothing is read from or written to disk. Built from a FrameDatasetConfig whose zarr_file_path names a store (one to_zarr wrote, for example), the store is read into memory once, at construction, and resampled there when the config carries resample_freq; the store is never written. This is the form quantlab.core.component.rebuild rebuilds from get_config(). Either way there are no raw files and no store of its own, so it has no build path (no from_raw_data, save or update), and a stream-mode factor refuses it, since a stream is fed live bars rather than a held panel. resample() returns a dataset holding the resampled panel, and to_zarr(path) writes the held panel to a new store.

Parameters:

datapandas.DataFrame, polars.DataFrame, xarray.Dataset or FrameDatasetConfig

The bars, as a long frame or a panel on (timestamp, symbol); or a config naming the Zarr store to read them from.

columnsmapping of str to str, optional

Renames the frame’s columns before conversion, {caller_name: name}. Not accepted with a config, whose store is already in the library’s names.

Raises:

TypeError

If data is not a pandas or polars frame, an xarray panel or a FrameDatasetConfig.

ValueError

If timestamp or symbol is missing after renaming, columns names an absent column, a (timestamp, symbol) pair repeats, or data is a config without zarr_file_path or comes with columns.

FileNotFoundError

If the config’s store does not exist.

Examples

>>> import pandas as pd
>>> from quantlab.dataset.memory import FrameDataset
>>> frame = pd.DataFrame({
...     "timestamp": pd.to_datetime(["2024-01-02", "2024-01-02", "2024-01-03"]),
...     "symbol": ["AAA", "BBB", "AAA"],
...     "close": [10.0, 20.0, 11.0],
... })
>>> ds = FrameDataset(frame)
>>> ds.panel("2024-01-02", "2024-01-03")["close"].values
array([[10., 20.],
       [11., nan]])
>>> ds.bar_before("2024-01-03", 1)
Timestamp('2024-01-02 00:00:00')

Read back from a store:

>>> import tempfile
>>> from pathlib import Path
>>> from quantlab.dataset.config import FrameDatasetConfig
>>> path = str(Path(tempfile.mkdtemp()) / "bars.zarr")
>>> _ = ds.to_zarr(path)
>>> FrameDataset(FrameDatasetConfig(zarr_file_path=path)).panel(
...     "2024-01-02", "2024-01-03")["close"].values
array([[10., 20.],
       [11., nan]])

config_cls

The config class of this dataset.

alias of FrameDatasetConfig

copy()

Return a copy with its own config, holding the same panel.

The panel is shared, not duplicated: no method of this dataset changes it.

Examples

>>> import pandas as pd
>>> from quantlab.dataset.memory import FrameDataset
>>> frame = pd.DataFrame({
...     "timestamp": pd.to_datetime(["2024-01-02", "2024-01-02", "2024-01-03"]),
...     "symbol": ["AAA", "BBB", "AAA"], "close": [10.0, 20.0, 11.0]})
>>> ds = FrameDataset(frame)
>>> other = ds.copy()
>>> other == ds, other.config is ds.config
(True, False)

head(n)

Return at most n rows of the held panel as a LazyFrame.

Parameters:

nint

Maximum number of rows.

Examples

>>> import pandas as pd
>>> from quantlab.dataset.memory import FrameDataset
>>> frame = pd.DataFrame({
...     "timestamp": pd.to_datetime(["2024-01-02", "2024-01-02", "2024-01-03"]),
...     "symbol": ["AAA", "BBB", "AAA"], "close": [10.0, 20.0, 11.0]})
>>> ds = FrameDataset(frame)
>>> ds.head(2).collect().columns
['timestamp', 'symbol', 'close']

persist_with_run(run_dir, store)

Write the held panel into run_dir and return a config reading it.

The panel belongs to no project store, so a backtest run directory keeps a copy at store, written by to_zarr. The returned config names it relative to the run directory, so the directory can be moved; resolve_run_config resolves it again when the run is rebuilt.

Parameters:

run_dirPath

The run directory being written.

storestr

Where the run keeps the copy, relative to run_dir.

Returns:

dict

This dataset’s config reading the copy, with its path relative to run_dir.

Raises:

FileExistsError

If the copy exists already.

Examples

>>> import tempfile
>>> from pathlib import Path
>>> import pandas as pd
>>> from quantlab.dataset.memory import FrameDataset
>>> frame = pd.DataFrame({
...     "timestamp": pd.to_datetime(["2024-01-02", "2024-01-02", "2024-01-03"]),
...     "symbol": ["AAA", "BBB", "AAA"], "close": [10.0, 20.0, 11.0]})
>>> run_dir = Path(tempfile.mkdtemp())
>>> FrameDataset(frame).persist_with_run(run_dir, "copies/prices.zarr")["zarr_file_path"]
'copies/prices.zarr'
>>> FrameDataset.resolve_run_config(
...     {"zarr_file_path": "copies/prices.zarr"}, run_dir)["zarr_file_path"] == str(
...     run_dir / "copies" / "prices.zarr")
True

resample(freq, how)

Return a dataset holding this panel resampled onto freq, in memory.

The bars are cut and aggregated as on a Zarr-backed dataset: UTC-clock buckets labelled at their start, one ResampleMethod per variable, NaN cells skipped. Nothing is written or read: there is no sibling store, not even beside a store this dataset was read from, and store_path is None. The copy is still a FrameDataset: it has no build path and a stream-mode factor refuses it. This dataset is not changed.

Parameters:

freqstr

A ResampleFrequency token, coarser than the held bars.

howmapping of str to str, or str

One ResampleMethod for every variable, or a {variable: method} mapping naming every variable.

Returns:

FrameDataset

A new dataset whose config carries resample_freq and resample_how.

Raises:

ValueError

If freq or a method is not a known token, freq is not coarser than the held bars, or how does not name every variable.

Examples

>>> import pandas as pd
>>> from quantlab.dataset.memory import FrameDataset
>>> frame = pd.DataFrame({
...     "timestamp": pd.date_range("2024-01-02", periods=4, freq="12h").repeat(2),
...     "symbol": ["AAA", "BBB"] * 4,
...     "close": [10.0, 20.0, 11.0, 21.0, 12.0, 22.0, 13.0, 23.0],
...     "volume": [1.0] * 8,
... })
>>> daily = FrameDataset(frame).resample("1d", {"close": "last", "volume": "sum"})
>>> panel = daily.panel("2024-01-02", "2024-01-03")
>>> panel["close"].values
array([[11., 21.],
       [13., 23.]])
>>> panel["volume"].values
array([[2., 2.],
       [2., 2.]])
>>> daily.store_path is None
True

classmethod resolve_run_config(config, run_dir)

Return config with a store named relative to run_dir made absolute.

A relative zarr_file_path is one persist_with_run recorded; it is resolved against run_dir and never against the working directory. An absolute path, or none, is left as it is. config is not modified.

Parameters:

configdict

A recorded FrameDataset config.

run_dirPath or None

The run directory the config was read from.

Returns:

dict

The config to construct the dataset from.

Raises:

ValueError

If the path is relative and run_dir is None.

Examples

>>> from quantlab.dataset.memory import FrameDataset
>>> FrameDataset.resolve_run_config(
...     {"zarr_file_path": "copies/prices.zarr"}, "/runs/WeightsVectorBt_1"
... )
{'zarr_file_path': '/runs/WeightsVectorBt_1/copies/prices.zarr'}
>>> FrameDataset.resolve_run_config(
...     {"zarr_file_path": "copies/prices.zarr"}, None)
Traceback (most recent call last):
ValueError: quantlab.dataset.memory.FrameDataset reads the store 'copies/prices.zarr', ...

property store_path

Return the store the panel was read from, or None.

None for a panel handed over at construction, and for a resampled dataset, whose bars are resampled in memory with no store of their own beside the source (ADR 0011, an exception to ADR 0002’s resampled-store cache).

Examples

>>> import pandas as pd
>>> from quantlab.dataset.memory import FrameDataset
>>> frame = pd.DataFrame({
...     "timestamp": pd.to_datetime(["2024-01-02", "2024-01-02", "2024-01-03"]),
...     "symbol": ["AAA", "BBB", "AAA"], "close": [10.0, 20.0, 11.0]})
>>> FrameDataset(frame).store_path is None
True

to_zarr(path)

Write the held panel to a new Zarr store and return a dataset reading it.

The panel is written as held: a resampled dataset writes its resampled bars, and the returned dataset’s config carries no resample fields. This dataset is not changed. A backtest run directory’s copies of held panels are written this way.

Parameters:

pathstr or os.PathLike

The store to create; it must not exist.

Returns:

FrameDataset

A dataset whose config names path, read back from it.

Raises:

FileExistsError

If path exists; a store is never overwritten.

Examples

>>> import tempfile
>>> from pathlib import Path
>>> import pandas as pd
>>> from quantlab.dataset.memory import FrameDataset
>>> frame = pd.DataFrame({
...     "timestamp": pd.to_datetime(["2024-01-02", "2024-01-02", "2024-01-03"]),
...     "symbol": ["AAA", "BBB", "AAA"], "close": [10.0, 20.0, 11.0]})
>>> path = Path(tempfile.mkdtemp()) / "bars.zarr"
>>> on_disk = FrameDataset(frame).to_zarr(path)
>>> on_disk.store_path == str(path)
True
>>> on_disk.panel("2024-01-03", "2024-01-03")["close"].values
array([[11., nan]])
>>> FrameDataset(frame).to_zarr(path)
Traceback (most recent call last):
FileExistsError: FrameDataset.to_zarr(): ... already exists; a store is never overwritten.