quantlab.dataset.base.BaseDataset

class quantlab.dataset.base.BaseDataset(config)

Bases: Component, ABC

The read interface shared by every dataset in quantlab.

Reading is a query: panel(start, end, symbols, variables) opens the store lazily per request and holds nothing, so one dataset object serves any number of requests and consumers; calendar, bar_before and bar_after count bars on the same calendar. panel is the one read seam of every leaf, a dataset reading a store (or a panel held in memory) of its own: a leaf says only how it opens its bars (_open_bars, by default the Zarr store at store_path). A view reads other datasets, its inputs(), and keeps no store. Every dataset declares which stores its panel reads (stores); the live day and the data fingerprint ask that declaration.

This class has no build path: a dataset converted from raw files into a store it owns derives from quantlab.dataset.raw.RawDataset (or RawMarketDataset), which adds from_raw_data, save, from_raw_data_chunked and update. A view or a panel held in memory derives from this class or MarketDataset and simply has none of them.

Parameters:

configBaseDatasetConfig

The dataset config. It is not modified: the dataset holds a normalised copy; see the config property.

Examples

Any store of (timestamp, symbol) bars reads through this class alone:

ds = BaseDataset(BaseDatasetConfig(zarr_file_path="data/membership.zarr"))
panel = ds.panel("2020-01-01", "2020-12-31")

The method examples below use DemoDataset, a RawMarketDataset subclass whose _raw_data_to_xr returns six business days of synthetic OHLCV bars for AAA, BBB and CCC, built with a DatasetConfig covering 2024-01-02 to 2024-01-05. minute_config is the same kind of config over a store of two days of 390 one-minute bars, used by the resample examples.

COLUMN_MAP = {}

The store’s own variable names mapped onto the names a merge and an export use: the shared names every factor programs against (open, high, low, close, volume, amount), or a name that keeps two datasets’ variables apart in a merge (SF1’s marketcap as sf1_marketcap). A variable not named here keeps its name.

DEFAULT_RESAMPLE_HOW = None

Each variable’s ResampleMethod for resample(freq) without how; None makes how required. A dataset whose variables each have one right aggregation (bars: first, max, min, last, sum) declares it here.

bar_after(date, n)

Return the bar n bars after date, or the last bar there is.

The forward counterpart of bar_before, on the same calendar: the bars after date are those stamped strictly later, and n=1 is the first of them. n=0 returns date itself. When fewer than n bars follow date the calendar’s last bar is returned, and date itself when none follow, so the result is never before date. A resampled dataset counts its resampled bars. Only the timestamps are read.

Parameters:

datestr, datetime.date or pd.Timestamp

The date to count forward from.

nint

How many bars to count forward; non-negative.

Returns:

pd.Timestamp

The timestamp of that bar.

Raises:

ValueError

If n is negative.

Examples

>>> ds.bar_after("2024-01-05", 1)  # Friday: the bar after is Monday
Timestamp('2024-01-08 00:00:00')
>>> ds.bar_after("2024-02-27", 5)  # the store ends on 2024-02-29
Timestamp('2024-02-29 00:00:00')

bar_before(date, n)

Return the bar n bars before date on this dataset’s calendar.

The calendar is the store’s own timestamps, so gaps such as weekends and holidays are skipped rather than counted. The bars before date are those stamped strictly earlier; n=1 is the last of them. n=0 returns date itself. A resampled dataset counts its resampled bars. Only the timestamps are read.

Parameters:

datestr, datetime.date or pd.Timestamp

The date to count back from.

nint

How many bars to count back; non-negative.

Returns:

pd.Timestamp

The timestamp of that bar.

Raises:

ValueError

If n is negative.

InsufficientHistoryError

If fewer than n bars exist before date; a subclass of ValueError carrying available and requested.

Examples

>>> ds.bar_before("2024-01-08", 1)  # Monday: the bar before is Friday
Timestamp('2024-01-05 00:00:00')
>>> ds.bar_before("2024-01-03", 5)
Traceback (most recent call last):
ValueError: DemoDataset.bar_before(): only 2 bar(s) exist before ...

calendar(start, end)

Return the bars from start to end, both inclusive, timestamps only.

The calendar bar_before and bar_after count on: the store’s own timestamps, or the resampled bars of a resampled dataset. Only the timestamps are read, so it is not a data read a DataRecorder records; panel is.

Parameters:

start, endstr, datetime.date or pd.Timestamp

The range. A date-only end includes every bar of that day.

Returns:

pd.DatetimeIndex

The sorted bar timestamps in the range.

Raises:

ValueError

If start is after end.

Examples

>>> ds.calendar("2024-01-05", "2024-01-08")  # Friday to Monday
DatetimeIndex(['2024-01-05', '2024-01-08'], dtype='datetime64[ns]', freq=None)

property class_name

Return the concrete class name, used in log and error messages.

Examples

>>> ds.class_name
'DemoDataset'

property config

Return the dataset config.

Examples

>>> ds.config.start_date, ds.config.end_date
('2024-01-02', '2024-01-05')
>>> ds.config.name
'__main__.DemoDataset'

copy()

Return a copy with its own config.

The config is deep-copied, so the copy shares no mutable state with this object, not even a kwargs dict.

Examples

>>> other = ds.copy()
>>> other == ds, other.config is ds.config
(True, False)

classmethod from_config(config, run_dir=None)

Rebuild the dataset from its saved config, resolved against run_dir.

resolve_run_config prepares the config first (a FrameDataset resolves a store recorded relative to the run directory), then the component rule rebuilds it (quantlab.core.component.Component).

Parameters:

configdict

The dict get_config() returned.

run_dirstr or os.PathLike, optional

The run directory the config was read from.

Returns:

BaseDataset

The rebuilt dataset.

Examples

>>> DemoDataset.from_config(ds.get_config()) == ds
True

head(n)

Return at most n rows of the store as a LazyFrame.

The store is opened from disk and the config’s date range is not used or changed, so this is a safe way to look at column names.

Parameters:

nint

Maximum number of rows.

Examples

>>> ds.head(2).collect().shape
(2, 8)

inputs()

Return the datasets this dataset’s panel reads through: () for a leaf.

A leaf answers panel from a store (or a panel held in memory) of its own; a view answers it by reading other datasets, its inputs, and keeps no store (a roster cut, a merge, a mapping). A view declares its inputs here; what a dataset merely holds in its config to write its store (MappedTradeBarDataset’s source view) is not an input. store_uses and stores follow the declaration.

Returns:

tuple of BaseDataset

The inputs, in order; empty for a leaf.

Examples

>>> ds.inputs()
()
>>> RosterDataset(ds, membership).inputs() == (ds, membership)
True

own_names(names)

Return this dataset’s own names of the shared variable names.

The inverse of COLUMN_MAP: a shared name some column is renamed to becomes that column, any other name stays. A reader passes the result as panel(variables=...) to read only the columns it exports under the shared names (a KunQuant factor’s data_columns).

Parameters:

namesiterable of str

Shared variable names.

Examples

>>> spot.own_names(["close", "volume"])
['Close', 'Volume']
>>> stock.own_names(["adjClose"])
['adjClose']

panel(start, end, symbols=None, variables=None)

Return the stored panel from start to end, both inclusive.

The store is opened lazily on every call: no variable is loaded until it is used, and the dataset holds nothing afterwards, so one dataset object can answer several requests with different ranges and serve several consumers at once. The config is not changed.

This is the dataset read seam, one implementation for every leaf dataset: a leaf only says how its bars are opened (_open_bars). Inside an open quantlab.runs.record.DataRecorder the request is logged after the cut to symbols and variables, and fingerprinted when the recorder closes, together with the store’s _build_settings; outside one nothing is recorded.

A date-only end such as "2024-01-05" includes every bar of that day. A resampled dataset answers on its resampled bars: from its own store when one has been saved (see store_path), otherwise by resampling the part of the source store the range needs, which loads that part into memory; the source bars read are what is recorded.

Parameters:

startstr, datetime.date or pd.Timestamp

First bar to include.

endstr, datetime.date or pd.Timestamp

Last bar to include.

symbolssequence, optional

Symbol labels to keep, in the order given, of the store’s own label type (integer permatickers on Sharadar stores). None keeps every symbol of the store.

variablessequence of str, optional

Variables to keep, in the order given; only those are read. None keeps every variable.

Returns:

xr.Dataset

The panel on (timestamp, symbol).

Raises:

ValueError

If start is after end.

KeyError

If a requested symbol or variable is not in the store.

FileNotFoundError

If the store does not exist.

Examples

>>> panel = ds.panel("2024-01-03", "2024-01-04", symbols=["BBB"])
>>> dict(panel.sizes)
{'timestamp': 2, 'symbol': 1}
>>> dict(ds.panel("2024-01-02", "2024-01-05").sizes)  # same object
{'timestamp': 4, 'symbol': 3}
>>> list(ds.panel("2024-01-02", "2024-01-05", variables=["close"]).data_vars)
['close']
>>> ds.panel("2024-01-02", "2024-01-05", symbols=["ZZZ"])
Traceback (most recent call last):
KeyError: "DemoDataset.panel(): symbol(s) ['ZZZ'] are not in the store (1 of 1 requested)."

persist_with_run(run_dir, store)

Write what a backtest run directory needs to rebuild this dataset.

A backtest run directory is written calling this once per dataset found anywhere in the backtester’s component tree, with store the path, relative to run_dir, where the run keeps a copy of it (named by the dataset’s component path); a dataset found again is not asked again. The return value is the config config.json records for the dataset, or None to record get_config() as it is. A dataset read from a project store needs nothing written: the store outlives the run, so the default writes nothing and returns None. A dataset whose panel belongs to no store (FrameDataset) writes a copy into run_dir and returns a config reading it.

Parameters:

run_dirPath

The run directory being written.

storestr

Where the run would keep a copy, relative to run_dir.

Returns:

dict or None

The config to record, or None for get_config().

Examples

>>> import tempfile
>>> from pathlib import Path
>>> from quantlab.dataset.config import DatasetConfig
>>> from quantlab.dataset.stock import StockDataset
>>> ds = StockDataset(DatasetConfig(
...     zarr_file_path="data/us_equity/1d/us_all.zarr",
...     raw_data_dir_path="downloads/us_equity/1d/us_all/tiingo",
...     market="us_equity", frequency="1d", vendor="tiingo",
... ))
>>> run_dir = Path(tempfile.mkdtemp())
>>> ds.persist_with_run(run_dir, "copies/prices.zarr") is None
True
>>> list(run_dir.iterdir())
[]

resample(freq, how=None)

Return a copy of this dataset whose panel is resampled onto freq.

The copy’s config carries resample_freq=freq and resample_how=how; panel() and bar_before() on it answer on the resampled bars, from the resampled store beside the source (store_path) once one is saved, otherwise by resampling the source store on read. This dataset is not changed. A view is not resampled: resample its inputs, then wrap them.

Parameters:

freqstr

A ResampleFrequency token, coarser than the panel’s bars.

howdict[str, str] or str, optional

One ResampleMethod for every variable, or a {variable: method} dict naming every variable. None uses DEFAULT_RESAMPLE_HOW.

Returns:

Self

A new dataset of the same class.

Raises:

ValueError

If the dataset is a view, if freq or a method is not a known token, or if how is omitted and the dataset declares no DEFAULT_RESAMPLE_HOW.

Examples

>>> minute = DemoDataset(minute_config)
>>> daily = minute.resample("1d", {"open": "first", "high": "max",
...                               "low": "min", "close": "last",
...                               "volume": "sum"})
>>> daily.panel("2024-01-02", "2024-01-03").sizes["timestamp"]
2
>>> minute.panel("2024-01-02", "2024-01-03").sizes["timestamp"]
780

A dataset that declares DEFAULT_RESAMPLE_HOW needs no how:

hourly = MassiveTradeBarDataset(config).resample("1h")

resample_labels(timestamps, freq)

Return the bar each source timestamp belongs to when resampled.

How this dataset cuts its bars onto freq: its own resample, a factor computed on it (Factor.resample) and a view over it all cut them here, so they agree. The default floors each timestamp on the UTC clock, labelling a bar at its start; that suits bars stamped at their open time and daily stores stamped at midnight. A dataset whose bars follow trading sessions, or are labelled at their end, overrides this (MassiveTradeBarDataset and NbboPanelDataset cut by session); a view answers as its inputs do.

Parameters:

timestampsnp.ndarray

The panel’s timestamps.

freqstr

A ResampleFrequency token.

Returns:

np.ndarray

One target timestamp per source timestamp.

Examples

>>> bars = pd.to_datetime(["2024-01-02 09:00", "2024-01-02 15:00", "2024-01-03 10:00"])
>>> ds.resample_labels(bars.values, "1d")
array(['2024-01-02T00:00:00.000000', '2024-01-02T00:00:00.000000',
       '2024-01-03T00:00:00.000000'], dtype='datetime64[us]')

classmethod resolve_run_config(config, run_dir)

Return a config config.json recorded, made usable to rebuild the dataset.

The loader calls this on the class a saved config names before constructing it, with run_dir the run directory the config was read from (None when the caller gave none). It is the inverse of persist_with_run: the default returns config unchanged, since a store path is used as written; FrameDataset resolves a store named relative to the run directory. config is not modified.

Parameters:

configdict

The recorded config of a dataset of this class.

run_dirPath or None

The run directory the config was read from.

Returns:

dict

The config to construct the dataset from.

Examples

>>> from quantlab.dataset.stock import StockDataset
>>> StockDataset.resolve_run_config({"zarr_file_path": "prices.zarr"}, None)
{'zarr_file_path': 'prices.zarr'}

shared_name_map(names)

Return the part of COLUMN_MAP that applies to names.

Parameters:

namesiterable of str

Variable or column names, such as a panel’s data_vars or a LazyFrame’s columns.

Examples

>>> spot.shared_name_map(["timestamp", "Close", "Volume"])
{'Close': 'close', 'Volume': 'volume'}

property store_path

Return the Zarr store this dataset keeps its own bars in, or None.

This is config.zarr_file_path for a dataset that is not resampled. A resampled dataset uses a store beside it with _resample_<freq> added to the name, so the resampled panel never overwrites the source bars; until that store is saved, its requests read the source store (see stores). A view keeps no store of its own: None.

Examples

>>> ds.store_path
'data/klines.zarr'
>>> ds.resample("1d", "last").store_path
'data/klines_resample_1d.zarr'
>>> RosterDataset(ds, membership).store_path is None
True

store_uses()

Return the stores this dataset uses itself: a leaf reads its store.

A leaf reads one store ("reads"): store_path, or the source store (config.zarr_file_path) while a resampled dataset resamples on read; a panel held in memory reads no store (a "reads" use without a path). A resampled dataset’s store is written from the source store, so that is a "source" use as well. A view uses no store itself: its inputs declare theirs. A component the dataset holds that is not one of its inputs (MappedTradeBarDataset’s source view, the ticker map of a PermatickerTradeBarDataset) is one it is only built from (BuiltFrom), so every store under it is a source, not read on a day. See Component.store_uses; nothing is opened.

Returns:

tuple of StoreUse or BuiltFrom

Examples

>>> ds.store_uses()
(StoreUse(path='data/klines.zarr', relation='reads', handle=None),)
>>> RosterDataset(ds, membership).store_uses()   # its inputs declare theirs
()

stored_symbols()

Return the symbol axis of the dataset’s store, in axis order.

Only the coordinate is read, never a data variable, and nothing is held on the dataset afterwards. The labels keep the store’s own type.

Raises:

ValueError

If the store at store_path does not exist.

Examples

>>> ds.stored_symbols()
['AAA', 'BBB', 'CCC']

stores()

Return the stores this dataset’s panel reads, in order.

The "reads" uses of the tree under the dataset, itself included (walk_store_uses over store_uses): a leaf’s store, or the source store while a resampled dataset resamples on read; a view’s inputs’ stores, each listed once; none for a panel held in memory. The read seam records the store it names, and the live day (quantlab.backtest.live.input_stores) checks these stores hold the bar it predicts. Nothing is opened.

Returns:

list of str

The store paths.

Examples

>>> ds.stores()
['data/klines.zarr']
>>> ds.resample("1d", "last").stores()  # no resampled store saved yet
['data/klines.zarr']
>>> RosterDataset(ds, membership).stores()
['data/klines.zarr', 'data/membership.zarr']

ticker_lookup()

Return the lookup that names this dataset’s symbols, or None.

A backtester and a membership report label symbols through it (see BaseBacktester.ticker_lookup). A dataset whose vendor records tickers beside its store overrides this (a CRSP store returns a CrspTickerLookup over its .crsp_tickers.json sidecar, a Sharadar price store a SharadarTickerLookup over its .sharadar_tickers.json). The default is None: no lookup applies, and the symbols are shown as they are.

Returns:

TickerLookup or None

The lookup, or None.

Examples

>>> from quantlab.dataset.config import DatasetConfig
>>> from quantlab.dataset.stock import StockDataset
>>> StockDataset(DatasetConfig(
...     zarr_file_path="data/us_equity/1d/us_all.zarr",
...     raw_data_dir_path="downloads/us_equity/1d/us_all/tiingo",
...     market="us_equity", frequency="1d", vendor="tiingo",
... )).ticker_lookup() is None
True

to_shared_names(panel)

Return panel with its variables renamed by COLUMN_MAP.

Names of COLUMN_MAP the panel does not hold are skipped. This is the renaming to_kunquant applies, and the one a merge applies to each of its inputs.

Parameters:

panelxr.Dataset

A panel of this dataset.

Returns:

xr.Dataset

The same panel under the shared names.

Examples

>>> spot = SpotKlineDataset(spot_config)   # Binance's Title-Case names
>>> list(spot.to_shared_names(spot.panel("2024-01-02", "2024-01-05")).data_vars)
['open', 'high', 'low', 'close', 'volume', 'amount']