quantlab.dataset.merged.MergedDataset

class quantlab.dataset.merged.MergedDataset(config)

Bases: MarketDataset

A read-only view merging several datasets into one panel.

panel(start, end, symbols) asks every input for the same range, renames each to the shared variable names, checks that no cell holds a value in two inputs, and outer-joins them. bar_before counts bars on the union of the inputs’ calendars. The merged view holds no store and no data, so it has no build path; its inputs are its inputs() and its stores() theirs. It is not resampled (resample refuses a view): build, save or resample its inputs instead.

Parameters:

configMergedDatasetConfig or sequence of BaseDataset

The datasets to merge, in order, or a config holding them.

Raises:

ValueError

If no dataset is given, or an input is not a dataset.

Examples

index holds S&P 500 members, etf holds SPY, both daily:

>>> merged = MergedDataset([index, etf])
>>> panel = merged.panel("2024-01-02", "2024-01-31")
>>> panel.sizes["symbol"] == index.panel("2024-01-02", "2024-01-31").sizes["symbol"] + 1
True

config_cls

The config class used to rebuild this dataset from a saved config.

alias of MergedDatasetConfig

copy()

Return a merged view over copies of the inputs.

Examples

>>> other = merged.copy()
>>> other == merged, other.datasets[0] is merged.datasets[0]
(True, False)

property datasets

The merged datasets, in order.

Examples

>>> MergedDataset([index, etf]).datasets == (index, etf)
True

head(n)

Return up to n rows of every input, under the shared names.

The rows are stacked, not merged, so a (timestamp, symbol) pair may repeat; the frame is meant for reading column names and dtypes.

Examples

>>> sorted(MergedDataset([prices, quotes]).head(2).collect_schema().names())
['amount', 'close', 'high', 'low', 'open', 'symbol', 'timestamp', 'volume']

inputs()

Return the merged datasets, every one read by panel.

Examples

>>> MergedDataset([index, etf]).inputs() == (index, etf)
True

own_names(names)

Return names as they are: the merged panel holds the shared names.

Examples

>>> MergedDataset([spot, held]).own_names(["close"])
['close']

panel(start, end, symbols=None, variables=None)

Return the merged panel from start to end, both inclusive.

Every input is asked for the range (and for those of symbols it holds), renamed to the shared names and outer-joined. The panel spans the union of the inputs’ timestamps and symbols, NaN where an input has no value. Unlike a single dataset’s panel it is loaded into memory, because finding a cell held twice reads every value.

The view records nothing in an open DataRecorder: each input does, asked only for its own names of variables.

Parameters:

start, endstr, datetime.date or pd.Timestamp

The range to return. A date-only end includes every bar of that day.

symbolssequence, optional

Symbol labels to keep, in the order given. None keeps every symbol of every input.

variablessequence of str, optional

Shared variable names to keep, in the order given. Each input reads only those it holds, and an input holding none is not read. None keeps every variable.

Returns:

xr.Dataset

The panel on (timestamp, symbol).

Raises:

ValueError

If start is after end, the inputs have different bar spacing, or a cell holds a value in more than one input; the message names the variable and both inputs.

KeyError

If a requested symbol or variable is in no input.

Examples

>>> dict(MergedDataset([prices, quotes]).panel("2024-01-02", "2024-01-05").sizes)
{'timestamp': 4, 'symbol': 3}
>>> MergedDataset([index, index_copy]).panel("2024-01-02", "2024-01-05")
Traceback (most recent call last):
ValueError: MergedDataset: variable 'close' holds a value in both ...

resample_labels(timestamps, freq)

Return the bar each timestamp belongs to, as every input cuts bars.

Raises:

ValueError

If two inputs cut the timestamps into different bars (a UTC clock and trading sessions, say).

stored_symbols()

Return the union of the inputs’ symbol axes, in the merged panel’s order.

Only the inputs’ symbol coordinates are read, never a data variable.

Examples

index holds AAPL and MSFT, etf holds SPY:

>>> MergedDataset([index, etf]).stored_symbols()
['AAPL', 'MSFT', 'SPY']

ticker_lookup()

Return the lookup naming the merged symbols through the inputs’ lookups.

The view has no store and so no sidecar of its own: each symbol is named by the first input, in order, whose lookup knows it, and by its id when none does. A merged CRSP index and ETF, or a bad-print masked view of a CRSP store, is labelled as its stores are.

Returns:

TickerLookup or None

The single input lookup when only one input names one, a lookup over all of them when several do, None when none does.

Examples

index and etf are CRSP datasets whose ticker sidecars name PERMNO 14593 AAPL and PERMNO 84398 SPY:

>>> MergedDataset([index, etf]).ticker_lookup().label([14593, 84398], date(2024, 1, 2))
['AAPL', 'SPY']

tradable_bars(prices, fill_column)

Mark which symbols can be traded at each bar: tradable for every input.

Each input market dataset judges this view’s panel by its own rule (MarketDataset.tradable_bars: a real fill price at the bar, plus what the input knows, such as SEP’s halted bars), and a bar is tradable only when every input says so. On a view that masks prices (BadPrintMaskedDataset) a masked fill price is therefore no fill price, and an input’s own rule still holds.

Parameters:

pricesxr.Dataset

A panel of this view on (timestamp, symbol).

fill_columnstr

The price orders fill at.

Returns:

xr.DataArray

Booleans on the panel’s (timestamp, symbol).

Examples

With sep a Sharadar SEP dataset, SIVB’s halted bars (volume 0, the last close repeated) stay untradable through the bad-print mask:

prices = BadPrintMaskedDataset([sep])
prices.tradable_bars(prices.panel("2023-03-10", "2023-03-28"), "adjOpen")