quantlab.dataset.massive.mapped_bars.MappedDayBarDataset

class quantlab.dataset.massive.mapped_bars.MappedDayBarDataset(config)

Bases: MappedPermatickerDataset

Massive’s Day aggregates on permatickers, adjusted by Massive’s own events (#353, #354).

The store massive_permaticker_day_bars_1d holds the raw open, high, low, close, volume and n_trades of every security the Ticker map places, and SEP’s adjusted variables chained from Massive’s own split and dividend events (ADR 0048):

• divCash and splitFactor, the events of each permaticker’s bar (quantlab.dataset.massive.day_bar_events: Massive’s /stocks/v1 dividends and splits placed by the Ticker map of their day; never historical_adjustment_factor);

• adjOpen, adjHigh, adjLow, adjClose, adjVolume and cumfacshr, by SEP’s rule (quantlab.dataset._support.adjustment), chained append-only: a day appended continues each permaticker from its stored chain and never rewrites a stored bar. A day mapped again because its map rows changed is chained again with every day after it.

A day is appended only once both REST pulls hold it (their watermarks), so its events are known when it is chained. Events left out are counted in <store>.unplaced.json (unplaced_path).

With sep set, update() checks each new day’s adjusted returns against SEP’s store: a permaticker and day whose split or dividend differs from SEP’s, or whose two returns differ by more than check_tolerance plus SEP’s rounding of its closes (to 0.001, to 0.01 below a dollar: SEP_TICKS) on a day both vendors have a bar on and the bar before, is written, with both vendors’ events, to the check file <store>.sep_check.json (check_path, disagreements()). A disagreement points at one vendor’s events; it never stops the build. Days after SEP’s last bar are checked by a later update.

Parameters:

configMappedDayBarsConfig

The view (source, a PermatickerDayBarDataset), the store and SEP.

Examples

With days a PermatickerDayBarDataset and sep SEP’s dataset built beforehand:

mapped = MappedDayBarDataset(MappedDayBarsConfig(source=days, sep=sep, zarr_file_path="mapped.zarr"))
panel = mapped.update().panel("2024-03-04", "2024-03-11")
panel["adjClose"].sel(symbol=101).values.tolist()   # a 2-for-1 split on 2024-03-06
# [100.0, 102.0, 102.0, 104.0, 106.0, 108.0]
[(d["symbol"], d["date"]) for d in mapped.disagreements()]
# [(303, '2024-03-07')]

BATCH_DAYS = 250

Session days mapped and appended per write: a whole-market day is about 0.6 MB, so a year goes in one write.

DEFAULT_RESAMPLE_HOW = {'close': 'last', 'high': 'max', 'low': 'min', 'n_trades': 'sum', 'open': 'first', 'volume': 'sum'}

resample(freq) aggregates each variable as the Day aggregates do.

SOURCE_CLS

alias of PermatickerDayBarDataset

property adjustment_path

<store>.adjustment.json: the last day whose adjusted variables are chained.

Examples

With mapped the MappedDayBarDataset of the class example:

mapped.adjustment_path.name
# 'mapped.zarr.adjustment.json'

property check_path

<store>.sep_check.json: the days where the adjusted return disagrees with SEP’s.

Examples

With mapped the MappedDayBarDataset of the class example:

mapped.check_path.name
# 'mapped.zarr.sep_check.json'

config_cls

The config class used to rebuild this dataset from a saved config.

alias of MappedDayBarsConfig

disagreements()

Return the check file’s disagreements with SEP, by date and permaticker.

Each is {"symbol", "date", "massive_return", "sep_return", "events_differ", "massive_divCash", "sep_divCash", "massive_splitFactor", "sep_splitFactor"}: the two adjusted returns and each vendor’s events as recorded, None for a value that does not exist (a return on a vendor’s first bar, an event the vendor does not record that day); empty without a check file.

Examples

With mapped the MappedDayBarDataset of the class example:

mapped.disagreements()[0]["sep_return"]
# -0.025

property unplaced_path

<store>.unplaced.json: the split and dividend records left out of the store.

Examples

With mapped the MappedDayBarDataset of the class example:

Path(mapped.unplaced_path).name
# 'mapped.zarr.unplaced.json'

update(end=None, workers=1)

Bring the store up to date, chain its adjusted variables and check them against SEP.

After the mapping (MappedPermatickerDataset.update), every day after the last one chained is chained from the stored state each permaticker ended in, and the days after the check file’s last are checked against SEP. A disagreement is recorded, never raised.

Parameters:

endstr, datetime.date or pd.Timestamp, optional

The last day to append; None for the last day every input holds.

workersint, default 1

Processes mapping batches of days ahead of the writes.

Returns:

Self

self, for chaining.

Examples

With mapped the MappedDayBarDataset of the class example:

mapped.update().stored_days()[-1]
# datetime.date(2024, 3, 11)

update_summary()

Return what the daily update reports: the disagreements with SEP and the events left out.

Examples

With mapped the MappedDayBarDataset of the class example:

mapped.update_summary()
# '1 disagreement(s) with SEP, checked through 2024-03-11; 1 event record(s) left out'