quantlab.dataset.massive.mapped_bars#

Stores on the Massive ticker mapped onto permatickers once and kept in a store of their own.

A PermatickerDataset view (quantlab.dataset.massive.permaticker_bars) maps a store’s ticker columns to permatickers through the Ticker map on every read: for each session day it reads that day’s map rows, picks the store’s ticker columns and reorders them. Over years of one-minute bars that is most of the cost of a read. MappedPermatickerDataset does the mapping once: its store holds the view’s panel, the same variables on (timestamp, symbol) with symbol the int64 permaticker, and panel reads the store. There is one per view: MappedTradeBarDataset keeps the Trade bars (massive_permaticker_bars_1m) and MappedDayBarDataset Massive’s Day aggregates (massive_permaticker_day_bars_1d, #353).

The store is the view’s cache, so it is only as current as the Ticker map it was mapped with. A sidecar, <store>.permaticker_map.json, records a digest of each stored day’s mapped rows (ticker, permaticker) and, per day, the number of stored tickers the map could not place (unplaced()): their bars are in no column, never guessed onto one. update() brings the store up to date in two steps:

a stored day whose map rows changed since it was written (a rebuilt map: a rule change, a Sharadar restatement) is mapped again and written over its own bars, in place;
the days the store on the Massive ticker and the Ticker map both hold after the store’s last are mapped and appended, BATCH_DAYS days per write.

A permaticker met for the first time is appended at the end of the symbol axis without rewriting what is stored (ADR 0032); over earlier days it reads as a column without a trade (NaN prices, zero volumes), as the view gives it. panel refuses to return a day whose map rows no longer match the sidecar, so a read never returns bars mapped by a map that has since changed; update() rewrites them.

Examples#

With bars the permaticker view of the whole-market Trade bar store and days that of the Day aggregate store:

mapped = MappedTradeBarDataset(MappedTradeBarsConfig(
    source=bars, zarr_file_path="/data/massive_permaticker_bars_1m.zarr",
))
mapped.update(workers=8)                       # build, or catch up
panel = mapped.panel("2020-03-02", "2020-03-03")
daily = mapped.resample("1d").panel("2020-03-02", "2020-03-06")
mapped_days = MappedDayBarDataset(MappedDayBarsConfig(
    source=days, zarr_file_path="/data/massive_permaticker_day_bars_1d.zarr",
)).update()

Classes

`MappedDayBarDataset`	Massive's Day aggregates on permatickers, adjusted by Massive's own events (#353, #354).
`MappedPermatickerDataset`	A permaticker view's panel, read from a store the view writes; see the module docstring.
`MappedTradeBarDataset`	Trade bars on permatickers, read from a store the view writes; see the module docstring.

Functions

`day_digests`	Return `{day: digest}` of map rows: a hash of each day's mapped `(ticker, permaticker)` pairs.
`map_days`	Return the view's panel of `first`..``last``, the map digests it was mapped with and the unplaced counts.