quantlab.dataset.sharadar.stock.SharadarStockDataset

class quantlab.dataset.sharadar.stock.SharadarStockDataset(config)

Bases: RawMarketDataset

Dense daily panel of raw OHLCV from a Sharadar price table, keyed by permaticker.

Parameters:

dataset_configSharadarDatasetConfig

raw_data_dir_path is <download-dir>/sharadar; table names the price table ("sep"); permatickers optionally restricts the conversion. The ticker-based symbols field must be unset.

Examples

>>> ds = SharadarStockDataset(config)
>>> ds.from_raw_data().save()
>>> SharadarStockDataset(config).panel("2024-01-02", "2024-01-05").symbol.dtype
dtype('int64')

config_cls

The config class used to rebuild this dataset from a saved config.

alias of SharadarDatasetConfig

corrections_path()

Return the corrections report beside the store (<store>.corrections.json).

Examples

>>> SharadarStockDataset(config).corrections_path().name
'sharadar_sep_1d.zarr.corrections.json'

diff(vendor_root=None)

Diff the store against a raw tier, report the differences, write nothing to the store.

The usual raw tier is a fresh full bulk pull into a separate download directory, so the store’s own raw tier stays the one it was built from:

for code in ("sep", "tickers", "actions"):
    client.bulk_table(code, "/data/quantlab/bulk_check")
differences = SharadarStockDataset(config).diff(
    "/data/quantlab/bulk_check/sharadar"
)

Every stored security is compared over the store’s dates (up to the raw tier’s watermark) on CORRECTION_VARIABLES, one calendar year at a time so memory stays bounded by a year of the panel. Securities the raw tier has but the store lacks are not compared. The store, its ledger and both raw tiers are left as they were; the report is rewritten at diff_path() on every run.

Parameters:

vendor_rootstr or Path, optional

The sharadar directory of the raw tier to compare with, holding the price table, TICKERS and ACTIONS. None uses the store’s own raw_data_dir_path (after a bulk pull has replaced it).

Returns:

list of dict

The differences, sorted by date, permaticker and variable. Each names table, permaticker, date and variable, with the stored and vendor values (None for no value).

Raises:

FileNotFoundError

If the store does not exist.

diff_path()

Return the bulk-diff report beside the store (<store>.diff.json).

Examples

>>> SharadarStockDataset(config).diff_path().name
'sharadar_sep_1d.zarr.diff.json'

ticker_lookup()

Return the lookup over the ticker sidecar beside the store.

A backtest labels this dataset’s permatickers through it. The file is read on first use; a store converted before sidecars existed reads as its ids until write_ticker_sidecar is run. None without a store path.

Examples

>>> SharadarStockDataset(config).ticker_lookup()
SharadarTickerLookup('/data/market/sharadar/sharadar_sep_1d/sharadar_sep_1d.zarr.sharadar_tickers.json')

ticker_sidecar_path()

Return the ticker sidecar beside the store (<store>.sharadar_tickers.json).

Examples

>>> SharadarStockDataset(config).ticker_sidecar_path().name
'sharadar_sep_1d.zarr.sharadar_tickers.json'

tradable_bars(prices, fill_column)

Mark which symbols can be traded at each bar: a real fill price and a real trade.

The default (MarketDataset.tradable_bars: a fill price at the bar), less the bars Sharadar carries forward through a halt: during a trading halt it repeats the last close with volume 0 (SIVB from 2023-03-13 to 2023-03-27, before its first OTC print). A bar with volume 0 whose close equals the previous bar’s close had no trade, so it is no fill price, and a holding there stays locked. Close and volume are read from the store, the previous bar included, so the judgement uses nothing later than the bar however few columns or bars prices holds.

Parameters:

pricesxr.Dataset

A panel of this dataset on (timestamp, symbol).

fill_columnstr

The price orders fill at.

Returns:

xr.DataArray

Booleans on the panel’s (timestamp, symbol).

Examples

A halted bar (volume 0, the previous close repeated) is untradable:

ds.tradable_bars(ds.panel("2023-03-10", "2023-03-28"), "adjOpen")

unmapped_path()

Return the report of raw rows left out for want of a permaticker (<store>.unmapped.json).

Written by every derivation that leaves rows out, and deleted by one that leaves none out; see quantlab.dataset.sharadar.permatickers.PermatickerResolver.left_out.

Examples

>>> SharadarStockDataset(config).unmapped_path().name
'sharadar_sep_1d.zarr.unmapped.json'

write_ticker_sidecar()

Write the ticker sidecar of the existing store from the raw tier, and return its path.

The store’s permatickers are named from config.raw_data_dir_path’s TICKERS and ACTIONS, as a conversion names them; the store itself is only read. scripts/sharadar/ticker_sidecar.py runs this for a store converted before sidecars existed.

Returns:

Path

ticker_sidecar_path().

Raises:

FileNotFoundError

If the store does not exist.

Examples

>>> SharadarStockDataset(config).write_ticker_sidecar().name
'sharadar_sep_1d.zarr.sharadar_tickers.json'