quantlab.dataset.sharadar.tables.scan_raw_table

quantlab.dataset.sharadar.tables.scan_raw_table(vendor_root, code, *, annotate=None)

Scan one raw table: its bulk pull overlaid with its later pulls.

TICKERS is read with its history (fold_tickers_history): a row the history holds and the current pull lacks is kept as of its last sighting, unless another security now holds its ticker in that table. The vendor drops a trading security’s row for a day now and then, and a security must not leave a panel, nor its rows their permaticker, for it.

A window pull is a complete copy of the table over its dates, so a row is kept only from the newest file covering its date: the bulk file’s rows of a window’s dates, and an older window’s rows of a newer window’s dates, are left out. An updated pull (updated_file) holds changed rows, so each of its rows replaces the earlier row with the same primary key, and the newest pull wins.

Parameters:

vendor_rootstr or Path

<download-dir>/sharadar.

codestr

The table’s code.

annotatecallable, optional

Called with each raw file and its scan before the files are combined, and returns the scan with a permaticker column (quantlab.dataset.sharadar.permatickers.PermatickerResolver.annotate): each file names its securities by the tickers of its own pull, so they are mapped file by file. An updated pull then replaces the earlier row of the same permaticker and key, so a security renamed between two pulls keeps one row per key (a row no permaticker was found for is keyed by its ticker).

Raises:

FileNotFoundError

If the table’s directory holds no parquet file.

Examples

>>> scan_raw_table("/data/downloads/sharadar", "tickers").collect_schema().names()[:3]
['table', 'permaticker', 'ticker']