quantlab.acquisition.sharadar.refresh#

Keep the Sharadar raw tier current: the daily table plan.

Each table is pulled the way its updates arrive:

  • WHOLE_TABLES whole, in bulk: TICKERS (new listings and ticker changes), INDICATORS (units and the 8-K event codes), SP500 and SF3A (small; SF3A’s newest quarter fills in place) and ACTIONS (the vendor records an action under its effective date, which can be weeks before it publishes it, so a date window would never see it);

  • WINDOW_TABLES as a trailing date window over REST, from the trading_days-th most recent raw date before each table’s watermark through end, so a late vendor correction to a recent day is seen;

  • UPDATED_TABLES as every row the vendor changed since the watermark (by lastupdated), or whole in bulk when that query fails: Sharadar stops a query after 15 seconds, and after a mass re-stamp of lastupdated the bulk zip is the cheaper copy of the changed rows.

A table Sharadar fails (one of TABLE_FAILURES: a refusal, an error status or a connection failure past the retries, a file not the table’s schema) is recorded as failed (RefreshedPart.failure) and the tables after it are still pulled; any other error, a bug of ours or a table never pulled in bulk, is raised. A missing key raises before any table is asked for.

SF3 and SF3B feed no store and are not refreshed; bulk_table pulls them. refresh_raw_tier is registered on SHARADAR_SOURCE; the stores are brought up to date afterwards by quantlab.dataset.sharadar.stores.

Examples#

Needs SHARADAR_API_KEY and the network:

from quantlab.acquisition.registry import refresh_raw_tier
from quantlab.acquisition.base import DataSourceRegistry

result = refresh_raw_tier(DataSourceRegistry.get("sharadar"), "/data/quantlab/downloads")
result.summary()

Functions

refresh_raw_tier

Pull the table plan into the raw tier under <download_dir>/sharadar/.
On this page
Show Source