quantlab.acquisition.sharadar.client.SharadarClient#

class quantlab.acquisition.sharadar.client.SharadarClient(transport=<function default_transport>, *, max_retries=5, backoff_seconds=2.0, sleep=<built-in function sleep>, download_workers=8, part_bytes=67108864)[source]#

Bases: object

Pull Sharadar tables into the raw tier through one transport function.

Parameters:

transportcallable, default default_transport: Sends one GET: transport(url, params=..., headers=...) returns an HttpResponse (quantlab.acquisition._support.http) and must not follow redirects.
max_retriesint, default 5: Retries after a 429 or 5xx before SharadarHttpError is raised.
backoff_secondsfloat, default 2.0: First wait when the response gives no Retry-After; it doubles on each retry.
sleepcallable, default time.sleep: Called with the seconds to wait between retries.
download_workersint, default 8: Threads downloading byte ranges of a bulk zip at once.
part_bytesint, default 64 MiB: Size of one byte range.

Examples

Tests pass a fake transport, which keeps every request offline:

client = SharadarClient(transport=fake_transport, sleep=lambda s: None)
client.bulk_table("indicators", download_dir)

RETRY_STATUSES = frozenset({429, 500, 502, 503, 504})#: Statuses that are retried after a back-off.

bulk_table(code, download_dir, *, years='full')[source]#

Pull a whole table as a bulk zip and write it as raw parquet.

The parquet replaces any earlier bulk pull of the table and is written only once the whole zip has been read and checked, so an interrupted pull leaves the previous file in place. When the pull was asked for is recorded beside it (tables.BULK_PULL_FILE), and a TICKERS pull also keeps a copy of itself as a snapshot (tables.tickers_snapshot_file): a raw file names securities by the tickers of its own pull, and is later mapped with the snapshot of its run (quantlab.dataset.sharadar.permatickers). Snapshots no raw file maps with any more are deleted.

Parameters:

codestr: The table’s code ("sep", "tickers", "indicators").
download_dirstr or Path: The download root; the table goes under sharadar/<code>/.
yearsstr, default “full”: The history tier of the bulk file: "5", "10" or "full". It must not exceed the subscription.

Returns:

Path: The parquet file written.

Raises:

KeyError: If the code is not a known table; no request is sent.
SharadarEntitlementError: If the key is missing or refused for this table.
SharadarHttpError: On any other error status, or a rate limit past the retries.
SharadarResponseError: If the zip holds other than one CSV, or its header is not the table’s schema (a ValueError).

Examples

Pull the five-year tier (needs the key and the network):

SharadarClient().bulk_table("sep", "/data/quantlab/downloads", years="5")

updated_table(code, download_dir, *, since=None, page_rows=10000)[source]#

Pull every row the vendor changed since a day, and write them as an updated pull.

The rows are those whose lastupdated is on or after since (the table’s watermark by default), in ticker order. Pages are cut at whole tickers: a full page drops its last ticker, and the next request starts from it (ticker.gte), so a ticker’s rows are never split across two pages whatever order the vendor gives them within it. The rows are checked against the schema and written as one updated file (quantlab.dataset.sharadar.tables.updated_file); each replaces the earlier row with the same primary key. Only then is the watermark moved to the day the pull started (US/Eastern), so the next pull asks again for that day: a row changed during the pull is pulled twice rather than missed.

Parameters:

codestr: The code of a table with a lastupdated column and a primary key ("sf1").
download_dirstr or Path: The download root holding sharadar/<code>/, with a bulk pull of the table already in it.
sincestr, date or None, default None: First lastupdated day to pull; None is the watermark.
page_rowsint, default 10000: Rows per request (Sharadar’s limit). One ticker’s changed rows must fit in a page.

Returns:

Path: The updated file written.

Raises:

KeyError: If the code is not a known table.
ValueError: If the table has no lastupdated column or primary key, has no watermark and no since is given, or one ticker fills a page.
SharadarResponseError: If a page’s header is not the table’s schema.
SharadarEntitlementError, SharadarHttpError: As for bulk_table.

Examples

Refresh SF1 each morning (needs the key and the network):

SharadarClient().updated_table("sf1", "/data/quantlab/downloads")

window_table(code, download_dir, *, through=None, trading_days=10, page_rows=10000)[source]#

Pull a table’s recent dates over REST and write them as a window file.

The window runs from the trading_days-th most recent date of the raw table on or before its watermark through through, every calendar day (a weekend answers no row). Each day is asked for separately and paged by page_rows in ticker order, so paging is stable. The rows are checked against the schema, written as one window file (see quantlab.dataset.sharadar.tables.window_file), and only then is the watermark moved to through: a failed pull leaves the raw tier and the watermark as they were.

Parameters:

codestr: A date-keyed table’s code ("sep", "actions").
download_dirstr or Path: The download root holding sharadar/<code>/, with a bulk pull of the table already in it.
throughstr, date or None, default None: Last day of the window; None is today in Sharadar’s time zone (US/Eastern).
trading_daysint, default 10: Raw dates re-pulled before the watermark, so a late vendor correction to a recent day is seen.
page_rowsint, default 10000: Rows per request (Sharadar’s limit).

Returns:

Path: The window file written.

Raises:

KeyError: If the code is not a known table.
ValueError: If the table has no date column, or no raw pull yet.
SharadarEntitlementError, SharadarHttpError: As for bulk_table.
SharadarResponseError: If a page’s header is not the table’s schema.

Examples

Refresh SEP and ACTIONS each morning (needs the key and the network):

client = SharadarClient()
for code in ("sep", "actions"):
    client.window_table(code, "/data/quantlab/downloads")