quantlab.acquisition.massive.rest.RestEndpoint#

class quantlab.acquisition.massive.rest.RestEndpoint(name, url, date_field, earliest=datetime.date(1970, 1, 1), params=<factory>, limit=5000, overlap_days=0, key='id', tiebreak='ticker', once_a_day=False, as_of_param=None, ids_from=None, id_field='composite_figi', order_param=False, timestamped=False, window_days=None)[source]#

Bases: object

One REST endpoint pulled into the raw tier.

Parameters:

namestr: The endpoint’s directory under <download-dir>/massive/rest/.
urlstr: The endpoint’s URL, without a query.
date_fieldstr or None: The yyyy-mm-dd (or ISO timestamp) field the pull filters with .gte/.lte and sorts on; None for a snapshot endpoint, pulled whole every time without a watermark.
earliestdate, default 1970-01-01: The first day a pull without a watermark asks for.
paramsdict, default empty: Fixed query parameters of every first page (filters of the endpoint’s own).
limitint, default 5000: Rows per page; the endpoint’s maximum.
overlap_daysint, default 0: Days before the watermark asked for again on every pull, for rows a vendor adds or revises after their day.
keystr or None, default “id”: The field identifying a record (raw.read_rest_records); None when the records have no such field, so a record is identified by all its fields.
tiebreakstr or None, default “ticker”: A second sort field, so pages of one day are in a stable order.
once_a_daybool, default False: For a snapshot endpoint: keep one snapshot a day. A pull on a day (in New York) that already has one asks for nothing, and the watermark is the day of the newest snapshot.
as_of_paramstr or None, default None: For a snapshot endpoint that serves the table as of a past day: the query parameter naming that day (date for the reference tickers). The table is then pulled whole as of every XNYS session after the watermark, from earliest, through yesterday in New York (or end), each session kept apart (raw.read_rest_as_of); the watermark is the last session kept.
ids_fromstr or None, default None: For an endpoint served one id at a time ({id} in url, the ticker events): the as-of endpoint whose tables give the ids. A pull reads that endpoint’s sessions after this one’s watermark and asks for every id (id_field) with no record kept yet, or whose value of the source’s key (the ticker) changed on one of them, against the session before; it writes one record per id (the vendor’s results object) and limit records a page file, and moves the watermark to the last session read once every id is asked. An id the vendor does not know (404) is skipped, and asked again with the next new session.
id_fieldstr, default “composite_figi”: The source records’ field giving an ids_from endpoint’s ids.
order_parambool, default False: Give the order in its own order parameter (sort=<date field>&order=asc), as the /vX/reference IPOs and the news take it, instead of sort=<date field>.asc,<tiebreak>.asc.
timestampedbool, default False: date_field holds a timestamp, not a day (news’ published_utc): a pull bounded by end asks for <date_field>.lt the day after it, so the whole of end is kept.
window_daysint or None, default None: For an endpoint that filters on date_field but will not sort on it (the 10-K sections sort on period_end only): the pull asks one window of this many days at a time, with no sort, and moves the watermark to a window’s last day only once all its pages are in, since its rows come in no date order. An interrupted window is asked again whole.

Examples

>>> RestEndpoint("short_volume", f"{REST_HOST}/stocks/v1/short-volume", "date").name
'short_volume'

property sort#

The sort parameter: the date field ascending, then the tiebreak.

Examples

>>> ENDPOINTS["splits"].sort
'execution_date.asc,ticker.asc'

property sort_params#

The query parameters ordering a date-ranged pull: sort, and order with order_param.

Examples

>>> ENDPOINTS["splits"].sort_params
{'sort': 'execution_date.asc,ticker.asc'}
>>> ENDPOINTS["ipos"].sort_params
{'sort': 'listing_date', 'order': 'asc'}