quantlab.acquisition.massive.refresh#
Keep Massive’s REST raw tier current: every endpoint, each from its watermark.
Each endpoint (quantlab.acquisition.massive.rest.ENDPOINTS) is pulled by
MassiveClient.pull: pages from the day after its watermark, written as the
vendor returned them under <download-dir>/massive/rest/<endpoint>/, the
watermark moved after every page. A first refresh backfills the whole history,
a later one asks only for what is new, and an interrupted one resumes where it
stopped. Every endpoint is tried even when one fails: an endpoint Massive
refuses (MassiveEntitlementError), whose pages kept failing past the
retries (MassiveTransportError) or came back unreadable
(MassiveResponseError) is recorded as failed (RefreshedPart.failure);
any other error, a bug of ours among them, is raised. A missing credential
raises before any endpoint is asked for.
The daily trade, minute and day aggregate files are not refreshed here: their
download runs beside the Trade bar conversion (scripts/massive/convert.py
--download), which converts each day while the next downloads, so a refresh
that downloaded them all first would hold every file on disk before
converting any. refresh_raw_tier is registered on MASSIVE_SOURCE.
Examples#
Needs MASSIVE_API_KEY, MASSIVE_S3_ACCESS_KEY_ID and the network:
from quantlab.acquisition.registry import refresh_raw_tier
from quantlab.acquisition.base import DataSourceRegistry
result = refresh_raw_tier(DataSourceRegistry.get("massive"), "/data/quantlab/downloads")
result.failed # the endpoints not refreshed
Functions
Pull the REST endpoints into the raw tier under |