quantlab.acquisition.massive.client#
Massive client: one trading day’s file from S3, and REST endpoints into the raw tier.
Massive (Stocks Developer, personal use) serves every SIP trade and its own
minute and day aggregates as one gzipped CSV per data type and trading day
in the S3 bucket flatfiles at files.massive.com (ADR 0030), and
reference data over REST at api.massive.com. MassiveClient makes a
day’s file present and verified in the raw tier
(quantlab.dataset.massive.raw) and stores the condition table beside
it, so a conversion needs no network.
Credentials come from the environment only: MASSIVE_API_KEY (REST, and
the S3 secret unless MASSIVE_S3_SECRET_ACCESS_KEY is set) and
MASSIVE_S3_ACCESS_KEY_ID. A missing one fails when the client is made,
before any transfer, naming the variables. The API key travels in an
Authorization header, never in a URL a log could print.
Two seams carry every transfer, and tests replace both: the object store
(size of a key and read of a byte range; the default signs S3
requests with boto3) and http_get (one REST GET returning an
HttpResponse of quantlab.acquisition._support.http; the default is
rest_transport). The client tells three failures apart:
MassiveNotPublishedError (no file for that day: a holiday, or today
before the vendor publishes), MassiveEntitlementError (the key was
refused, or the day is outside the plan’s window) and
MassiveTransportError (the network, throttling included), which alone is
retried after a back-off that doubles on each attempt. Every ranged GET is
conditioned on the ETag its HEAD read: an object replaced mid-download
answers 412 (MassiveObjectChangedError), and the day starts over at once
from a fresh HEAD.
pull keeps one REST endpoint (quantlab.acquisition.massive.rest) in
the raw tier: it pages on the endpoint’s date field from the day after its
watermark, follows every next_url (stripped of any key), writes each page
as the vendor returned it and moves the watermark after each page, so a rerun
asks only for what is new and an interrupted backfill resumes. HTTP 429 and
5xx are retried after the same back-off, or after the Retry-After the
response gives; 401 and 403 raise
MassiveEntitlementError naming the endpoint.
A file is downloaded in streams byte ranges of part_bytes at once, each
written at its offset of <name>.part; a range whose stream breaks is
asked again for only the bytes it still lacks
(quantlab.acquisition._support.ranged_download). The whole file is then
checked against the vendor’s size and decoded to the end of its gzip, and
only then renamed into place. download_days runs over the XNYS sessions
of a date range, oldest first, with several files in flight, and records a
watermark per data type, so an interrupted run resumes where it stopped.
One file is fetched at a time, with all streams, while up to
files - 1 earlier ones are being verified: decoding a day of trades
takes a core about twice as long as fetching it (docs/user-guide/massive.md lists
the measured rates).
Examples#
Needs the credentials and the network:
client = MassiveClient()
client.condition_table("/data/quantlab/downloads")
client.pull("splits", "/data/quantlab/downloads")
client.download_day("trades", date(2024, 11, 29), "/data/quantlab/downloads")
for day in client.download_days("minute_aggs", date(2016, 10, 11), date(2016, 12, 30),
"/data/quantlab/downloads"):
print(day.day, day.published)
Classes
What |
|
Download Massive's daily files and condition table into the raw tier. |
|
The three secrets, read from the environment; |
|
The S3 seam: the size of a key, and the bytes of a range of it in chunks. |
|
What |
|
|
Functions
Send one REST GET; the client's default |