quantlab.acquisition.massive.raw_tier

The writers of Massive’s raw tier: REST pages and watermarks.

MassiveClient (quantlab.acquisition.massive.client) downloads day files and pulls REST endpoints into <download-dir>/massive/ and records them with these writers. The layout (raw_file, rest_page_file, rest_pending_dir, …) and every reader of it (read_watermark, read_rest_watermark, read_rest_records, rest_frame, …) stay in quantlab.dataset.massive.raw: the dataset layer reads the raw tier, only the acquisition layer writes it.

Both watermarks are dated watermarks (quantlab.utils.watermark) labelled here: a day-file type’s (<massive>/<type>/_watermark.json) holds data_type and through, a REST endpoint’s (<massive>/rest/<endpoint>/_watermark.json) endpoint, field and through; field is "snapshot" for a snapshot endpoint. The client resumes either the day after it (quantlab.utils.watermark.resume_from) and never moves a REST watermark back.

Examples

>>> import tempfile
>>> from datetime import date
>>> from quantlab.dataset.massive.raw import read_rest_watermark
>>> root = tempfile.mkdtemp()
>>> write_rest_watermark(root, "splits", "execution_date", date(2024, 3, 14))
>>> read_rest_watermark(root, "splits")
datetime.date(2024, 3, 14)

Functions

write_day_file_watermark	Record that a data type's downloads are complete through through.
write_rest_page	Write one page's records, one JSON object per line, atomically.
write_rest_watermark	Record that a REST endpoint's rows are pulled through through on field.