quantlab.dataset.massive.raw#

Massive’s raw tier: where the daily files live, and how they are read.

Massive (Stocks Developer) serves every SIP trade of the US market, and its own minute and day aggregates, as one gzipped CSV per data type and trading day in the S3 bucket flatfiles at files.massive.com (ADR 0030). The raw tier keeps those files exactly as downloaded, under the Massive directory of the download root:

<download-dir>/massive/trades/2024/2024-11-29.csv.gz
<download-dir>/massive/minute_aggs/2024/2024-11-29.csv.gz
<download-dir>/massive/trades/_watermark.json
<download-dir>/massive/conditions/conditions_20261009T120000000000.json

and beside them each data type’s watermark (the day a run over several days is complete through) and the trade-condition table, one snapshot per download run.

REST pulls (MassiveClient.pull) keep each page as the vendor returned it, one JSON-lines file per page, under rest/<endpoint>/<YYYY>/ (the pull’s year), named by the pull’s UTC stamp and the page’s number, with the endpoint’s watermark beside them:

<download-dir>/massive/rest/splits/2026/20261016T120000000000_000001.jsonl
<download-dir>/massive/rest/splits/_watermark.json

A row may be in several files (a page re-read past the watermark); read_rest_records keeps each record’s newest copy. A snapshot endpoint (the Float table) has the same layout, each snapshot every page of one pull (rest_snapshots, read_rest_snapshot); a table kept as of each session (the reference tickers) has one directory a session:

<download-dir>/massive/rest/tickers/2024/2024-03-11/20261016T120000000000_000001.jsonl

read by rest_as_of_days and read_rest_as_of. Pages of a snapshot wait in rest/<endpoint>/_pending/ until it is whole; nothing reads it. The client (quantlab.acquisition.massive.client) writes this tree, with the page and watermark writers of quantlab.acquisition.massive.raw_tier, and quantlab.dataset.massive.trade_bars reads it; both take every path from here. This module only reads.

A trade file holds one row per SIP trade, sorted by ticker; its columns are TRADE_COLUMNS. Times are nanoseconds since the Unix epoch, UTC. conditions is a comma-separated list of condition ids (empty for a regular trade), each looked up in the condition table.

Examples#

>>> raw_file("/data/downloads/massive", "trades", date(2024, 11, 29))
PosixPath('/data/downloads/massive/trades/2024/2024-11-29.csv.gz')
>>> s3_key("trades", date(2024, 11, 29))
'us_stocks_sip/trades_v1/2024/11/2024-11-29.csv.gz'

Functions

`conditions_file`	Return the path of the condition-table snapshot pulled at `pulled_at` (UTC).
`latest_conditions`	Return the newest condition-table snapshot of the raw tier.
`raw_days`	Return the trading days whose file of `data_type` is in the raw tier, ascending.
`raw_file`	Return where the raw tier keeps one data type's file for one trading day.
`read_aggregates`	Read one minute- or day-aggregate file, with `AGGREGATE_COLUMNS` types.
`read_conditions`	Return the trade conditions of one snapshot and their consolidated update rules.
`read_rest_as_of`	Return an endpoint's table as of `day`, in the vendor's order, verbatim.
`read_rest_records`	Return one REST endpoint's records, each record's newest copy once.
`read_rest_snapshot`	Return the records of one snapshot, in the vendor's order, verbatim.
`read_rest_watermark`	Return the day a REST endpoint's rows are pulled through, or `None`.
`read_trades`	Read the given columns of one trade file, with `TRADE_COLUMNS` types.
`read_watermark`	Return the day a data type's downloads are complete through, or `None`.
`rest_as_of_days`	Return the days an endpoint's table is kept as of, ascending.
`rest_as_of_dir`	Return the directory of an endpoint's table as of `day` (`RestEndpoint.as_of_param`).
`rest_dir`	Return the directory of one REST endpoint's pages and watermark.
`rest_frame`	Return one REST endpoint's records as a frame of the fields in `schema`.
`rest_page_file`	Return the path of page `page` (from 1) of the pull started at `pulled_at` (UTC).
`rest_page_files`	Return every page file of one REST endpoint, oldest pull first, pages in order.
`rest_pending_dir`	Return where a snapshot's pages wait until the snapshot is whole.
`rest_snapshots`	Return the pull times (UTC) of a snapshot endpoint's whole snapshots, oldest first.
`rest_watermark_file`	Return the path of a REST endpoint's watermark, beside its pages.
`s3_key`	Return the S3 key of one data type's file for one trading day.
`watermark_file`	Return the path of a data type's watermark, in its directory.