Massive#

Massive (Stocks Developer) is quantlab’s vendor of intraday US-equity trades and of several REST tables (dividends, news, short interest, filings). It serves every SIP trade, and its own minute and day aggregates, as one gzipped CSV per data type and trading day over a rolling ten-year window. quantlab keeps those files as they are and builds its own Trade bars from the trades.

Massive names everything by ticker, and a ticker is not a security: one ticker can belong to different securities on different days. Massive stores therefore keep the Massive ticker as their symbol axis, and the Ticker map says which Sharadar permaticker traded under each ticker on each day. Everything read as securities goes through it.

Credentials#

Downloading needs the Massive credentials, read only from the environment:

export MASSIVE_API_KEY=<your-api-key>
export MASSIVE_S3_ACCESS_KEY_ID=<your-s3-access-key-id>
# optional; defaults to MASSIVE_API_KEY
export MASSIVE_S3_SECRET_ACCESS_KEY=<your-s3-secret>

A missing variable raises MassiveCredentialError when the client is made, before any transfer. Converting and reading a store need no credential and no network. The Ticker map needs the Sharadar raw tier’s TICKERS, ACTIONS, SEP and SFP (see Sharadar).

The raw tier#

Daily files#

from datetime import date

from quantlab.acquisition.massive.client import MassiveClient

client = MassiveClient()
client.condition_table("/data/quantlab/downloads")
client.download_day("trades", date(2016, 11, 25), "/data/quantlab/downloads")
client.download_day("minute_aggs", date(2016, 11, 25), "/data/quantlab/downloads")

The files land under <download-dir>/massive/:

massive/trades/2016/2016-11-25.csv.gz
massive/minute_aggs/2016/2016-11-25.csv.gz
massive/conditions/conditions_<UTC stamp>.json

A file is fetched in parallel byte ranges, a broken range resumes from the bytes it lacks, and the whole file is checked against the vendor’s size and decoded to the end before it is renamed into place. Every range is requested on the condition that the object still has the ETag its size was read with: a file the vendor replaces mid-download is started over from a fresh size and ETag. A day without a file raises MassiveNotPublishedError (a holiday, or a day the vendor has not published yet); a day outside the plan’s window raises MassiveEntitlementError.

A range of days goes through download_days, which asks only for XNYS sessions and yields one DayDownload per session in date order, so a caller can convert day d while d+1 downloads:

for done in client.download_days("minute_aggs", date(2016, 10, 11), date(2016, 12, 30),
                                 "/data/quantlab/downloads"):
    print(done.day, done.published, done.fetched_bytes, done.seconds)

Each data type’s watermark (massive/<data type>/_watermark.json) moves to each published day while no earlier day of the run is missing, so an interrupted run resumes where it stopped. Raw files are never deleted: Massive serves only a rolling ten years, so a deleted day could not be fetched again.

REST pulls#

The REST endpoints are kept in the same raw tier, one directory per endpoint, by MassiveClient.pull. Each endpoint is a RestEndpoint of quantlab.acquisition.massive.rest.ENDPOINTS:

Date-ranged: splits, dividends, short_interest, short_volume, 8k_disclosures, news, ipos, and the filing tables 10k_sections, 8k_text, risk_factors, form_3, form_4 and 13f.
Snapshots, once a day: float and disclosure_taxonomy.
As of each session: tickers (the reference tickers listed that day), and ticker_events (each composite FIGI’s ticker changes).

from quantlab.acquisition.massive.client import MassiveClient
from quantlab.dataset.massive.raw import read_rest_records, read_rest_watermark

client = MassiveClient()
done = client.pull("dividends", "/data/quantlab/downloads")
print(done.pages, done.rows, done.through)

dividends = read_rest_records("/data/quantlab/downloads/massive", "dividends")
read_rest_watermark("/data/quantlab/downloads/massive", "dividends")

Each page is kept as the vendor returned it, one JSON object per line, under massive/rest/<endpoint>/, with the endpoint’s watermark beside the pages. A pull asks for the days after the watermark, less the endpoint’s overlap (days asked for again for rows the vendor adds or revises late). The watermark moves after every page and never passes yesterday in New York, so a rerun asks only for what is new and an interrupted backfill resumes from the last page it wrote. read_rest_records() keeps each record’s newest copy. The filing tables have no record id, so they are read with key=None (a record is named by all its fields); snapshots are read with read_rest_snapshot(), and the reference tickers with read_rest_as_of().

quantlab.acquisition.massive.refresh.refresh_raw_tier() pulls every endpoint (or those of endpoints=), tries each one even when another fails, and lists the failed ones in its result’s failed. The daily update calls it through quantlab.acquisition.registry.refresh_raw_tier(). The first pull of the reference tickers, the news and the filing text is a long backfill; pass end= to run it in steps.

The API key goes in an Authorization: Bearer header only, never in a URL, a log line or a raw file.

Trade bars#

MassiveTradeBarDataset builds bars from every trade, on the Massive ticker axis. The shared one-minute store is market/massive/massive_trade_bars_1m.

from quantlab.dataset.config import MassiveTradeBarsDatasetConfig
from quantlab.dataset.massive.trade_bars import MassiveTradeBarDataset

config = MassiveTradeBarsDatasetConfig(
    zarr_file_path="/data/quantlab/zarrs/massive_trade_bars_1m.zarr",
    raw_data_dir_path="/data/quantlab/downloads/massive",
    start_date="2016-11-25",
    end_date="2016-11-25",
)
MassiveTradeBarDataset(config).from_raw_data_chunked(granularity="day")
panel = MassiveTradeBarDataset(config).panel("2016-11-25", "2016-11-26")

variable	what it is
`open`, `high`, `low`, `close`	prices of the trades eligible for the matching update rule
`volume`, `n_trades`	size and count of the volume-eligible trades
`dollar_volume`	price times size of the volume-eligible trades; VWAP is `dollar_volume / volume`
`buy_volume`, `sell_volume`	volume signed by the tick rule
`offexchange_volume`	volume reported through a TRF
`oddlot_volume`	volume of trades under 100 shares, fractional ones included
`open_auction_price`, `open_auction_volume`	the opening cross, on the bar that holds it
`close_auction_price`, `close_auction_volume`	the closing cross, on the session’s last bar

Counting. Each trade counts towards high/low, open/close and volume separately, by the SIP’s consolidated update rules for its conditions. Corrected and cancelled trades, and trades with an unknown condition, are dropped.
Time. Bars are cut on sip_timestamp, right-closed and labelled at their end: the bar labelled 14:31 UTC covers trades after 14:30 up to and including 14:31. The session is XNYS regular hours, half days included.
Auctions. The closing auction reaches the SIP after the close, so it is in no OHLCV; it sets close_auction_price and close_auction_volume on the session’s last bar, which therefore holds information published after its label. A daily resample’s close_auction_price is the official close.
Empty bars. A bar without an eligible trade has NaN prices and zero volumes.
Symbols. Every ticker is kept, spelled as the trade file spells it (BRK.A). A new day’s new tickers are appended at the end of the symbol axis without rewriting stored chunks.

Each converted day is checked against Massive’s own minute aggregates (check_minute_bars()); the result goes in the sidecar <store>.massive_stats.json with the day’s trade counts. Differences are reported, never raised.

The store grows one day at a time: update with a config whose range holds the next day appends it, so a backfill and a daily update are the same call. A whole-market day is tens of millions of trades, so convert one day per window.

from dataclasses import replace

day = replace(config, start_date="2016-11-28", end_date="2016-11-28")
MassiveTradeBarDataset(day).update(granularity="day")

The bar interval, session window, roster and counting rules are the store’s identity; a conversion into a store recorded with other settings is refused. Any interval from 1s to 30m converts directly into its own store, and tickers restricts a conversion to a roster so finer bars stay small:

fine = replace(config, zarr_file_path="/data/quantlab/zarrs/massive_trade_bars_1s.zarr",
               bar_interval="1s", tickers=("AAPL", "MSFT"),
               start_date="2016-11-25", end_date="2016-11-25")

Coarser bars come from resample, cut per session and labelled at their end; "1d" is one bar per session:

hourly = MassiveTradeBarDataset(config).resample("1h").panel("2016-11-25", "2016-11-26")

Day aggregates#

MassiveDayBarDataset keeps Massive’s own daily bars (open, high, low, close unadjusted, volume, n_trades) as a store on the Massive ticker axis, market/massive/massive_day_bars_1d:

from quantlab.dataset.config import MassiveDayBarsDatasetConfig
from quantlab.dataset.massive.day_bars import MassiveDayBarDataset

days = MassiveDayBarDataset(MassiveDayBarsDatasetConfig(
    zarr_file_path="/data/quantlab/market/massive/massive_day_bars_1d/massive_day_bars_1d.zarr",
    raw_data_dir_path="/data/quantlab/downloads/massive",
    start_date="2016-10-10",
))
days.update()                                   # build, or append the new days
panel = days.panel("2024-11-26", "2024-11-29")  # symbol: Massive tickers

The Ticker map#

TickerMap says, for each day and each ticker of that day’s day aggregates, which Sharadar permaticker traded under it, or none, with the evidence. It is the only way from Massive tickers to Sharadar’s securities. The shared map is market/massive/massive_ticker_map, one Parquet file per day.

from datetime import date

from quantlab.dataset.config import TickerMapConfig
from quantlab.dataset.massive.ticker_map import TickerMap

ticker_map = TickerMap(TickerMapConfig(
    path="/data/quantlab/market/massive/massive_ticker_map/massive_ticker_map",
    massive_dir="/data/quantlab/downloads/massive",
    sharadar_dir="/data/quantlab/downloads/sharadar",
))
ticker_map.append(date(2020, 3, 2))   # one more day
ticker_map.rebuild()                   # every day of the raw tier again
rows = ticker_map.read("2020-03-02", "2020-03-02")

A row has date, ticker, permaticker (null when unmapped), evidence, reason, both vendors’ closes and rules_version. The rules:

The candidates are every Sharadar holder of the ticker (SEP’s and SFP’s securities, from TICKERS, the ACTIONS ticker changes and relatedtickers).
A candidate matches when its unadjusted close that day is within max(1%, 0.0051) of Massive’s day close.
One match is chosen (price). Several: the one whose span covers the day (price+span), else none.
No match: a candidate whose span covers the day and whose close is off by a ratio of its later splits is chosen (split_corrected), since Sharadar sometimes folds a later split into its unadjusted close.
Otherwise the ticker is unmapped, with the reason.

A day is never mapped before Sharadar’s closes of it are in. rebuild maps every day again, for a change of rules or a Sharadar restatement; the trades never need converting again.

Reading bars as securities#

PermatickerTradeBarDataset reads the Trade bar store through the Ticker map and returns bars on the int64 permaticker axis, a drop-in dataset for factors, models and backtests:

from quantlab.dataset.massive.permaticker_bars import PermatickerTradeBarDataset

bars = PermatickerTradeBarDataset(MassiveTradeBarDataset(config), ticker_map)
panel = bars.panel("2020-03-02", "2020-03-03")                 # symbol: int64 permatickers
hourly = bars.resample("1h").panel("2020-03-02", "2020-03-03")

For each day, an unmapped ticker is dropped, two tickers of one permaticker keep the busier, and a security that changed ticker is one column across the change. The view has no store of its own.

Mapping on every read is most of the cost of reading years of minute bars, so MappedTradeBarDataset maps once and keeps the result in its own store, market/massive/massive_permaticker_bars_1m. Read a universe from it through RosterDataset.

from quantlab.core.store_folder import load_component
from quantlab.dataset.config import MappedTradeBarsConfig
from quantlab.dataset.massive.mapped_bars import MappedTradeBarDataset

mapped = MappedTradeBarDataset(MappedTradeBarsConfig(
    source=bars,   # the PermatickerTradeBarDataset above
    zarr_file_path="/data/quantlab/market/massive/massive_permaticker_bars_1m/massive_permaticker_bars_1m.zarr",
))
mapped.update(workers=16)                              # build, or catch up
panel = mapped.panel("2020-03-02", "2020-03-03")
daily = mapped.resample("1d").panel("2020-03-02", "2020-03-06")

mapped = load_component("/data/quantlab/market/massive/massive_permaticker_bars_1m")   # the shared one

The store is only as current as the map it was mapped with. update first maps again every stored day whose map rows changed, then appends the new days. panel refuses a range holding a day whose map rows changed since it was mapped, so a read never returns bars from an outdated map.

The day aggregates are kept the same way: PermatickerDayBarDataset is the view and MappedDayBarDataset keeps it as market/massive/massive_permaticker_day_bars_1d:

from quantlab.core.store_folder import load_component
from quantlab.dataset.config import MappedDayBarsConfig
from quantlab.dataset.massive.mapped_bars import MappedDayBarDataset
from quantlab.dataset.massive.permaticker_bars import PermatickerDayBarDataset

view = PermatickerDayBarDataset(days, ticker_map)   # the MassiveDayBarDataset above
mapped = MappedDayBarDataset(MappedDayBarsConfig(
    source=view,
    sep=load_component("/data/quantlab/market/sharadar/sharadar_sep_1d"),   # checked against; optional
    zarr_file_path="/data/quantlab/market/massive/massive_permaticker_day_bars_1d/massive_permaticker_day_bars_1d.zarr",
))
mapped.update()                                     # build, or append the new days
panel = mapped.panel("2024-11-26", "2024-11-29")    # symbol: int64 permatickers
mapped.unplaced()                                   # {day: tickers the map could not place}
mapped.disagreements()                              # the check file: days whose adjusted return is not SEP's

This store also carries SEP’s adjusted variables (divCash, splitFactor, adjOpen, adjHigh, adjLow, adjClose, adjVolume, cumfacshr, with SEP’s meaning), chained from Massive’s own splits and dividends, so a reader can swap SEP for Massive’s daily bars by changing one dataset. The chain is append-only: a new event never rewrites a stored bar. With sep set, every update compares each new day’s events and adjusted return with SEP’s, allowing for SEP’s price rounding, and writes the disagreements to <store>.sep_check.json; a disagreement never stops the build. SEP stays the daily price of research, backtests and trading.

REST panels on permatickers#

Six REST tables become daily panels on Sharadar’s permaticker axis and SEP’s trading calendar. Each row is placed through the Ticker map on its own day (an ex-date, a settlement date, a publication day), by TickerPlacement, so a reused ticker sends each row to the security that held it that day. A row the map cannot place is never guessed: it is left out and counted in <store>.unplaced.json.

Dataset	Store	Variables
`MassiveDividendDataset`	`massive_dividends_1d`	`dividend_declared`, `dividend_days_to_ex` (declaration to ex-date), `dividend_cash` (ex-date), `split_ratio`
`MassiveShortInterestDataset`	`massive_short_interest_1d`	`short_interest`, `short_avg_daily_volume`, `days_to_cover`, shown from settlement + 8 trading days
`MassiveShortVolumeDataset`	`massive_short_volume_1d`	`short_volume`, `short_total_volume`, `short_exempt_volume`, `short_volume_ratio` (from 2024-02)
`MassiveNewsDataset`	`massive_news_1d`	`news_count`, `news_positive`, `news_negative`, `news_neutral`
`MassiveDisclosureDataset`	`massive_disclosures_1d`	one boolean per primary category of Massive’s 8-K taxonomy (from 2020-01)
`MassiveIpoDataset`	`massive_ipos_1d`	`ipo_offer_price`, `ipo_offer_size`, `ipo_days_announced`, on the listing day

They share MassiveRestPanel. from_data_root returns the shared panel of <data-dir>/market/massive/<store>/, rebuilt from its component.json, or configured on the shared layout and written there the first time:

from quantlab.dataset.massive.news import MassiveNewsDataset

news = MassiveNewsDataset.from_data_root("/data/quantlab", "/data/quantlab/downloads")
news.update()
panel = news.panel("2024-01-02", "2024-12-31")

A store is built through the last day that SEP, every endpoint it reads and the Ticker map all reach, so a store built day by day equals one built at once. Timing rules that matter for signals:

News counts on the trading day of its publication when published before that session’s actual close (13:00 on an early close), and on the next trading day otherwise.
8-K disclosures show on the first trading day after the filing date, since an 8-K is often filed after the close.
Declared dividends show on the declaration date’s bar even when declared after the close; read them one bar back for a signal formed at the close.
Amounts are as declared: Massive’s split-adjusted amounts and adjustment factors are never read, so a later event never rewrites the panel’s history.

MassiveTenKChangesDataset builds massive_10k_changes_1d from the 10k_sections pull: the similarity of each 10-K’s business and risk-factor sections to the filer’s report of the year before (Cohen, Malloy and Nguyen 2020, “Lazy Prices”). Its records are keyed by CIK, so it maps filers through TICKERS rather than the Ticker map.