quantlab.dataset.crsp.CrspStockDataset

class quantlab.dataset.crsp.CrspStockDataset(dataset_config)

Bases: StockDataset

Dense daily panel built from the CRSP raw files, keyed by PERMNO.

The class reuses StockDataset’s scan of the raw tree (the crsp_daily directory under the vendor root, month= hive partitions, the single-vendor check, date-window filters and has_raw_data). It overrides how the axes and each window’s dense panel are built, because both must use the adjusted derivation (the frame of computed panel variables) rather than the raw rows: the raw symbol column is the PERMNO as a string, and every derived variable is computed here.

Each conversion also writes two sidecars next to the store, the filter report and the ticker table (see filter_report_path and ticker_sidecar_path).

Parameters:

dataset_configCrspDatasetConfig

Must have frequency="1d" and vendor="wrds". The inherited ticker-based symbols field must be unset; use permnos to convert only some securities. reference_dir points to the downloaded CRSP reference tables, security_filter chooses which security types to keep, and roster_universe optionally names an index whose members are kept regardless of the filter.

Examples

Build the store from raw files already downloaded from WRDS (the paths follow the layout the download writes under the data root), then read it back as a panel whose symbol axis is the integer PERMNO:

>>> config = CrspDatasetConfig(
...     zarr_file_path="/data/crsp.zarr",
...     raw_data_dir_path="/data/downloads/us_equity/1d/wrds_crsp/wrds",
...     reference_dir="/data/downloads/us_equity/1d/wrds_crsp/_reference",
...     start_date="2008-01-01",
...     end_date="2020-12-31",
... )
>>> ds = CrspStockDataset(config)
>>> ds.from_raw_data().save()
>>> panel = CrspStockDataset(config).panel("2008-01-01", "2020-12-31")
>>> panel.symbol.values.tolist()
[14593, 80599]
>>> panel["ret"].sel(symbol=80599).to_pandas().dropna().tail(2)
timestamp
2008-09-17   -0.566667
2008-09-18   -0.600000
Name: ret, dtype: float64

DATA_TYPE = 'crsp_daily'

The data type whose directory under the vendor root holds the raw files; the same value the WRDS capability and the CRSP acquisition name.

EXTRA_VARIABLES = ('permco', 'ret', 'retx', 'shrout', 'market_cap', 'bid', 'ask', 'prc_is_bidask', 'is_delisting', 'numtrd', 'cumfacpr', 'cumfacshr', 'facprc', 'close_trade')

Variables added beyond the Tiingo twelve. Defined on the class so a subclass can narrow or extend the set without changing the module constant.

TIINGO_VARIABLES = ('open', 'high', 'low', 'close', 'volume', 'adjOpen', 'adjHigh', 'adjLow', 'adjClose', 'adjVolume', 'divCash', 'splitFactor')

The twelve variables of a Tiingo daily panel, in TiingoColumns.EOD order. Taken from that constant, so this panel always matches Tiingo’s variables exactly.

config_cls

The config class the module loader rebuilds this dataset with.

alias of CrspDatasetConfig

filter_report_path()

Return the path of the filter report written next to the store.

Examples

>>> ds.filter_report_path()
PosixPath('/data/crsp.zarr.crsp_filter_report.json')

ticker_lookup()

Return the lookup over the ticker sidecar beside the store.

The backtester labels this dataset’s PERMNOs through it. The file is read on first use, so a store converted before the sidecar existed still gets a lookup; it falls back to the PERMNO digits. A dataset with no store has no sidecar and gets None.

Examples

>>> ds.ticker_lookup()
CrspTickerLookup('/data/crsp.zarr.crsp_tickers.json')

ticker_sidecar_path()

Return the path of the ticker sidecar written next to the store.

Examples

>>> ds.ticker_sidecar_path()
PosixPath('/data/crsp.zarr.crsp_tickers.json')