quantlab.acquisition.wrds.crsp.WrdsCrspDailyAcquisition#

class quantlab.acquisition.wrds.crsp.WrdsCrspDailyAcquisition(config)[source]#

Bases: SymbolBatchAcquisition

Download CRSP Stock v2 daily bars from crsp_a_stock.dsf_v2.

The symbols of this acquisition are PERMNOs written as digit strings, for example "14593" (Apple). The work is split into pages, and one page is one calendar year for one batch of PERMNOs. _fetch_page reads the year named by the page token (the window’s first year when there is no token) and returns the following year as the next token. The shared SymbolBatchAcquisition base class runs the batch loop, writes the parquet files (shards), records finished pages and resumes interrupted runs.

The raw tier is written under .../wrds_crsp/wrds/crsp_daily/month=YYYY-MM/, with one row per (permno, dlycaldt) exactly as CRSP serves it: no derived price, no adjusted series and no filtering. The symbol column holds the PERMNO as a string, and an Int64 permno column is stored beside it. A page is refused, not repaired, when it repeats a key, contains a PERMNO that was not requested, or holds a row dated outside the page.

Before any batch runs, the acquisition checks that every symbol is a PERMNO, that the account can read crsp_a_stock, that the window does not extend past the last day of the current CRSP release (or clips it when kwargs["clip_to_product_end"] is set), and that the raw tier on disk was built from the same release. Batches are fetched by max_workers threads, each on its own pooled connection; a value above MAX_WORKERS (the session’s connection cap) is refused.

Parameters:

configAcquisitionConfig: Built by build_config. It must have frequency="1d" and kwargs["data_type"] == "crsp_daily".

Examples

Needs WRDS_USERNAME and a ~/.pgpass entry; the first query opens the connection:

cfg = WrdsCrspDailyAcquisition.build_config(
    ("14593", "10107"), start_date="2020-08-01", end_date="2020-08-31"
)
acq = WrdsCrspDailyAcquisition(cfg).download()
report = acq.coverage_report()

Shards land under .../wrds_crsp/wrds/crsp_daily/month=2020-08/, watermarks under .../wrds_crsp/_watermarks/wrds/crsp_daily/ and the vintage stamp at .../wrds_crsp/_vintage/wrds.json.

CREDENTIAL_ENV_VARS = ('WRDS_USERNAME',)#: The environment variables whose values the request policy hides (each becomes <NAME REDACTED>). A subclass fills this from module-level constants, not from its HTTP client class, so a test double that replaces the client cannot turn the redaction off. An empty tuple means the vendor uses no credentials.

CRSP_COLUMNS = ('permno', 'hdrcusip', 'permco', 'siccd', 'nasdissuno', 'yyyymmdd', 'sharetype', 'securitytype', 'securitysubtype', 'usincflg', 'issuertype', 'primaryexch', 'conditionaltype', 'tradingstatusflg', 'dlycaldt', 'dlydelflg', 'dlyprc', 'dlyprcflg', 'dlycap', 'dlycapflg', 'dlyprevprc', 'dlyprevprcflg', 'dlyprevdt', 'dlyprevcap', 'dlyprevcapflg', 'dlyret', 'dlyretx', 'dlyreti', 'dlyretmissflg', 'dlyretdurflg', 'dlyorddivamt', 'dlynonorddivamt', 'dlyfacprc', 'dlydistretflg', 'dlyvol', 'dlyclose', 'dlylow', 'dlyhigh', 'dlybid', 'dlyask', 'dlyopen', 'dlynumtrd', 'dlymmcnt', 'dlyprcvol', 'dlycumfacpr', 'dlycumfacshr', 'cusip', 'ticker', 'exchangetier', 'shrout')#: The 50 dsf_v2 columns this class reads, in server order. Fixed so the SELECT, and therefore every shard, is identical for every page. If the server stops reporting a column the page fails; nothing is filled with nulls.

DATA_TYPE = 'crsp_daily'#: The only data type this class serves.

DEFAULT_BATCH_SIZE = 200#: PERMNOs per query. An S&P 500 history covers about 1,100 PERMNOs, so 200 gives about six batches.

DEFAULT_MAX_WORKERS = 4#: Concurrent batch fetches, each on its own pooled connection. Overridable through kwargs["max_workers"], up to MAX_WORKERS.

DEFAULT_SUBDIR = 'wrds_crsp'#: The raw tier lives under downloads/us_equity/1d/{DEFAULT_SUBDIR}/wrds.

DEFAULT_VERIFY_PAGE_COUNTS = True#: Count each page with the same WHERE before copying it, and fail the page if the copied row count differs. Overridable through kwargs["verify_page_counts"].

GLOBAL_STOP_ERRORS = (<class 'quantlab.acquisition.wrds.taq.WrdsSessionError'>, <class 'quantlab.acquisition.wrds.taq.WrdsEntitlementError'>, <class 'psycopg2.OperationalError'>, <class 'psycopg2.InterfaceError'>)#: Exceptions meaning the shared session or the account is unusable.

MAX_WORKERS = 6#: The most workers a run may use: the session’s connection cap.

RAW_COLUMNS = ('timestamp', 'symbol', 'vendor', 'permno', 'hdrcusip', 'permco', 'siccd', 'nasdissuno', 'yyyymmdd', 'sharetype', 'securitytype', 'securitysubtype', 'usincflg', 'issuertype', 'primaryexch', 'conditionaltype', 'tradingstatusflg', 'dlycaldt', 'dlydelflg', 'dlyprc', 'dlyprcflg', 'dlycap', 'dlycapflg', 'dlyprevprc', 'dlyprevprcflg', 'dlyprevdt', 'dlyprevcap', 'dlyprevcapflg', 'dlyret', 'dlyretx', 'dlyreti', 'dlyretmissflg', 'dlyretdurflg', 'dlyorddivamt', 'dlynonorddivamt', 'dlyfacprc', 'dlydistretflg', 'dlyvol', 'dlyclose', 'dlylow', 'dlyhigh', 'dlybid', 'dlyask', 'dlyopen', 'dlynumtrd', 'dlymmcnt', 'dlyprcvol', 'dlycumfacpr', 'dlycumfacshr', 'cusip', 'ticker', 'exchangetier', 'shrout')#: The shard column projection and order.

RAW_SCHEMA = {'conditionaltype': String, 'cusip': String, 'dlyask': Float64, 'dlybid': Float64, 'dlycaldt': Date, 'dlycap': Float64, 'dlycapflg': String, 'dlyclose': Float64, 'dlycumfacpr': Float64, 'dlycumfacshr': Float64, 'dlydelflg': String, 'dlydistretflg': String, 'dlyfacprc': Float64, 'dlyhigh': Float64, 'dlylow': Float64, 'dlymmcnt': Int64, 'dlynonorddivamt': Float64, 'dlynumtrd': Int64, 'dlyopen': Float64, 'dlyorddivamt': Float64, 'dlyprc': Float64, 'dlyprcflg': String, 'dlyprcvol': Float64, 'dlyprevcap': Float64, 'dlyprevcapflg': String, 'dlyprevdt': Date, 'dlyprevprc': Float64, 'dlyprevprcflg': String, 'dlyret': Float64, 'dlyretdurflg': String, 'dlyreti': Float64, 'dlyretmissflg': String, 'dlyretx': Float64, 'dlyvol': Float64, 'exchangetier': String, 'hdrcusip': String, 'issuertype': String, 'nasdissuno': Int64, 'permco': Int64, 'permno': Int64, 'primaryexch': String, 'securitysubtype': String, 'securitytype': String, 'sharetype': String, 'shrout': Int64, 'siccd': Int64, 'symbol': String, 'ticker': String, 'timestamp': Datetime(time_unit='us', time_zone=None), 'tradingstatusflg': String, 'usincflg': String, 'vendor': String, 'yyyymmdd': Int64}#: timestamp is Datetime("us") at midnight, the same type as the Tiingo daily data, so panels built from either vendor line up.

REFERENCE_DIR_NAME = '_reference'#: The reference tier’s directory name, a sibling of the raw root.

SAMPLE_LIMIT = 10#: How many offending values a page refusal lists. Bounded because a bad page can be bad in every row, and the message is stored in a JSON failure file that a person has to read.

VENDOR = 'wrds'#: Which vendor this subclass fetches from. Set by every subclass. It is written into every shard as a literal vendor column and hashed into every PageLedger.batch_key, so changing it changes the files.

VINTAGE_DIR_NAME = '_vintage'#: The vintage stamp’s directory name, a sibling of both roots.

classmethod build_config(symbols, start_date=None, end_date=None, kwargs=None, subdir='wrds_crsp')[source]#

Build the AcquisitionConfig for a CRSP daily download.

This is the config_factory of the registry’s crsp_daily capability. The raw directory is .../{subdir}/wrds and the watermarks go to the sibling .../{subdir}/_watermarks/wrds, both under get_data_root() / "downloads" / "us_equity" / "1d". No credential goes into the config.

Parameters:

symbolsiterable: PERMNOs; stored as strings.
start_date, end_datestr or None, default None: The window, inclusive, as ISO dates.
kwargsdict or None, default None: Extra options, such as clip_to_product_end or verify_page_counts. data_type is set to "crsp_daily".
subdirstr, default “wrds_crsp”: Directory under downloads/us_equity/1d that holds this raw tier. Use a new one to start a fresh tier for a new CRSP release.

Returns:

AcquisitionConfig: The config for WrdsCrspDailyAcquisition.

Raises:

ValueError: If kwargs["data_type"] is set to anything but "crsp_daily".

Examples

>>> cfg = WrdsCrspDailyAcquisition.build_config(
...     ("14593", "10107"), start_date="2020-08-01", end_date="2020-08-31"
... )
>>> cfg.kwargs
{'data_type': 'crsp_daily'}

cfg.raw_data_dir_path is '<data root>/downloads/us_equity/1d/wrds_crsp/wrds'.

classmethod reference_dir_for(config)[source]#

Return the reference tier directory, .../{subdir}/_reference.

The reference tier holds CRSP’s lookup tables (security names, delistings, distributions, index membership); see quantlab.acquisition.wrds.crsp_reference. It sits beside the raw directory, never inside it, because the dataset’s conversion reads every parquet file below the raw directory. The path is derived from the config rather than from get_data_root(), so a config that points at a custom directory keeps both tiers together.

Parameters:

configAcquisitionConfig: The acquisition’s config.

Returns:

pathlib.Path: The reference directory.

Examples

With cfg from build_config:

WrdsCrspDailyAcquisition.reference_dir_for(cfg)
# PosixPath('<data root>/downloads/us_equity/1d/wrds_crsp/_reference')

classmethod resolve_window(session, start, end, *, clip)[source]#

Query the release’s last day and fit the window to it.

See window_for_product_end for the rules. _run does not call this method: it queries the last day once and uses the answer both for the window and for the vintage check.

Parameters:

sessionWrdsSession: The shared WRDS session.
start, endstr or date: The requested window, inclusive.
clipbool: Whether to shorten a window that ends past the release’s last day instead of refusing it.

Returns:

tuple of (date, date, date or None): (start, effective_end, clipped_to), where clipped_to is the release’s last day if the window was shortened and None otherwise.

Examples

Needs a live WrdsSession:

WrdsCrspDailyAcquisition.resolve_window(
    session, "2020-01-01", "2026-06-30", clip=True
)

classmethod vintage_path_for(config)[source]#

Return the vintage stamp path, .../{subdir}/_vintage/wrds.json.

The vintage stamp is a small JSON file that records which CRSP release the raw tier was built from. It sits beside the raw directory and the watermark directory, never inside either. Every *.json file under the watermark directory is read as one symbol’s progress, so a stamp there would look like an extra PERMNO, and the dataset’s conversion reads every file below the raw directory.

Parameters:

configAcquisitionConfig: The acquisition’s config.

Returns:

pathlib.Path: The stamp’s path.

Examples

With cfg from build_config:

WrdsCrspDailyAcquisition.vintage_path_for(cfg)
# PosixPath('<data root>/downloads/us_equity/1d/wrds_crsp/_vintage/wrds.json')

classmethod window_for_product_end(product_end, start, end, *, clip)[source]#

Fit a window to an already known last day of the CRSP release.

crsp_a_stock is updated once a year, so no data exists after its last day (the product end). A start after it is always refused. An end after it is refused unless clip is set; then the window ends at the product end and the third element reports that date. Refusing is preferred to returning an empty result, which would look exactly like a symbol list with no data. This method opens no connection.

Parameters:

product_endstr or date: The release’s last day, as returned by CrspQueries.product_end.
start, endstr or date: The requested window, inclusive.
clipbool: Whether to shorten a window that ends past product_end instead of refusing it.

Returns:

tuple of (date, date, date or None): (start, effective_end, clipped_to), where clipped_to is product_end if the window was shortened and None otherwise.

Raises:

CrspProductEndError: If start is after product_end, or if end is and clip is false.

Examples

>>> WrdsCrspDailyAcquisition.window_for_product_end(
...     "2025-12-31", "2020-01-01", "2026-06-30", clip=True
... )
(datetime.date(2020, 1, 1), datetime.date(2025, 12, 31),
 datetime.date(2025, 12, 31))
>>> WrdsCrspDailyAcquisition.window_for_product_end(
...     "2025-12-31", "2020-01-01", "2024-12-31", clip=False
... )
(datetime.date(2020, 1, 1), datetime.date(2024, 12, 31), None)