quantlab.dataset.crsp.membership.CrspMembership#

class quantlab.dataset.crsp.membership.CrspMembership(reference)[source]#

Bases: object

Point-in-time index membership from one CRSP reference directory.

It holds no database connection; it only reads parquet files on disk.

Parameters:

referenceCrspReference: The downloaded CRSP reference tables.

Attributes:

referenceCrspReference: The reference tables given to the constructor.
reportdict: What the most recent permno_intervals call excluded, clipped, tolerated or could not link. Its keys are reset on every call.

Examples

>>> from quantlab.dataset.crsp.membership import CrspMembership
>>> from quantlab.dataset.crsp.reference import CrspReference
>>> ref = CrspReference("data/downloads/us_equity/1d/wrds_crsp/_reference")
>>> membership = CrspMembership(ref)
>>> membership.permnos_in_range("comp_nasdaq100", "2010-01-01", "2020-12-31")
['14542', '90319']
>>> membership.report["clipped_to_product_end"]
2

INDEXES = ('crsp_sp500', 'comp_nasdaq100')#: Every index this class serves, in the order the command line lists them.

LINK_GAP_TOLERANCE_DAYS = 4#: A gap of at most this many calendar days between two links of one (gvkey, iid) is treated as a joint in the link table rather than a hole in the membership (a long weekend plus a holiday is four days without trading). Longer gaps count as unlinked.

LINK_TYPES = ('LC', 'LU', 'LS')#: The CCM link types that state a real gvkey-to-PERMNO match. NR (“no research”) and NU rows have a null lpermno and are not links.

NASDAQ100 = 'comp_nasdaq100'#: Compustat’s Nasdaq-100 membership, linked to PERMNOs through CCM.

NDX_GVKEYX = '000208'#: The Compustat index id (gvkeyx in comp.idx_index) of the Nasdaq 100.

PIT_COVERAGE_START = {'comp_nasdaq100': '1995-01-01', 'crsp_sp500': '1925-12-31'}#: The earliest date each index has data for. The S&P date is where the CRSP index history starts. The Nasdaq-100 date is where Compustat’s history is cut off, not where the index began: spells that start on 1995-01-01 may really have started earlier.

SP500 = 'crsp_sp500'#: CRSP’s own S&P 500 membership.

SP500_INDNO = 1000500#: The S&P 500’s index number in dsp500list_v2. The table also holds other indexes; a row with any other indno is ignored.

permno_intervals(index, *, allow_unlinked=False, window=None)[source]#

Return membership intervals as (permno, start_date, end_date).

There is one row per continuous membership of one PERMNO, sorted, with both ends inclusive. end_date is never null and never later than CrspReference.product_end. Adjacent spells are merged.

allow_unlinked and window only affect the Nasdaq-100. By default, a Nasdaq-100 spell with membership days that no CCM link covers raises an error, because dropping those days would silently remove a real member. allow_unlinked=True records them in report["unlinked"] instead. window limits the error to uncovered days inside the window, where a member would actually be lost; it never filters the rows returned.

Parameters:

indexstr: "crsp_sp500" or "comp_nasdaq100".
allow_unlinkedbool, default False: Record unlinked Nasdaq-100 days instead of raising.
windowtuple of date or str, optional: (start, end) as dates or ISO strings. Only limits which unlinked days raise.

Returns:

pl.DataFrame: Columns permno (Int64), start_date and end_date (Date).

Raises:

ValueError: If index is unknown, if window ends before it starts, if Nasdaq-100 membership days inside the window have no PERMNO and allow_unlinked is false, or if a source table holds invalid or conflicting rows.

Examples

>>> membership.permno_intervals("crsp_sp500")
shape: (1, 3)
┌────────┬────────────┬────────────┐
│ permno ┆ start_date ┆ end_date   │
│ ---    ┆ ---        ┆ ---        │
│ i64    ┆ date       ┆ date       │
╞════════╪════════════╪════════════╡
│ 14593  ┆ 1982-11-18 ┆ 2025-12-31 │
└────────┴────────────┴────────────┘

permnos_in_range(index, start_date, end_date, *, allow_unlinked=False)[source]#

Return every PERMNO whose membership overlaps [start_date, end_date].

This is the list of securities (the roster) a download over the whole window fetches. The test is overlap, not full coverage (start_date <= end and end_date >= start), so a security that left the index during the window is still included; dropping it would cause survivorship bias.

The order is numeric, as quantlab.utils.symbol_axis.sort_symbol_axis defines it, not alphabetical: "7000" sorts before "14593".

The window is passed on to permno_intervals, so a Nasdaq-100 roster only fails for unlinked days inside this window.

Parameters:

indexstr: "crsp_sp500" or "comp_nasdaq100".
start_datedate or str: Window start, as a date or ISO string.
end_datedate or str: Window end, inclusive.
allow_unlinkedbool, default False: Record unlinked Nasdaq-100 days instead of raising.

Returns:

list of str: PERMNOs as strings, in numeric order.

Raises:

ValueError: If start_date is after end_date, or for the reasons permno_intervals raises.

Examples

>>> membership.permnos_in_range("crsp_sp500", "2020-01-01", "2020-12-31")
['14593']
>>> membership.permnos_in_range("crsp_sp500", "1970-01-01", "1980-12-31")
[]

report#: What the last permno_intervals call excluded, clipped, tolerated or could not link.