quantlab.dataset.crsp.reference.ReferenceTableSpec#

class quantlab.dataset.crsp.reference.ReferenceTableSpec(name, schema, table, columns, dtypes)[source]#

Bases: object

One reference table: where it lives on the WRDS server and its schema.

Attributes:

namestr: Local name of the table; also the parquet file stem.
schemastr: Server schema (database namespace), e.g. "crsp_a_stock".
tablestr: Table name on the server.
columnstuple of str: Column names in server order, so a SELECT built from them can be compared one by one with the server’s information_schema.
dtypesdict of str to pl.DataType: The polars type each column is read as.

Examples

>>> from quantlab.dataset.crsp.reference import REFERENCE_TABLES_BY_NAME
>>> spec = REFERENCE_TABLES_BY_NAME["dsp500list_v2"]
>>> spec.schema, spec.table
('crsp_a_indexes', 'dsp500list_v2')
>>> spec.columns
('permno', 'indno', 'mbrstartdt', 'mbrenddt', 'mbrflg', 'indfam')

cast(frame)[source]#

Cast an all-string frame, as a database COPY produces, to this spec’s types.

Dates are parsed with strict=False, so a malformed or empty field becomes null instead of failing the whole table; the live tables have real nulls in every date column. Columns missing from frame are added as all-null, so a table downloaded before a column existed still reads with the current schema.

Parameters:

framepl.DataFrame: A frame whose columns are all String.

Returns:

pl.DataFrame: A frame holding exactly columns, in order, typed by dtypes.

Examples

>>> import polars as pl
>>> raw = pl.DataFrame(
...     {"permno": ["14593"], "indno": ["1000500"],
...      "mbrstartdt": ["1982-11-18"], "mbrenddt": [None],
...      "mbrflg": ["NORM"]},
... )
>>> spec.cast(raw).schema
Schema({'permno': Int64, 'indno': Int64, 'mbrstartdt': Date,
        'mbrenddt': Date, 'mbrflg': String, 'indfam': Int64})