quantlab.universe.UniverseCatalog

class quantlab.universe.UniverseCatalog(config)

Bases: object

Point-in-time reference table of US equity universes.

Builds one table with the columns symbol, category, start_date, end_date and end_date_is_inferred from every roster fetcher in ROSTER_FETCHERS and every index membership fetcher in MEMBERSHIP_FETCHERS. A category names one universe, such as "us_all" or "sp500_constituent". The table is saved as parquet through PlBackend and answers point-in-time queries. Dates are ISO YYYY-MM-DD strings throughout and are compared as text, which gives the right order for that format.

A walk-forward backtest, which steps through time and rebalances as it goes, must call get_symbols_as_of on each rebalance date rather than once at setup. Otherwise the symbol list reflects later dates, and the backtest uses information it could not have had (look-ahead bias).

Parameters:

configUniverseConfig

Where the table is saved (output_path) and where fetchers cache change logs (cache_dir).

Examples

Build the table from the live sources and persist it:

config = UniverseConfig(
    output_path="data/reference/universe.parquet",
    cache_dir="data/reference/_cache",
)
UniverseCatalog(config).build().save()  # downloads from Tiingo/Wikipedia

Load a table already on disk and query it (here a three-row table written by hand):

>>> pl.DataFrame({
...     "symbol": ["AAPL", "MSFT", "OLD1"],
...     "category": ["us_all"] * 3,
...     "start_date": ["1980-12-12", "1986-03-13", "1980-01-01"],
...     "end_date": [None, None, "1997-06-30"],
...     "end_date_is_inferred": [False] * 3,
... }).write_parquet(config.output_path)
>>> catalog = UniverseCatalog.load(config)
>>> catalog.get_symbols_as_of("us_all", "2020-01-01")
['AAPL', 'MSFT']

CATALOG_COLUMNS = ('symbol', 'category', 'start_date', 'end_date', 'end_date_is_inferred')

Column order of the saved table. end_date_is_inferred marks an end date that reconstruct_intervals inferred rather than read from a source.

MEMBERSHIP_FETCHERS = (<class 'quantlab.universe.SP500MembershipFetcher'>, <class 'quantlab.universe.Nasdaq100MembershipFetcher'>)

The index membership fetchers this catalog uses. Listing a fetcher here gives its category a coverage start date: build takes the category from cls.CATEGORY and the queries take the start date from cls.PIT_COVERAGE_START. Roster fetchers do not belong here; they have listing dates but no coverage start.

ROSTER_FETCHERS = (<class 'quantlab.universe.NasdaqUniverseFetcher'>, <class 'quantlab.universe.USEquityUniverseFetcher'>)

The exchange roster fetchers this catalog uses, looped over in build like MEMBERSHIP_FETCHERS. Kept separate because a roster has no coverage start, so a query before a stock listed is answered from the roster’s own dates instead of raising. Each fetcher downloads the ticker directory itself, which costs a couple of seconds per build.

build(allow_stale=False)

Fetch every category and hold the combined table in memory.

Rosters come first, then index memberships. Each frame is reduced to CATALOG_COLUMNS, in that order, before the frames are joined. Nothing is written to disk until save.

Parameters:

allow_stalebool, default False

Whether to accept a membership fetcher that fell back to its cached change-log snapshot. Off by default, because a stale table saved by save looks exactly like a fresh one and would freeze the universe at the cache date without anyone noticing.

Returns:

Self

self, so build and save chain.

Raises:

ValueError

If a membership fetcher is stale and allow_stale is false.

Examples

>>> catalog = UniverseCatalog(config).build()  # downloads every source
>>> catalog.save()

get_symbols_as_of(category, as_of_date)

Return every symbol whose interval contains as_of_date.

This is point-in-time membership on one day, the query a walk-forward backtest should make at each rebalance. To download a whole window of history, use get_symbols_in_range. The result has no duplicates and is sorted ascending, for the same reason as there.

Parameters:

categorystr

A token from known_categories.

as_of_datestr

ISO date. For an index category it must not be before that index’s coverage start.

Returns:

list[str]

Sorted symbols without duplicates.

Raises:

ValueError

For an unknown category, a non-ISO date, or a date before the category’s coverage start.

Examples

>>> catalog.get_symbols_as_of("us_all", "1990-01-01")
['AAPL', 'MSFT', 'OLD1']
>>> catalog.get_symbols_as_of("sp500_constituent", "1970-01-01")
Traceback (most recent call last):
...
ValueError: Cannot answer sp500_constituent membership before 1976-07-01 ...

get_symbols_in_range(category, start_date, end_date)

Return every symbol whose interval overlaps [start_date, end_date].

A row overlaps when start_date <= end and end_date is null or >= start. A symbol that delisted inside the window is therefore kept; dropping it would be exactly the survivorship bias this table exists to remove. Use this query to download a whole window of history; use get_symbols_as_of at each rebalance of a walk-forward backtest.

The result has no duplicates and is sorted ascending. Callers rely on the order, for example to take the first N symbols with --limit, so it must depend only on which symbols match and not on the row order of the parquet file.

Parameters:

categorystr

A token from known_categories.

start_datestr

ISO date, inclusive. For an index category it must not be before that index’s coverage start. An earlier date is refused, not moved forward, because moving it would return a shorter roster without saying so.

end_datestr

ISO date, inclusive.

Returns:

list[str]

Sorted symbols without duplicates.

Raises:

ValueError

For an unknown category, a non-ISO date, or a start_date before the category’s coverage start.

Examples

>>> catalog.get_symbols_in_range("us_all", "1995-01-01", "2000-12-31")
['AAPL', 'MSFT', 'OLD1']

known_categories()

Return every category this catalog can answer for.

This is the union of both fetcher lists, so a category cannot exist without a fetcher, and every listed fetcher’s category is known.

Examples

>>> sorted(catalog.known_categories())
['nasdaq100_constituent', 'nasdaq_all', 'sp500_constituent', 'us_all']

classmethod load(config)

Return a catalog whose table is read from config.output_path.

Parameters:

configUniverseConfig

The same config the table was built with; only output_path is read here.

Examples

>>> catalog = UniverseCatalog.load(config)
>>> sorted(catalog.known_categories())
['nasdaq100_constituent', 'nasdaq_all', 'sp500_constituent', 'us_all']

save()

Write the table built by build to config.output_path.

Parent directories are created as needed.

Returns:

Self

self.

Raises:

ValueError

If any known category has no rows in the table.

Examples

>>> UniverseCatalog(config).build().save()  # downloads, then writes