quantlab.universe.IndexMembershipFetcher#

class quantlab.universe.IndexMembershipFetcher(cache_dir)[source]#

Bases: ABC

Base class for rebuilding the dates each stock was in an index.

The result is a set of membership intervals: one row per stay in the index, with the date the stock joined and the date it left (null while it is still a member). An index is described by data: a subclass sets the class constants below and implements fetch_anchor, and inherits the change-log parse, the rebuild algorithm and the caching of fetch_changes.

Two sources are combined. The anchor is a snapshot of today’s members and is trusted for who is a member now. The change log is Wikipedia’s dated table of additions and removals and is trusted for when membership changed. reconstruct_intervals replays the log forward and reconciles it with the anchor.

A subclass sets ANCHOR_URL, CHANGES_URL, PIT_COVERAGE_START (the earliest date the change log covers; point-in-time queries before it are refused), CACHE_FILENAME (the per-index change-log snapshot under cache_dir), INDEX_LABEL, CATEGORY, EXPECTED_SOURCE_HEADER (the flattened header that identifies the change-log table) and DATE_HEADER. CHANGES_TABLE_ATTRS is optional.

Three safety checks live on this base so every index gets them. The change-log table is picked by matching its header against EXPECTED_SOURCE_HEADER and its columns are read by name, so a reordered header raises instead of quietly swapping additions and removals. A live table with fewer rows than the cached snapshot is treated as a parse failure, because the log only grows. On any fetch or parse failure the cached snapshot is returned and the cache file is left as it was.

Parameters:

cache_dirstr: Directory holding the cached change-log snapshot.

Attributes:

changes_are_stalebool: Whether the last fetch_changes result came from the cache instead of a live fetch.
changes_source_asofstr or None: The date or time the last fetch_changes result is current to.

Examples

>>> fetcher = SP500MembershipFetcher(cache_dir="data/reference/_cache")
>>> intervals = fetcher.build_intervals()  # downloads anchor and change log
>>> intervals.columns
['symbol', 'start_date', 'end_date', 'end_date_is_inferred']

ADDED_TICKER_HEADER = 'Added Ticker'#: The two ticker columns consumed. Both live sources word them identically; they are still validated against EXPECTED_SOURCE_HEADER.

CATEGORY#: The catalog category this index is stored under. Typed as the literal so an unknown token is a type error rather than a runtime surprise.

CHANGES_TABLE_ATTRS = None#: Optional pd.read_html(attrs=...) pre-filter for a page that marks its change-log table with an id or class.

DATE_HEADER#: The EXPECTED_SOURCE_HEADER entry carrying the effective date.

EXPECTED_SOURCE_HEADER#: Exact flattened header (see _flatten_header) of the change-log table. It both selects the table and names the columns read from it.

build_intervals()[source]#

Fetch the anchor and change log and return the rebuilt intervals.

Returns:

pl.DataFrame: The frame from reconstruct_intervals.

Examples

>>> fetcher = Nasdaq100MembershipFetcher(cache_dir="data/reference/_cache")
>>> intervals = fetcher.build_intervals()  # downloads anchor and change log
>>> intervals.columns
['symbol', 'start_date', 'end_date', 'end_date_is_inferred']

abstractmethod fetch_anchor()[source]#

Return today’s index members as a [symbol, date_added] frame.

date_added may be entirely null when the source has no such column; reconstruct_intervals then uses PIT_COVERAGE_START for those symbols.

Examples

A subclass whose source is a CSV with Symbol and Date added columns:

def fetch_anchor(self) -> pl.DataFrame:
    response = requests.get(self.ANCHOR_URL, timeout=30)
    response.raise_for_status()
    data = pl.read_csv(io.StringIO(response.text))
    data = data.rename({"Symbol": "symbol", "Date added": "date_added"})
    return data.select(["symbol", "date_added"])

fetch_changes()[source]#

Fetch the change log, validate it and update the cached snapshot.

On success the parsed table is written to CACHE_FILENAME under the cache directory, atomically so an interrupted write cannot leave a half-written file. On any fetch or parse failure, or when the live table has fewer rows than the cached one, the cached snapshot is returned instead, the cache file is left as it was, and changes_are_stale is set so the catalog can refuse to save the result.

Returns:

pl.DataFrame: A frame with columns effective_date, added_ticker and removed_ticker.

Raises:

RuntimeError: If the live fetch failed and no cached snapshot exists.

Examples

>>> fetcher = SP500MembershipFetcher(cache_dir="data/reference/_cache")
>>> changes = fetcher.fetch_changes()  # downloads from Wikipedia
>>> changes.columns
['effective_date', 'added_ticker', 'removed_ticker']
>>> fetcher.changes_are_stale
False

reconstruct_intervals(anchor, changes)[source]#

Replay the change log forward and reconcile it with the anchor.

The anchor is trusted for who is a member today; the change log is trusted for when membership changed. Three disagreements between them are resolved, each with a warning:

A stay still open at the end of the log whose symbol is not in the anchor is closed at the last date the log covers and marked end_date_is_inferred=True, because the log never says when it ended.
A symbol whose last event is a removal but which the anchor still lists is re-opened from that removal date, because the log is missing a re-addition.
An anchor member with no event in the log is opened at its date_added, or at PIT_COVERAGE_START when that is null.

A removal with no earlier addition opens at PIT_COVERAGE_START, and a duplicate addition keeps the earlier start date.

Parameters:

anchorpl.DataFrame: A [symbol, date_added] frame as returned by fetch_anchor.
changespl.DataFrame: A frame as returned by fetch_changes, with None where a row has no ticker on one side.

Returns:

pl.DataFrame: A frame with columns symbol, start_date, end_date (null for a current member) and end_date_is_inferred.

Examples

>>> fetcher = SP500MembershipFetcher(cache_dir="/tmp/cache")
>>> anchor = pl.DataFrame(
...     {"symbol": ["AAA", "BBB"], "date_added": ["1999-01-01", None]}
... )
>>> changes = pl.DataFrame(
...     {
...         "effective_date": ["2010-05-03", "2015-09-21"],
...         "added_ticker": ["BBB", "DDD"],
...         "removed_ticker": ["ZZZ", None],
...     }
... )
>>> fetcher.reconstruct_intervals(anchor, changes).sort("symbol")
shape: (4, 4)
┌────────┬────────────┬────────────┬──────────────────────┐
│ symbol ┆ start_date ┆ end_date   ┆ end_date_is_inferred │
│ ---    ┆ ---        ┆ ---        ┆ ---                  │
│ str    ┆ str        ┆ str        ┆ bool                 │
╞════════╪════════════╪════════════╪══════════════════════╡
│ AAA    ┆ 1999-01-01 ┆ null       ┆ false                │
│ BBB    ┆ 2010-05-03 ┆ null       ┆ false                │
│ DDD    ┆ 2015-09-21 ┆ 2015-09-21 ┆ true                 │
│ ZZZ    ┆ 1976-07-01 ┆ 2010-05-03 ┆ false                │
└────────┴────────────┴────────────┴──────────────────────┘