factor_agent.evaluator

The Evaluator: the only module that reads data.

It is fixed to one segment when it is made. evaluate compiles all of a round’s graphs as one KunQuant graph and computes it on the whole-market dataset (the segment plus the warm-up bars before it). A candidate of an intraday family runs as its stage split (factor_agent.gate.stages): the round’s session columns are computed once on the whole-market intraday dataset over the same days, all in one KunQuant function, a subtree shared by two candidates once, a chunk of sessions at a time (factor_agent.session.SessionColumns), then its daily graph runs with the others; a day without a session (the warm-up before the intraday data begins, say) has no session value, so a daily window reaching back before it is missing and counted against coverage. Each candidate is then scored by the campaign’s Scoring: ranked across the whole market (KunQuant Rank), neutralised when the scoring neutralises, and masked to the members of day t, or kept on every symbol without a membership. A reference or known factor given as a component is scored the same way; one given as a store (a *_neutral store, say) is read as stored and masked (Scoring.read). A batch that does not compile or run is bisected until the failing graphs are found; they become CompileFailure and the rest are evaluated.

Only dates whose label lies entirely inside the segment are scored (the purge of segments), and on each date only the members with a label. The statistics of a candidate:

• IC: the daily rank IC against the forward label, its mean and the Newey-West t of the mean.

• Correlation: the daily correlation of normal scores (below) with every factor of the reference pool (the reference set and the factors accepted so far), averaged over dates; the pool factor with the largest absolute mean is the nearest. It is the van der Waerden rank correlation, within a few hundredths of Spearman’s.

• Nearest known factor: the same correlation with every output of the known factors, on a fixed sample of the scored dates (every KNOWN_SAMPLE_EVERY-th, counting from the first); the known factor with the largest absolute mean is the nearest. The known factors are read as the reference set is, once, on first use, and only their normal scores on the sampled dates are kept, in a panel of their own: they never enter the reference pool or the partial-IC regression.

• Partial IC: the rank IC of the residual of the candidate’s normal scores from a daily least-squares regression on the normal scores of every pool factor (with an intercept), and its Newey-West t.

• Deciles: the mean label of each tenth of the candidate’s daily cross section, lowest values first, averaged over dates.

• Coverage: the share of the day’s scored members on which the graph itself (before the rank and the neutralisation) has a value, averaged over the dates with any scored member. It is measured before any fill, on the graph with every SetInfOrNanToValue replaced by its input (Graph.without_fills), so a fill does not raise it; the scored values come from the graph as written.

For a candidate refused for its coverage, trace_coverage computes the nodes of its graph one by one, unranked, and gives each one’s coverage and the share on which it is infinite, so the refusal can say which step loses the values.

Normal scores: on each date a factor’s values are ranked among its own finite symbols and mapped to the standard normal quantiles of their ranks; a missing value scores 0, the middle rank. A candidate that smooths or rescales a pool factor is then a linear function of it plus noise, so the regression removes what the pool factor explains, which ranks would not.

The reference pool grows as candidates are admitted: evaluate hands each candidate’s metrics to an admit callback in order, and a candidate it admits joins the pool before the next one is scored, so two candidates of one round are never both accepted when the second says nothing new.

join puts factors in the pool or the tried trials without scoring them, as a resume puts back the factors of the rounds already written. With a store, every column that joins the pool or the tried trials is written to it as the frame holds it (one file per graph and kind, named by the hash of the graph), and join takes a column from the last batch evaluate computed, then from the store, and computes only the rest; a column read from the store is the one a computation gives, bit for bit. The store is fingerprinted by what scored its columns (the caller’s store_key, the segment, the purge and the scored dates, symbols and rows); one that does not match is emptied. It is not fingerprinted by the code: a resume after a fix keeps the values its trials were judged against.

Classes

CompileFailure	The candidate's graph did not compile or run; the error KunQuant gave.
Contribution	A pool factor's part in a group of pool factors.
DailyIC	A candidate's daily rank IC and rank IC of its residual (the partial IC) on each day of the label, NaN where unscored.
Evaluator	Score graphs on one segment; see the module docstring.
Metrics	A candidate's training-segment statistics; see the module docstring.
NodeCoverage	One node of a graph on the scored member days (Evaluator.trace_coverage).

Functions

coverage	The share of each date's rows where values (before ranking) is finite, averaged over the dates with rows.
decile_returns	The mean label of each decile of the daily cross section, averaged over dates.