quantlab.model.evaluation#

Evaluation: score a trained unit’s predictions against its raw labels.

A model and an ensemble are both scored here, after training, by one rule: evaluate takes the unit’s prediction panel (a model’s predict_panel over its collected panel, an ensemble’s combined prediction), the panel of the labels’ raw values and the unit’s train / validation / test segments, returns the metrics and writes the unit’s two evaluation files, ic_series.csv and test_predictions.zarr. It needs no model instance, so predictions can be scored without training anything.

Scoring rules, one set for every unit:

Every label is scored on every non-empty segment. The first label’s keys are {split}_{metric}, every other label’s {split}_{label}_{metric}.
The IC family (ic, rank_ic, icir, rank_icir, see ic_panel_metrics) always, and rank_autocorrelation, the mean of rank_autocorrelation_series over the segment: how much the prediction’s order persists from one bar to the next, which bounds the turnover of a book that follows it.
The error metrics mse, rmse, mae and r2 only for a label whose prediction is on its own scale (label scale "raw"): a standardized prediction is in other units than the label.
qlike and variance_ratio (volatility_level_metrics) where scores_volatility_level holds: a volatility label predicted raw.
whatever extra_metrics returns for a label and split, when given: the caller’s own metrics on the same bars and symbols (an ensemble adds how much its members agree, member_correlation). evaluate itself knows no ensemble.

The panel metrics. MSE, RMSE, MAE, R2, IC and RankIC, and the level metrics of a volatility prediction, are defined here too.

The return models score their predictions with these functions. Predictions and targets are [T, S] panels (time by symbol), and every function follows the same conventions: inputs are cast to float64 and must share a shape; only cells where both prediction and target are finite take part in any sum or ranking; and an empty set of usable cells yields NaN without raising a RuntimeWarning.

The IC (information coefficient) measures how well predictions order the symbols. Cross-sectional IC is the Pearson correlation between prediction and target across the symbols of one timestamp, averaged over time. RankIC ranks each row first (average ranks on ties) and then computes IC on the ranks, so it is a per-timestamp Spearman correlation and is not dominated by outliers. Both are fully vectorised, with no per-row Python loop, since a panel can hold tens of thousands of timestamps.

The per-timestamp values behind both means are available as series (cross_sectional_ic_series, cross_sectional_rank_ic_series), NaN on a skipped row. The ICIR (information ratio of the IC) divides the mean of the valid values of such a series by their sample standard deviation (information_ratio), so it measures how stable a signal is, not only how strong.

The IC of a volatility prediction measures only how well it ranks the symbols’ volatility. A consumer that uses the predicted level, such as a covariance built from it, also needs the level right, which volatility_level_metrics scores: the QLIKE loss and the ratio of realised to predicted variance.

Classes

Segments

The bars of a unit's train, validation and test segments.

Functions

`cross_sectional_ic`	Return the mean over time of the per-row Pearson correlation.
`cross_sectional_ic_series`	Return the per-row Pearson correlation of a panel, NaN on a skipped row.
`cross_sectional_rank_ic`	Return the cross-sectional IC computed on per-row ranks.
`cross_sectional_rank_ic_series`	Return the per-row IC computed on per-row ranks, NaN on a skipped row.
`evaluate`	Score a unit's predictions of every label and write its evaluation files.
`ic_panel_metrics`	Return the four IC metrics keyed `ic, rank_ic, icir, rank_icir`.
`information_ratio`	Return the mean of the finite values of `series` over their sample std.
`mae`	Return the mean absolute error over the jointly finite cells, or NaN if none.
`mse`	Return the mean squared error over the jointly finite cells, or NaN if none.
`r2`	Return the coefficient of determination `1 - SS_res / SS_tot`.
`rank_autocorrelation_series`	Return each row's rank correlation with the row `lag` rows before, NaN where there is none.
`regression_panel_metrics`	Return the eight panel metrics keyed `mse, rmse, mae, r2, ic, rank_ic, icir, rank_icir`.
`rmse`	Return `sqrt(mse(pred, target))`, or NaN when the MSE is undefined.
`scores_volatility_level`	Return whether a label's predictions get `volatility_level_metrics`.
`volatility_level_metrics`	Return the level metrics of a volatility prediction, `qlike` and `variance_ratio`.