quantlab.model.evaluation.evaluate#

quantlab.model.evaluation.evaluate(predictions, truth, *, labels, label_scales, segments, test_bounds, run_dir, extra_metrics=None)[source]#

Score a unit’s predictions of every label and write its evaluation files.

Each label’s prediction is taken on its segment’s bars and on the symbols of truth, so a predicted symbol without truth is ignored and a missing prediction counts as missing.

Parameters:

predictionsxr.Dataset: One variable per label name on (timestamp, symbol).
truthxr.Dataset or Mapping[str, xr.Dataset]: The labels’ raw values, one variable per label name on (timestamp, symbol), covering every segment’s bars; or each label’s own panel holding it (an ensemble scores a label on the symbols of the first member predicting it).
labelsMapping[str, object]: Label name to the label object predicting it (read for its kind), in label order: the first key is the first label.
label_scalesMapping[str, str]: Label name to its prediction scale, "raw" or "standardized".
segmentsSegments or Mapping[str, Segments]: The unit’s segments, or each label’s own (an ensemble scores a label on the segments of the first member predicting it).
test_boundstuple: The unit’s (test_start, test_end); test_predictions.zarr keeps the test bars inside them.
run_dirPath or str or None: Directory receiving ic_series.csv and test_predictions.zarr; None writes nothing.
extra_metricscallable, optional: extra_metrics(label, timestamps, symbols) is called for every label and scored split with the bars and symbols that split is scored on, and the metrics it returns are added under the same keys as the others ({split}_{key}, prefixed by the label after the first). An ensemble adds member_correlation this way.

Returns:

dict[str, float]: The metrics, NaN where a metric is undefined.

Notes

Written into run_dir:

ic_series.csv, columns split, timestamp, ic, rank_ic, rank_autocorrelation: the first label’s per-bar IC, rank IC and rank autocorrelation behind its {split}_ic / {split}_icir / {split}_rank_autocorrelation, one row per bar of each scored split in SPLITS order; a bar where the IC and the rank IC are both undefined has no row.
test_predictions.zarr: every label’s prediction on the first label’s test bars inside test_bounds; not written when there are none.

Examples

>>> metrics = evaluate(
...     predictions, truth, labels={"ret": label}, label_scales={"ret": "raw"},
...     segments=segments, test_bounds=("2024-01-31", "2024-02-09"), run_dir=None,
... )
>>> sorted(metrics)[:4]
['test_ic', 'test_icir', 'test_mae', 'test_mse']