quantlab.model.ensemble.BaseEnsemble#

class quantlab.model.ensemble.BaseEnsemble(members)[source]#

Bases: Component, ABC

Base class of an ensemble that combines the predictions of several models.

Subclass it, build the members, declare config_cls (a dataclass whose member fields are declared with component()) with a config property building it, and override from_config to construct the ensemble from the rebuilt fields; override _combine to replace the per-label rule (see the module docstring for every hook).

Parameters:

memberssequence: At least two models (BaseModel instances). A label two members predict must have the same config in both, and their test windows must overlap.

Attributes:

memberslist: The member models, in member order.

Raises:

ValueError: If there are fewer than two members, two members predict a label of one name with different configs, or the members’ test windows do not overlap.

Examples

A concrete ensemble passes its members to this constructor; given a SeedEnsemble of a model over one forward-return label:

>>> len(ensemble.members), ensemble.label_delays
(3, (1,))

assemble(runs, save_dir=None)[source]#

Assemble an ensemble unit from models trained separately and return its run.json.

Nothing is trained: run k, a "model" unit, is copied into member_{k}/ of a new {class}_trial_{timestamp} directory under save_dir, which becomes an "ensemble" unit that load and a backtest read as if train() had written it. Run k must be a model of member k’s class, trained on member k’s variables (check_checkpoint) and, when the ensemble varies seeds, under member k’s seed; every run must have the same test window and have fitted no bar of it.

Only the test segment is scored, from the members’ saved test predictions and the labels over the test window (no feature is read): the unit has test_* metrics only, and its assembled_from names each source run and its code record.

Parameters:

runssequence of Path or str: One trained model unit per member, in member order: its directory, run.json or checkpoint.
save_dirPath or str, optional: Where the unit’s directory is created; model_save_dir (the first member’s) by default.

Returns:

Path: Absolute path of the unit’s run.json, the ensemble’s load-mode checkpoint.

Raises:

ValueError: If the runs do not match the members as described, before anything is written.

Examples

>>> checkpoint = ModelEnsemble([xgb, mlp]).assemble([xgb_run, mlp_run], "models")
>>> sorted(TrainedRun.open(checkpoint).metrics)[:2]
['test_ic', 'test_icir']

assemble_cv(runs, save_dir=None)[source]#

Assemble a walk-forward unit of ensembles from walk-forward runs trained separately.

Run k is member k’s own walk-forward run (train_cv). Fold i of the result is the ensemble unit assemble would write from fold i of every run, so every run must have the same folds, and in each fold the same test window; the fold means of their test_* metrics are cv_mean. A backtester’s run_cv replays the result with the ensemble as its model.

Parameters:

runssequence of Path or str: One walk-forward run per member, in member order.
save_dirPath or str, optional: Where the unit’s directory is created; model_save_dir by default.

Returns:

TrainedRun: The "walk_forward" unit, whose folds are "ensemble" units.

Raises:

ValueError: If a run is not a walk-forward run, the runs’ folds differ, or a fold’s runs do not match the members as assemble requires, before anything is written.

Examples

>>> cv = ModelEnsemble([xgb, gats]).assemble_cv([xgb_cv.path, gats_cv.path])
>>> cv.kind, cv.folds[0].kind
('walk_forward', 'ensemble')

check_checkpoint(path)[source]#

Check an ensemble unit and every member checkpoint in it, loading nothing.

The unit, read through TrainedRun, must hold as many members as the ensemble has, each with this ensemble’s member class and seed, and every member checkpoint must pass that member’s check_checkpoint.

Parameters:

pathPath or str: The unit’s run.json that train() returned, or its directory.

Returns:

TrainedRun: The ensemble unit.

Raises:

FileNotFoundError: If path does not exist.
ValueError: If the unit cannot be opened or does not match this ensemble’s members, or a member’s own check fails.

Examples

>>> ensemble.check_checkpoint(checkpoint).kind
'ensemble'

check_hyperparameters()[source]#

Check every member’s hyperparameters, in member order.

Raises:

ValueError: From the first member whose hyperparameters are invalid.

Examples

>>> ensemble.check_hyperparameters()  # valid members return None

property class_name#

The ensemble’s class name, used in directory names and messages.

Examples

>>> ensemble.class_name
'SeedEnsemble'

collect()[source]#

Load every member’s features and labels into its data backend.

The members’ reads (_collect) are recorded once, by a DataRecorder keyed by component path within the ensemble, and written into the run.json of the ensemble or walk-forward unit train() or train_cv() writes next; its members and folds record none.

Returns:

Self: The ensemble itself, for chaining.

Examples

>>> ensemble.collect() is ensemble
True
>>> sorted(ensemble.training_record)[:1]
['model.factors.0.dataset']

property fitted_train_bounds#

The training window actually fitted, covering every member’s.

The earliest member start and the latest member end of the members’ fitted_train_bounds, so a bar after the end was fitted by no member. Known after train() or load().

Raises:

RuntimeError: If the members have been neither trained nor loaded.

Examples

>>> ensemble.fitted_train_bounds
('2024-01-01', '2024-01-31T00:00:00.000000000')

property label_delays#

Each label’s delay in bars, in the order of labels.

Examples

>>> ensemble.label_delays
(1,)

property label_scales#

Each label name’s prediction scale: "raw" or "standardized".

A label averaged over several members is "standardized" (the default combine is in per-bar z-score units); a label only one member predicts keeps that member’s scale.

Examples

>>> ensemble.label_scales
{'fwd_ret_1': 'standardized'}

property labels#

The union of the members’ label objects, in first-appearance order.

A label several members predict appears once, as the first member predicting it holds it.

Examples

>>> [label.get_factor_names() for label in ensemble.labels]
[('fwd_ret_1',)]

load(path)[source]#

Restore every member from an ensemble unit.

Every member checkpoint is checked (see check_checkpoint) before any is loaded.

Parameters:

pathPath or str: The unit’s run.json that train() returned, or its directory.

Returns:

Self: The ensemble itself, for chaining.

Raises:

FileNotFoundError: If path does not exist.
ValueError: As check_checkpoint.

Examples

>>> SeedEnsemble(model, [0, 1, 2]).load(checkpoint).members[0].model is not None
True

property model_save_dir#

The directory train() creates the ensemble directory in.

The first member’s config.model_save_dir.

Examples

>>> ensemble.model_save_dir.name
'models'

predict_window(start, end)[source]#

Predict every bar from start to end by combining the members.

The members’ predictions are combined by _combine: by default a label several members predict is the equal-weight mean of their per-bar z-scores and a label one member predicts is its prediction. Every member must be trained or loaded.

Parameters:

start, endstr: First and last bar to predict, inclusive.

Returns:

xr.Dataset: One variable per label name on (timestamp, symbol).

Examples

>>> out = ensemble.predict_window("2024-02-12", "2024-03-11")
>>> list(out.data_vars), out.sizes["timestamp"]
(['fwd_ret_1'], 21)

provenance()[source]#

What the ensemble’s top unit records: training_record and the code record.

Examples

>>> sorted(ensemble.collect().provenance())
['code', 'data_fingerprint']

property purge_bars#

The largest purge_bars among the members.

train_cv lays its folds out with it, so no member’s fitted label reads a test bar; each member still purges its own window by its own.

Examples

>>> ensemble.purge_bars
2

property test_bounds#

The test window (test_start, test_end) every member tests on.

The intersection of the members’ test windows: the latest start and the earliest end.

Raises:

ValueError: If the members’ test windows do not overlap.

Examples

>>> ensemble.test_bounds
('2024-02-05', '2024-02-09')

property tracker#

The first member’s tracker, through which the CV summary run is opened.

The ensemble has no tracker of its own; its members’ runs go through their own trackers.

Examples

>>> ensemble.tracker is ensemble.members[0].tracker
True

property tracking_project#

The first member’s tracking_project, so the summary sits beside the members’ runs.

Examples

>>> ensemble.tracking_project
'SeededHead'

train()[source]#

Train every member into one ensemble unit and return its run.json.

Every member’s hyperparameters are checked first. Then a new {class}_trial_{timestamp} directory under model_save_dir is filled by _train_into: member k is trained, in order, into the "model" unit member_{k}/ under a tracking run {MemberClass}_member_{k} of its own tracker, grouped by the directory’s name, and reseeds its generators from its own random_seed right before it trains. The directory then gets the evaluation files of the combined prediction (ic_series.csv, test_predictions.zarr, see _evaluate) and last, atomically, run.json, which makes it an "ensemble" unit (quantlab.runs.trained_run) recording the members, the windows and the metrics. If a member or the ensemble evaluation fails, the error propagates, no run.json is written and the files already written stay. Call collect() first.

Returns:

Path: Absolute path of the unit’s run.json, the ensemble’s load-mode checkpoint.

Examples

>>> checkpoint = ensemble.collect().train()
>>> run = TrainedRun.open(checkpoint)
>>> run.checkpoint == checkpoint, run.path.name.startswith("SeedEnsemble_trial_")
(True, True)
>>> sorted(p.name for p in run.path.iterdir())
['ic_series.csv', 'member_0', 'member_1', 'member_2', 'run.json', 'test_predictions.zarr']
>>> sorted(run.metrics)
['test_ic', 'test_icir', 'test_member_correlation', 'test_rank_ic', 'test_rank_icir', 'train_ic', 'train_icir', 'train_member_correlation', 'train_rank_ic', 'train_rank_icir']

property train_bounds#

The training window (train_start, train_end) covering every member’s.

The earliest member start and the latest member end, so a bar after train_end was seen in training by no member.

Examples

>>> ensemble.train_bounds
('2024-01-01', '2024-02-02')

train_cv(train_periods, expanding=False, test_periods=None)[source]#

Run a walk-forward cross-validation of the ensemble and return the walk-forward unit.

The procedure is a model’s, train_walk_forward (in quantlab.model.walk_forward_training), with the ensemble as the trained unit: the folds are laid out over walk_forward_bars (the first member’s), sliding or, with expanding=True, growing from the first fold’s start, and each training window loses its last L bars, L being purge_bars, the largest among the members. Every member’s hyperparameters are checked once, before any directory is created.

A new {class}_trial_{timestamp} directory under model_save_dir becomes a "walk_forward" unit, laid out as a model’s. Fold i is trained by train_fold into the "ensemble" unit fold_{i}/, as train() writes its directory, member k under the tracking run {MemberClass}_fold_{i}_member_{k}. Folds train one after another, and afterwards every member keeps the dates it was configured with. The fold means of the ensemble metrics, keyed cv_mean_{key}, and cv_n_folds go to the summary of a separate {class}_cv_summary run, opened through tracker (the first member’s) in the members’ project and group, and into the unit’s run.json, which a backtester’s run_cv replays with the ensemble as its model. Call collect() first.

Parameters:

train_periodsint: Number of timestamps in the first fold’s training segment, and in every fold’s when sliding.
expandingbool, default False: Train every fold from the first fold’s start instead of sliding a fixed-length window.
test_periodsint, optional: Number of timestamps in each fold’s test segment, and the step from one fold to the next. Defaults to train_periods // 5.

Returns:

TrainedRun: The "walk_forward" unit, whose folds are "ensemble" units with the fold’s ensemble metrics ({split}_ic, {split}_rank_ic, {split}_icir, {split}_rank_icir, {split}_member_correlation).

Raises:

ValueError: If test_periods is below 1, or is not given and train_periods is below 5, no timestamp falls inside the date range, the purge leaves a fold no training bar, or a member’s hyperparameters are invalid.

Examples

>>> cv = ensemble.collect().train_cv(train_periods=30)
>>> cv.kind, len(cv.folds), cv.folds[0].kind
('walk_forward', 8, 'ensemble')
>>> [member.seed for member in cv.folds[0].members]
[0, 1, 2]

train_fold(fold, run_dir, group)[source]#

Train one walk-forward fold of every member into run_dir, then restore their dates.

Every member’s config gets the fold’s dates before the purge (fold_config; each member purges them itself), and _train_into writes the "ensemble" unit, member k under the tracking run {MemberClass}_fold_{i}_member_{k}. Every member’s own config is put back afterwards, also when training raises.

Parameters:

foldFold: The fold to train.
run_dirPath or str: The fold’s unit directory, fold_{i}/ of the trial.
groupstr: Tracking group of the members’ runs, the trial directory’s name.

Examples

>>> ensemble.collect().train_fold(fold, tmp / "fold_0", group="trial")
>>> TrainedRun.open(tmp / "fold_0").kind
'ensemble'

property training_record#

What the last collect() read, by component path; empty before it.

Examples

>>> sorted(ensemble.collect().training_record)[:1]
['model.factors.0.dataset']

walk_forward_bars()[source]#

The first member’s walk_forward_bars: one fold geometry for every member.

Examples

>>> len(ensemble.collect().walk_forward_bars())
30