quantlab.factor.base.Factor#

class quantlab.factor.base.Factor(config)[source]#

Bases: Component, ABC

Backend-agnostic base class for factors and labels.

A Factor holds a config whose dataset attribute is the dataset it reads from. Asked for a date range, it answers compute(start, end) from its inputs or read(start, end) from its own store, which build(start, end) writes and extend(end) lengthens. None of these holds a panel or changes a config, the factor’s or its dataset’s, so one dataset object can feed several factors. The returned panel is what the model layer consumes as features; to use a factor as a label, wrap it in quantlab.label.forward.Forward. Subclasses implement _get_factor_names and _compute_panel (both backends here do).

Assigning config runs the property setter, which sets name and resolves factor_names when they are not pinned, so factor names are known as soon as the object is constructed, before anything is computed.

Parameters:

configBaseFactorConfig: The factor config. Its dataset field is the dataset it reads from, or a list of datasets, which the factor merges into one MergedDataset (see quantlab.dataset.merged).

Attributes:

data_backendXrBackend: Holds the last bar a stream-mode factor computed; empty otherwise.

Examples

>>> factor = MyFactor(config)
>>> factor.get_factor_names()          # known before anything is computed
>>> panel = factor.compute("2024-02-01", "2024-02-29")
>>> factor.build("2024-01-01", "2024-06-30")
>>> panel = factor.read("2024-02-01", "2024-02-29")

PEAK_COPIES = 4#: How many times its own 8 bytes a variable takes at the peak of a computation: the copies, intermediate results and buffers made on the way. A rough, conservative figure; see cell_bytes.

analyze(start, end, factor_names=None, frets=None, output_dir=None, quantiles=5, data_strategy='cal', plot=True)[source]#

Report how well this factor predicts forward returns, alphalens style.

Every analyzed factor variable is paired with every variable of every forward-return label (fret). For each pair the factor and fret panels are joined on their common timestamps and symbols, then quantlab.analysis.factor_report.FactorAnalyzer computes the information analysis (per-period Spearman IC, its mean, std, IR, t-statistic, p-value, skew, kurtosis and monthly means), the returns analysis (mean forward return per factor quantile, top-minus-bottom spread, cumulative returns) and the turnover analysis (quantile turnover, lag-1 factor rank autocorrelation), and draws one composite matplotlib figure unless plot is False.

The factor and every fret are asked for their panels from start to end: computed from their inputs with compute(start, end) under data_strategy="cal", read from their stores with read(start, end) under "read".

Parameters:

start, endstr, datetime.date or pd.Timestamp: The range to analyze, both inclusive.
factor_nameslist of str, optional: Variables of this factor to analyze. All of get_factor_names() when None.
fretslist of Forward: Forward-return labels, for example quantlab.label.predefined.fret.Return or any factor wrapped in quantlab.label.forward.Forward; each one’s panel gives the forward returns and its span_bars() the horizon. Required.
output_dirstr, optional: When given, the directory is created and summary.json, summary.csv, ic.csv, monthly_ic.csv, quantile_returns.csv, turnover.csv, one <factor>__<fret>.png per pair and config.json are written there, plus factor_correlation.csv, factor_correlation_pairs.csv, factor_clusters.csv and factor_correlation.png when two or more factor variables are analyzed. config.json holds {"factor": ..., "frets": [...]}, each rebuildable with rebuild. With plot=False no PNG is written. When None, nothing is written.
quantilesint, default 5: Number of equal-count factor buckets per timestamp.
data_strategy{“cal”, “read”}, default “cal”: Whether the panels are computed or read from the stores build wrote, for the factor and every fret alike.
plotbool, default True: Draw the figures: to PNG files in output_dir, else into figures. False draws none, which is most of the time a report over many factors takes.

Returns:

FactorAnalysis: pairs (metrics per "<factor>__<fret>"), figures (matplotlib figures, same keys), the headline metrics per pair through summary() and tidy tables through summary_table(), ic_table(), quantile_returns_table(), turnover_table() and monthly_ic_table(); with two or more factor variables also correlation, their FactorCorrelation.

Raises:

ValueError: If frets is empty, data_strategy is neither "cal" nor "read", a factor name is unknown, a fret’s most common bar spacing differs from the factor’s, or the panels share no cells.

Examples

factor is a Momentum (momentum_5) over eight symbols, and fwd a quantlab.label.predefined.fret.Return with n_forward_periods=1 over the same symbols and dates, its adjOpen 0.99 times the momentum’s Close:

>>> result = factor.analyze(
...     "2024-02-01", "2024-02-29", frets=[fwd], quantiles=4,
...     output_dir="data/analysis/momentum",
... )
>>> list(result.pairs)
['momentum_5__ret_1']
>>> cols = ["factor", "fret", "ic_mean", "ic_t_stat", "mean_spread"]
>>> result.summary_table()[cols].round(4)
       factor   fret  ic_mean  ic_t_stat  mean_spread
0  momentum_5  ret_1  -0.0279    -0.4301      -0.0014
>>> sorted(os.listdir("data/analysis/momentum"))
['config.json', 'ic.csv', 'momentum_5__ret_1.png', 'monthly_ic.csv', 'quantile_returns.csv', 'summary.csv', 'summary.json', 'turnover.csv']
>>> result = factor.analyze(
...     "2024-02-01", "2024-02-29", frets=[fwd], quantiles=4,
...     output_dir="data/analysis/momentum_tables", plot=False,
... )
>>> result.figures, sorted(os.listdir("data/analysis/momentum_tables"))
({}, ['config.json', 'ic.csv', 'monthly_ic.csv', 'quantile_returns.csv', 'summary.csv', 'summary.json', 'turnover.csv'])

append_rows(rows, end)[source]#

Append rows, the bars after the recorded range up to end, and move its end.

The write half of extend, for rows computed elsewhere; the axes widen to fit as in extend, and empty rows (no bar in the range) only move the recorded end.

Parameters:

rowsxr.Dataset: The factor’s outputs on the bars after the recorded end, up to end.
endstr, datetime.date or pd.Timestamp: The new end of the store’s range.

Returns:

Self: self, for chaining.

Raises:

ValueError: As extendable_range.

Examples

>>> factor.append_rows(factor.compute("2024-02-01", "2024-02-29"), "2024-02-29").store_range()
('2024-01-01', '2024-02-29')

bar_after(date, n)[source]#

Return the bar n bars after date on the factor calendar, or the last bar there is.

The forward counterpart of bar_before, with BaseDataset.bar_after’s rules: the bars after date are those stamped strictly later, n=1 is the first of them, n=0 returns date itself, and when fewer than n bars follow the calendar’s last bar is returned, date itself when none follow, so the result is never before date. A resampled factor counts its resampled bars. Only timestamps are read.

Parameters:

datestr, datetime.date or pd.Timestamp: The date to count forward from.
nint: How many bars to count forward; non-negative.

Returns:

pd.Timestamp: The timestamp of that bar.

Raises:

ValueError: If n is negative.

Examples

>>> daily = minute.resample("1d", "last")  # five business days from 2024-01-02
>>> daily.bar_after("2024-01-05", 1)  # Friday: the bar after is Monday
Timestamp('2024-01-08 00:00:00')
>>> daily.bar_after("2024-01-04", 10)  # the bars end on 2024-01-08
Timestamp('2024-01-08 00:00:00')

bar_before(date, n)[source]#

Return the bar n bars before date on the factor calendar.

The bars before date are those stamped strictly earlier; n=1 is the last of them and n=0 returns date itself, as BaseDataset.bar_before does. A factor that is not resampled answers with its dataset’s; a resampled factor counts its resampled bars (calendar), so a daily factor computed from minute bars moves by trading days. Only timestamps are read.

Parameters:

datestr, datetime.date or pd.Timestamp: The date to count back from.
nint: How many bars to count back; non-negative.

Returns:

pd.Timestamp: The timestamp of that bar.

Raises:

ValueError: If n is negative.
InsufficientHistoryError: If fewer than n bars exist before date; a subclass of ValueError carrying available and requested, both in the factor’s bars.

Examples

>>> daily = minute.resample("1d", "last")  # five business days from 2024-01-02
>>> daily.bar_before("2024-01-08", 1)  # Monday: the bar before is Friday
Timestamp('2024-01-05 00:00:00')
>>> daily.bar_before("2024-01-03", 3)
Traceback (most recent call last):
quantlab.dataset.base.InsufficientHistoryError: Momentum.bar_before(): only 1 bar(s) exist before '2024-01-03' in the panel held in memory resampled to '1d', but 3 were requested.

build(start, end)[source]#

Compute start to end and write it as the factor store.

The store at store_path is replaced by compute(start, end), its symbol axis in sort_symbol_axis order as extend keeps it, and the range is recorded beside it, in <store_path>.range.json, for store_range, read and extend. The factor holds nothing afterwards and no config changes. Only the store’s owner builds it (see owns_store). The store and its range are written through quantlab.backend.ranged.RangedStore, as a factor risk model’s stores are.

Parameters:

start, endstr, datetime.date or pd.Timestamp: The range the store covers.

Returns:

Self: self, for chaining.

Raises:

ValueError: If the factor has no store path or does not own its store.

Examples

>>> factor.build("2024-01-01", "2024-03-31").store_range()
('2024-01-01', '2024-03-31')

calendar(start, end)[source]#

Return the factor’s bars from start to end, both inclusive, timestamps only.

The factor calendar: the bars of the factor’s output panel, which bar_before and bar_after count on (ADR 0051). A factor that is not resampled answers with its dataset’s calendar (BaseDataset.calendar). A factor made with resample answers with its resampled bars: the dataset’s bars over the range, bucketed the way the dataset cuts them (BaseDataset.resample_labels) and labelled as the resampled panel labels them. Only timestamps are read, so a DataRecorder records nothing.

Parameters:

start, endstr, datetime.date or pd.Timestamp: The range. A date-only end includes every bar of that day.

Returns:

pd.DatetimeIndex: The sorted bar timestamps in the range.

Raises:

ValueError: If start is after end.

Examples

A momentum on one-minute bars, and the same factor resampled to daily (five business days of four minute bars each):

>>> minute.calendar("2024-01-04", "2024-01-05")
DatetimeIndex(['2024-01-04 09:30:00', '2024-01-04 09:31:00',
               '2024-01-04 09:32:00', '2024-01-04 09:33:00',
               '2024-01-05 09:30:00', '2024-01-05 09:31:00',
               '2024-01-05 09:32:00', '2024-01-05 09:33:00'],
              dtype='datetime64[ns]', freq=None)
>>> daily = minute.resample("1d", "last")
>>> daily.calendar("2024-01-04", "2024-01-08")  # Thursday to Monday
DatetimeIndex(['2024-01-04', '2024-01-05', '2024-01-08'], dtype='datetime64[ns]', freq=None)

cell_bytes()[source]#

Return the bytes one bar of one symbol takes: at peak, and as output.

The output is 8 bytes per factor output. The peak counts every input variable the factor reads (_input_variables; as many as the outputs when it reads all of them) and every output at 8 bytes, PEAK_COPIES times over. A chunked build (quantlab.factor.chunking) sizes its chunks with it.

Returns:

tuple[int, int]: (peak, output) bytes per bar and symbol.

Examples

>>> factor.cell_bytes()   # one input column, one output
(64, 8)

property class_name#

Bare class name, used in log and error messages.

Examples

>>> factor.class_name
'Momentum'

compute(start, end)[source]#

Compute the factor from start to end, both inclusive.

The inputs are requested from the dataset from warmup_bars bars before start up to end, so rolling operators are warm on the first requested bar; the result holds only the requested range. A resampled factor is computed on its dataset’s own bars and returned on its resampled bars. Neither this factor’s config nor its dataset’s config changes, and the factor holds nothing afterwards.

If the dataset holds fewer than warmup_bars bars before start, a UserWarning states the shortfall in bars and the computation starts from the first bar there is.

Parameters:

start, endstr, datetime.date or pd.Timestamp: The range to return. A date-only end includes every bar of that day.

Returns:

xr.Dataset: The factor panel on (timestamp, symbol), in memory.

Raises:

ValueError: If start is after end.

Examples

>>> panel = factor.compute("2024-02-01", "2024-02-10")  # warmup_bars=20
>>> dict(panel.sizes)
{'timestamp': 10, 'symbol': 8}

With only two bars before start, the call warns:

UserWarning: Momentum.compute(): 20 warm-up bar(s) are needed
before '2024-01-03' but the dataset holds only 2; the first
bars are short by 18 bar(s) of warm-up.

property config#

The factor’s normalised config: name and factor_names filled in.

Examples

>>> factor.config is config
False
>>> factor.config.name, config.name
('quantlab.factor.predefined.momentum.Momentum', None)

copy()[source]#

Return a copy with its own config, components and empty backend.

Every component the config holds (its dataset, a wrapped factor, a roster, a benchmark, each dataset of a many-field) is copied with its own copy(), and the rest of the config is deep-copied, so the copy shares no mutable state with this factor, not even a kwargs dict.

Examples

>>> other = factor.copy()
>>> other == factor, other.config.dataset is factor.config.dataset
(True, False)

extend(end)[source]#

Append the bars after the recorded range, up to end.

The bars are computed with compute, so they are warmed from the dataset’s history, then appended to the store; its timestamp, symbol and variable axes widen to fit, and cells created by widening are filled from _widen_fill_values(). The recorded range then ends at end.

A resampled factor appends resampled bars. compute reads every source bar a resampled bar covers, past the range’s ends too, so each stored bar is whole and a store built in pieces equals one built at once: a minute factor’s daily store can be built a quarter at a time.

Parameters:

endstr, datetime.date or pd.Timestamp: The new end of the store’s range.

Returns:

Self: self, for chaining.

Raises:

ValueError: If the factor has no store path, does not own its store (see owns_store), the store has no recorded range, or the recorded range already reaches end.

Examples

>>> factor.build("2024-01-01", "2024-03-31").store_range()
('2024-01-01', '2024-03-31')
>>> factor.extend("2024-04-30").store_range()
('2024-01-01', '2024-04-30')

extendable_range(end)[source]#

Return the recorded range, refusing an extend to end.

extend’s refusals, before anything is computed: a caller that computes the rows elsewhere asks this first, then hands them to append_rows.

Parameters:

endstr, datetime.date or pd.Timestamp: The end an extend would reach.

Returns:

tuple of str: The recorded (start, end).

Raises:

ValueError: If the factor has no store path, does not own its store, the store has no recorded range, or the recorded range already reaches end.

Examples

>>> factor.extendable_range("2024-03-31")
('2024-01-01', '2024-02-29')
>>> factor.extendable_range("2024-02-10")
Traceback (most recent call last):
ValueError: Momentum.extend(): the store at ... already covers 2024-01-01 to 2024-02-29; ...

get_factor_names()[source]#

Return the names of the columns this factor produces.

Examples

>>> factor.get_factor_names()
('momentum_20',)

keep_compiled()[source]#

Keep whatever the factor compiles across compute calls inside the block.

Does nothing here; FactorKunQuant keeps its compiled library, which it otherwise looks up again on every call (compile_graphs compiles a graph only once). A computation split into many calls (quantlab.factor.chunking) runs inside it.

Examples

>>> with factor.keep_compiled():
...     january = factor.compute("2024-01-01", "2024-01-31")
...     february = factor.compute("2024-02-01", "2024-02-29")

property num_factors#

Number of columns this factor produces.

Examples

>>> factor.num_factors
1

output_symbols()[source]#

Return the symbols compute outputs, without computing anything.

The symbol axis of config.dataset’s store here. A factor whose dataset is not where its symbols come from (a wrapper reading its exposures, or keeping a roster) overrides it. quantlab.factor.chunking sizes its chunks on it.

Examples

>>> factor.output_symbols()[:2]
['S0USDT', 'S1USDT']

owns_store()[source]#

Whether the factor may write its store: it holds exactly its outputs.

A store is written only by its owner, the factor whose outputs are every variable of it; a factor pinned to some of them reads it but never writes it (ADR 0029). A store that does not exist yet belongs to whoever writes it first.

Examples

>>> owner.owns_store(), owner_pinned_to_one_output.owns_store()
(True, False)

read(start, end, symbols=None)[source]#

Return the factor store from start to end, both inclusive.

The stored panel is opened lazily; the factor holds nothing afterwards. The range must lie inside the one recorded beside the store by build and extend (see store_range). A resampled factor reads its own store when one has been built, otherwise it resamples on read as a resampled dataset’s panel does (quantlab.backend.resample.resample_on_read): the part of the source factor’s store the range needs, cut to symbols first, is aggregated, and those source bars are what is recorded.

Only the factor’s own outputs (get_factor_names()) are returned, so a factor pinned to some outputs of a store shared with a factor of more outputs reads just those (ADR 0029). symbols is a pure query, like BaseDataset.panel’s; a model reads a store on a roster through quantlab.factor.predefined.roster.RosterFactor.

This is the factor read seam: inside an open quantlab.runs.record.DataRecorder the request, its symbols included, is logged and fingerprinted, over every returned variable, when the recorder closes (over the source bars, keyed by the source store when the factor is not mapped, for a resample on read).

Parameters:

start, endstr, datetime.date or pd.Timestamp: The range to return. A date-only end includes every bar of that day.
symbolssequence, optional: Symbol labels to keep, in the order given, of the store’s own type. None keeps every symbol of the store.

Returns:

xr.Dataset: The panel on (timestamp, symbol).

Raises:

ValueError: If start is after end, the store has no recorded range, the recorded range does not contain the requested one, or the store lacks one of the factor’s outputs or of symbols.

Examples

>>> factor.build("2024-01-01", "2024-03-31").store_range()
('2024-01-01', '2024-03-31')
>>> dict(factor.read("2024-02-01", "2024-02-10").sizes)
{'timestamp': 10, 'symbol': 8}
>>> panel = factor.read("2024-02-01", "2024-02-10", symbols=["S3USDT", "S1USDT"])
>>> panel["symbol"].values.tolist()
['S3USDT', 'S1USDT']
>>> factor.read("2024-03-20", "2024-04-10")
Traceback (most recent call last):
ValueError: Momentum.read(): the store at ... covers 2024-01-01 to 2024-03-31, ...

refuse_unless_owner(method)[source]#

Raise unless the factor owns its store (see owns_store): the guard of every write.

build, extend, write_rows, append_rows and extendable_range run it; a caller that computes rows elsewhere runs it before computing anything. A view overrides it.

Parameters:

methodstr: The write being refused, named in the message ("build" or "extend").

Raises:

ValueError: If the store at store_path exists and its variables are not exactly get_factor_names().

Examples

>>> owner.refuse_unless_owner("build")   # the owner passes
>>> owner_pinned_to_one_output.refuse_unless_owner("build")
Traceback (most recent call last):
ValueError: Momentum.build(): the store at ... holds [...], not just this factor's outputs [...]; ...

resample(freq, how)[source]#

Return a copy of this factor whose panel is resampled onto freq.

The factor is still computed on its dataset’s own bars; only the computed panel is aggregated, so a minute-bar factor becomes a daily one without changing what it measures. The copy’s config carries resample_freq=freq and resample_how=how; compute, read and build on it answer on the resampled bars. This factor and its dataset are not changed.

A resampled factor cannot stream. build() and extend() write to store_path, the resampled store beside the source. The bars are cut the way the dataset cuts them (see BaseDataset.resample_labels).

Parameters:

freqstr: A ResampleFrequency token, coarser than the dataset’s bars.
howdict[str, str] or str: One ResampleMethod for every factor variable, or a {variable: method} dict naming every one.

Returns:

Self: A new factor of the same class.

Raises:

ValueError: If freq or a method is not a known token. Whether freq is coarser than the bars and how names every variable is checked when a panel is requested.

Examples

>>> daily = factor.resample("1d", "last")      # factor on minute bars
>>> factor.compute("2024-01-02", "2024-01-03").sizes["timestamp"]
780
>>> daily.compute("2024-01-02", "2024-01-03").sizes["timestamp"]
2

rewind(end)[source]#

Drop the bars after end and end the recorded range there: the inverse of extend.

The bars up to end stay as they are and the symbol and variable axes keep every label, so the next extend computes the dropped bars again (quantlab.backend.ranged.RangedStore.rewind). Only the factor owning its store may rewind it (refuse_unless_owner).

Parameters:

endstr, datetime.date or pd.Timestamp: The new end of the store’s range, inside it and before its end.

Returns:

Self: self, for chaining.

Raises:

ValueError: If the factor has no store path, does not own its store, the store has no recorded range, or end is not inside the recorded range and before its end.

Examples

>>> factor.build("2024-01-01", "2024-03-31").rewind("2024-02-29").store_range()
('2024-01-01', '2024-02-29')
>>> factor.extend("2024-03-31").store_range()
('2024-01-01', '2024-03-31')

property store_path#

Return the Zarr store this factor reads and writes.

This is config.file_path for a factor that is not resampled. A resampled factor uses a store beside it with _resample_<freq> added to the name, so the resampled panel never overwrites the source bars. None when config.file_path is None.

Examples

>>> factor.store_path
'data/factors/momentum.zarr'
>>> factor.resample("1d", "last").store_path
'data/factors/momentum_resample_1d.zarr'

store_range()[source]#

Return the (start, end) the store was built for, if recorded.

None when the store was not written by build.

Examples

>>> factor.build("2024-01-01", "2024-03-31").store_range()
('2024-01-01', '2024-03-31')

store_uses()[source]#

Return the store this factor uses itself: its store, owned or only read.

A factor with a store declares it with itself as the RangedStoreHandle (store_range(), extend), so the live day and the data update extend a factor and a factor risk model’s stores in one loop: "owns" when it may write the store (owns_store), "reads" when it is pinned to some of the store’s variables. A factor without a store declares none; a view (FactorView) declares the wrapped store. See Component.store_uses.

Examples

>>> [(use.relation, use.handle is factor) for use in factor.store_uses()]
[('owns', True)]
>>> [use.relation for use in owner_pinned_to_one_output.store_uses()]
['reads']
>>> factor_without_file_path.store_uses()
()

stored_symbols()[source]#

Return the symbol axis of the store read reads, in axis order.

Only the coordinate is read, never a data variable. A resampled factor without a store of its own reads the source factor’s store, as read does. The labels keep the store’s own type.

Examples

>>> factor.build("2024-01-01", "2024-03-31").stored_symbols()[:2]
['S0USDT', 'S1USDT']

property warmup_bars#

Bars of history compute reads before the requested start.

Counted on the input dataset’s own calendar, so weekends and holidays are skipped, not counted. It is config.warmup_bars.

Examples

>>> factor.warmup_bars
20

write_rows(rows, start, end)[source]#

Replace the store with rows, computed over start to end, and record the range.

The write half of build, for rows computed elsewhere: a chunk computed in a worker process (ChunkedBuilder). The rows are written as build writes compute(start, end), through the same quantlab.backend.ranged.RangedStore and the same owner guard (refuse_unless_owner), so write_rows then append_rows leave the store build then extend would.

Parameters:

rowsxr.Dataset: The factor’s outputs on (timestamp, symbol) over the range.
start, endstr, datetime.date or pd.Timestamp: The range rows cover, recorded as the store’s range.

Returns:

Self: self, for chaining.

Raises:

ValueError: If the factor has no store path or does not own its store.

Examples

>>> factor.write_rows(factor.compute("2024-01-01", "2024-01-31"), "2024-01-01", "2024-01-31").store_range()
('2024-01-01', '2024-01-31')