quantlab.factor.chunking.ChunkedBuilder

class quantlab.factor.chunking.ChunkedBuilder(factor, granularity=None, workers=1)

Bases: object

Build and extend factor’s store one chunk of time at a time.

The chunks are one TimeChunkPlanner period each, cut on the factor’s dataset’s own bars. With granularity set every range is cut that way; with None the range is computed whole when it fits in memory_budget(), and otherwise cut at the coarsest granularity whose largest chunk, warm-up included, fits. What a bar of a symbol takes is the factor’s cell_bytes(), and the symbols are the ones it outputs (output_symbols()), not its dataset’s.

The builder is a tool, not a component: it has no config and no store, and build and extend return the factor, whose store a model then reads. workers processes compute chunks at once (parallel_workers); see the module docstring.

Parameters:

factorFactor

The factor whose store is built; it must own its store (Factor.owns_store). It may be resampled.

granularitystr, optional

One TimeChunkPlanner granularity for every chunk; None chooses the coarsest one whose chunks fit in memory.

workersint, default 1

Worker processes computing chunks at once; the factor’s own threads (njobs) run in each. 1 computes them one after another here.

Raises:

ValueError

If factor does not own its store (the factor’s own refuse_unless_owner text: a view has no store of its own, a pinned factor does not hold every variable of it), granularity is unknown or finer than the factor’s resampled bars, or workers is not a positive integer. The ownership refusal names build whichever of build and extend follows.

Examples

With momentum a Momentum factor over three symbols’ daily prices, January to March 2024, and its store not built yet:

>>> builder = ChunkedBuilder(momentum, granularity="month")
>>> builder.build("2024-01-01", "2024-02-15").store_range()
('2024-01-01', '2024-02-15')
>>> builder.extend("2024-03-31") is momentum
True
>>> momentum.store_range()
('2024-01-01', '2024-03-31')
>>> ChunkedBuilder(RosterFactor(RosterConfig(factor=momentum, roster=roster)))
Traceback (most recent call last):
ValueError: RosterFactor.build(): a view has no store of its own; ...

build(start, end)

Build the factor’s store from start to end, a chunk at a time.

With one worker the first chunk is the factor’s build and every later one its extend; with more, the chunks computed in the workers are written with write_rows and append_rows. See Factor.build.

Parameters:

start, endstr, datetime.date or pd.Timestamp

The range to build, both inclusive.

Returns:

Factor

The factor, for chaining (.store_range(), .read(...)).

Raises:

ValueError

As Factor.build, or as parallel_workers.

MemoryError

As plan.

Examples

>>> builder.build("2024-01-01", "2024-03-31").store_range()
('2024-01-01', '2024-03-31')

property class_name

Return the builder’s class name, used in its messages.

Examples

>>> builder.class_name
'ChunkedBuilder'

compute(start, end)

Compute the factor from start to end, a chunk at a time if needed.

The panel is allocated once, on the factor’s own bars (Factor.calendar), and each chunk is written into it, so only one chunk’s computation is in memory beside it. Nothing is written to the store.

Parameters:

start, endstr, datetime.date or pd.Timestamp

The range to return, both inclusive.

Raises:

MemoryError

If the output alone would not fit in memory_budget(): build the store instead and read it a range at a time.

Examples

>>> dict(builder.compute("2024-01-10", "2024-03-31").sizes)
{'timestamp': 82, 'symbol': 3}

extend(end)

Append the bars after the store’s recorded range, a chunk at a time.

Every chunk is the factor’s extend, or, with more than one worker, its append_rows; see Factor.extend. After a build that failed part way, this resumes it.

Parameters:

endstr, datetime.date or pd.Timestamp

The new end of the store’s range.

Returns:

Factor

The factor, for chaining.

Raises:

ValueError

As Factor.extend, or as parallel_workers.

Examples

>>> builder.build("2024-01-01", "2024-01-20").store_range()
('2024-01-01', '2024-01-20')
>>> builder.extend("2024-03-31").store_range()
('2024-01-01', '2024-03-31')

property factor

Return the factor whose store the builder writes.

Examples

>>> builder.factor is momentum
True

property granularity

Return the granularity every chunk is cut at; None chooses from the memory available.

Examples

>>> builder.granularity
'month'

parallel_workers(chunks)

Return how many worker processes compute chunks: workers, cut to fit.

No more than there are chunks, and no more than the largest chunk’s peak (cell_bytes, warm-up included) fits in memory_budget() side by side; a cut below workers is logged.

Raises:

ValueError

If more than one worker would run and the factor does not rebuild from its config.

Examples

>>> _, chunks = builder.plan("2024-01-01", "2024-03-31")
>>> ChunkedBuilder(momentum, granularity="month", workers=8).parallel_workers(chunks)
3

plan(start, end)

Return the granularity and the (first, last) bar of every chunk of a range.

The granularity is None when the range is one chunk because it fits whole. With workers above 1 and no granularity set, the coarsest granularity that cuts at least workers chunks whose largest fits workers times in memory_budget() is chosen, so every worker has a chunk; when none does, the plan is the one-worker plan.

Parameters:

start, endstr, datetime.date or pd.Timestamp

The range, both inclusive.

Raises:

MemoryError

If one chunk of the finest granularity would not fit either.

Examples

>>> granularity, chunks = builder.plan("2024-01-01", "2024-03-31")
>>> granularity, chunks[0]
('month', (Timestamp('2024-01-01 00:00:00'), Timestamp('2024-01-31 00:00:00')))
>>> ChunkedBuilder(momentum).plan("2024-01-01", "2024-03-31")[0] is None  # fits whole
True

property workers

Return the worker processes asked to compute chunks at once.

Examples

>>> builder.workers
1