quantlab.factor.chunking

Chunked build: a factor’s store built and extended one chunk of time at a time.

A factor over a long range of many symbols may not fit in memory at once: an Alpha158 store of every listed stock over twenty years is larger than the machine. ChunkedBuilder splits the range into chunks of one calendar period each, each computed with its own warm-up, and writes them into the factor’s own store in time order. Joined, they equal the whole range computed at once, to within floating-point tolerance, provided the factor’s warmup_bars is enough for its rolling operators.

A chunked build is not a factor (ADR 0043): it writes only through the store owner’s public entries (build, extend, write_rows, append_rows, extendable_range, refuse_unless_owner), so only the owner writes the store (ADR 0029), and nothing reads through it; a model reads the factor itself. A factor that does not own its store (a view, a factor pinned to some of its store’s outputs) is refused when handed to the builder, before anything is computed. A build that fails part way leaves the finished chunks and their recorded range, and extend resumes from there. compute returns one panel in memory, filled chunk by chunk when only the computation would not fit, and refuses when the output alone would not.

A factor is chunked along time only: a cross-sectional operator needs every symbol of a bar at once.

A resampled factor is chunked on its source bars, in chunks no finer than its resampled bars, so every chunk holds whole resampled bars: a minute factor’s daily store is built a month at a time. Its output is sized on the factor calendar (Factor.calendar, ADR 0051): the panel compute fills is allocated on the factor’s own bars, daily for a minute-to-daily factor, while the chunks and their peak are counted on the dataset’s bars the factor reads.

With workers above 1 the chunks are computed in that many worker processes at once, each rebuilding the factor from its own config, and written in time order by this process through the factor’s write_rows and append_rows, which write them as its build and extend would, behind the same owner guard. The workers are a joblib.Parallel (loky) fan-out, like the repository’s others.

Classes

ChunkedBuilder

Build and extend factor's store one chunk of time at a time.

Functions

memory_budget	Return the bytes a chunk may take: MEMORY_FRACTION of the memory available.
release_memory	Hand the heap memory freed by the last chunk back to the system.