quantlab.runs.record.DataRecorder#

class quantlab.runs.record.DataRecorder(keys=(), expected=None, owner='DataRecorder')[source]#

Bases: object

Record the data a run reads, hash it once per request, compare on close.

A run opens one around everything it reads: a trained unit, a backtest run, a run_cv fold. While it is the innermost open recorder, every read through BaseDataset.panel, Factor.read and a factor risk model’s RiskStore.read is logged as a request; nested recorders log to the innermost only. Only leaf datasets record (a store or a held panel): a MergedDataset records nothing itself, its inputs do. A risk store is recorded under its model’s key and the store (risk_model.estimate), its reads merged into one request over the first to the last bar read (record_read’s reread_range).

When the context closes, each distinct request is read once more and hashed with _dataset_fingerprint over the requested variables (every variable of the panel when none were requested). records then maps each key to its requests, in the order first read; an in-memory dataset hashes the panel it holds. With expected given, the records are compared with it per key and request by digest alone; a changed, missing or extra request or key logs a warning showing the ranges and sizes as explanation, and nothing is raised.

If the body raises, what was read so far is hashed and compared partially: a request or key not read yet is not reported, every warning carries _PARTIAL_NOTE, and the original error propagates. The diagnostic never raises and never replaces that error: a failure of the hashing or the comparison itself only logs a warning, and records keeps what was hashed before it.

Parameters:

keysiterable of (object, str) pairs, optional: The key each dataset, factor or risk model is recorded under, matched by identity; a source given twice takes its first key. An unmapped source is recorded under "<ClassName>:<store path>", so two unmapped in-memory datasets share "FrameDataset:None"; map them.
expectedmapping, optional: A records of an earlier run to compare with on close.
ownerstr: The name that opens every warning.

Attributes:

recordsdict: {key: [entry, ...]}, filled when the context closes. Each entry is a _dataset_fingerprint record plus request: the first and last bar read (start, end), the requested symbols and variables (None for all); a source that reports its build settings adds settings, folded into digest.

Examples

prices is a StockDataset over a business-day store from 2024-01-01:

>>> with DataRecorder(keys=[(prices, "price_dataset")]) as recorder:
...     _ = prices.panel("2024-01-02", "2024-01-31", variables=["adjClose"])
>>> entry = recorder.records["price_dataset"][0]
>>> entry["request"]["variables"], entry["n_timestamps"]
(['adjClose'], 22)

Comparing a rebuilt run whose store was restated (restated holds the same symbols and bars, other prices) logs, through loguru:

with DataRecorder(keys=[(restated, "price_dataset")], expected=recorder.records):
    restated.panel("2024-01-02", "2024-01-31", variables=["adjClose"])

DataRecorder: data fingerprint mismatch for 'price_dataset', request
2024-01-02..2024-01-31, variables ['adjClose']: digest differs: values of
['adjClose'] changed (expected
2024-01-02T00:00:00 to 2024-01-31T00:00:00, 22 timestamps x 6 symbols, got
...). The data changed since the expected run; continuing