quantlab.dataset.raw.ConversionResult#

class quantlab.dataset.raw.ConversionResult(zarr_path, ledger_path, granularity, pinned_symbols, windows_planned, windows_written, windows_skipped, rows_written, peak_window_bytes, predicted_peak_bytes=None, resumed=False, cancelled=False, rebuild_rolled_back=False)[source]#

Bases: object

Summary of one chunked conversion from raw files to a Zarr store.

RawDataset.from_raw_data_chunked stores an instance on the dataset’s last_chunk_result attribute when it finishes, so a caller can report what the run did. The attributes are documented inline below.

Examples

>>> ds = DemoDataset(config).from_raw_data_chunked(granularity="month")
>>> result = ds.last_chunk_result
>>> result.windows_written, result.rows_written, result.resumed
(1, 6, False)

cancelled = False#: Whether a CancelToken stopped the loop between windows. Without this flag a cancelled run would look like a run that had less work to do.

granularity#: Window granularity the run planned on ("year", "quarter", …).

ledger_path#: Path of the ledger file (a JSON sidecar next to the store) that records completed windows.

peak_window_bytes#: Largest in-memory size (window.nbytes) of any window this run built; None when no window was written.

pinned_symbols#: Number of symbols on the pinned symbol axis, the fixed list of symbols for the whole date range that every window is written on.

predicted_peak_bytes = None#: The caller’s own estimate of the peak, passed through unchanged; usually None.

rebuild_rolled_back = False#: Whether an on_new_listing="rebuild" run was cancelled and rolled back, restoring the previous store and ledger. Only True together with cancelled. When set, windows_written and rows_written describe work that no longer exists on disk.

resumed = False#: Whether at least one window was skipped, meaning this run continued an earlier one.

rows_written#: Rows appended along the append dimension, summed over written windows.

windows_planned#: Number of windows the planner produced over the observed timestamps.

windows_skipped#: Number of windows skipped because the ledger already recorded them.

windows_written#: Number of windows this run built and appended.

zarr_path#: The Zarr store that was written (config.zarr_file_path).