quantlab.acquisition.registry.convert#

quantlab.acquisition.registry.convert(descriptor, dataset_config, *, data_type=None, granularity='year', on_new_listing='refuse', predicted_peak_bytes=None, reporter=None, cancel=None)[source]#

Convert an already downloaded raw tier into a Zarr store, in-process.

This is the second step after a download. It reads the raw parquet tier and writes the store through the capability’s dataset_cls, one time window at a time. It uses no vendor client and no credential. The capability is looked up by (dataset_config.market, dataset_config.frequency, data_type).

No memory check runs here. A window too large for memory fails with an out-of-memory error, so choose the window size (granularity) yourself. predicted_peak_bytes is only copied into the result for reporting.

Parameters:

descriptorSourceDescriptor: The source whose raw tier is being converted.
dataset_configDatasetConfig: Where the raw tier is and where the store goes.
data_typestr or None, default None: Picks one capability when the vendor serves several at this (market, frequency).
granularitystr, default “year”: Length of each conversion window, such as "year" or "quarter".
on_new_listingstr, default “refuse”: What to do with a symbol that is not on the store’s fixed symbol axis; passed to from_raw_data_chunked.
predicted_peak_bytesint or None, default None: The caller’s own memory estimate, copied into the result.
reporterProgressReporter or None, default None: Receives a ProgressEvent per window.
cancelCancelToken or None, default None: Checked between windows. Windows already written stay in the store and in its record of finished windows, so a later call with the same config resumes at the first unwritten window and reports resumed=True.

Returns:

ConversionResult: The result the dataset stored on last_chunk_result, with predicted_peak_bytes filled in.

Raises:

ValueError: In three cases: the descriptor serves no such capability (the message lists what it does serve); several capabilities match with different dataset classes (pass data_type); or the matching capability has no dataset_cls, because its data is a stream of irregularly timed events that no panel can hold.

Examples

Needs a raw tier already downloaded by download_symbols():

from quantlab.dataset.config import DatasetConfig
from quantlab.acquisition.base import DataSourceRegistry
from quantlab.acquisition.registry import convert

source = DataSourceRegistry.get("tiingo")
dataset_config = DatasetConfig(
    raw_data_dir_path="data/us_equity/1d/nasdaq_data/tiingo",
    zarr_file_path="data/us_equity/1d/tiingo_1d.zarr",
    market="us_equity", frequency="1d", vendor="tiingo",
    start_date="2024-01-01", end_date="2024-05-31",
)
result = convert(source, dataset_config, granularity="year")
result.windows_written

A capability the vendor serves but no dataset can express is refused without touching disk:

>>> from quantlab.acquisition.base import DataSourceRegistry
>>> from quantlab.dataset.config import DatasetConfig
>>> tick_config = DatasetConfig(
...     raw_data_dir_path="data/alpaca", zarr_file_path="data/out.zarr",
...     market="us_equity",
...     frequency="tick", vendor="alpaca",
...     start_date="2024-01-01", end_date="2024-01-31",
... )
>>> convert(DataSourceRegistry.get("alpaca"), tick_config,
...         data_type="quotes")
Traceback (most recent call last):
...
ValueError: Alpaca Market Data: no raw-to-Zarr conversion exists for ...