quantlab.dataset.stock.StockDataset#

class quantlab.dataset.stock.StockDataset(dataset_config)[source]#

Bases: RawMarketDataset

US-equity dataset over a single vendor’s hive-partitioned raw tree.

The raw root must be the vendor’s own directory and hold only that vendor’s files (shards). Two vendors under one root would silently merge into a blended price series, so the root name, the vendor column and the parquet schema are all checked before any row is used. Daily (1d) data is partitioned by month, minute (1m) data by trading-session date, and tick data by date and symbol. A vendor that names its data types (bars, quotes, trades, crsp_daily, nbbo) keeps each one in its own directory beneath the vendor root, and the scan starts there; kwargs["data_type"] names it, which the registry’s convert() fills in from the capability. Tick data has no dense-panel form and can only be read with _scan_raw.

Parameters:

dataset_configDatasetConfig: Paths, market, frequency, vendor and date range of the dataset. For tick data, kwargs["data_type"] must be set ("quotes" or "trades" for Alpaca).

Examples

>>> config = DatasetConfig(
...     raw_data_dir_path="downloads/us_equity/1d/us_all/tiingo",
...     zarr_file_path="data/us_equity/1d/us_all.zarr",
...     market="us_equity",
...     frequency="1d",
...     vendor="tiingo",
...     start_date="2024-01-01",
...     end_date="2024-12-31",
... )
>>> StockDataset(config).from_raw_data_chunked(granularity="quarter")
>>> panel = StockDataset(config).panel("2024-01-01", "2024-12-31")

DERIVED_HIVE_KEYS = ('month', 'date')#: Hive keys that only describe the partition and are dropped after the scan. symbol is left out on purpose: in the tick layout the directory name is the only place the symbol is stored.

HIVE_SCHEMA_BY_FREQUENCY = {'1d': {'month': String}, '1m': {'date': Date}, 'tick': {'date': Date, 'symbol': String}}#: Hive key dtypes per frequency, with the same keys as RAW_HIVE_KEYS. Always passed to pl.scan_parquet explicitly: polars would infer a numeric-looking value such as symbol=8686 as an integer, and a string filter on it would then silently match nothing.

RAW_SHARD_SUFFIX = '.pqt'#: Filename suffix of one raw shard. The scan glob and has_raw_data both use it, so they always look at the same set of files.

SESSION_DATE_SLACK = datetime.timedelta(days=1)#: How far the intraday date filter widens the window at each edge. The date= key is a session date in the exchange’s time zone, while window edges are naive UTC datetimes, and the two can differ by up to a day. One day of slack reads at most two extra partitions, which the timestamp filter then trims exactly. Without it, the end of the window’s first session could be silently lost.

has_raw_data()[source]#

Return whether the raw tree holds at least one shard to convert.

This is the one place that checks whether raw data exists. _scan_raw uses it to tell a missing root apart from a window that simply has no rows, and the ingest scripts call it to refuse a conversion before it starts. It looks inside the data type’s directory (_scan_root), which a plain check of config.raw_data_dir_path would miss. It only lists files and opens no parquet file.

Returns:

bool: True if at least one .pqt shard exists below the scan root.

Examples

>>> ds = StockDataset(config)
>>> ds.has_raw_data()  # the vendor root exists but holds no shard
False
>>> ds.has_raw_data()  # once a month=2024-01/part.pqt shard lands
True