quantlab.acquisition.symbol_batch#
The symbol-batched download engine: SymbolBatchAcquisition.
SymbolBatchAcquisition is the abstract base of one download family: a
vendor queried one symbol batch at a time over a date window (Tiingo, Alpaca,
FRED, the retired WRDS CRSP and TAQ downloads). It is not the base of every
download. Sharadar’s whole tables and Massive’s daily files and REST
endpoints are pulled by their own clients, and every vendor the daily update
keeps current registers a RawTierRefresh (quantlab.acquisition.refresh)
instead (ADR 0049). The descriptions of every vendor (SourceDescriptor,
Capability, DataSourceRegistry) live in quantlab.acquisition.base.
A subclass declares VENDOR and RAW_COLUMNS and implements
_fetch_page, which issues one request and returns one page of rows. The
base class supplies everything around that request: splitting the symbol list
into batches, following pagination, fetching batches concurrently on worker
threads, isolating a failure to the batch that caused it, stopping the whole
run when the vendor’s request quota runs out, retrying a request the server
asked to repeat, honouring a caller’s cancel request, reporting progress, and
removing credentials from every captured error message. The retry and the
scrubbing are the shared RequestPolicy
(quantlab.acquisition._support.request_policy), so this family waits as
the Sharadar and Massive clients do. BatchOutcome summarises one batch and AcquisitionResult
one whole download.
A few terms recur throughout the module. A shard is one parquet file
holding the rows of one page for one partition. Shards are laid out as a
hive tree, where each directory level is named key=value (for example
month=2024-01/part-....pqt), so a reader can skip whole directories by
key. A watermark is the last date a symbol is known to be downloaded
through; it is stored in a small per-symbol JSON file called a sidecar,
which lets refresh() fetch only what is missing. The page ledger
records which pages of a batch have landed, so an interrupted batch resumes
from the next page instead of starting over.
This layer writes raw files only. Converting them into the project’s
canonical panel, an xarray.Dataset indexed by timestamp and
symbol, is the matching dataset class’s job. Coverage bookkeeping lives
in quantlab.acquisition._support.coverage, the page ledger in
quantlab.acquisition._support.page_ledger and progress events in
quantlab.utils.progress.
Classes
Outcome of one |
|
Summary of what one |
|
Abstract base of the symbol-batched downloads: a roster over a window, resumable. |