quantlab.acquisition.alpaca.AlpacaAcquisition#

class quantlab.acquisition.alpaca.AlpacaAcquisition(config)[source]#

Bases: SymbolBatchAcquisition

Download Alpaca Market Data through the shared SymbolBatchAcquisition base.

Alpaca serves many symbols per request and splits large answers into pages, so _fetch_page sends one request and returns (rows, next_page_token). The base class loops over the pages, writes the shards and keeps the page ledger.

Data types. One value, returned by _data_type, selects the endpoint, the field names and the shard columns. frequency="1d" and "1m" fetch bars. frequency="tick" fetches quotes or trades and requires kwargs["data_type"] to say which, with no default. Tick rows are written exactly as received: nothing between the request and the shard resamples, groups or deduplicates them.

Subscription tiers. Alpaca’s free (Basic) plan and its paid Algo Trader Plus plan both serve history since 2016 with the same fields. The free plan withholds the latest 15 minutes, which a historical backfill never asks for. The difference that matters is the historical request rate: 200 requests per minute on the free plan against 10,000 on the paid one, a factor of 50. At 200 per minute a full-market daily backfill takes about 8 minutes and a full-market minute backfill about 50 hours and 358 GB. The practical question is therefore how much data a run asks for, and the volume check in quantlab.universe makes the user answer it before a run starts.

Whether the free plan may use SIP data is unresolved. SIP (Securities Information Processor) is the consolidated feed covering every US exchange; IEX is a single exchange’s feed. Alpaca’s own documentation does not settle whether the free plan can request historical SIP data, and this class takes no position. One reading is yes, for data older than 15 minutes: the FAQ says end must be at least 15 minutes old to query SIP data without a subscription, and the plan table reads the same way. The other reading is IEX only: the data-sources table says IEX is the only feed available without a subscription. Three vendor pages support the first reading and one the second; that is a vote count, not evidence. The difference is material, because IEX carries roughly 2.5% of consolidated volume, and tick data limited to IEX would be unusable for research. So feed is a config.kwargs option with no in-code default: when it is unset the parameter is left out of the request and the vendor picks the best feed the account allows. Only a request made with a real key can settle the question.

Corporate actions. This class has no corporate-actions method, endpoint or constant. If one is ever added it must not replace the delisting information in Tiingo’s supported_tickers.csv: Alpaca’s Corporate Actions API leaves out delistings and reorganisations, and relying on it would bring back survivorship bias.

Credentials. Only the market-data endpoints are used; there is no trading or broker API and no paper/live switch, because the market-data API does not distinguish the two. APCA_API_KEY_ID and APCA_API_SECRET_KEY are read from the environment inside _AlpacaMarketDataClient and are never put on a config, because AcquisitionConfig is saved as JSON beside model checkpoints and a key stored there would end up on disk. Their values are removed from every captured message.

Parameters:

configAcquisitionConfig: The acquisition config. Alpaca-specific options go in config.kwargs: data_type (required for frequency="tick"), feed, adjustment (default "raw"), asof and page_limit (default 10,000), plus the base class’s tuning parameters.

Raises:

ValueError: If data_type, feed or adjustment is not an accepted value.
RuntimeError: If the credentials are missing from the environment.

Examples

Needs APCA_API_KEY_ID and APCA_API_SECRET_KEY exported:

from quantlab.acquisition.config import AcquisitionConfig

cfg = AcquisitionConfig(
    market="us_equity", frequency="1d", vendor="alpaca",
    raw_data_dir_path="downloads/nasdaq_data/alpaca",
    watermark_path="downloads/nasdaq_data/_watermarks/alpaca",
    symbols=("AAPL",), start_date="2024-01-02",
    end_date="2024-01-03",
)
acq = AlpacaAcquisition(cfg).download()
print(acq.coverage_report())

Trades at full resolution use frequency="tick" together with kwargs={"data_type": "trades"}.

ASOF_NO_MAPPING = '-'#: Copy of the module-level constant; see its comment.

CREDENTIAL_ENV_VARS = ('APCA_API_KEY_ID', 'APCA_API_SECRET_KEY')#: The two credentials the request policy hides (as <NAME REDACTED>) before a message reaches a log line or the failure manifest. Built from the module-level constants, not from the client class that tests replace. Alpaca sends credentials in headers rather than in the URL, but a requests exception can still expose exc.request.headers.

DEFAULT_BATCH_SIZE = 100#: Symbols per request. Alpaca does not document a maximum, so this is a cautious practical value rather than a verified limit; override it through config.kwargs["batch_size"].

ENDPOINT_MAP = {'bars': '/stocks/bars', 'quotes': '/stocks/quotes', 'trades': '/stocks/trades'}#: Data type to endpoint path. The response keys its rows by the same name, so one value selects both the request and the response field.

FEED_VALUES = frozenset({'boats', 'delayed_sip', 'iex', 'otc', 'overnight', 'sip'})#: Accepted values for the request options a caller may set. An unknown value raises instead of being sent, because the vendor rejects some and silently ignores others, and an ignored value means a run that asks for split-adjusted bars and stores raw ones. The lists come from alpaca-py 0.44.0.

FIELD_MAP = {'c': 'close', 'h': 'high', 'l': 'low', 'n': 'trade_count', 'o': 'open', 't': 'timestamp', 'v': 'volume', 'vw': 'vwap'}#: Alpaca’s single-letter bar fields to this project’s column names. A mistake here is easy to miss: swapping o and c would produce plausible-looking data forever.

FIELD_MAP_BY_DATA_TYPE = {'bars': {'c': 'close', 'h': 'high', 'l': 'low', 'n': 'trade_count', 'o': 'open', 't': 'timestamp', 'v': 'volume', 'vw': 'vwap'}, 'quotes': {'ap': 'ask_price', 'as': 'ask_size', 'ax': 'ask_exchange', 'bp': 'bid_price', 'bs': 'bid_size', 'bx': 'bid_exchange', 'c': 'conditions', 't': 'timestamp', 'z': 'tape'}, 'trades': {'c': 'conditions', 'i': 'trade_id', 'p': 'price', 's': 'size', 't': 'timestamp', 'x': 'exchange', 'z': 'tape'}}#: Data type to its field map. FIELD_MAP is the bars entry.

INTRADAY_START_SUFFIX = 'T00:00:00Z'#

Suffixes that turn an intraday window’s dates into full timestamps; see _window_bounds.

Alpaca documents start and end as RFC 3339 timestamps. A bare YYYY-MM-DD is clear for daily bars but not for minute or tick data, where a bare end most likely means 00:00:00Z. That would leave out the whole final session while the watermark still recorded the date as covered. Nine fractional digits match the nanosecond tick timestamps.

OPTIONAL_COLUMNS_BY_DATA_TYPE = {'bars': (), 'quotes': ('conditions',), 'trades': ('conditions',)}#: Per data type, the columns the vendor may leave out of every row of a page; conditions is optional in the response. Any other missing column means the field map no longer matches the response, and _fetch_page raises rather than filling it with nulls: an all-null price column would look like symbols that never traded.

QUOTE_FIELD_MAP = {'ap': 'ask_price', 'as': 'ask_size', 'ax': 'ask_exchange', 'bp': 'bid_price', 'bs': 'bid_size', 'bx': 'bid_exchange', 'c': 'conditions', 't': 'timestamp', 'z': 'tape'}#: Alpaca’s quote fields to this project’s column names. c means conditions on a quote and close on a bar, which is why each data type has its own map.

RATE_LIMIT_HEADERS = ('X-RateLimit-Limit', 'X-RateLimit-Remaining', 'X-RateLimit-Reset')#: Headers Alpaca may send with a 429, read for logging only. A missing header is recorded as missing, never as a default.

property RAW_COLUMNS#

Return the shard columns, in order, for this run’s data type.

A property rather than a class attribute because the columns depend on which endpoint the run reads. The values come from RAW_COLUMNS_BY_DATA_TYPE; for bars they are timestamp, symbol, vendor, open, high, low, close, volume, trade_count and vwap.

RAW_COLUMNS_BY_DATA_TYPE = {'bars': ('timestamp', 'symbol', 'vendor', 'open', 'high', 'low', 'close', 'volume', 'trade_count', 'vwap'), 'quotes': ('timestamp', 'symbol', 'vendor', 'bid_exchange', 'bid_price', 'bid_size', 'ask_exchange', 'ask_price', 'ask_size', 'conditions', 'tape'), 'trades': ('timestamp', 'symbol', 'vendor', 'exchange', 'price', 'size', 'trade_id', 'conditions', 'tape')}#: Shard columns, in order, per data type. Each begins with ("timestamp", "symbol", "vendor") followed by only that endpoint’s own fields, so a directory scan always sees one schema and a trade never carries empty bid/ask columns.

property RAW_SCHEMA#

Return the column dtypes for this run’s data type.

For bars timestamp is pl.Datetime("us"); for quotes and trades it is pl.Datetime("ns").

RAW_SCHEMA_BY_DATA_TYPE = {'bars': {'close': Float64, 'high': Float64, 'low': Float64, 'open': Float64, 'symbol': String, 'timestamp': Datetime(time_unit='us', time_zone=None), 'trade_count': Float64, 'vendor': String, 'volume': Float64, 'vwap': Float64}, 'quotes': {'ask_exchange': String, 'ask_price': Float64, 'ask_size': Float64, 'bid_exchange': String, 'bid_price': Float64, 'bid_size': Float64, 'conditions': List(String), 'symbol': String, 'tape': String, 'timestamp': Datetime(time_unit='ns', time_zone=None), 'vendor': String}, 'trades': {'conditions': List(String), 'exchange': String, 'price': Float64, 'size': Float64, 'symbol': String, 'tape': String, 'timestamp': Datetime(time_unit='ns', time_zone=None), 'trade_id': Int64, 'vendor': String}}#

Explicit column dtypes per data type. They build an empty page (so a response with no rows still has a readable symbol column), type columns a sparse response leaves out, and cast a full page so every shard has the same schema.

The timestamp unit matters. Bars use microseconds; quotes and trades use nanoseconds because that is what Alpaca sends. Parsing them at microseconds would lose their ordering below one microsecond and create false (timestamp, symbol) ties in tick data, which is never deduplicated.

SESSION_TIME_ZONE = 'America/New_York'#

The time zone whose calendar date names the intraday date= directories.

Timestamps are stored as UTC without a time zone attached, like every other timestamp in the project; only the directory key is converted. The US regular session runs 14:30 to 21:00 UTC and extended hours cross UTC midnight, so a UTC date would file the last hours of each session under the next day and make a one-day query wrong at both edges.

SORT = 'asc'#: Always asc, never read from an option; see _fetch_page.

TICK_DATA_TYPES = ('quotes', 'trades')#: The values kwargs["data_type"] accepts under frequency="tick". There is no default: quotes and trades share one vendor directory and differ only by their data-type directory beneath it, so a guess would file one as the other with the wrong columns.

TIMEFRAME_MAP = {'1d': '1Day', '1m': '1Min'}#: Maps the project’s frequency value to Alpaca’s timeframe parameter, using the vendor’s documented spelling. Adding a bar size is a one-line change here.

TRADE_FIELD_MAP = {'c': 'conditions', 'i': 'trade_id', 'p': 'price', 's': 'size', 't': 'timestamp', 'x': 'exchange', 'z': 'tape'}#: Alpaca’s trade fields to this project’s column names.

VENDOR = 'alpaca'#: Which vendor this subclass fetches from. Set by every subclass. It is written into every shard as a literal vendor column and hashed into every PageLedger.batch_key, so changing it changes the files.