quantlab.acquisition.wrds.taq.WrdsTaqNbboAcquisition#

class quantlab.acquisition.wrds.taq.WrdsTaqNbboAcquisition(config)[source]#

Bases: SymbolBatchAcquisition

Download WRDS TAQ complete_nbbo day tables.

The work is split into pages, and one page is one trading day’s table for one batch of symbols. _fetch_page reads the day named by the page token (the window’s first trading day when there is no token) and returns the next trading day’s ISO date as the next token. The shared SymbolBatchAcquisition base class runs the batch loop, writes the parquet files (shards), records finished pages and resumes interrupted runs.

The raw tier is written under .../wrds/nbbo/date=/symbol=/ (the nbbo directory is the data type’s own; below it the directories are named key=value, a layout called hive partitioning), with one row per NBBO record for the full trading day, unfiltered. The position at which each row arrived is stored as wrds_row_ord before anything else touches the frame, because before 2018 the tables have no sequence number and the server’s order is the only way to order records with equal timestamps. TAQ’s time_m is New York local clock time. timestamp is the same instant in UTC, stored without a time zone at nanosecond resolution, and the date= directory is the New York trading date.

Symbols use a dot for share classes (BRK.B); the hyphenated form (BRK-B) is refused. Batches are fetched by max_workers threads, each on its own pooled connection; a value above MAX_WORKERS (the session’s connection cap) is refused.

Parameters:

configAcquisitionConfig: Built by build_config. It must have frequency="tick" and kwargs["data_type"] == "nbbo".

Examples

Needs WRDS_USERNAME and a ~/.pgpass entry; the first query opens the connection:

cfg = WrdsTaqNbboAcquisition.build_config(
    ("AAPL", "MSFT"), start_date="2024-01-24", end_date="2024-01-25"
)
acq = WrdsTaqNbboAcquisition(cfg).download()
report = acq.coverage_report()

Shards land under .../wrds_taq/wrds/nbbo/date=2024-01-24/symbol=AAPL/ and the watermarks under .../wrds_taq/_watermarks/wrds/nbbo/.

CREDENTIAL_ENV_VARS = ('WRDS_USERNAME',)#: The environment variables whose values the request policy hides (each becomes <NAME REDACTED>). A subclass fills this from module-level constants, not from its HTTP client class, so a test double that replaces the client cannot turn the redaction off. An empty tuple means the vendor uses no credentials.

DEFAULT_BATCH_SIZE = 25#: Symbols per day-table query. Overridable through kwargs["batch_size"].

DEFAULT_MAX_WORKERS = 4#: Concurrent batch fetches, each on its own pooled connection. Overridable through kwargs["max_workers"], up to MAX_WORKERS.

DEFAULT_SUBDIR = 'wrds_taq'#: The raw tier lives under downloads/us_equity/tick/{DEFAULT_SUBDIR}/wrds.

DEFAULT_VERIFY_PAGE_COUNTS = True#: Count every page with the same WHERE before copying it, and fail the page if the copied row count differs. Overridable through kwargs["verify_page_counts"]. On the server the count takes well under a second per (day, batch).

GLOBAL_STOP_ERRORS = (<class 'quantlab.acquisition.wrds.taq.WrdsSessionError'>, <class 'quantlab.acquisition.wrds.taq.WrdsEntitlementError'>, <class 'psycopg2.OperationalError'>, <class 'psycopg2.InterfaceError'>)#: Exceptions meaning the shared session or the account is unusable.

MAX_WORKERS = 6#: The most workers a run may use: the session’s connection cap.

RAW_COLUMNS = ('timestamp', 'symbol', 'vendor', 'taq_date', 'time_m', 'time_m_nano', 'sym_root', 'sym_suffix', 'qu_cond', 'natbbo_ind', 'qu_source', 'nbbo_qu_cond', 'best_bid', 'best_bidsizeshares', 'best_ask', 'best_asksizeshares', 'wrds_row_ord')#: The shard columns and their order. TAQ’s date is stored as taq_date, because the shard writer adds its own date column for the date= directory and would otherwise overwrite the raw one.

SESSION_TIME_ZONE = 'America/New_York'#: TAQ’s time_m is New York clock time, and the date= directory is the New York trading date.

SUFFIX_DELIMITER = '.'#: BRK.B is root BRK with suffix B, the notation used by the index-constituent symbol lists.

TAQ_COLUMNS = ('date', 'time_m', 'time_m_nano', 'sym_root', 'sym_suffix', 'qu_cond', 'natbbo_ind', 'qu_source', 'nbbo_qu_cond', 'best_bid', 'best_bidsizeshares', 'best_ask', 'best_asksizeshares')#: The TAQ columns this class reads. time_m_nano exists only from 2018-01-02 and is optional; every other column is required.

TICK_DATA_TYPES = ('nbbo',)#: The only data type this vendor serves under frequency="tick".

VENDOR = 'wrds'#: Which vendor this subclass fetches from. Set by every subclass. It is written into every shard as a literal vendor column and hashed into every PageLedger.batch_key, so changing it changes the files.

classmethod build_config(symbols, start_date=None, end_date=None, kwargs=None, subdir='wrds_taq')[source]#

Build the AcquisitionConfig for a WRDS NBBO download.

This is the config_factory of the registry’s nbbo capability. The raw directory is .../{subdir}/wrds and the watermarks go to the sibling .../{subdir}/_watermarks/wrds, both under get_data_root() / "downloads" / "us_equity" / "tick". No credential goes into the config.

Parameters:

symbolsiterable of str: Tickers in dot notation, for example "BRK.B".
start_date, end_datestr or None, default None: The window, inclusive, as ISO dates.
kwargsdict or None, default None: Extra options, such as batch_size or verify_page_counts. data_type is set to "nbbo".
subdirstr, default “wrds_taq”: Directory under downloads/us_equity/tick that holds this raw tier.

Returns:

AcquisitionConfig: The config for WrdsTaqNbboAcquisition.

Raises:

ValueError: If kwargs["data_type"] is set to anything but "nbbo".

Examples

>>> cfg = WrdsTaqNbboAcquisition.build_config(
...     ("AAPL", "MSFT"), start_date="2024-01-24", end_date="2024-01-25"
... )
>>> cfg.kwargs
{'data_type': 'nbbo'}

cfg.raw_data_dir_path is '<data root>/downloads/us_equity/tick/wrds_taq/wrds'.

classmethod pair_to_symbol(root, suffix)[source]#

Join a TAQ (sym_root, sym_suffix) pair back into a dotted symbol.

Parameters:

rootstr: The symbol root.
suffixstr or None: The share-class suffix; None or "" means none.

Returns:

str: ROOT or ROOT.SUFFIX.

Examples

>>> WrdsTaqNbboAcquisition.pair_to_symbol("BRK", "B")
'BRK.B'
>>> WrdsTaqNbboAcquisition.pair_to_symbol("AAPL", None)
'AAPL'

classmethod symbol_to_pair(symbol)[source]#

Split a dotted symbol into its TAQ (sym_root, sym_suffix) pair.

A hyphenated symbol such as BRK-B is refused rather than guessed at: the project’s ticker pattern accepts both separators, and a query for sym_root = 'BRK-B' would silently return nothing. More than one dot, or an empty root or suffix, is refused too.

Parameters:

symbolstr: ROOT or ROOT.SUFFIX.

Returns:

tuple of (str, str or None): The root and the suffix, or None when there is no suffix.

Raises:

ValueError: If the symbol contains a hyphen or is not ROOT or ROOT.SUFFIX.

Examples

>>> WrdsTaqNbboAcquisition.symbol_to_pair("BRK.B")
('BRK', 'B')
>>> WrdsTaqNbboAcquisition.symbol_to_pair("AAPL")
('AAPL', None)