quantlab.acquisition.wrds.taq#
Download of NYSE TAQ quote data from WRDS, and the shared WRDS connection.
WRDS (Wharton Research Data Services) is a university-run service that
serves licensed financial databases through a PostgreSQL server. One of them
is NYSE TAQ (“Trade and Quote”), which records every trade and quote on US
exchanges with millisecond timestamps or finer. This module downloads TAQ’s
NBBO records. The NBBO (National Best Bid and Offer) is the highest bid and
the lowest ask across all US exchanges; TAQ stores one record each time
either side changes. WRDS stores one table per trading day, named
taqm_{YYYY}.complete_nbbo_{YYYYMMDD}.
The module holds three things. WrdsSession is the read-only database
session that every WRDS product in this package uses: a small pool of
connections to a fixed WRDS host, opened on demand and capped at
MAX_CONNECTIONS, so concurrent download workers each query on their own
connection while a WRDS account’s connection limit is respected. It reads
the username from the WRDS_USERNAME environment variable and leaves the
password to libpq (the PostgreSQL client library), which reads it from the
~/.pgpass file. WRDS protects logins with Duo two-factor authentication;
in practice a .pgpass login sends no prompt, but the session still never
reconnects after a failure, so a broken run stops and is resumed by hand.
WrdsTaqNbboAcquisition downloads the day tables. The work is split into
pages, and one page is one trading day for one batch of symbols, fetched
with a plain COPY (SELECT ... WHERE ...). Each NBBO record becomes one
row of the raw tier (the parquet files on disk), unfiltered and unsorted,
and the order in which the server sent the rows is kept in the
wrds_row_ord column.
Sorting, de-duplicating and resampling the raw records happen later, in
quantlab.dataset.nbbo. The wrds package from PyPI is not used: its
connection asks for input interactively and it may open extra connections,
each of which can send a Duo prompt. The vendor’s registry entry is defined
in quantlab.acquisition.wrds, not in this file.
Classes
A pool of read-only PostgreSQL connections to WRDS, opened on first use. |
|
Download WRDS TAQ |
Functions
Return the trading days in |