quantlab.acquisition.massive.client.S3ObjectStore#

class quantlab.acquisition.massive.client.S3ObjectStore(credentials, *, max_connections=16, read_timeout=30.0, client=None, today=None)[source]#

Bases: object

ObjectStore over Massive’s S3 endpoint, signed by boto3.

Massive answers 404 for a missing day inside the plan’s window (a holiday) and 403 for every day outside it, at both ends: before the oldest day, and after the newest published one (today before it is published, the future). On a 403, size lists the key’s month: a day after the newest listed day, or a recent day of a month with nothing listed yet, is not published yet; any other is outside the plan’s window; a refused listing means the credentials were refused.

Parameters:

credentialsMassiveCredentials: The S3 key id and secret.
max_connectionsint, default 16: The connection pool’s size; at least the streams in flight.
read_timeoutfloat, default 30: Seconds a stream may send nothing before it is given up as stalled; the range then resumes from the bytes it wrote.
clientbotocore S3 client, optional: A client to use instead of one made from credentials.
todaycallable, optional: Today’s date in New York; the system clock’s when omitted.

Examples

Needs the credentials and the network:

store = S3ObjectStore(MassiveCredentials.from_env(), max_connections=32)

etag(key)[source]#

The ETag size last read for key (it changes when the object is replaced), else None.

The ranged download’s identity: a .part left for another ETag starts over.

Examples

Needs the credentials and the network:

store = S3ObjectStore(MassiveCredentials.from_env())
key = s3_key("trades", date(2016, 11, 25))
store.size(key), store.etag(key)

read(key, start, stop, if_match=None)[source]#

Yield the bytes start up to stop of the object in chunks from one ranged GET.

With if_match (an ETag) the GET is conditioned on it: an object replaced since raises MassiveObjectChangedError (S3’s 412).

Examples

Needs the credentials and the network:

store = S3ObjectStore(MassiveCredentials.from_env())
head = b"".join(store.read(s3_key("trades", date(2016, 11, 25)), 0, 1024))

size(key)[source]#

Return the object’s size from a HEAD request.

Examples

Needs the credentials and the network:

store = S3ObjectStore(MassiveCredentials.from_env())
store.size(s3_key("trades", date(2016, 11, 25)))  # 217911421