quantlab.utils.watermark

Dated watermark files: the day through which some raw data is complete.

A dated watermark is a small JSON file beside raw data saying the day through which that data is complete. It holds through (YYYY-MM-DD) and any labels its writer adds to say what it is the watermark of; the file is written with indent=2 and sorted keys. What the labels are and where the file lives is the owner’s layout: the reader and the writer of each raw tier pass them. This module knows no vendor.

resume_from and holds are the two rules every day-indexed pull shares: resume the day after the watermark (less an overlap), and skip what the watermark already holds.

Examples

>>> import tempfile
>>> from pathlib import Path
>>> path = Path(tempfile.mkdtemp()) / WATERMARK_FILE
>>> write_watermark(path, date(2024, 1, 11), {"name": "prices"})
>>> print(path.read_text())
{
  "name": "prices",
  "through": "2024-01-11"
}
>>> read_watermark(path)
datetime.date(2024, 1, 11)

Functions

holds	Return whether a watermark already reaches day.
read_watermark	Return the day the dated watermark in path reaches, or None without a file.
resume_from	Return the first day a pull asks for: the day after the watermark, less an overlap.
write_watermark	Write a dated watermark reaching through to path atomically.