daily

A small aggregator that, once a day, pulls a fixed set of public readings — Madison lake levels, Mendota buoy weather, a basket of market prices, and the UWCU 30-year mortgage rate — writes the new rows to BigQuery, renders an HTML digest with 90-day sparkline charts, and emails it via Resend. Runs on Cloud Run, kicked off by Cloud Scheduler.

The code is intentionally small (one Python package, no plugin auto-discovery, no framework abstraction). Adding a new source is two files and a YAML block.

Status

Live and running daily in the palmer-daily GCP project. Cloud Build owns the deploy pipeline; every merge to main auto-triggers a build via the GitHub App trigger, which runs tests → builds the linux/amd64 image → pushes to Artifact Registry → rolls a new Cloud Run revision. All five data sources are wired and populated in the digest. Schedule: 7am America/Chicago via Cloud Scheduler.

What it collects

Configured in sources.yaml:

• lake_mendota — Lake Mendota level (ft) and Tenney lock percent open, via lwrd.danecounty.gov JSON endpoints (/api/LakeLevels/Current, /api/Dams/GetDams).
• mendota_buoy — Surface water temp, 10 m water temp, wind speed, wind direction, via the SSEC MetObs API (metobs.ssec.wisc.edu/api/data.json, site=mendota&inst=buoy). Water temps are stored in °C (the upstream unit) and rendered in °F in the email; wind speed is converted m/s → mph in the collector.
• markets — 12 tickers (S&P 500, NASDAQ, gold, oil, BTC, FX pairs, DXY, 10y yield, VIX) via yfinance.
• uwcu_mortgage_30y — UWCU 30-year fixed mortgage rate (%), scraped from the public rates page.
• portfolio — A manually-maintained holdings.yaml at the repo root lists share counts and cash per account. Daily run multiplies shares by current yfinance prices and emits one reading per account plus a grand total. The email renders the total in the main metrics table and the accounts in a sub-table below; a red banner appears when any account's as_of date is older than stale_after_days (default 60).

To add or remove a metric, edit sources.yaml and redeploy. No code change needed unless the source type itself is new (see "Adding a new source" below). To update portfolio balances, edit holdings.yaml (bumping as_of, cash_usd, and positions) and redeploy.

Quick start

python -m venv .venv && source .venv/bin/activate
make install   # editable install + dev deps (pytest, ruff, mypy, respx)
make lint      # ruff check + format check + mypy
make test      # pytest with 80% coverage floor
make dev       # FastAPI on http://127.0.0.1:3501 with --reload

Trigger a one-off run while make dev is up:

curl -X POST http://127.0.0.1:3501/run

The dev server returns the run summary as JSON; the digest email goes out via Resend if RESEND_API_KEY is set, otherwise the orchestrator logs the rendered HTML and skips delivery.

Architecture

sources.yaml ──► config.py ──┐
                             ▼
                       orchestrator.py ─► collectors/* ──► MetricReading list
                             │                                    │
                             │                                    ▼
                             │                            storage/bigquery.py
                             │                                    │
                             ▼                                    ▼
                       email/render.py ◄────────────── storage.query_history
                             │
                             ▼
                       email/send.py (Resend)

Source layout (src/daily/):

• main.py — FastAPI app, /run and /healthz endpoints
• orchestrator.py — one-run pipeline: load config → run collectors in a thread pool → write readings → query history → render → send
• config.py — load + validate sources.yaml into pydantic models
• collectors/
  • base.py — abstract Collector (one method: fetch() -> list[MetricReading])
  • __init__.py — explicit REGISTRY mapping type strings to classes
  • one module per source (dane_county_lakes.py, mendota_buoy.py, market_yfinance.py, uwcu_mortgage.py)
• storage/
  • schema.py — BigQuery column constants
  • bigquery.py — BigQueryStorage (ensure_table, write_readings, query_history)
• email/
  • render.py — Jinja2 templates → HTML digest
  • sparkline.py — matplotlib → inline PNG attachments
  • send.py — Resend client wrapper
• observability/logging.py — structured JSON logs for Cloud Logging
• models.py — MetricReading, SourceRunResult dataclasses

A collector that returns per-metric error readings is the normal way to report individual failures — a metric whose value is None and whose error is non-None renders as ⚠ <reason> in the email without poisoning the rest of the source. Only whole-source HTTP failures should raise.

Adding a new source

Two cases. Both end with an entry in sources.yaml.

1. The source uses an existing collector type

If you want another lake from dane_county_lakes or another ticker from market_yfinance, just edit sources.yaml. For lakes, valid metric keys are mendota_level_ft / monona_level_ft / waubesa_level_ft / kegonsa_level_ft and tenney_open_pct / babcock_open_pct / lafollette_open_pct. For markets, any Yahoo Finance ticker works.

2. The source needs a new collector type

1. Pick a type_key for sources.yaml (e.g. usgs_river_gauge).

2. Write src/daily/collectors/<your_type>.py: subclass Collector, set type_key, implement fetch() returning a list[MetricReading]. Look at dane_county_lakes.py for a JSON-API style example, or uwcu_mortgage.py for an HTML-scrape style. Return per-metric error readings rather than raising — see MetricReading in models.py.

3. Register it in src/daily/collectors/__init__.py (REGISTRY dict) and add the new type_key to KNOWN_COLLECTOR_TYPES in config.py.

4. Capture a fixture under tests/fixtures/ and add it to the _SOURCES table in scripts/refresh_fixtures.py so future-you can re-pull when the upstream layout shifts.

5. Write tests/collectors/test_<your_type>.py using respx to mock httpx.get. Cover the happy path, individual metric failures, and full HTTP failure. See tests/collectors/test_dane_county_lakes.py for the pattern.

6. Add to sources.yaml with your type and metrics list.

The 80% coverage floor is enforced; per-source tests must land in the same commit as the collector code.

Fixtures

The scrape collectors are tested against captured fixtures, not the live internet (CI doesn't have outbound network to flaky third parties). When a source's layout or API shape changes and a test starts failing in a way that looks like reality moved underneath it, refresh the fixture:

python scripts/refresh_fixtures.py                      # all four
python scripts/refresh_fixtures.py --source uwcu_mortgage
python scripts/refresh_fixtures.py --dry-run            # show what would change

The script writes into tests/fixtures/. Diff the result before committing — a layout change must be a deliberate update, not silent test drift. If the source dropped or renamed a field, update the collector and its tests in the same commit.

make smoke runs refresh_fixtures.py --dry-run to exercise outbound network without overwriting anything; CI does not run it.

Configuration cheat sheet

sources.yaml is the single user-editable file. Schema:

email:
  to: someone@example.com
  from: onboarding@resend.dev
  subject_template: "Daily digest — {date}"
  timezone: America/Chicago

history:
  sparkline_days: 90        # 1–3650; how far back the sparkline reaches

sources:
  - name: <source slug>
    type: <one of dane_county_lakes | mendota_buoy | market_yfinance | uwcu_mortgage | portfolio>
    enabled: true
    url: <optional, source-dependent>
    metrics:                # used by lakes, buoy, uwcu
      - { key: <metric slug>, label: "Display label", unit: "ft" }
    tickers:                # used by market_yfinance only
      sp500: { symbol: "^GSPC", label: "S&P 500", unit: "USD" }
    holdings_file: holdings.yaml   # portfolio only — path relative to cwd
    stale_after_days: 60           # portfolio only — banner threshold

Cross-source metric keys must be unique; the loader enforces this.

The portfolio source additionally reads holdings.yaml:

accounts:
  - name: HSA Account
    mask: "763"          # stable identifier; do not rename
    type: HSA
    as_of: 2026-05-13    # date you last verified the snapshot
    cash_usd: 20.89
    positions:
      - { symbol: VOO, shares: 3 }
      - { symbol: AVUV, shares: 14 }

Deployment

Local Docker is no longer required. Cloud Build owns the pipeline:

make submit

This uploads the working tree (filtered by .gcloudignore) to Cloud Build, runs cloudbuild.yaml: tests → docker build --platform=linux/amd64 → push to Artifact Registry → gcloud run services update to roll the new revision. A failing test suite blocks the image push.

Watch a build:

gcloud builds log $(gcloud builds list --project=palmer-daily --limit=1 \
    --format='value(id)') --project=palmer-daily

Manually trigger a daily run:

gcloud scheduler jobs run daily-digest --location=us-central1 --project=palmer-daily

Cloud Run logs (including the structured JSON entries from observability/logging.py) land in Cloud Logging under run.googleapis.com/stdout.

Local dev port

make dev listens on http://127.0.0.1:3501. Override with make dev LOCAL_PORT=4000. The Dockerfile defaults to the same 3501 so docker run without -p mapping listens on 3501; in Cloud Run, PORT=8080 is injected and overrides the default.

Documents

• Design spec
• Cost estimate
• Implementation plan
• CLAUDE.md — orientation for Claude sessions working in this repo