Suite 45 — ingestion pipeline correctness fixes (45.1-45.4, 45.7-45.10)

[aryavenkatesan]

Nine of ten suite-45 tickets, orchestrated in four batches. 752 tests green, ruff clean, migrations linear (0019 → 0020 → 0021).

What's here

Ticket	Change
45.1	prompt_suffix never applied in any scheduled run — new ingestion/prompt_dispatch.py
45.2	TIME_ZONE UTC → America/New_York — digest emails showed every event 4–5h late
45.3	Reject cancelled events pipeline-wide at the standardizer chokepoint
45.4	EventSource.default_town to recover skipped_no_town losses
45.7	blocked_reason/blocked_since + a quarantined health level
45.8	manage.py probe_sources — reachability from the real prod egress path
45.9	Stale scrape_all_sources_task docstring
45.10	Internal-only rename raw_start → raw_start_datetime

Findings that changed the tickets

45.2's premise was unverified in the ticket — I reproduced it first. A 9:00 PM ET event rendered Thursday, August 6 · 1:00 AM: wrong time and wrong weekday. USE_TZ=True means storage stays UTC, so no data migration. Swept for double-correction sites; found none.

45.1's root cause was narrower than described. The batch functions already resolved prompt_suffix correctly — the bug was purely that tasks.py and ingest_events.py called them bare. Also: RawEvent.source is non-nullable, so the "orphaned rows" concern was theoretical; the sweep is in as defense-in-depth.

45.3: cancelled is deliberately excluded from CANDIDATE_STATUSES. Unlike skipped_no_town (structural, won't reverse), organizers un-cancel and resubmit. As a dedupe anchor it would silently match the legitimate resubmission as a duplicate and permanently suppress a real event — so it follows the rejected precedent instead.

45.4 keeps resolve_town's fallback-not-override semantics (the PR #42 regression). The fallback resolves per staged event inside the publish loop rather than being threaded from the task, since a batch spans multiple sources.

45.8 hardens beyond spec: a WAF captcha page isn't empty, so it would parse to 0 items and misreport as ok. Any non-2xx is now refused.

Not in this PR

• 45.5 — executed directly against prod (source id 8 deactivated). Evidence inverted the ticket's guess: across 765 raw events id 8 had 0 unique source_uids vs id 6's 9, making id 8 the strict subset. Deactivated, not deleted, so SourceRun history survives.
• 45.6 — not built, needs an owner decision. Research overturned the presumed fix: Eventbrite's public events/search API was discontinued Dec 2019 (org-owned events only), so an official integration covers 0 of 4 sources, not 3 of 4. Their WAF captcha also plausibly enforces their own anti-scraping ToS.

Deploy notes

Migration 0021 is a RenameField, and deploy runs migrate before up -d — so for ~30–60s the old containers run old code against the new column. /api/events/direct-submit and the admin RawEvent list would 500 in that window; the public /events/ API doesn't touch RawEvent. A pre-migrate pg_dump is taken automatically.

Post-deploy ops (Neon data — no migration ships these):

1. Set default_town on prod sources, then manage.py reopen_skipped_towns (18 stranded rows).
2. Set blocked_reason on the 4 WAF-blocked sources.
3. Run probe_sources in scrape-worker — backend has no Chromium.

🤖 Generated with Claude Code