fix(ingestion): force_town as fallback, scraper time fixes, Wake Forest town - #42
Merged
Conversation
Confirmed against both local dev and prod_readonly: none of the three existed yet, only Apex/Bynum/Carrboro/Chapel Hill/Durham/Pittsboro/Siler City did. Raleigh and Cary were already valid broadcast routing localities, but events.Town is a separate, isolated vocabulary. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Extract `resolve_town()` and use it everywhere a town is decided. The devtools "force town" selector previously flattened every staged event to the operator's choice, which silently mislabeled correct per-event guesses — a Raleigh-focused source occasionally surfacing a Cary event had that event rewritten to Raleigh. Now the forced town only fills in where Gemini's own guess doesn't resolve to a known `Town` (empty, or a place we don't track). The publish-stage result snapshot drops its `town=town` filter for the same reason: events that published under a correctly-guessed town still belong in the run's results. Co-Authored-By: Claude <noreply@anthropic.com>
…local The ChamberMate API's `startDateTime`/`endDateTime` carry no offset, and the chamber's own event copy claims "Time zone: America/New_York" — but the values are already UTC. A ribbon-cutting whose description reads "Time of Ceremony: 4:30" (pm) ships `20:00:00`, i.e. 4:30pm EDT converted to UTC. Attaching `America/New_York` double-applied the offset and published every event from this source roughly four hours late. Co-Authored-By: Claude <noreply@anthropic.com>
Some recurring listings give a start with no meridiem at all ("7:30 until we
run out of comics"). Anchor a bare-hour pattern to the start of the string so
it can't grab a stray number elsewhere in the text (e.g. "through 8/23/2026"
on an ongoing-exhibit listing), and only trust it for hours 1-7, which are
unambiguously PM for this venue set. Genuinely ambiguous market hours
("12-6", "11-3") still return None rather than guess.
Co-Authored-By: Claude <noreply@anthropic.com>
Secondary-market pages (StubHub, Vivid Seats, SeatGeek, viagogo, …) carry marked-up prices well above face value. Tell Gemini to ignore them, and to return -1 rather than averaging when the only price signal is a resale page or a very wide spread with no clearly-labeled face value. Co-Authored-By: Claude <noreply@anthropic.com>
Uses instant rather than smooth scrolling — smooth-scroll animations get dropped by the browser when the tab is backgrounded or unfocused. Co-Authored-By: Claude <noreply@anthropic.com>
Idempotent `get_or_create`, following 0023's pattern. Co-Authored-By: Claude <noreply@anthropic.com>
Classification + build pass over ~48 candidate Triangle-area sources, with per-source verdicts, the confirmed ICS feed list, and the platform gotchas worth carrying into the next batch. Status section reflects the PR #41 merge and the five sources wired up in prod. Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
aryavenkatesan
added a commit
that referenced
this pull request
Aug 3, 2026
….7-45.10) Nine of ten suite-45 tickets. 752 tests green, ruff clean, migrations linear (0019 -> 0020 -> 0021). 45.1 prompt_suffix never applied in any scheduled run — the batch functions already resolved it correctly; the bug was purely that tasks.py and ingest_events.py called them bare. New ingestion/prompt_dispatch.py loops per source (all sources, not just active=True, so deactivated sources' unprocessed rows aren't stranded) and sweeps leftovers. 45.2 TIME_ZONE UTC -> America/New_York. Reproduced first: a 9:00 PM ET event rendered "Thursday, August 6 - 1:00 AM" — wrong time and wrong weekday. USE_TZ=True keeps storage UTC, so no data migration. Swept for double-correction sites; found none. 45.3 Reject cancelled events pipeline-wide at the standardizer chokepoint, matching title only (a description reading "rain date if cancelled" must not trip it). Guard runs before the Gemini call. `cancelled` is deliberately excluded from CANDIDATE_STATUSES: organizers un-cancel and resubmit, so anchoring would silently suppress the legitimate resubmission. 45.4 EventSource.default_town, resolved per staged event inside the publish loop rather than threaded from the task — a batch spans multiple sources. resolve_town keeps its fallback-not-override semantics (PR #42 regression). 45.7 EventSource.blocked_reason/blocked_since + a `quarantined` health level. Quarantined sources keep polling; refused -> ok is how we learn 45.6 landed. 45.8 manage.py probe_sources — fetch-only reachability from the real prod egress path, which the DEBUG-gated devtools playground structurally cannot test. Surfaces WAF headers; non-2xx is always `refused` so a captcha page can't parse to 0 items and misreport as ok. 45.9 scrape_all_sources_task docstring stated a stale roster; now states the rule, plus the scrape-worker vs backend container distinction. 45.10 Internal-only rename raw_start/raw_end -> raw_start_datetime/ raw_end_datetime, removing the third naming. Event.date untouched — no API break. Staging->published boundary now documented in ingestion/models.py. 45.5 was executed directly against prod (source id 8 deactivated; evidence showed it a strict subset of id 6, inverting the ticket's guess). 45.6 is not built and needs an owner decision — see STATE.md. Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #41. Ships the outstanding ingestion working-tree changes plus the two Town seed migrations, so the migrations actually reach the prod image.
Why this matters for the deploy
Migration
0024_seed_wake_forest_townwas untracked and0023was committed but unpushed. Migrations are baked into the image byCOPY . .and applied by the one-shotmigratecompose service — an unpushed migration is never pulled on the VM, never copied into the image, andmigratereports "nothing to apply" with no error. This PR closes that gap.Note
0023seeds Raleigh/Cary/Morrisville, which already exist in prod — it'sget_or_create, so it's an idempotent no-op there and only matters for dev/fresh databases.0024(wake-forest) is the one that actually changes prod.Changes
force_townbecomes a fallback, not an override. Extractsresolve_town()and uses it everywhere a town is decided. The devtools "force town" selector previously flattened every staged event to the operator's choice, silently mislabeling correct per-event guesses — a Raleigh-focused source occasionally surfacing a Cary event had that event rewritten to Raleigh. Now the forced town only fills in where Gemini's guess doesn't resolve to a knownTown. The publish-stage snapshot drops itstown=townfilter for the same reason.morrisvillechambertimestamps are UTC, not wall-clock local. The ChamberMate API carries no offset and the chamber's own copy claimsAmerica/New_York, but the values are already UTC — a ribbon-cutting described as "4:30" (pm) ships20:00:00. AttachingAmerica/New_Yorkdouble-applied the offset and published every event from this source ~4 hours late.downtownraleighrecovers bare leading hours. Some listings give a start with no meridiem ("7:30 until we run out of comics"). Anchored to string start so it can't grab a stray number (e.g. "through 8/23/2026"), and trusted only for hours 1-7. Genuinely ambiguous market hours ("12-6") still returnNone.Standardizer distrusts resale-marketplace pricing. StubHub/Vivid Seats/SeatGeek et al. carry marked-up prices; return
-1rather than averaging a wide spread with no clear face value.Devtools playground scrolls results into view on completion (instant, not smooth — smooth-scroll gets dropped when the tab is backgrounded).
Docs: adds the 8-1 source buildout handoff, with its status section corrected to reflect the #41 merge and the five sources wired up in prod.
Test plan
ruff check ingestion/ devtools/ events/— cleanorigin/main:0022 → 0023 → 0024Prod state verified before this PR
Sources 23-27 are already live in Neon (they're rows, not image content — nothing to bake), the running image already contains all 27 scrapers, and Towns
raleigh/cary/morrisvillealready exist. Those five sources showlast_polled=Nonebecause prod runsINGEST_SHARD_COUNT=3withn = day_of_year % 3— they come up across the next three daily runs.🤖 Generated with Claude Code