Suite 36 complete + Apex/Durham town coverage - #38
Merged
Conversation
A staged event whose town had no `Town` row stayed `approved` forever and re-logged a warning on every pipeline run — 27 rows on prod (Apex 15, Durham 12) burning 27 unactionable log lines per run. That is how Siler City's dropped events went unnoticed for weeks (35.14). Adds `skipped_no_town`, set at the skip point, so the row leaves the `approved` queryset and the warning fires exactly once. Added to `deduplicator.CANDIDATE_STATUSES` so a re-scrape collapses onto the existing row instead of inserting a fresh one each poll. Coverage fixes are no longer free: `manage.py reopen_skipped_towns` re-opens rows once a `Town` lands. Migration backfills the 27 existing rows. Includes the devtools funnel fix in the same commit deliberately — the new status fell into no funnel bucket, silently breaking the documented "buckets sum to raw" invariant and dropping 27 rows out of the monitor. Splitting these would plant a commit with failing devtools tests. The monitor now surfaces a `no_town` bucket with an explanatory note, which is what 36.1 wanted: `raw > 0, published = 0` now says why. The reconcile test sums buckets dynamically so a future unwired bucket fails instead of passing silently. Co-Authored-By: Claude <noreply@anthropic.com>
All 15 had an empty `source_name`, so they predate the ingestion pipeline — hand-entered rows whose town was never resolved. 11 of 15 are one recurring "Senior ..." series at Covenant Place. Mapping: Fair Game Beverage Company, bmc brewing and Chatham YMCA -> pittsboro; Covenant Place -> carrboro (Carrboro Parks & Rec programming, per product decision, though the building's postal address is Chapel Hill). The Jordan Lake spring-cleanup row is left NULL on purpose: the recreation area spans Wake and Chatham with access points in different towns, and a wrong town puts an event on the wrong hub page. `Event.town` stays nullable — ingestion legitimately produces unmatched towns (36.1). Co-Authored-By: Claude <noreply@anthropic.com>
Root cause of the hourly `healthcheck.service` failure. django-celery-beat holds `PeriodicTask.last_run_at` in memory and writes it only when `Scheduler._do_sync()` runs. Both sync triggers were defeated: - `CELERY_BEAT_MAX_LOOP_INTERVAL = 6h` becomes `Scheduler.max_interval` (celery/beat.py:253-256), overriding django-celery-beat's DEFAULT_MAX_INTERVAL of 5s. `Service.start()` (beat.py:645-652) calls `_do_sync()` only after `time.sleep(interval)`, so beat can sleep 6h between sync opportunities. - `apply_async`'s finally (beat.py:416-418) is gated on `should_sync()`, whose task-count clause is dead because `beat_sync_every` defaults to 0. A task firing shortly after a prior sync flushes nothing. Measured on prod: a write made at 18:00:00 was still absent 4h24m later and appeared the instant celerybeat was restarted (shutdown's `finally: self.sync()` flushed it) — 12:00:01/runs=9 -> 18:00:00.021/runs=10. The task was sent AND executed on time; only the bookkeeping was late. Fix is `CELERY_BEAT_SYNC_EVERY = 1`, forcing a flush after every send. Costs one UPDATE per fire and deliberately does not touch the loop interval, which exists to reduce Neon serverless wake-ups. Note this means PR #36's 7h staleness window was not "too tight" — it was comparing against a column that lagged by up to 6h. Widening it to 13h (the original 36.5) would have turned a true positive into a pass and buried this bug. Co-Authored-By: Claude <noreply@anthropic.com>
A staleness window must exceed the task's period, so every task had a
blind spot at least one period wide — a missed weekly digest rode along
for ~8 days. Now `_task_freshness` asks the schedule itself:
expected_next = now + schedule.remaining_estimate(last_run_at)
`DEFAULT_STALENESS_HOURS` is demoted to (a) the must-exist task set, which
still catches a seeded task vanishing from the schedule (2026-07-21
outage), and (b) a fallback window for interval-backed tasks only.
Grace is keyed to `CELERY_BEAT_SYNC_EVERY`: 5 minutes when it is set (the
persistence lag is gone — see 36.4), automatically re-widening to
MAX_LOOP_INTERVAL + 1h if it is ever removed. Evaluated at call time so
`@override_settings` can reach both branches.
Also fixes a bug the tightened grace exposed. `TzAwareCrontab` converts
`last_run_at` into the schedule's tz inside `is_due()` (tzcrontab.py:39)
but inherits `remaining_estimate`/`remaining_delta` unmodified from
celery, which read `.hour`/`.minute`/`.isoweekday()` off whatever tzinfo
the datetime carries (celery/schedules.py:577-605). Calling
`remaining_estimate()` with UTC datetimes therefore compared
America/New_York crontab fields against UTC clock fields — a permanent
4-5h error every day for ingest-events-daily, scrape-sources-daily and
weekly-digest-sunday. Not a DST-only bug; it reproduces mid-June. The old
7h grace had been silently absorbing it. Fixed by converting both
operands via `.astimezone(schedule.tz)`, mirroring `is_due()`, rather than
widening grace back out.
Detection latency: ~8 days -> ~5 min for a missed weekly digest.
Co-Authored-By: Claude <noreply@anthropic.com>
36.1–36.4 built and green. 36.5 recorded as won't-fix: widening the orphan-recovery staleness window to 13h would have converted a true positive into a pass and buried 36.4's root cause. OUTBOX.md carries the full change blocks (gitignored, so it does not travel with this commit). Note the board still shows every suite back to 17 as _(pending)_ — the desktop app has never synced. Co-Authored-By: Claude <noreply@anthropic.com>
ingest_direct_submission's out-of-coverage branch did a bare return None, leaving the StagedEvent stuck unpublished forever. Set status=skipped_no_town before returning (mirroring publish_all_approved's sweep path from 36.1) so the row is terminal and serves as a dedupe anchor for re-submits. reopen_skipped_towns already reopens these (no source filter). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ndow (36.6) /devtools/monitor warned healthy sources (Carrboro, The Plant NC) whose events are live on the site but whose surviving StagedEvent anchors predate suite 34's delete->flip change, so the windowed published bucket reads 0 forever. Add an un-windowed all-time live-Event count per source (keyed by source_name; Event has no source FK or ingest timestamp), downgrade the 'none published in window' warn when a source has live events all-time, and render a '0 in window; N live all-time' note mirroring _raw_zero_note. Uses Count(pk) since Event's PK is uuid, not id. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Seed `apex` and `durham` Town rows (migration 0018) so the ~27 host/scraper submissions Gemini geocodes just over the Chatham county line stop dead-ending as `skipped_no_town` and publish to the site. `durham` is already a valid broadcast routing region, so this aligns event coverage with broadcast. Migrate the 36.1/36.7 out-of-coverage test sentinels off Apex/Durham (now covered) to Greensboro/Charlotte, which remain genuinely outside the service area. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Completes suite 36 (ingestion funnel dead-ends) and expands town coverage to Apex + Durham. Branches off
fix/suite-36-beat-persistence, so it carries all of 36.1–36.7 plus the coverage change. Supersedes #37.What's here
Suite 36 (36.1–36.7)
skipped_no_townstatus for out-of-coverage rows + monitor noteCELERY_BEAT_SYNC_EVERY = 1so beat persistslast_run_atpromptlysource_name;Count("pk")sinceEvent's PK isuuid)skipped_no_towninstead of a silent perpetual holdApex + Durham coverage
0018_seed_apex_durham_townsseeds the twoTownrows so the ~27 host/scraper submissions Gemini geocodes just over the Chatham county line stop dead-ending asskipped_no_townand publish.durhamis already a valid broadcast routing region, so this aligns event coverage with broadcast.Post-merge (deploy auto-runs on push to main)
The
deployjob applies migrations on prod. After it lands I'll runreopen_skipped_towns+publish_all_approved()on prod to flush the queued Apex/Durham events onto the site.Verification (local)
ruff check/ruff format --check/mypyclean🤖 Generated with Claude Code