feat: add Civico provider for Westminster Council#22
Open
fsargent wants to merge 2 commits into
Open
Conversation
Adds a new `civico` provider class that paginates Civico's archived-meetings JSON API (www.civico.net/api/{authority}/{offset}/archived) and ingests meeting metadata. UIDs are prefixed (civico-{authority}-{id}) to stay globally unique across authorities, since Civico's integer id is only per-tenant. `get_meetings` is currently a pass-through: Westminster's Civico tenant publishes no captions (live_transcription is 0; no subtitle track in DASH/HLS manifests), so transcript search will return no Westminster results until a future ASR pipeline is added. To register Westminster after deploy: POST /meetings/add_provider?provider=civico POST /meetings/add_authority?authority=westminster&provider=civico&nice_name=Westminster Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Per-authority INFO logs (build index, fetch transcripts, store) plus top-level start/complete, so 'tail -f' shows clear ingest progress. Per-authority try/except + logger.exception so one provider's failure doesn't terminate the whole batch silently. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
civicoprovider that paginateswww.civico.net/api/{authority}/{offset}/archivedand ingests meeting metadata for Civico-hosted councils (Westminster is the first tenant).civico-{authority}-{id}so Civico's per-tenant integer IDs stay globally unique alongside PublicI.get_meetingsis a pass-through for now: Westminster's Civico tenant publishes no captions (live_transcriptionis0; no subtitle track in DASH/HLS manifests), so transcript search returns no Westminster results until an ASR pipeline lands.load_meetingssotail -fshows per-authority progress, and wraps each authority in try/except so one provider's failure no longer terminates the whole batch silently.To register Westminster after deploy:
Test plan
https://www.civico.net/api/westminster/0/archivedand confirm pagination terminates at the empty page (24 meetings total).load_meetings('all')locally; verify 24 meetings land in SQLite with validdatetime/unixtime/link.live_transcriptionis0for every Westminster meeting (no transcripts ingested, as expected).🤖 Generated with Claude Code