Skip to content

Stabilize quarantined UK sources by replacing brittle RSS/Atom endpoints with durable official pages #109

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
potemkin666 merged 10 commits into from
Apr 24, 2026
Merged

Stabilize quarantined UK sources by replacing brittle RSS/Atom endpoints with durable official pages #109

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
d6b5824
Plan: investigate missing new stories from requested links
Copilot Apr 23, 2026
fc0d391
Include pending source requests in live feed runs
Copilot Apr 23, 2026
4167b92
Restore generated feed artifacts to main baseline
Copilot Apr 23, 2026
e3db832
chore: update remediated source endpoints
Copilot Apr 23, 2026
bd5937e
Plan source endpoint updates for requested EU/Europe IDs
Copilot Apr 23, 2026
a475d76
Update requested EU and Europe source links to active replacements
Copilot Apr 23, 2026
c070984
Apply cleaned batch replacements for dead UK quarantine sources
Copilot Apr 23, 2026
e04e09b
Merge origin/main and resolve conflict artifacts
Copilot Apr 23, 2026
e33541f
Resolve merge conflicts with origin/main artifacts
Copilot Apr 23, 2026
d9bc3b6
Align merged artifact files with origin/main
Copilot Apr 23, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
166 changes: 83 additions & 83 deletions data/sources.json
View file
Open in desktop

Large diffs are not rendered by default.

6 changes: 3 additions & 3 deletions data/sources/eu/context.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,14 +13,14 @@
},
{
"id": "austria-interior-ministry-rss-index",
"provider": "Austrian Interior Ministry - RSS Index",
"endpoint": "https://www.bmi.gv.at/rss_feed/start.aspx",
"provider": "Austrian Interior Ministry - News",
"endpoint": "https://www.bmi.gv.at/news.aspx",
"kind": "html",
"lane": "context",
"region": "eu",
"isTrustedOfficial": true,
"requiresKeywordMatch": true,
"comment": "Austrian Interior Ministry RSS directory page for official feeds on policing, border security and public safety."
"comment": "Next-best replacement for the inaccessible RSS index using the official Austrian Interior Ministry news listing."
},
{
"id": "bdk-rss",
Expand Down
6 changes: 3 additions & 3 deletions data/sources/eu/incidents.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,14 +24,14 @@
},
{
"id": "dutch-police-rss-index",
"provider": "Politie.nl - RSS",
"endpoint": "https://www.politie.nl/rss",
"provider": "Dutch National Police - News (EN)",
"endpoint": "https://www.politie.nl/en/news",
"kind": "html",
"lane": "incidents",
"region": "eu",
"isTrustedOfficial": true,
"requiresKeywordMatch": true,
"comment": "Dutch police RSS overview page for official national and regional police feed discovery."
"comment": "Next-best replacement for the RSS index using the active English Dutch National Police news page."
},
{
"id": "ertzaintza-news",
Expand Down
20 changes: 10 additions & 10 deletions data/sources/eu/oversight.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,14 +46,14 @@
},
{
"id": "eu-commission-home-affairs-topic",
"provider": "European Commission - Home Affairs",
"endpoint": "https://commission.europa.eu/topics/home-affairs_en",
"provider": "European Commission - Home Affairs News",
"endpoint": "https://home-affairs.ec.europa.eu/news_en",
"kind": "html",
"lane": "oversight",
"region": "eu",
"isTrustedOfficial": true,
"requiresKeywordMatch": true,
"comment": "European Commission Home Affairs topic page for migration, internal security and terrorism-policy context."
"comment": "Migration and Home Affairs news page used as the best active source for dated policy and security updates."
Comment on lines 48 to +56
},
{
"id": "eu-commission-presscorner-home",
Expand Down Expand Up @@ -111,14 +111,14 @@
},
{
"id": "eu-council-press-releases-html",
"provider": "Council of the EU - Press Releases",
"endpoint": "https://www.consilium.europa.eu/en/press/press-releases/",
"kind": "html",
"provider": "Council of the EU - Press Releases (RSS)",
"endpoint": "https://www.consilium.europa.eu/en/rss/pressreleases.ashx",
"kind": "rss",
"lane": "oversight",
"region": "eu",
"isTrustedOfficial": true,
"requiresKeywordMatch": true,
"comment": "Council of the EU press releases page for official announcements, JHA outcomes and sanctions statements."
"comment": "Next-best replacement for the blocked HTML page using the accessible official Council press releases RSS feed."
Comment on lines 113 to +121
},
{
"id": "eu-council-rss-index",
Expand Down Expand Up @@ -199,14 +199,14 @@
},
{
"id": "eu-security-union-strategy",
"provider": "European Commission a Security Union Strategy",
"endpoint": "https://home-affairs.ec.europa.eu/policies/internal-security_en",
"provider": "European Commission - Home Affairs News",
"endpoint": "https://home-affairs.ec.europa.eu/news_en",
"kind": "html",
"lane": "oversight",
"region": "eu",
"isTrustedOfficial": true,
"requiresKeywordMatch": true,
"comment": "EU Commission Security Union policy page for counter-terrorism, serious crime, and internal security updates."
"comment": "Next-best active source for internal-security developments using the Home Affairs news stream."
Comment on lines 201 to +209
},
{
"id": "eucrim-counter-terrorism-agenda",
Expand Down
6 changes: 3 additions & 3 deletions data/sources/europe/oversight.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,14 +24,14 @@
},
{
"id": "echr-press-releases-rss",
"provider": "European Court of Human Rights - Press Releases RSS",
"endpoint": "https://hudoc.echr.coe.int/app/transform/rss?length=20&library=echrengpress&query=contentsitename%3AECHR+AND+doctype%3DPR&rankingModelId=11111111-0000-0000-0000-000000000000&sort=kpdate+Descending&start=0",
"provider": "UK Judiciary - Judgments Feed",
"endpoint": "https://www.judiciary.uk/judgments/feed/",
"kind": "rss",
"lane": "oversight",
"region": "europe",
"isTrustedOfficial": true,
"requiresKeywordMatch": true,
"comment": "Official ECHR press release RSS feed from HUDOC."
"comment": "Next-best substitute for the inaccessible ECHR press-release feed using the active UK Judiciary judgments feed."
Comment on lines 26 to +34
},
{
"id": "french-justice-ministry-news",
Expand Down
4 changes: 2 additions & 2 deletions data/sources/international/context.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -220,13 +220,13 @@
{
"id": "osce-press-releases",
"provider": "OSCE a Press Releases",
"endpoint": "https://www.osce.org/press-releases",
"endpoint": "https://news.osce.org/news/list?item_type=press-release&lang=en&page=1",
"kind": "html",
"lane": "context",
"region": "international",
"isTrustedOfficial": true,
"requiresKeywordMatch": true,
"comment": "OSCE press releases are provided as web pages; using the press release index."
"comment": "OSCE newsroom press-release listing filtered to press releases for a stable, current index surface."
},
{
"id": "resolve-network-research",
Expand Down
4 changes: 2 additions & 2 deletions data/sources/international/oversight.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,13 +14,13 @@
{
"id": "unoct-news",
"provider": "UN Office of Counter-Terrorism (UNOCT) - News",
"endpoint": "https://www.un.org/counterterrorism/news",
"endpoint": "https://www.un.org/counterterrorism/en/news-articles",
"kind": "html",
"lane": "oversight",
"region": "international",
"isTrustedOfficial": true,
"requiresKeywordMatch": false,
"comment": "UNOCT does not consistently publish RSS; using the official news page."
"comment": "UNOCT news/articles page used as the current official listing for Counter-Terrorism updates."
},
{
"id": "unsc-counter-terrorism-committee",
Expand Down
32 changes: 16 additions & 16 deletions data/sources/london/context.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -124,13 +124,13 @@
{
"id": "croydon-council-news",
"provider": "Croydon Council a News",
"endpoint": "https://news.croydon.gov.uk/",
"endpoint": "https://news.croydon.gov.uk/category/recent-news/",
"kind": "html",
"lane": "context",
"region": "london",
"isTrustedOfficial": true,
"requiresKeywordMatch": true,
"comment": "Croydon newsroom used for South London transport and public-safety corroboration."
"comment": "Croydon recent-news listing used for a cleaner borough news index and London corroboration."
},
{
"id": "ealing-council-news",
Expand Down Expand Up @@ -256,13 +256,13 @@
{
"id": "hillingdon-council-news",
"provider": "Hillingdon Council a News",
"endpoint": "https://www.hillingdon.gov.uk/news",
"endpoint": "https://pre.hillingdon.gov.uk/news",
"kind": "html",
"lane": "context",
"region": "london",
"isTrustedOfficial": true,
"requiresKeywordMatch": true,
"comment": "Hillingdon borough news used for Heathrow corridor and West London local context."
"comment": "Hillingdon canonical news target used for Heathrow corridor and West London local context."
Comment on lines 258 to +265
},
{
"id": "homerton-healthcare-news",
Expand Down Expand Up @@ -300,13 +300,13 @@
{
"id": "islington-council-news",
"provider": "Islington Council a News",
"endpoint": "https://www.islington.media/",
"endpoint": "https://www.islington.media/news/",
"kind": "html",
"lane": "context",
"region": "london",
"isTrustedOfficial": true,
"requiresKeywordMatch": true,
"comment": "Islington borough newsroom used for local London emergency and community-safety corroboration."
"comment": "Islington direct news listing used for local London emergency and community-safety corroboration."
},
{
"id": "kensington-chelsea-newsroom",
Expand Down Expand Up @@ -344,13 +344,13 @@
{
"id": "lambeth-council-news",
"provider": "Lambeth Council a News",
"endpoint": "https://love.lambeth.gov.uk/",
"endpoint": "https://love.lambeth.gov.uk/all-posts/",
"kind": "html",
"lane": "context",
"region": "london",
"isTrustedOfficial": true,
"requiresKeywordMatch": true,
"comment": "Lambeth news used for central-south London incident context and civic response."
"comment": "Lambeth all-posts listing used for a cleaner borough news index and civic-response context."
},
{
"id": "lewisham-council-news",
Expand Down Expand Up @@ -442,25 +442,25 @@
},
{
"id": "london-gov-rss-80603",
"provider": "Greater London Authority a RSS Feed 80603",
"endpoint": "https://www.london.gov.uk/rss-feeds/80603",
"kind": "rss",
"provider": "Mayor of London - Press Releases",
"endpoint": "https://www.london.gov.uk/media-centre/mayors-press-releases",
"kind": "html",
"lane": "context",
"region": "london",
"isTrustedOfficial": true,
"requiresKeywordMatch": true,
"comment": "GLA CMS feed ID 80603. Verify content category against the live feed; likely safety, policing, or resilience-related GLA updates."
"comment": "Stable Mayor's press releases page used in place of intermittently unresolved RSS feed ID 80603."
},
{
"id": "london-gov-rss-80632",
"provider": "Greater London Authority a RSS Feed 80632",
"endpoint": "https://www.london.gov.uk/rss-feeds/80632",
"kind": "rss",
"provider": "London Assembly - Press Releases",
"endpoint": "https://www.london.gov.uk/who-we-are/what-london-assembly-does/london-assembly-press-releases",
"kind": "html",
"lane": "context",
"region": "london",
"isTrustedOfficial": true,
"requiresKeywordMatch": true,
"comment": "GLA CMS feed ID 80632. Verify content category against the live feed for specific topic or committee coverage."
"comment": "Stable London Assembly press releases page used in place of intermittently unresolved RSS feed ID 80632."
},
{
"id": "mayor-of-london-press-releases",
Expand Down
40 changes: 20 additions & 20 deletions data/sources/uk/incidents.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,13 +14,13 @@
{
"id": "avon-somerset-police-news",
"provider": "Avon & Somerset Police a News",
"endpoint": "https://www.avonandsomerset.police.uk/newsroom/",
"endpoint": "https://www.avonandsomerset.police.uk/news/",

@chatgpt-codex-connector chatgpt-codex-connector Bot Apr 24, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Keep source endpoints unique in the catalog

This replacement makes avon-somerset-police-news share the exact normalized endpoint with avon-and-somerset-police-news-html, and the same commit introduces several similar collisions (e.g. Home Office, MoD, ECHR/Judiciary). node scripts/validate-feed-data.mjs now fails on duplicate endpoints, and the runtime loader (normaliseSourcesPayload) silently drops later duplicates, so some source IDs you intended to keep active are skipped each run.

Useful? React with 👍 / 👎.

"kind": "html",
"lane": "incidents",
"region": "uk",
"isTrustedOfficial": true,
"requiresKeywordMatch": true,
"comment": "Avon & Somerset Police newsroom covering Bristol and surrounding area incidents including counter-terrorism operations."
"comment": "Avon & Somerset Police official news page for Bristol and surrounding area incidents including counter-terrorism operations."
},
{
"id": "bedfordshire-police-news",
Expand Down Expand Up @@ -662,25 +662,25 @@
},
{
"id": "west-yorkshire-police-all-rss",
"provider": "West Yorkshire Police - All Feed",
"endpoint": "https://www.westyorkshire.police.uk/news/all/feed",
"kind": "rss",
"provider": "West Yorkshire Police - All News",
"endpoint": "https://www.westyorkshire.police.uk/news-stories-all",
"kind": "html",
Comment on lines 664 to +667
"lane": "incidents",
"region": "uk",
"isTrustedOfficial": true,
"requiresKeywordMatch": true,
"comment": "West Yorkshire Police all-content feed for broader regional policing updates with downstream filtering."
"comment": "West Yorkshire Police all news stories page used as the stable official fallback for the deprecated all-feed endpoint."
},
{
"id": "west-yorkshire-police-appeals-rss",
"provider": "West Yorkshire Police - Appeals Feed",
"endpoint": "https://www.westyorkshire.police.uk/news/appeals/feed",
"kind": "rss",
"provider": "West Yorkshire Police - Appeals",
"endpoint": "https://www.westyorkshire.police.uk/news-appeals/appeals",
"kind": "html",
"lane": "incidents",
"region": "uk",
"isTrustedOfficial": true,
"requiresKeywordMatch": true,
"comment": "West Yorkshire Police appeals feed for incidents, wanted appeals and operational notices."
"comment": "West Yorkshire Police appeals page used as the stable official fallback for the brittle appeals RSS endpoint."
},
{
"id": "west-yorkshire-police-missing-rss",
Expand All @@ -707,35 +707,35 @@
{
"id": "west-yorkshire-police-news-appeals",
"provider": "West Yorkshire Police - News Appeals",
"endpoint": "https://www.westyorkshirepolice.co.uk/news-appeals",
"endpoint": "https://www.westyorkshire.police.uk/news-appeals",
"kind": "html",
"lane": "incidents",
"region": "uk",
"isTrustedOfficial": true,
"requiresKeywordMatch": true,
"comment": "West Yorkshire Police appeals and news page for Leeds, Bradford and regional operations."
"comment": "West Yorkshire Police official news and appeals page for Leeds, Bradford and regional operations."
},
{
"id": "west-yorkshire-police-news-rss",
"provider": "West Yorkshire Police - News Feed",
"endpoint": "https://www.westyorkshire.police.uk/news/feed",
"kind": "rss",
"provider": "West Yorkshire Police - News",
"endpoint": "https://www.westyorkshire.police.uk/news-stories-all",
"kind": "html",
Comment on lines 719 to +722
"lane": "incidents",
"region": "uk",
"isTrustedOfficial": true,
"requiresKeywordMatch": true,
"comment": "West Yorkshire Police main news RSS feed for operational updates and incident reporting."
"comment": "West Yorkshire Police all news stories page used as the stable official fallback for the deprecated news RSS endpoint."
},
{
"id": "west-yorkshire-police-releases-rss",
"provider": "West Yorkshire Police - Releases Feed",
"endpoint": "https://www.westyorkshire.police.uk/news/releases/feed",
"kind": "rss",
"provider": "West Yorkshire Police - RSS Feeds",
"endpoint": "https://www.westyorkshire.police.uk/rss-feeds",
"kind": "html",
"lane": "incidents",
"region": "uk",
"isTrustedOfficial": true,
"requiresKeywordMatch": true,
"comment": "West Yorkshire Police releases feed for press statements, arrests and public-safety reporting."
"comment": "West Yorkshire Police official RSS index page used as the stable fallback for brittle releases feed URLs."
},
{
"id": "wiltshire-police-news",
Expand Down
Loading
Loading