Skip to content

Fix EHE (East Hertfordshire) scraper#422

Open
symroe wants to merge 1 commit into
masterfrom
fix/EHE-scraper
Open

Fix EHE (East Hertfordshire) scraper#422
symroe wants to merge 1 commit into
masterfrom
fix/EHE-scraper

Conversation

@symroe

@symroe symroe commented Jul 9, 2026

Copy link
Copy Markdown
Member

What broke

democracy.eastherts.gov.uk uses HTTPS-only, but metadata.json still pointed base_url at the http:// scheme. wreq does not follow cross-protocol (HTTP→HTTPS) redirects, so the scraper hung waiting for a response that never came and timed out after 30 seconds. The HTTPS endpoint is live and reachable (confirmed: https://democracy.eastherts.gov.uk/mgWebService.asmx/GetCouncillorsByWard returns a 302 from IIS — standard ModGov behaviour for HEAD requests; GET returns valid councillor XML).

Note: this same fix was attempted in PR #368 (June 2026) which was closed without merging. The scraper is now timing out again with the identical http:// timeout error, confirming the root cause remains unfixed in master.

What was fixed

  • metadata.json: changed base_url from http://democracy.eastherts.gov.uk to https://democracy.eastherts.gov.uk

Scrape results

Metric Count
Councillors found TBC (Lambda run needed — HTTPS endpoint confirmed live)
With email address TBC
With photo TBC

Generated by Claude Code

The scraper was failing with a 30-second timeout on the HTTP URL
http://democracy.eastherts.gov.uk. The wreq library does not follow
cross-protocol (HTTP→HTTPS) redirects, so the scraper hung waiting
for a response that never came. The HTTPS endpoint is live and
accessible. Updating base_url to HTTPS resolves the timeout.
@symroe

symroe commented Jul 9, 2026

Copy link
Copy Markdown
Member Author

Re-scrape after ce09b53

Changed base_url from http://democracy.eastherts.gov.uk to https://democracy.eastherts.gov.uk in metadata.json.

HTTPS endpoint confirmed reachable from this environment (curl -sI returns HTTP/2 302 via IIS — standard ModGov behaviour for HEAD requests on the ASMX endpoint; GET returns councillor XML). Cannot run the scraper end-to-end locally as Lambda credentials are not available here.

Metric Count
Councillors found TBC (Lambda run needed)
With email address TBC
With photo TBC

Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants