Fix ELN (East Lothian) scraper#410
Open
symroe wants to merge 1 commit into
Open
Conversation
eastlothian.gov.uk returns HTTP 403 to wreq's Firefox133 TLS fingerprint but serves the page correctly to browser clients (confirmed via WebFetch). The scraper has been failing intermittently since June 21 with 403 and 'list index out of range' errors — the latter caused by the WAF returning a challenge page instead of the councillor listing, leaving selectors with nothing to parse. Switching to http_lib="playwright" uses headless Chromium whose TLS fingerprint passes the WAF, matching the fix pattern used for HAL, TON, MON, HNS, and other councils with the same behaviour.
Member
Author
Re-scrape after 05357e5Initial push: added The WAF blocks all non-Chromium HTTP clients from this environment (same block that Lambda's wreq receives), so a live scrape count cannot be obtained pre-merge. The page is confirmed to load correctly in a browser client (valid LocalGov Drupal councillor listing with ward accordions,
Counts will be updated here after the next Lambda run confirms the fix. Generated by Claude Code |
symroe
added a commit
that referenced
this pull request
Jul 4, 2026
fife.gov.uk returns HTTP 403 to wreq's Firefox133 TLS fingerprint but serves the councillors page correctly to browser clients. The WAF drops the connection when it detects the non-browser TLS client-hello. This is the same WAF fingerprint block pattern seen on ELN (PR #410), HAL (#400), TON (#402), and others. Adding http_lib = "playwright" switches from wreq to headless Chromium, whose Chrome TLS fingerprint passes the WAF. The scraper has been failing for 10 consecutive days (2026-06-23 through 2026-07-02).
symroe
pushed a commit
that referenced
this pull request
Jul 10, 2026
…int block www.cne-siar.gov.uk returns HTTP 403 to wreq's Firefox133 TLS fingerprint. The existing verify_requests=False addresses a certificate trust issue but does not bypass the WAF's TLS fingerprint check. Adding http_lib="playwright" switches to headless Chromium's Chrome TLS fingerprint, which passes the WAF and allows the councillors page and individual detail pages to load correctly. Same fix pattern as ELN (#410), FIF (#419), and other recently fixed councils.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What broke
eastlothian.gov.ukreturns HTTP 403 to wreq's Firefox133 TLS fingerprint but serves the councillors page correctly to browser clients (verified: WebFetch returns valid HTML with ward accordion structure). The scraper has failed repeatedly since at least June 21 with two symptoms: explicitStatus(403)errors, andlist index out of rangeerrors on days the WAF serves a challenge page instead of real HTML (the.view-id-councillors .view-contentcontainer selector finds nothing on a challenge page, leaving an empty list that then causes a parse error). The scraper passed on June 27–29 — likely due to IP rotation in Lambda — then failed again on June 30.What was fixed
http_lib = "playwright"to the Scraper class — headless Chromium's TLS fingerprint is not blocked by the WAF, and it renders the LocalGov Drupal page correctly before handing the DOM to BeautifulSoup. Same fix pattern as HAL (Fix HAL (Halton) scraper #400), TON (Fix TON (Tonbridge and Malling) scraper #402), MON (Fix MON (Monmouthshire) scraper #401), HNS (Fix HNS (Hounslow) scraper #405), and EAT (Fix EAT (Eastleigh) scraper #409).Scrape results
The WAF blocks all non-Chromium clients from this environment so a pre-merge count cannot be obtained here. Counts will be confirmed after the next Lambda run.
Generated by Claude Code