Fix PEM (Pembrokeshire) scraper#406
Open
symroe wants to merge 1 commit into
Open
Conversation
…nt block After PR #395 migrated the URL from http:// to https://, the scraper still fails with connection reset by peer (OS error 104, ECONNRESET) on mgenglish.pembrokeshire.gov.uk:443. Testing confirms: - TCP connection establishes successfully (port 443 reachable) - TLS handshake is RST'd by the server when using wreq's Firefox133 fingerprint - Python ssl can establish TLS 1.3 but HTTP request then receives unexpected EOF This is a WAF TLS fingerprint block. Switching to headless Chromium's TLS fingerprint bypasses the WAF, same fix as HAL, TON, MON, HNS.
Member
Author
Re-scrape after a37a55fAdded Cannot run the scraper from this environment — all non-Chrome TLS fingerprints (wreq Firefox133, Python ssl, httpx) receive either a TCP RST or unexpected EOF from the server. TCP port 443 is reachable, and TLS 1.3 negotiates successfully with Python's ssl module, but the HTTP request then receives an unexpected EOF. Playwright's Chrome TLS fingerprint should clear this WAF check from Lambda.
Generated by Claude Code |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What broke
PR #395 (merged June 27) migrated
mgenglish.pembrokeshire.gov.ukfrom HTTP to HTTPS, but the scraper has continued failing on June 28 withconnection reset by peer(OS error 104, ECONNRESET). Investigation shows:The pattern is the same as HAL, TON, MON, and HNS — WAF TLS fingerprint blocking that playwright's Chrome fingerprint bypasses.
What was fixed
http_lib = "playwright"— headless Chromium's TLS fingerprint is not blocked by the WAF, allowing the ModGov XML endpoint to be reached.Scrape results
Cannot confirm counts from this environment (server rejects all non-Chrome TLS fingerprints including our proxy IP). Counts will be confirmed on the next Lambda run.
Generated by Claude Code