SGC (South Gloucestershire) scraper failing — TCP connection timeout, IP-level block on Lambda

[symroe]

Error (2026-06-29)

is_connect error: wreq::Error { kind: Connect, source: ConnectError("tcp connect error", 62.64.208.91:443, Os { code: 110, kind: TimedOut, message: "Connection timed out" }) }

Consecutive failures: 10 days (June 18–27).

History

SGC has been failing on and off since May with different error modes:

• SGC (South Gloucestershire) scraper failing — proxy returns 407 Proxy Authentication Required #306 (closed): use_proxy = True → proxy returned 407
• SGC (South Gloucestershire) scraper failing — TCP connection timeout, server unreachable #336 (closed): TCP timeout after proxy was removed
• SGC (South Gloucestershire) scraper failing — ModGov server returning 503 #341 (closed 2026-06-26): Server returning 503

Issue #341 was closed yesterday as "completed", but the scraper is still failing today — now with a TCP connection timeout rather than a 503.

Investigation

From our environment (non-Lambda IP, through HTTPS proxy):

• https://council.southglos.gov.uk/ → 403 Forbidden ("Access denied")
• https://council.southglos.gov.uk/mgWebService.asmx/GetCouncillorsByWard → 403 Forbidden

From Lambda (direct, no proxy):

• TCP connect timeout to 62.64.208.91:443 — SYN not acknowledged

The server is running (our proxy gets a 403, not a timeout), but Lambda's IP range (3.x.x.x) appears to be blocked at the TCP firewall level. council.southglos.gov.uk resolves to 62.64.208.91 which is an OVH/Ionos-hosted IP — the council may have a firewall rule specifically denying AWS IP ranges.

Fix patterns ruled out

1. HTTPS migration — already HTTPS
2. verify_requests = False — TCP connect fails before TLS; cert verification isn't the issue
3. http_lib = "playwright" — Playwright runs inside Lambda with the same blocked IP; won't help for a TCP-level firewall block
4. URL changes — DNS resolves fine; the URL hasn't moved, the server is actively blocking Lambda IPs
5. Proxy — previous proxy (SGC (South Gloucestershire) scraper failing — proxy returns 407 Proxy Authentication Required #306) returned 407; proxy infrastructure issue is separate

What needs to happen

This requires an infrastructure fix, not a scraper code change:

• Either the Lambda IP range needs to be allowlisted with South Gloucestershire's hosting provider
• Or the scraper needs to route through a working proxy (the proxy infrastructure issue from SGC (South Gloucestershire) scraper failing — proxy returns 407 Proxy Authentication Required #306 would need to be resolved first)
• Or the council needs to be contacted to allowlist the Lambda IP range

[symroe]

Still failing — 2026-07-06

Same TCP connect timeout, same host IP (62.64.208.91:443):

is_connect error: wreq::Error { kind: Request, uri: https://council.southglos.gov.uk/mgWebService.asmx/GetCouncillorsByWard, source: Error { kind: Connect, source: Some(ConnectError("tcp connect error", 62.64.208.91:443, Os { code: 110, kind: TimedOut, message: "Connection timed out" })), connect_info: None } }

Run: 2026-07-04T20:27:15 UTC (duration 15.8s). No change in error pattern — still a Lambda-IP-level TCP firewall block. Infrastructure fix still needed.

---

Generated by Claude Code

[symroe]

Still failing — 2026-07-07

Latest error (2026-07-05):

is_connect error: wreq::Error { kind: Request, uri: https://council.southglos.gov.uk/mgWebService.asmx/GetCouncillorsByWard, source: Error { kind: Connect, source: Some(ConnectError("tcp connect error", 62.64.208.91:443, Os { code: 110, kind: TimedOut, message: "Connection timed out" })), connect_info: None } }

Duration: 15.7 s (TCP connect timeout). The behaviour is unchanged — Lambda's IP is being dropped at the TCP firewall before TLS can be negotiated. No new fix patterns available; still needs an infrastructure-level solution (Lambda IP allowlisting or working egress proxy).

---

Generated by Claude Code

[symroe]

Still failing — 2026-07-08

Same TCP connection timeout, same IP (62.64.208.91:443), same error:

is_connect error: wreq::Error { kind: Request, uri: https://council.southglos.gov.uk/mgWebService.asmx/GetCouncillorsByWard, source: Error { kind: Connect, source: Some(ConnectError("tcp connect error", 62.64.208.91:443, Os { code: 110, kind: TimedOut, message: "Connection timed out" })), connect_info: None } }

Run: 2026-07-06T20:27:32 UTC, duration 15.8 s, status code 1. No change in behaviour — the TCP-level firewall block on Lambda's IP range remains in place. No scraper code fix will resolve this.

---

Generated by Claude Code

[symroe]

2026-07-10 update: SGC still failing with the same error — TCP connection timed out after 110 seconds to https://council.southglos.gov.uk/mgWebService.asmx/GetCouncillorsByWard. No code-level fix is possible; this remains an infrastructure-level block (Lambda IP range blocked at the TCP firewall). No new PR opened.

---

Generated by Claude Code

[bolivian-peru]

Your diagnosis looks exactly right, and there's a second signal in it worth pulling out because it points at the fix:

• Lambda → TCP timeout (SYN dropped). The council's host is behind an ACL that silently drops packets from AWS IP space. Nothing at the request layer (Playwright, UA, verify_requests) can help — you never complete the handshake, so there's no request to tune.
• Your non-Lambda proxy → 403 "Access denied", not a timeout. This is the useful part: that path did connect, so it's not the same firewall rule. A 403 at the app layer means that proxy's exit IP is also on a blocklist — almost certainly because it's another datacenter IP (most cheap HTTPS proxies exit from OVH/Hetzner/DO ranges, which Modern.gov / council WAFs flag by ASN reputation).

So both failures are the same underlying thing — datacenter-IP reputation — hitting you at two different layers. That's why swapping one datacenter egress (Lambda) for another datacenter egress (the old proxy) just moved you from a TCP drop to an app-layer 403.

The thing that clears both is an egress IP that isn't in datacenter space at all. A UK residential or mobile IP is essentially never on a council's AWS/OVH blocklist, and it's a plain env-level change — no scraper code touched. Quick confirm before committing to anything: route that one SOAP call (GetCouncillorsByWard) through any UK residential exit and check whether you get 200 vs the 403 your current proxy returns. If it's 200, the IP class was the whole problem.

One note on your use_proxy history (#306's 407): a 407 is the proxy rejecting your auth (missing/expired creds), not the target blocking you — so the LGSF proxy plumbing itself is fine, it just needs a working upstream with a residential UK exit.

---

Disclosure: I'm on the Proxies.sx team, so flagging the bias up front — and honestly the point that matters here is vendor-neutral: any working residential UK exit clears this, ours included. For a councillor list you're moving kilobytes, so cost is a rounding error at our per-GB rate. If it's handy, we do UK mobile (real 4G/5G) and residential; the drop-in is a client.proxies.sx account → API key → a sticky UK IP that slots into your existing proxy setting with no code change. The takeaway is just: the exit needs to be non-datacenter.