[Bug]

[josibartholomew]

Description

KBs encounter a 403 Forbidden HTTP status code when attempting to fetch content from https://osp.finance.harvard.edu/subrecipient-vs-contractor-guidance.

Steps to Reproduce

1. Go to a knowledge base
2. Click on "Add URL"
3. Paste https://osp.finance.harvard.edu/subrecipient-vs-contractor-guidance
4. Observe 403 error

Expected Behavior

I expected the Harvard URL to work the same as other URLs, where the web crawler goes to the provided URL and successfully extracts text.

Actual Behavior

Encountered a 403 error.

Screenshots

Environment

Browser: Chrome Version 149.0.7827.201 (Official Build) (arm64)
OS: macOS Tahoe Version 26.5.1

[umzcio]

I was able to replicate this bug and this is Harvard's CDN refusing the request, not a Vandalizer bug.

The 403 comes from Akamai's edge, before the request reaches Harvard's application.
The response body is Akamai's standard block page:

Access Denied — You don't have permission to access
"http://osp.finance.harvard.edu/subrecipient-vs-contractor-guidance" on this server.
Reference #18.xxxxx — https://errors.edgesuite.net/...

It is not our client identity. Four different clients, all → 403:

Client	Result
no User-Agent	403
python-httpx/0.27.0 UA	403
full Chrome UA (curl)	403
real headless Chromium (Playwright), from the app container	403

That last row is the decisive one. Akamai commonly distinguishes real browsers from
"curl with a spoofed UA" via TLS/HTTP2 fingerprinting — but a genuine Chromium engine is
refused identically. So this is not something a different UA, extra headers, or the
JS-rendering fallback can work around.

It's also not scoped to this page or to us. https://osp.finance.harvard.edu/
(site root) returns 403 as well, and outbound HTTPS from the same host is healthy
(example.com → 200, ecfr.gov → 302). That rules out egress/DNS problems on the
Vandalizer side.

Most likely cause: Akamai blocking by source IP / network reputation (datacenter
ranges), not by client behavior. The page loads fine in a browser on a residential
connection, which is consistent with that.

Suggested resolution: close as external / won't fix. The error message users already
see is accurate and gives the right remediation ("download the page in your browser and
upload it directly").

One small copy suggestion: "This usually means the site blocks automated tools like
Vandalizer" slightly implies we're at fault, when a stock Chrome browser from the same
host is refused too. Something like "The site's CDN refused this connection" would be
more accurate.

Adjacent finding — not the cause here, but worth its own issue

In web_fetcher.py, resp.raise_for_status() runs before the Playwright fallback, so a
403 short-circuits and the browser path is never attempted. For this site it makes no
difference (real Chromium is also refused), but sites that block httpx while permitting
a real browser would never get the fallback. Low priority; separate from this issue.

Happy to supply the full Akamai reference ID privately if anyone wants to raise it with
Harvard IT — omitting it here since it encodes the requesting host.