**fix(harness): route 202/499 and JS-shell pages through jina reader**

[michaelatamuk]

Paired: GitHub #524 ↔ GitCode !2322

What type of PR is this?
/kind bugfix

---

What does this PR do / why do we need it

This PR expands the fetch_webpage fallback logic so that JS-rendered pages (HTTP 202) and paywalled/anti‑bot pages (HTTP 499) are routed through the jina.ai reader, which performs server‑side rendering.
Without this fix, the tool returned empty shells or hard errors for these pages, even though the reader can often retrieve real content.

This resolves **Bug #523 **: fetch_webpage returns empty/errors for JS-rendered and paywalled pages (202 / 499).

---

Problem

Non‑technical

Many modern websites do not return meaningful HTML to a plain HTTP GET:

• JS-rendered sites (e.g., weatherspark.com) return 202 and an empty HTML shell
• Paywalled / anti‑bot sites (e.g., climate-data.org) return 499 “Pay for usage”

The agent sees either:

• Status: 202 / Content: [empty]
• or a hard failure: HTTP 499; response body: Pay for usage

even though the page has real content when opened in a browser.

Technical

fetch_webpage:

• performs a raw GET
• extracts title/text
• falls back to the jina.ai reader only on 401/403/429

Two common failure modes were not recognized:

1. 202 soft-block / JS shell

   • body present
   • no extractable title/text
   • returned as empty instead of falling back

2. 499 paywall

   • treated as a hard error
   • no fallback attempted

This caused repeated failures and prevented the agent from accessing real page content.

---

Solution

Non‑technical

Route both 202 and 499 responses through the jina.ai reader, and also fall back when HTML extraction yields no meaningful content.
This recovers real page text for JS-rendered and paywalled sites whenever the reader can fetch them.

Technical

The fix adds:

1. 202 and 499 to the fallback status set
2. Defense in depth:
   After HTML extraction, if the response body was non-empty but produced no title and no text, treat it as a JS-rendered shell and fall back to the reader.

This ensures:

• 202 → reader
• 499 → reader
• JS shell → reader
• genuine empty 200 → still returned as empty

---

Expected Impact

• JS-rendered pages (weatherspark) now return real content instead of [empty]
• Paywalled pages (climate-data.org) now return reader content instead of hard errors
• More reliable web content retrieval
• No behavior change for genuinely empty 200 responses
• Fixes issue #2851

---

Validation

• python -m py_compile openjiuwen/harness/tools/web/fetch_webpage.py passes
• Manual tests confirm correct fallback behavior for 202 and 499 cases

Linked Closing Issues:

• Fixes [Bug]: fetch_webpage returns empty/errors for JS-rendered and paywalled pages (202 / 499) #523

[openjiuwen-collaboration-bot]

head_sha: e89b71b64a92ae3d271e40c6d5ee924dac233adc

变更摘要

此 PR 扩展了 WebFetchWebpageTool._fetch_or_fallback 方法的降级逻辑，将原本仅对 401/403/429 触发 jina.ai reader 回退的机制，扩展为同时覆盖 HTTP 202（JS 渲染的空白壳页面）和 HTTP 499（付费墙/反爬页面）。此外增加了防御性检查：当 HTML 提取后既无标题也无正文、但响应体非空时（即典型的 JS 渲染壳），同样回退到 reader 进行服务端渲染，从而恢复对这些页面的真实内容获取能力。

主要改动

• 扩展 fallback 状态码集合：将 {401, 403, 429} 改为 {202, 401, 403, 429, 499}，使 JS 软拦截（202）和付费墙（499）直接路由至 _fetch_via_jina_reader，而非报错或返回空内容。
• 新增 JS 空壳检测与回退：在 HTML 内容类型分支中，当 _extract_main_text_from_html 返回的 title 和 text 均为空字符串且原始 body 非空时，判定为 JS 渲染壳页面，随即调用 _fetch_via_jina_reader 获取服务端渲染后的真实文本。
• 更新 _fetch_or_fallback 文档字符串：补充了对 202 软拦截、499 付费墙以及 JS 空壳回退行为的说明，明确各状态码和空内容场景的处理策略。

[openjiuwen-collaboration-bot]

head_sha: e89b71b64a92ae3d271e40c6d5ee924dac233adc

代码审查

✅ 未发现问题

[openjiuwen-collaboration-bot]

head_sha: e89b71b64a92ae3d271e40c6d5ee924dac233adc

任务名称	结果	日志操作
静态检查	✅SUCCESS	点此跳转
禁用词扫描	N/A	N/A
防投毒检查	✅SUCCESS	点此跳转
开源合规检查	✅SUCCESS	点此跳转
UT测试	✅SUCCESS	点此跳转
ST测试	N/A	N/A
build 编译包	N/A	N/A
ruff codecheck	✅SUCCESS	N/A