[Bug]: fetch_webpage returns empty/errors for JS-rendered and paywalled pages (202 / 499)

[michaelatamuk]

Checklist

• I have searched for related issues but did not get the expected help.
• This bug has not been fixed in the latest version.
• Please note that if your bug description lacks corresponding environment information and a minimal reproducible demo, it will be difficult for us to reproduce and resolve the issue, which will reduce the possibility of receiving feedback, or even lead to the issue being closed.

🐞 Detailed Description of the Problem

fetch_webpage performs a plain HTTP GET and only falls back to the jina.ai reader (which renders pages server-side) on 401/403/429. Two common cases are broken:

1. JS-rendered pages (e.g., weatherspark.com) respond with HTTP 202 and an empty HTML shell. The tool reports Status: 202 / Content: [empty], even though the page has real content when opened in a browser.

2. Paywalled / anti-bot sites (e.g., climate-data.org) respond with HTTP 499 and Pay for usage. The tool hard-fails with

   [ERROR]: web page fetch failed ... reason='HTTP 499; response body: Pay for usage'
   

Reproduction

1. fetch_webpage on
   https://weatherspark.com/m/57830/6/Average-Weather-in-June-in-Amsterdam-Netherlands (weatherspark.com in Bing)
   → Status: 202, Content: [empty]

2. fetch_webpage on
   https://en.climate-data.org/europe/spain/barcelona/barcelona-395/ (en.climate-data.org in Bing)
   → Hard error: HTTP 499 “Pay for usage”

Expected

Both cases should fall back to the jina.ai reader so the agent receives real page content (when the reader can retrieve it) instead of an empty result or a hard error.

Detailed Environment Information Description

Jiuwenswarm

Additional Information

No response

Version Information

v0.2.4beta

[openjiuwen-collaboration-bot]

注意

此 Issue 需要负责人。 如果没有指派时， iamcandiceguo 是默认的负责人