fix: harden HTML adapters with live E2E coverage

CHANGELOG.md

@@ -44,9 +44,10 @@ tags:
   credentials; `quality_escalation`, `paid_first`, and `free_only` are explicit
   alternatives. Default routing selects only the first configured provider in
   the candidate order; multiple optional providers require explicit selection.
-- Bounded live E2E behind explicit `LIVE_E2E=true` authorization, a maximum of
-  two network operations, a 10-second minimum interval, cleared optional
-  credentials, and a one-attempt search budget.
+- Bounded live E2E behind explicit `LIVE_E2E=true` authorization, selectable
+  DDG/Bing/Baidu/Yandex search scenarios, a maximum of five network operations,
+  a 10-second minimum interval, cleared optional credentials, and a one-attempt
+  search budget. Challenge responses stop subsequent probes.
 
 ### Runtime and quality infrastructure
 
@@ -106,6 +107,10 @@ tags:
   request-local Undici proxy transport with per-engine overrides, credential
   redaction, cancellation propagation, and no ambient proxy-variable pickup.
   Development-only scripts are no longer shipped in the npm artifact.
+- Hardened the Bing, Baidu, and Yandex zero-key HTML adapters with the shared
+  request-local transport, Cheerio result parsing, explicit per-engine proxy
+  settings, and structured `parse_error`/challenge handling for unexpected
+  successful HTML pages. Valid zero-result pages remain valid empty results.
 - Added a native DuckDuckGo Web representation using the page-issued,
   exact-allowlisted preload URL before HTML/Lite fallback. Added structured
   adapter errors and immediate provider cooldown for DDG/Sogou bot challenges.

HANDOVER.md

@@ -48,6 +48,9 @@ tags:
   Lite 只在 HTML HTTP 202 后、同一 deadline 内尝试一次。
 - DDG/Sogou 共享 request-local Undici 代理 transport；只读取显式引擎配置或
   `USE_PROXY=true` + `PROXY_URL`，不读取 ambient proxy，也不泄露凭证。
+- Bing/Baidu/Yandex 现复用同一 request-local transport，并支持
+  `BING_PROXY_URL`、`BAIDU_PROXY_URL`、`YANDEX_PROXY_URL`；HTML 200 响应若无
+  已识别搜索面会返回 `parse_error`，合法空结果仍保持为空成功。
 - DDG/Sogou 反爬统一返回 `bot_challenge` 并进入有界冷却。
 - `ProviderCooldownStore` 和 `SearchCache` 都以小型 store interface 解耦；
   内存是默认实现，本地持久化必须显式启用并对损坏/过期数据 fail open。
@@ -89,9 +92,21 @@ tags:
   HTTP 202 challenge；不要从该出口继续探测或捕获质量 fixture。
 - 不使用指纹轮换、挑战规避或高频重试来获取 DDG/Sogou 结果。
 
+### 2026-07-28 live E2E
+
+- `tests/e2e/basic-search.e2e.ts` 现在覆盖 DDG、Bing、Baidu、Yandex 的
+  独立搜索场景；`LIVE_E2E_ENGINES` 可选择引擎，challenge 会停止后续探测。
+- 受控运行显示上游状态随出口/时段变化：一次运行中 Bing、Baidu 非空通过，Yandex
+  被最终 URL `showcaptchafast` 识别为 `bot_challenge`；最终 commit 的复核中 Bing
+  通过、Baidu 返回 `bot_challenge`，Yandex 按规则未继续探测。
+- 这证明了 MCP stdio 到这些适配器的真实链路和 challenge 保留行为，不证明 Baidu/Yandex
+  的稳定可用性，也不构成多引擎搜索质量声明。复现示例：
+  `LIVE_E2E=true LIVE_E2E_ENGINES=bing,baidu,yandex LIVE_E2E_MAX_REQUESTS=3`
+  后运行 `npm run test:e2e:live`；PowerShell 需要使用 `$env:` 设置变量。
+
 ## 当前验证
 
-- 默认离线门禁：73 个测试文件，742 passed，2 个联网 E2E 按设计 skipped。
+- 默认离线门禁：73 个测试文件，754 passed，5 个 live E2E 按设计 skipped。
 - TypeScript/Windows build、能力矩阵漂移、冻结 Token benchmark 和 bootstrap
   quality benchmark：通过；bootstrap 仍不具备质量声明资格。
 - Lint：0 errors、0 warnings；`npm run lint` 通过 `--max-warnings 0`

README.md

@@ -192,7 +192,7 @@ settings cover the common deployment choices:
 | Choose spend policy | `SEARCH_PROVIDER_MODE`, `PAID_ENGINE_ORDER` |
 | Reduce response tokens | `OUTPUT_STYLE=compact`, `MAX_FULL_RESULTS`, `SNIPPET_LENGTH`, `EVIDENCE_BUDGET_CHARS` |
 | Restrict tools or engines | `ENABLED_TOOLS`, `DISABLED_TOOLS`, `ALLOWED_ENGINES`, `DENIED_ENGINES` |
-| Use an explicit proxy | `DUCKDUCKGO_PROXY_URL`, `SOGOU_PROXY_URL`, or `USE_PROXY=true` with `PROXY_URL` |
+| Use an explicit proxy | `DUCKDUCKGO_PROXY_URL`, `SOGOU_PROXY_URL`, `BING_PROXY_URL`, `BAIDU_PROXY_URL`, or `YANDEX_PROXY_URL`; alternatively `USE_PROXY=true` with `PROXY_URL` |
 | Persist the exact-result cache | `SEARCH_CACHE_DIRECTORY`, `SEARCH_CACHE_TTL_MS`, `SEARCH_CACHE_MAX_ENTRIES` |
 | Enable optional semantic processing | `SEMANTIC_DEDUP`, `SEMANTIC_RERANK`, `DEDUP_THRESHOLD`, `RERANK_TOP_K` |
 

README_zh.md

@@ -182,7 +182,7 @@ Wiby 使用官方 JSON API，是无需账号和 API Key 的真实零密钥来源
 | 选择费用策略 | `SEARCH_PROVIDER_MODE`、`PAID_ENGINE_ORDER` |
 | 减少响应 Token | `OUTPUT_STYLE=compact`、`MAX_FULL_RESULTS`、`SNIPPET_LENGTH`、`EVIDENCE_BUDGET_CHARS` |
 | 限制工具或引擎 | `ENABLED_TOOLS`、`DISABLED_TOOLS`、`ALLOWED_ENGINES`、`DENIED_ENGINES` |
-| 使用显式代理 | `DUCKDUCKGO_PROXY_URL`、`SOGOU_PROXY_URL`，或 `USE_PROXY=true` 配合 `PROXY_URL` |
+| 使用显式代理 | `DUCKDUCKGO_PROXY_URL`、`SOGOU_PROXY_URL`、`BING_PROXY_URL`、`BAIDU_PROXY_URL` 或 `YANDEX_PROXY_URL`；也可使用 `USE_PROXY=true` 配合 `PROXY_URL` |
 | 持久化精确结果缓存 | `SEARCH_CACHE_DIRECTORY`、`SEARCH_CACHE_TTL_MS`、`SEARCH_CACHE_MAX_ENTRIES` |
 | 开启可选语义处理 | `SEMANTIC_DEDUP`、`SEMANTIC_RERANK`、`DEDUP_THRESHOLD`、`RERANK_TOP_K` |
 

docs/architecture.md

@@ -148,6 +148,8 @@ semantic dedup/rerank，再以 post-semantic display basket 决定是否跳过
 运行状态只在首次需要时创建，并缓存到进程生命周期:
 
 - **代理连接池**: DDG/Sogou 首次代理请求时按脱敏配置创建
+- **HTML 引擎 transport**: Bing/Baidu/Yandex 复用同一 request-local 代理 seam；
+  每个引擎可通过显式环境变量覆盖
 - **引擎健康状态**: 首次失败后缓存降级结果
 - **Rate limiter**: 首次调用时创建，后续复用
 
@@ -162,6 +164,10 @@ DDG/Sogou 的 CAPTCHA、202 challenge 和 `/antispider/` 会转换为结构化
 `bot_challenge`。健康控制面立即暂停该 provider 一小时；到期后才允许新探测，
 避免在已知受限的网络出口上继续消耗延迟和上游配额。
 
+Bing、Baidu 和 Yandex 的 HTML adapter 使用 Cheerio 解析结果卡片。HTTP 200
+但没有已识别搜索面会转换为 `parse_error`；有搜索面但没有结果仍是合法空结果，
+避免把结构变更伪装成成功。它们继续复用编排层的限速、重试、健康和冷却策略。
+
 ### 5. 运行控制面不占默认工具槽位
 
 运行状态通过 MCP Resource 和 HTTP probe 暴露，而不是再注册一个默认可见的

scripts/run-live-e2e.mjs

@@ -15,7 +15,10 @@ if (process.env.LIVE_E2E !== 'true') {
   process.exit(2);
 }
 
-const maxRequests = boundedInteger('LIVE_E2E_MAX_REQUESTS', 2, 1, 2);
+const LIVE_ENGINE_NAMES = ['duckduckgo', 'bing', 'baidu', 'yandex'];
+validateEngineSelection(process.env.LIVE_E2E_ENGINES);
+
+const maxRequests = boundedInteger('LIVE_E2E_MAX_REQUESTS', 5, 1, 5);
 const minIntervalMs = boundedInteger(
   'LIVE_E2E_MIN_INTERVAL_MS',
   10_000,
@@ -46,6 +49,17 @@ function boundedInteger(name, fallback, minimum, maximum) {
   return parsed;
 }
 
+function validateEngineSelection(rawValue) {
+  if (rawValue === undefined) return;
+  const engines = rawValue.split(',').map(engine => engine.trim());
+  if (engines.length === 0 || engines.some(engine => !LIVE_ENGINE_NAMES.includes(engine))) {
+    console.error(
+      `LIVE_E2E_ENGINES must contain one or more supported engines: ${LIVE_ENGINE_NAMES.join(', ')}`,
+    );
+    process.exit(2);
+  }
+}
+
 function run(command, args, environment = {}) {
   return new Promise((resolve, reject) => {
     const child = spawn(command, args, {

src/engines/baidu.ts

@@ -1,7 +1,16 @@
-import { SearchResult, type EngineSearchOptions } from '../types.js';
-import { decodeHTMLTags } from '../infrastructure/html-utils.js';
-import { withTimeout } from '../infrastructure/abort.js';
+import * as cheerio from 'cheerio';
+import type { AnyNode } from 'domhandler';
+
 import { logger } from '../infrastructure/logger.js';
+import type { EngineSearchOptions, SearchResult } from '../types.js';
+import {
+  createHtmlParseError,
+  fetchSearchHtml,
+  normalizeHtmlText,
+  resolveHtmlResultUrl,
+} from './html-search.js';
+
+const BAIDU_SEARCH_URL = 'https://www.baidu.com/s';
 
 export const baiduProvider = {
   id: 'baidu' as const,
@@ -10,136 +19,113 @@ export const baiduProvider = {
   languages: ['zh'],
 };
 
-export async function searchBaidu(query: string, limit: number = 10, options?: EngineSearchOptions): Promise<SearchResult[]> {
+export async function searchBaidu(
+  query: string,
+  limit: number = 10,
+  options?: EngineSearchOptions,
+): Promise<SearchResult[]> {
   try {
-    const url = `https://www.baidu.com/s?wd=${encodeURIComponent(query)}&rn=${limit}`;
-    const res = await fetch(url, {
+    const url = new URL(BAIDU_SEARCH_URL);
+    url.searchParams.set('wd', query);
+    url.searchParams.set('rn', String(limit));
+    const html = await fetchSearchHtml('baidu', url, {
+      signal: options?.signal,
       headers: {
-        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36',
+        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) '
+          + 'AppleWebKit/537.36 (KHTML, like Gecko) '
+          + 'Chrome/120.0.0.0 Safari/537.36',
         'Accept': 'text/html,application/xhtml+xml',
         'Accept-Language': 'zh-CN,zh;q=0.9',
       },
-      signal: withTimeout(options?.signal, 10000),
     });
 
-    if (!res.ok) {
-      if (options?.throwOnError) throw new Error(`Baidu HTTP ${res.status}`);
-      logger.warn({ status: res.status }, 'Baidu HTTP error');
-      return [];
+    const results = parseBaiduHTML(html, limit);
+    if (!hasBaiduSearchSurface(html) || (hasBaiduResultCards(html) && results.length === 0)) {
+      throw createHtmlParseError('baidu');
     }
-
-    const html = await res.text();
-    return parseBaiduHTML(html, limit);
+    return results;
   } catch (error) {
     options?.signal?.throwIfAborted();
     if (options?.throwOnError) throw error;
     const msg = error instanceof Error ? error.message : String(error);
-    if (msg.includes('timeout')) {
-      logger.warn('Baidu search timed out');
-    } else {
-      logger.warn({ err: msg.slice(0, 200) }, 'Baidu search failed');
-    }
+    logger.warn({ err: msg.slice(0, 200) }, 'Baidu search failed');
     return [];
   }
 }
 
-/**
- * Extract a snippet from a Baidu result block using multiple fallback patterns.
- *
- * Patterns tried in order:
- *   1. <div class="c-abstract"> or <span class="c-abstract"> — classic Baidu snippet
- *   2. <span class="content-right_*"> — new-style Baidu snippet
- *   3. Any <span> containing 20–200 chars of meaningful text
- */
-function extractBaiduSnippet(block: string): string {
-  // Pattern 1: c-abstract div or span (classic Baidu snippet)
-  const abstractMatch = block.match(/<(?:div|span)[^>]*class="[^"]*c-abstract[^"]*"[^>]*>([\s\S]*?)<\/(?:div|span)>/i);
-  if (abstractMatch) {
-    const text = decodeHTMLTags(abstractMatch[1]);
-    if (text) return text;
-  }
-
-  // Pattern 2: content-right_* class (new-style Baidu snippet)
-  const contentRightMatch = block.match(/<(?:div|span)[^>]*class="[^"]*content-right_[^"]*"[^>]*>([\s\S]*?)<\/(?:div|span)>/i);
-  if (contentRightMatch) {
-    const text = decodeHTMLTags(contentRightMatch[1]);
-    if (text) return text;
-  }
-
-  // Pattern 3: any <span> with 20-200 chars of meaningful text
-  const spanRegex = /<span[^>]*>([\s\S]*?)<\/span>/g;
-  let spanMatch: RegExpExecArray | null;
-  while ((spanMatch = spanRegex.exec(block)) !== null) {
-    const text = decodeHTMLTags(spanMatch[1]);
-    if (text.length >= 20 && text.length <= 200) {
-      return text;
-    }
-  }
-
-  return '';
-}
-
-/**
- * Split Baidu HTML into result blocks around <h3><a href="..."> headers.
- * Returns one block per search result, spanning from slightly before the
- * h3 tag to just before the next h3 (or end of HTML).
- */
-function getResultBlocks(html: string): string[] {
-  const h3Regex = /<h3[^>]*><a[^>]*href="([^"]+)"[^>]*>([\s\S]*?)<\/a><\/h3>/g;
-  const h3Positions: number[] = [];
-
-  let match: RegExpExecArray | null;
-  while ((match = h3Regex.exec(html)) !== null) {
-    h3Positions.push(match.index);
-  }
-
-  if (h3Positions.length === 0) return [];
-
-  const blocks: string[] = [];
-  for (let i = 0; i < h3Positions.length; i++) {
-    const start = h3Positions[i];
-    const end = i + 1 < h3Positions.length ? h3Positions[i + 1] : html.length;
-    blocks.push(html.slice(start, end));
-  }
-
-  return blocks;
-}
-
-/**
- * Parse Baidu search result HTML into structured SearchResult objects.
- *
- * Uses a block-based approach: the HTML is split at <h3> result headers,
- * then each block is processed for title, URL, and snippet independently.
- * Snippet extraction tries three fallback patterns (c-abstract,
- * content-right_*, and generic spans).
- */
 export function parseBaiduHTML(html: string, limit: number = 10): SearchResult[] {
+  const $ = cheerio.load(html);
   const results: SearchResult[] = [];
-  const blocks = getResultBlocks(html);
-
-  if (blocks.length === 0) return results;
-
-  for (const block of blocks) {
-    if (results.length >= limit) break;
-
-    const h3Match = block.match(/<h3[^>]*><a[^>]*href="([^"]+)"[^>]*>([\s\S]*?)<\/a><\/h3>/);
-    if (!h3Match) continue;
-
-    const url = h3Match[1];
-    const title = decodeHTMLTags(h3Match[2]);
-
-    if (!url || !title || url.includes('baidu.com')) continue;
-
-    const snippet = extractBaiduSnippet(block);
-
+  const seenUrls = new Set<string>();
+  const cards = $('#content_left .c-container, #content_left .result, .c-container, .result');
+
+  cards.each((_, element) => {
+    if (results.length >= limit) return;
+
+    const card = $(element);
+    const titleLink = card.find('h3 a[href], h2 a[href], a[href]').first();
+    const rawUrl = card.attr('data-landurl')
+      ?? card.attr('mu')
+      ?? titleLink.attr('href')
+      ?? '';
+    const url = resolveHtmlResultUrl(
+      rawUrl,
+      BAIDU_SEARCH_URL,
+    );
+    const title = normalizeHtmlText(titleLink.text());
+    if (!title || !isExternalBaiduUrl(url) || seenUrls.has(url)) return;
+
+    const snippet = extractBaiduSnippet(card);
+    seenUrls.add(url);
     results.push({
       title,
       url,
       snippet,
       source: 'baidu',
       engines: ['baidu'],
     });
-  }
+  });
 
   return results;
 }
+
+function extractBaiduSnippet(card: cheerio.Cheerio<AnyNode>): string {
+  const knownSnippet = card
+    .find('.c-abstract, [class*="content-right_"], .c-color-text, .f13')
+    .first();
+  const knownText = normalizeHtmlText(knownSnippet.text());
+  if (knownText) return knownText;
+
+  let fallback = '';
+  card.find('span').each((_, element) => {
+    if (fallback) return;
+    const text = normalizeHtmlText(card.find(element).text());
+    if (text.length >= 20 && text.length <= 200) fallback = text;
+  });
+  return fallback;
+}
+
+function hasBaiduSearchSurface(html: string): boolean {
+  const $ = cheerio.load(html);
+  const resultContainer = $('#content_left').first();
+  return hasBaiduResultCards(html)
+    || (resultContainer.length > 0
+      && resultContainer.children().length === 0
+      && normalizeHtmlText(resultContainer.text()) === '');
+}
+
+function hasBaiduResultCards(html: string): boolean {
+  return cheerio.load(html)(
+    '#content_left .c-container, #content_left .result, .c-container, .result',
+  ).length > 0;
+}
+
+function isExternalBaiduUrl(url: string): boolean {
+  if (!url) return false;
+  try {
+    const hostname = new URL(url).hostname.toLowerCase();
+    return hostname !== 'baidu.com' && !hostname.endsWith('.baidu.com');
+  } catch {
+    return false;
+  }
+}