From 456d500f851d1f14351dd30649518b31edbf21c2 Mon Sep 17 00:00:00 2001 From: lennney Date: Tue, 28 Jul 2026 13:39:08 +0800 Subject: [PATCH 1/5] fix: harden HTML adapters with live E2E coverage --- CHANGELOG.md | 11 +- HANDOVER.md | 17 +- README.md | 2 +- README_zh.md | 2 +- docs/architecture.md | 6 + scripts/run-live-e2e.mjs | 2 +- src/engines/baidu.ts | 190 +++++++++------------ src/engines/bing.ts | 115 ++++++++----- src/engines/html-search.ts | 205 +++++++++++++++++++++++ src/engines/yandex.ts | 99 +++++++---- src/infrastructure/doctor.ts | 18 ++ src/infrastructure/engine-http.ts | 10 +- tests/e2e/basic-search.e2e.ts | 140 +++++++++++++++- tests/engines/baidu.test.ts | 52 ++++++ tests/engines/bing.test.ts | 79 ++++++++- tests/engines/yandex.test.ts | 81 ++++++++- tests/infrastructure/doctor.test.ts | 21 +++ tests/infrastructure/engine-http.test.ts | 21 +++ 18 files changed, 870 insertions(+), 201 deletions(-) create mode 100644 src/engines/html-search.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index d63e864..942de88 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -44,9 +44,10 @@ tags: credentials; `quality_escalation`, `paid_first`, and `free_only` are explicit alternatives. Default routing selects only the first configured provider in the candidate order; multiple optional providers require explicit selection. -- Bounded live E2E behind explicit `LIVE_E2E=true` authorization, a maximum of - two network operations, a 10-second minimum interval, cleared optional - credentials, and a one-attempt search budget. +- Bounded live E2E behind explicit `LIVE_E2E=true` authorization, selectable + DDG/Bing/Baidu/Yandex search scenarios, a maximum of five network operations, + a 10-second minimum interval, cleared optional credentials, and a one-attempt + search budget. Challenge responses stop subsequent probes. ### Runtime and quality infrastructure @@ -106,6 +107,10 @@ tags: request-local Undici proxy transport with per-engine overrides, credential redaction, cancellation propagation, and no ambient proxy-variable pickup. Development-only scripts are no longer shipped in the npm artifact. +- Hardened the Bing, Baidu, and Yandex zero-key HTML adapters with the shared + request-local transport, Cheerio result parsing, explicit per-engine proxy + settings, and structured `parse_error`/challenge handling for unexpected + successful HTML pages. Valid zero-result pages remain valid empty results. - Added a native DuckDuckGo Web representation using the page-issued, exact-allowlisted preload URL before HTML/Lite fallback. Added structured adapter errors and immediate provider cooldown for DDG/Sogou bot challenges. diff --git a/HANDOVER.md b/HANDOVER.md index 6513925..2aa7fff 100644 --- a/HANDOVER.md +++ b/HANDOVER.md @@ -48,6 +48,9 @@ tags: Lite 只在 HTML HTTP 202 后、同一 deadline 内尝试一次。 - DDG/Sogou 共享 request-local Undici 代理 transport；只读取显式引擎配置或 `USE_PROXY=true` + `PROXY_URL`，不读取 ambient proxy，也不泄露凭证。 +- Bing/Baidu/Yandex 现复用同一 request-local transport，并支持 + `BING_PROXY_URL`、`BAIDU_PROXY_URL`、`YANDEX_PROXY_URL`；HTML 200 响应若无 + 已识别搜索面会返回 `parse_error`，合法空结果仍保持为空成功。 - DDG/Sogou 反爬统一返回 `bot_challenge` 并进入有界冷却。 - `ProviderCooldownStore` 和 `SearchCache` 都以小型 store interface 解耦； 内存是默认实现，本地持久化必须显式启用并对损坏/过期数据 fail open。 @@ -89,9 +92,21 @@ tags: HTTP 202 challenge；不要从该出口继续探测或捕获质量 fixture。 - 不使用指纹轮换、挑战规避或高频重试来获取 DDG/Sogou 结果。 +### 2026-07-28 live E2E + +- `tests/e2e/basic-search.e2e.ts` 现在覆盖 DDG、Bing、Baidu、Yandex 的 + 独立搜索场景；`LIVE_E2E_ENGINES` 可选择引擎，challenge 会停止后续探测。 +- 受控运行显示上游状态随出口/时段变化：一次运行中 Bing、Baidu 非空通过，Yandex + 被最终 URL `showcaptchafast` 识别为 `bot_challenge`；最终 commit 的复核中 Bing + 通过、Baidu 返回 `bot_challenge`，Yandex 按规则未继续探测。 +- 这证明了 MCP stdio 到这些适配器的真实链路和 challenge 保留行为，不证明 Baidu/Yandex + 的稳定可用性，也不构成多引擎搜索质量声明。复现示例： + `LIVE_E2E=true LIVE_E2E_ENGINES=bing,baidu,yandex LIVE_E2E_MAX_REQUESTS=3` + 后运行 `npm run test:e2e:live`；PowerShell 需要使用 `$env:` 设置变量。 + ## 当前验证 -- 默认离线门禁：73 个测试文件，742 passed，2 个联网 E2E 按设计 skipped。 +- 默认离线门禁：73 个测试文件，754 passed，5 个 live E2E 按设计 skipped。 - TypeScript/Windows build、能力矩阵漂移、冻结 Token benchmark 和 bootstrap quality benchmark：通过；bootstrap 仍不具备质量声明资格。 - Lint：0 errors、0 warnings；`npm run lint` 通过 `--max-warnings 0` diff --git a/README.md b/README.md index 6e231b8..deb2c0d 100644 --- a/README.md +++ b/README.md @@ -192,7 +192,7 @@ settings cover the common deployment choices: | Choose spend policy | `SEARCH_PROVIDER_MODE`, `PAID_ENGINE_ORDER` | | Reduce response tokens | `OUTPUT_STYLE=compact`, `MAX_FULL_RESULTS`, `SNIPPET_LENGTH`, `EVIDENCE_BUDGET_CHARS` | | Restrict tools or engines | `ENABLED_TOOLS`, `DISABLED_TOOLS`, `ALLOWED_ENGINES`, `DENIED_ENGINES` | -| Use an explicit proxy | `DUCKDUCKGO_PROXY_URL`, `SOGOU_PROXY_URL`, or `USE_PROXY=true` with `PROXY_URL` | +| Use an explicit proxy | `DUCKDUCKGO_PROXY_URL`, `SOGOU_PROXY_URL`, `BING_PROXY_URL`, `BAIDU_PROXY_URL`, or `YANDEX_PROXY_URL`; alternatively `USE_PROXY=true` with `PROXY_URL` | | Persist the exact-result cache | `SEARCH_CACHE_DIRECTORY`, `SEARCH_CACHE_TTL_MS`, `SEARCH_CACHE_MAX_ENTRIES` | | Enable optional semantic processing | `SEMANTIC_DEDUP`, `SEMANTIC_RERANK`, `DEDUP_THRESHOLD`, `RERANK_TOP_K` | diff --git a/README_zh.md b/README_zh.md index 0313778..b5b86a7 100644 --- a/README_zh.md +++ b/README_zh.md @@ -182,7 +182,7 @@ Wiby 使用官方 JSON API，是无需账号和 API Key 的真实零密钥来源 | 选择费用策略 | `SEARCH_PROVIDER_MODE`、`PAID_ENGINE_ORDER` | | 减少响应 Token | `OUTPUT_STYLE=compact`、`MAX_FULL_RESULTS`、`SNIPPET_LENGTH`、`EVIDENCE_BUDGET_CHARS` | | 限制工具或引擎 | `ENABLED_TOOLS`、`DISABLED_TOOLS`、`ALLOWED_ENGINES`、`DENIED_ENGINES` | -| 使用显式代理 | `DUCKDUCKGO_PROXY_URL`、`SOGOU_PROXY_URL`，或 `USE_PROXY=true` 配合 `PROXY_URL` | +| 使用显式代理 | `DUCKDUCKGO_PROXY_URL`、`SOGOU_PROXY_URL`、`BING_PROXY_URL`、`BAIDU_PROXY_URL` 或 `YANDEX_PROXY_URL`；也可使用 `USE_PROXY=true` 配合 `PROXY_URL` | | 持久化精确结果缓存 | `SEARCH_CACHE_DIRECTORY`、`SEARCH_CACHE_TTL_MS`、`SEARCH_CACHE_MAX_ENTRIES` | | 开启可选语义处理 | `SEMANTIC_DEDUP`、`SEMANTIC_RERANK`、`DEDUP_THRESHOLD`、`RERANK_TOP_K` | diff --git a/docs/architecture.md b/docs/architecture.md index 28def6a..07beb6b 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -148,6 +148,8 @@ semantic dedup/rerank，再以 post-semantic display basket 决定是否跳过 运行状态只在首次需要时创建，并缓存到进程生命周期: - **代理连接池**: DDG/Sogou 首次代理请求时按脱敏配置创建 +- **HTML 引擎 transport**: Bing/Baidu/Yandex 复用同一 request-local 代理 seam； + 每个引擎可通过显式环境变量覆盖 - **引擎健康状态**: 首次失败后缓存降级结果 - **Rate limiter**: 首次调用时创建，后续复用 @@ -162,6 +164,10 @@ DDG/Sogou 的 CAPTCHA、202 challenge 和 `/antispider/` 会转换为结构化 `bot_challenge`。健康控制面立即暂停该 provider 一小时；到期后才允许新探测， 避免在已知受限的网络出口上继续消耗延迟和上游配额。 +Bing、Baidu 和 Yandex 的 HTML adapter 使用 Cheerio 解析结果卡片。HTTP 200 +但没有已识别搜索面会转换为 `parse_error`；有搜索面但没有结果仍是合法空结果， +避免把结构变更伪装成成功。它们继续复用编排层的限速、重试、健康和冷却策略。 + ### 5. 运行控制面不占默认工具槽位 运行状态通过 MCP Resource 和 HTTP probe 暴露，而不是再注册一个默认可见的 diff --git a/scripts/run-live-e2e.mjs b/scripts/run-live-e2e.mjs index 60552bf..040b801 100644 --- a/scripts/run-live-e2e.mjs +++ b/scripts/run-live-e2e.mjs @@ -15,7 +15,7 @@ if (process.env.LIVE_E2E !== 'true') { process.exit(2); } -const maxRequests = boundedInteger('LIVE_E2E_MAX_REQUESTS', 2, 1, 2); +const maxRequests = boundedInteger('LIVE_E2E_MAX_REQUESTS', 5, 1, 5); const minIntervalMs = boundedInteger( 'LIVE_E2E_MIN_INTERVAL_MS', 10_000, diff --git a/src/engines/baidu.ts b/src/engines/baidu.ts index 6208b25..3169acc 100644 --- a/src/engines/baidu.ts +++ b/src/engines/baidu.ts @@ -1,7 +1,16 @@ -import { SearchResult, type EngineSearchOptions } from '../types.js'; -import { decodeHTMLTags } from '../infrastructure/html-utils.js'; -import { withTimeout } from '../infrastructure/abort.js'; +import * as cheerio from 'cheerio'; +import type { AnyNode } from 'domhandler'; + import { logger } from '../infrastructure/logger.js'; +import type { EngineSearchOptions, SearchResult } from '../types.js'; +import { + createHtmlParseError, + fetchSearchHtml, + normalizeHtmlText, + resolveHtmlResultUrl, +} from './html-search.js'; + +const BAIDU_SEARCH_URL = 'https://www.baidu.com/s'; export const baiduProvider = { id: 'baidu' as const, @@ -10,128 +19,63 @@ export const baiduProvider = { languages: ['zh'], }; -export async function searchBaidu(query: string, limit: number = 10, options?: EngineSearchOptions): Promise { +export async function searchBaidu( + query: string, + limit: number = 10, + options?: EngineSearchOptions, +): Promise { try { - const url = `https://www.baidu.com/s?wd=${encodeURIComponent(query)}&rn=${limit}`; - const res = await fetch(url, { + const url = new URL(BAIDU_SEARCH_URL); + url.searchParams.set('wd', query); + url.searchParams.set('rn', String(limit)); + const html = await fetchSearchHtml('baidu', url, { + signal: options?.signal, headers: { - 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36', + 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) ' + + 'AppleWebKit/537.36 (KHTML, like Gecko) ' + + 'Chrome/120.0.0.0 Safari/537.36', 'Accept': 'text/html,application/xhtml+xml', 'Accept-Language': 'zh-CN,zh;q=0.9', }, - signal: withTimeout(options?.signal, 10000), }); - if (!res.ok) { - if (options?.throwOnError) throw new Error(`Baidu HTTP ${res.status}`); - logger.warn({ status: res.status }, 'Baidu HTTP error'); - return []; + if (!hasBaiduSearchSurface(html)) { + throw createHtmlParseError('baidu'); } - - const html = await res.text(); return parseBaiduHTML(html, limit); } catch (error) { options?.signal?.throwIfAborted(); if (options?.throwOnError) throw error; const msg = error instanceof Error ? error.message : String(error); - if (msg.includes('timeout')) { - logger.warn('Baidu search timed out'); - } else { - logger.warn({ err: msg.slice(0, 200) }, 'Baidu search failed'); - } + logger.warn({ err: msg.slice(0, 200) }, 'Baidu search failed'); return []; } } -/** - * Extract a snippet from a Baidu result block using multiple fallback patterns. - * - * Patterns tried in order: - * 1.
or — classic Baidu snippet - * 2. — new-style Baidu snippet - * 3. Any containing 20–200 chars of meaningful text - */ -function extractBaiduSnippet(block: string): string { - // Pattern 1: c-abstract div or span (classic Baidu snippet) - const abstractMatch = block.match(/<(?:div|span)[^>]*class="[^"]*c-abstract[^"]*"[^>]*>([\s\S]*?)<\/(?:div|span)>/i); - if (abstractMatch) { - const text = decodeHTMLTags(abstractMatch[1]); - if (text) return text; - } - - // Pattern 2: content-right_* class (new-style Baidu snippet) - const contentRightMatch = block.match(/<(?:div|span)[^>]*class="[^"]*content-right_[^"]*"[^>]*>([\s\S]*?)<\/(?:div|span)>/i); - if (contentRightMatch) { - const text = decodeHTMLTags(contentRightMatch[1]); - if (text) return text; - } - - // Pattern 3: any with 20-200 chars of meaningful text - const spanRegex = /]*>([\s\S]*?)<\/span>/g; - let spanMatch: RegExpExecArray | null; - while ((spanMatch = spanRegex.exec(block)) !== null) { - const text = decodeHTMLTags(spanMatch[1]); - if (text.length >= 20 && text.length <= 200) { - return text; - } - } - - return ''; -} - -/** - * Split Baidu HTML into result blocks around

headers. - * Returns one block per search result, spanning from slightly before the - * h3 tag to just before the next h3 (or end of HTML). - */ -function getResultBlocks(html: string): string[] { - const h3Regex = /]*>]*href="([^"]+)"[^>]*>([\s\S]*?)<\/a><\/h3>/g; - const h3Positions: number[] = []; - - let match: RegExpExecArray | null; - while ((match = h3Regex.exec(html)) !== null) { - h3Positions.push(match.index); - } - - if (h3Positions.length === 0) return []; - - const blocks: string[] = []; - for (let i = 0; i < h3Positions.length; i++) { - const start = h3Positions[i]; - const end = i + 1 < h3Positions.length ? h3Positions[i + 1] : html.length; - blocks.push(html.slice(start, end)); - } - - return blocks; -} - -/** - * Parse Baidu search result HTML into structured SearchResult objects. - * - * Uses a block-based approach: the HTML is split at

result headers, - * then each block is processed for title, URL, and snippet independently. - * Snippet extraction tries three fallback patterns (c-abstract, - * content-right_*, and generic spans). - */ export function parseBaiduHTML(html: string, limit: number = 10): SearchResult[] { + const $ = cheerio.load(html); const results: SearchResult[] = []; - const blocks = getResultBlocks(html); - - if (blocks.length === 0) return results; - - for (const block of blocks) { - if (results.length >= limit) break; - - const h3Match = block.match(/]*>]*href="([^"]+)"[^>]*>([\s\S]*?)<\/a><\/h3>/); - if (!h3Match) continue; - - const url = h3Match[1]; - const title = decodeHTMLTags(h3Match[2]); - - if (!url || !title || url.includes('baidu.com')) continue; - - const snippet = extractBaiduSnippet(block); - + const seenUrls = new Set(); + const cards = $('#content_left .c-container, #content_left .result, .c-container, .result'); + + cards.each((_, element) => { + if (results.length >= limit) return; + + const card = $(element); + const titleLink = card.find('h3 a[href], h2 a[href], a[href]').first(); + const rawUrl = card.attr('data-landurl') + ?? card.attr('mu') + ?? titleLink.attr('href') + ?? ''; + const url = resolveHtmlResultUrl( + rawUrl, + BAIDU_SEARCH_URL, + ); + const title = normalizeHtmlText(titleLink.text()); + if (!title || !isExternalBaiduUrl(url) || seenUrls.has(url)) return; + + const snippet = extractBaiduSnippet(card); + seenUrls.add(url); results.push({ title, url, @@ -139,7 +83,39 @@ export function parseBaiduHTML(html: string, limit: number = 10): SearchResult[] source: 'baidu', engines: ['baidu'], }); - } + }); return results; } + +function extractBaiduSnippet(card: cheerio.Cheerio): string { + const knownSnippet = card + .find('.c-abstract, [class*="content-right_"], .c-color-text, .f13') + .first(); + const knownText = normalizeHtmlText(knownSnippet.text()); + if (knownText) return knownText; + + let fallback = ''; + card.find('span').each((_, element) => { + if (fallback) return; + const text = normalizeHtmlText(card.find(element).text()); + if (text.length >= 20 && text.length <= 200) fallback = text; + }); + return fallback; +} + +function hasBaiduSearchSurface(html: string): boolean { + const $ = cheerio.load(html); + return $('#content_left').length > 0 + || $('.c-container, .result').length > 0; +} + +function isExternalBaiduUrl(url: string): boolean { + if (!url) return false; + try { + const hostname = new URL(url).hostname.toLowerCase(); + return hostname !== 'baidu.com' && !hostname.endsWith('.baidu.com'); + } catch { + return false; + } +} diff --git a/src/engines/bing.ts b/src/engines/bing.ts index 66c8eac..df9cca5 100644 --- a/src/engines/bing.ts +++ b/src/engines/bing.ts @@ -1,7 +1,15 @@ -import { SearchResult, type EngineSearchOptions } from '../types.js'; -import { decodeHTMLTags } from '../infrastructure/html-utils.js'; -import { withTimeout } from '../infrastructure/abort.js'; +import * as cheerio from 'cheerio'; + import { logger } from '../infrastructure/logger.js'; +import type { EngineSearchOptions, SearchResult } from '../types.js'; +import { + createHtmlParseError, + fetchSearchHtml, + normalizeHtmlText, + resolveHtmlResultUrl, +} from './html-search.js'; + +const BING_SEARCH_URL = 'https://www.bing.com/search'; export const bingProvider = { id: 'bing' as const, @@ -10,62 +18,83 @@ export const bingProvider = { languages: ['en', 'zh'], }; -export async function searchBing(query: string, limit: number = 10, options?: EngineSearchOptions): Promise { +export async function searchBing( + query: string, + limit: number = 10, + options?: EngineSearchOptions, +): Promise { try { - const url = `https://www.bing.com/search?q=${encodeURIComponent(query)}&count=${limit}`; - const res = await fetch(url, { + const url = new URL(BING_SEARCH_URL); + url.searchParams.set('q', query); + url.searchParams.set('count', String(limit)); + const html = await fetchSearchHtml('bing', url, { + signal: options?.signal, headers: { - 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36', + 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) ' + + 'AppleWebKit/537.36 (KHTML, like Gecko) ' + + 'Chrome/120.0.0.0 Safari/537.36', 'Accept': 'text/html,application/xhtml+xml', 'Accept-Language': 'en-US,en;q=0.9,zh-CN;q=0.8', }, - signal: withTimeout(options?.signal, 10000), }); - if (!res.ok) { - if (options?.throwOnError) throw new Error(`Bing HTTP ${res.status}`); - logger.warn({ status: res.status }, 'Bing HTTP error'); - return []; + if (!hasBingSearchSurface(html)) { + throw createHtmlParseError('bing'); } - - const html = await res.text(); - return parseBingResults(html, limit); + return parseBingHTML(html, limit); } catch (error) { options?.signal?.throwIfAborted(); if (options?.throwOnError) throw error; const msg = error instanceof Error ? error.message : String(error); - if (msg.includes('timeout')) { - logger.warn('Bing search timed out'); - } else { - logger.warn({ err: msg.slice(0, 200) }, 'Bing search failed'); - } + logger.warn({ err: msg.slice(0, 200) }, 'Bing search failed'); return []; } } -function parseBingResults(html: string, limit: number): SearchResult[] { +export function parseBingHTML(html: string, limit: number = 10): SearchResult[] { + const $ = cheerio.load(html); const results: SearchResult[] = []; - - // Parse Bing HTML results - // Pattern:

  • TITLE

    SNIPPET

    • - const resultRegex = /
  • [\s\S]*?

    ]*>([\s\S]*?)<\/a><\/h2>[\s\S]*?]*>([\s\S]*?)<\/p>/g; - - let match; - while ((match = resultRegex.exec(html)) && results.length < limit) { - const url = match[1]; - const title = decodeHTMLTags(match[2]); - const snippet = decodeHTMLTags(match[3]); - - if (url && title) { - results.push({ - title, - url, - snippet: snippet || '', - source: 'bing', - engines: ['bing'], - }); - } - } - + const seenUrls = new Set(); + + $('#b_results li.b_algo, li.b_algo').each((_, element) => { + if (results.length >= limit) return; + + const card = $(element); + const titleLink = card.find('h2 a[href], a[href]').first(); + const url = resolveHtmlResultUrl( + titleLink.attr('href') ?? '', + BING_SEARCH_URL, + ); + const title = normalizeHtmlText(titleLink.text()); + if (!title || !isExternalBingUrl(url) || seenUrls.has(url)) return; + + const snippet = normalizeHtmlText( + card.find('.b_caption p, .b_snippet, p').first().text(), + ); + seenUrls.add(url); + results.push({ + title, + url, + snippet, + source: 'bing', + engines: ['bing'], + }); + }); + return results; } +function hasBingSearchSurface(html: string): boolean { + const $ = cheerio.load(html); + return $('#b_results').length > 0 + || $('li.b_algo').length > 0; +} + +function isExternalBingUrl(url: string): boolean { + if (!url) return false; + try { + return new URL(url).hostname.toLowerCase() !== 'bing.com' + && !new URL(url).hostname.toLowerCase().endsWith('.bing.com'); + } catch { + return false; + } +} diff --git a/src/engines/html-search.ts b/src/engines/html-search.ts new file mode 100644 index 0000000..7ef41be --- /dev/null +++ b/src/engines/html-search.ts @@ -0,0 +1,205 @@ +import * as cheerio from 'cheerio'; + +import { withTimeout } from '../infrastructure/abort.js'; +import { + fetchForEngine, + type ProxyAwareEngine, +} from '../infrastructure/engine-http.js'; +import { EngineAdapterError } from './engine-error.js'; + +export type HtmlSearchEngine = Extract< + ProxyAwareEngine, + 'bing' | 'baidu' | 'yandex' +>; + +export interface HtmlSearchFetchOptions { + signal?: AbortSignal; + timeoutMs?: number; + headers?: HeadersInit; +} + +const DEFAULT_TIMEOUT_MS = 10_000; +const CHALLENGE_COOLDOWN_MS = 60 * 60 * 1000; +const DEFAULT_RATE_LIMIT_COOLDOWN_MS = 30_000; +const MAX_RATE_LIMIT_COOLDOWN_MS = 5 * 60_000; + +/** Fetch an HTML search page through the shared, explicitly configured transport. */ +export async function fetchSearchHtml( + engine: HtmlSearchEngine, + input: string | URL, + options: HtmlSearchFetchOptions = {}, +): Promise { + const response = await fetchForEngine(engine, input, { + headers: options.headers, + signal: withTimeout(options.signal, options.timeoutMs ?? DEFAULT_TIMEOUT_MS), + }); + const html = await response.text(); + + if (!response.ok) { + throw createHttpError(engine, response, html); + } + if (looksLikeBotChallenge(html) || looksLikeBotChallengeUrl(response.url)) { + throw createBotChallenge(engine); + } + return html; +} + +/** Create a stable adapter error when a successful page has no known search surface. */ +export function createHtmlParseError(engine: HtmlSearchEngine): EngineAdapterError { + return new EngineAdapterError( + 'parse_error', + `${engine} returned HTML without a recognized search result surface`, + { + retryable: false, + suggestion: 'Use another provider while the response parser is checked', + }, + ); +} + +/** Normalize DOM text without changing the result content semantics. */ +export function normalizeHtmlText(value: string): string { + return value.replace(/\s+/g, ' ').trim(); +} + +/** Resolve a result link while rejecting non-web URLs. */ +export function resolveHtmlResultUrl(rawUrl: string, baseUrl: string | URL): string { + if (!rawUrl.trim()) return ''; + try { + const url = new URL(rawUrl, baseUrl); + return ['http:', 'https:'].includes(url.protocol) ? url.toString() : ''; + } catch { + return ''; + } +} + +/** Detect common anti-bot interstitials without treating result snippets as challenges. */ +function looksLikeBotChallenge(html: string): boolean { + const $ = cheerio.load(html); + const title = normalizeHtmlText($('title').first().text()); + const metaDescription = normalizeHtmlText( + $('meta[name="description"]').first().attr('content') ?? '', + ); + const leadingText = normalizeHtmlText($('body').first().text()).slice(0, 800); + const titleAndMeta = `${title} ${metaDescription}`.toLowerCase(); + const leadingChallengeText = leadingText.toLowerCase(); + const titleMarkers = [ + 'captcha', + 'verify you are human', + 'unusual traffic', + 'access denied', + 'robot check', + 'smartcaptcha', + 'security verification', + '安全验证', + '验证码', + '访问异常', + '请输入验证码', + '反爬', + 'я не робот', + 'проверка безопасности', + ]; + if (titleMarkers.some(marker => titleAndMeta.includes(marker))) return true; + + return [ + 'verify you are human', + 'unusual traffic', + 'robot check', + 'smartcaptcha', + 'security verification', + 'checking your browser before redirecting', + '安全验证', + '验证码', + '访问异常', + '请输入验证码', + '反爬', + 'я не робот', + 'проверка безопасности', + ].some(marker => leadingChallengeText.includes(marker)); +} + +function looksLikeBotChallengeUrl(url: string): boolean { + const normalized = url.toLowerCase(); + return normalized.includes('/showcaptcha') + || normalized.includes('/captcha') + || normalized.includes('/verification'); +} + +function createBotChallenge(engine: HtmlSearchEngine): EngineAdapterError { + return new EngineAdapterError( + 'bot_challenge', + `${engine} returned an anti-bot challenge`, + { + retryable: false, + cooldownMs: CHALLENGE_COOLDOWN_MS, + suggestion: 'Wait for the provider cooldown or use another network exit', + }, + ); +} + +function createHttpError( + engine: HtmlSearchEngine, + response: Response, + html: string, +): EngineAdapterError { + if (looksLikeBotChallenge(html)) return createBotChallenge(engine); + + if (response.status === 429) { + return new EngineAdapterError( + 'rate_limited', + `${engine} rate limit reached`, + { + retryable: false, + cooldownMs: parseRetryAfter(response), + suggestion: 'Use another provider or retry after the cooldown expires', + }, + ); + } + if (response.status === 408 || response.status === 425) { + return new EngineAdapterError( + 'timeout', + `${engine} returned transient HTTP ${response.status}`, + { + retryable: true, + suggestion: 'Retry within the shared request budget or use another provider', + }, + ); + } + if (response.status >= 500) { + return new EngineAdapterError( + 'upstream_5xx', + `${engine} returned HTTP ${response.status}`, + { + retryable: response.status !== 501, + suggestion: 'Use another provider or retry later', + }, + ); + } + return new EngineAdapterError( + 'upstream_4xx', + `${engine} returned HTTP ${response.status}`, + { + retryable: false, + suggestion: 'Use an explicit provider proxy or another provider', + }, + ); +} + +function parseRetryAfter(response: Response): number { + const retryAfter = response.headers.get('retry-after')?.trim(); + if (!retryAfter) return DEFAULT_RATE_LIMIT_COOLDOWN_MS; + + const seconds = Number(retryAfter); + if (Number.isFinite(seconds) && seconds >= 0) { + return Math.min( + Math.max(Math.ceil(seconds * 1_000), 1_000), + MAX_RATE_LIMIT_COOLDOWN_MS, + ); + } + + const retryAt = Date.parse(retryAfter); + if (!Number.isFinite(retryAt)) return DEFAULT_RATE_LIMIT_COOLDOWN_MS; + return Math.min( + Math.max(retryAt - Date.now(), 1_000), + MAX_RATE_LIMIT_COOLDOWN_MS, + ); +} diff --git a/src/engines/yandex.ts b/src/engines/yandex.ts index 0d2f578..c6b1a26 100644 --- a/src/engines/yandex.ts +++ b/src/engines/yandex.ts @@ -1,7 +1,15 @@ -import { SearchResult, type EngineSearchOptions } from '../types.js'; -import { decodeHTMLTags } from '../infrastructure/html-utils.js'; -import { withTimeout } from '../infrastructure/abort.js'; +import * as cheerio from 'cheerio'; + import { logger } from '../infrastructure/logger.js'; +import type { EngineSearchOptions, SearchResult } from '../types.js'; +import { + createHtmlParseError, + fetchSearchHtml, + normalizeHtmlText, + resolveHtmlResultUrl, +} from './html-search.js'; + +const YANDEX_SEARCH_URL = 'https://yandex.com/search/'; export const yandexProvider = { id: 'yandex' as const, @@ -10,56 +18,64 @@ export const yandexProvider = { languages: ['ru', 'en', 'auto'], }; -export async function searchYandex(query: string, limit: number = 10, options?: EngineSearchOptions): Promise { +export async function searchYandex( + query: string, + limit: number = 10, + options?: EngineSearchOptions, +): Promise { try { - const url = `https://yandex.com/search/?text=${encodeURIComponent(query)}`; - const res = await fetch(url, { + const url = new URL(YANDEX_SEARCH_URL); + url.searchParams.set('text', query); + const html = await fetchSearchHtml('yandex', url, { + signal: options?.signal, headers: { - 'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36', + 'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) ' + + 'AppleWebKit/537.36 (KHTML, like Gecko) ' + + 'Chrome/120.0.0.0 Safari/537.36', + 'Accept': 'text/html,application/xhtml+xml', 'Accept-Language': 'en-US,en;q=0.9', }, - signal: withTimeout(options?.signal, 10000), }); - if (!res.ok) { - if (options?.throwOnError) throw new Error(`Yandex HTTP ${res.status}`); - logger.warn({ status: res.status }, 'Yandex HTTP error'); - return []; + if (!hasYandexSearchSurface(html)) { + throw createHtmlParseError('yandex'); } - - const html = await res.text(); return parseYandexHTML(html, limit); } catch (error) { options?.signal?.throwIfAborted(); if (options?.throwOnError) throw error; const msg = error instanceof Error ? error.message : String(error); - if (msg.includes('abort') || msg.includes('timeout')) { - logger.warn('Yandex search timed out'); - } else { - logger.warn({ err: msg.slice(0, 200) }, 'Yandex search failed'); - } + logger.warn({ err: msg.slice(0, 200) }, 'Yandex search failed'); return []; } } -function parseYandexHTML(html: string, limit: number): SearchResult[] { +export function parseYandexHTML(html: string, limit: number = 10): SearchResult[] { + const $ = cheerio.load(html); const results: SearchResult[] = []; + const seenUrls = new Set(); - const blockRegex = /]*class="[^"]*serp-item[^"]*"[^>]*>([\s\S]*?)<\/li>/gi; - let match; + $('li.serp-item, .serp-item').each((_, element) => { + if (results.length >= limit) return; - while ((match = blockRegex.exec(html)) !== null && results.length < limit) { - const block = match[1]; - const titleMatch = block.match(/]*>[\s\S]*?]*href="([^"]+)"[^>]*>([\s\S]*?)<\/a>/i); - if (!titleMatch) continue; - - const url = titleMatch[1]; - const title = decodeHTMLTags(titleMatch[2]); - if (!url || !title || url.includes('yandex')) continue; - - const snippetMatch = block.match(/]*class="[^"]*text-container[^"]*"[^>]*>([\s\S]*?)<\/div>/i); - const snippet = snippetMatch ? decodeHTMLTags(snippetMatch[1]) : ''; + const card = $(element); + const titleLink = card + .find('h2 a[href], h3 a[href], a.organic__url[href], a[href]') + .first(); + const url = resolveHtmlResultUrl( + titleLink.attr('href') ?? '', + YANDEX_SEARCH_URL, + ); + const title = normalizeHtmlText(titleLink.text()); + if (!title || !isExternalYandexUrl(url) || seenUrls.has(url)) return; + const snippet = normalizeHtmlText( + card + .find('.text-container, .OrganicTextContentSpan, .TextContainer, p') + .first() + .text(), + ); + seenUrls.add(url); results.push({ title, url, @@ -67,7 +83,22 @@ function parseYandexHTML(html: string, limit: number): SearchResult[] { source: 'yandex', engines: ['yandex'], }); - } + }); return results; } + +function hasYandexSearchSurface(html: string): boolean { + const $ = cheerio.load(html); + return $('.serp-list, li.serp-item, .serp-item, #search-result').length > 0; +} + +function isExternalYandexUrl(url: string): boolean { + if (!url) return false; + try { + const hostname = new URL(url).hostname.toLowerCase(); + return hostname !== 'yandex.com' && !hostname.endsWith('.yandex.com'); + } catch { + return false; + } +} diff --git a/src/infrastructure/doctor.ts b/src/infrastructure/doctor.ts index 1180009..8729f1d 100644 --- a/src/infrastructure/doctor.ts +++ b/src/infrastructure/doctor.ts @@ -48,6 +48,9 @@ export interface DoctorReport { | 'zero-key-search' | 'duckduckgo-proxy' | 'sogou-proxy' + | 'bing-proxy' + | 'baidu-proxy' + | 'yandex-proxy' | 'semantic-flags' | 'request-budget' | 'provider-cooldown-store' @@ -115,6 +118,21 @@ export function createDoctorReport( ...inspectEngineProxyConfiguration('sogou', environment), required: false, }, + { + id: 'bing-proxy', + ...inspectEngineProxyConfiguration('bing', environment), + required: false, + }, + { + id: 'baidu-proxy', + ...inspectEngineProxyConfiguration('baidu', environment), + required: false, + }, + { + id: 'yandex-proxy', + ...inspectEngineProxyConfiguration('yandex', environment), + required: false, + }, { id: 'semantic-flags', ...semanticFlags, diff --git a/src/infrastructure/engine-http.ts b/src/infrastructure/engine-http.ts index 6fed5ec..8cd9ac9 100644 --- a/src/infrastructure/engine-http.ts +++ b/src/infrastructure/engine-http.ts @@ -6,7 +6,12 @@ import { type RequestInit as UndiciRequestInit, } from 'undici'; -export type ProxyAwareEngine = 'duckduckgo' | 'sogou'; +export type ProxyAwareEngine = + | 'duckduckgo' + | 'sogou' + | 'bing' + | 'baidu' + | 'yandex'; export type ProxyConfigurationStatus = 'present' | 'missing' | 'invalid'; export interface EngineProxyInspection { @@ -17,6 +22,9 @@ export interface EngineProxyInspection { const ENGINE_PROXY_ENV: Record = { duckduckgo: 'DUCKDUCKGO_PROXY_URL', sogou: 'SOGOU_PROXY_URL', + bing: 'BING_PROXY_URL', + baidu: 'BAIDU_PROXY_URL', + yandex: 'YANDEX_PROXY_URL', }; const DEFAULT_PROXY_URL = 'http://127.0.0.1:7890'; const proxyAgents = new Map(); diff --git a/tests/e2e/basic-search.e2e.ts b/tests/e2e/basic-search.e2e.ts index 2b60aa2..7821a17 100644 --- a/tests/e2e/basic-search.e2e.ts +++ b/tests/e2e/basic-search.e2e.ts @@ -18,18 +18,38 @@ const EXPECTED_SERVER_VERSION = process.env.E2E_EXPECTED_SERVER_VERSION || '3.2. // with a clear message rather than a suite-level failure. const E2E_SKIP = !existsSync(SERVER_PATH); const LIVE_NETWORK_E2E = process.env.RUN_LIVE_NETWORK_E2E === 'true'; -const liveNetworkIt = LIVE_NETWORK_E2E ? it : it.skip; +type LiveSearchEngine = 'duckduckgo' | 'bing' | 'baidu' | 'yandex'; +const DEFAULT_LIVE_E2E_ENGINES: readonly LiveSearchEngine[] = [ + 'duckduckgo', + 'bing', + 'baidu', + 'yandex', +]; +const LIVE_E2E_ENGINES = new Set( + (process.env.LIVE_E2E_ENGINES ?? DEFAULT_LIVE_E2E_ENGINES.join(',')) + .split(',') + .map(engine => engine.trim()) + .filter((engine): engine is LiveSearchEngine => + DEFAULT_LIVE_E2E_ENGINES.includes(engine as LiveSearchEngine)), +); const LIVE_REQUEST_LIMIT = Number.parseInt( - process.env.LIVE_E2E_MAX_REQUESTS || '2', + process.env.LIVE_E2E_MAX_REQUESTS || '5', 10, ); -const liveExtractIt = LIVE_NETWORK_E2E && LIVE_REQUEST_LIMIT >= 2 ? it : it.skip; +const LIVE_E2E_INCLUDE_EXTRACT = process.env.LIVE_E2E_INCLUDE_EXTRACT !== 'false'; +const liveNetworkIt = LIVE_NETWORK_E2E && LIVE_E2E_ENGINES.has('duckduckgo') ? it : it.skip; +const liveExtractIt = LIVE_NETWORK_E2E + && LIVE_E2E_INCLUDE_EXTRACT + && LIVE_REQUEST_LIMIT >= 2 + ? it + : it.skip; const LIVE_MIN_INTERVAL_MS = Number.parseInt( process.env.LIVE_E2E_MIN_INTERVAL_MS || '10000', 10, ); let liveRequests = 0; let lastLiveRequestAt = 0; +let liveProbeStopReason: string | undefined; async function claimLiveRequest(): Promise { if (liveRequests >= LIVE_REQUEST_LIMIT) { @@ -229,6 +249,34 @@ function waitForStartup(ms: number = 500): Promise { }); } + function skipIfLiveProbesStopped(skip: (reason?: string) => void): boolean { + if (!liveProbeStopReason) return false; + skip(`Live probes stopped after ${liveProbeStopReason}`); + return true; + } + + function assertLiveEngineResults( + response: JsonRpcResponse, + engine: LiveSearchEngine, + ): void { + expect(response).toHaveProperty('result'); + expect(response).not.toHaveProperty('error'); + + const result = response.result as Record; + const structuredContent = result.structuredContent as Record; + const failures = Array.isArray(structuredContent.partialFailures) + ? structuredContent.partialFailures as Array> + : []; + const challenge = failures.find(failure => failure.type === 'bot_challenge'); + if (challenge) { + liveProbeStopReason = `${engine} returned bot_challenge: ${String(challenge.message ?? 'challenge')}`; + throw new Error(`[E2E] Stopping live probes: ${liveProbeStopReason}`); + } + + expect(structuredContent.results).toEqual(expect.any(Array)); + expect((structuredContent.results as unknown[]).length).toBeGreaterThan(0); + } + it('responds to initialize with server info', async () => { proc = spawnServer(); reader = createMessageReader(proc, 15000); @@ -296,8 +344,8 @@ function waitForStartup(ms: number = 500): Promise { liveNetworkIt('calls free_search and returns results', async () => { await claimLiveRequest(); proc = spawnServer(); - // The bounded live smoke selects DDG explicitly and permits one adapter - // attempt. It does not fan out across the free-provider set. + // Each bounded live smoke selects one adapter explicitly and permits one + // adapter attempt. It does not fan out across the free-provider set. reader = createMessageReader(proc, 50000); await waitForStartup(500); @@ -318,6 +366,7 @@ function waitForStartup(ms: number = 500): Promise { }); const response = await reader.readMessage(); + assertLiveEngineResults(response as JsonRpcResponse, 'duckduckgo'); expect(response).toHaveProperty('jsonrpc', '2.0'); expect(response).toHaveProperty('id', 3); expect(response).toHaveProperty('result'); @@ -346,7 +395,86 @@ function waitForStartup(ms: number = 500): Promise { expect((textContent as Record).text).toBeTruthy(); }, 60000); - liveExtractIt('calls free_extract and returns content', async () => { + it.runIf(LIVE_NETWORK_E2E && LIVE_E2E_ENGINES.has('bing'))('calls free_search through Bing and returns results', async ({ skip }) => { + if (skipIfLiveProbesStopped(skip)) return; + await claimLiveRequest(); + proc = spawnServer(); + reader = createMessageReader(proc, 50000); + await waitForStartup(500); + + await initialize(proc, reader); + + sendMessage(proc, { + jsonrpc: '2.0', + id: 5, + method: 'tools/call', + params: { + name: 'free_search', + arguments: { + query: 'OpenAI', + count: 3, + engines: ['bing'], + }, + }, + }); + + assertLiveEngineResults(await reader.readMessage() as JsonRpcResponse, 'bing'); + }, 60000); + + it.runIf(LIVE_NETWORK_E2E && LIVE_E2E_ENGINES.has('baidu'))('calls free_search through Baidu and returns results', async ({ skip }) => { + if (skipIfLiveProbesStopped(skip)) return; + await claimLiveRequest(); + proc = spawnServer(); + reader = createMessageReader(proc, 50000); + await waitForStartup(500); + + await initialize(proc, reader); + + sendMessage(proc, { + jsonrpc: '2.0', + id: 6, + method: 'tools/call', + params: { + name: 'free_search', + arguments: { + query: '人工智能', + count: 3, + engines: ['baidu'], + }, + }, + }); + + assertLiveEngineResults(await reader.readMessage() as JsonRpcResponse, 'baidu'); + }, 60000); + + it.runIf(LIVE_NETWORK_E2E && LIVE_E2E_ENGINES.has('yandex'))('calls free_search through Yandex and returns results', async ({ skip }) => { + if (skipIfLiveProbesStopped(skip)) return; + await claimLiveRequest(); + proc = spawnServer(); + reader = createMessageReader(proc, 50000); + await waitForStartup(500); + + await initialize(proc, reader); + + sendMessage(proc, { + jsonrpc: '2.0', + id: 7, + method: 'tools/call', + params: { + name: 'free_search', + arguments: { + query: 'OpenAI', + count: 3, + engines: ['yandex'], + }, + }, + }); + + assertLiveEngineResults(await reader.readMessage() as JsonRpcResponse, 'yandex'); + }, 60000); + + liveExtractIt('calls free_extract and returns content', async ({ skip }) => { + if (skipIfLiveProbesStopped(skip)) return; await claimLiveRequest(); proc = spawnServer(); reader = createMessageReader(proc, 20000); diff --git a/tests/engines/baidu.test.ts b/tests/engines/baidu.test.ts index 912a9d1..c36efcd 100644 --- a/tests/engines/baidu.test.ts +++ b/tests/engines/baidu.test.ts @@ -53,6 +53,39 @@ describe('Baidu engine', () => { global.fetch = originalFetch; } }); + + it('reports an unexpected 200 HTML shape instead of a successful empty result', async () => { + const originalFetch = global.fetch; + try { + global.fetch = async () => new Response( + '百度一下changed', + { status: 200, headers: { 'Content-Type': 'text/html' } }, + ); + + await expect(searchBaidu('test query', 5, { throwOnError: true })) + .rejects.toMatchObject({ + failureType: 'parse_error', + retryable: false, + }); + } finally { + global.fetch = originalFetch; + } + }); + + it('keeps a valid zero-result search page as an empty success', async () => { + const originalFetch = global.fetch; + try { + global.fetch = async () => new Response( + '百度一下
    ', + { status: 200, headers: { 'Content-Type': 'text/html' } }, + ); + + await expect(searchBaidu('没有匹配结果', 5, { throwOnError: true })) + .resolves.toEqual([]); + } finally { + global.fetch = originalFetch; + } + }); }); describe('parseBaiduHTML snippet extraction', () => { @@ -145,6 +178,25 @@ describe('parseBaiduHTML snippet extraction', () => { expect(results[0].snippet).toBe('Real snippet text here.'); }); + it('uses Baidu card landing URLs when result links are redirect URLs', () => { + const html = ` +
    +

    Landing Result

    +
    Landing page snippet.
    +
    +
    +

    Entity Result

    +
    Entity page snippet.
    +
    +`; + + const results = parseBaiduHTML(html, 10); + expect(results.map(result => result.url)).toEqual([ + 'https://example.com/landing', + 'https://example.com/entity', + ]); + }); + it('respects the limit parameter', () => { const makeBlock = (i: number) => `

    Result ${i}

    diff --git a/tests/engines/bing.test.ts b/tests/engines/bing.test.ts index cdb6b8c..5757d8e 100644 --- a/tests/engines/bing.test.ts +++ b/tests/engines/bing.test.ts @@ -1,5 +1,9 @@ import { describe, it, expect } from 'vitest'; -import { searchBing, bingProvider } from '../../src/engines/bing.js'; +import { + bingProvider, + parseBingHTML, + searchBing, +} from '../../src/engines/bing.js'; describe('Bing engine', () => { it('has correct provider metadata', () => { @@ -54,4 +58,77 @@ describe('Bing engine', () => { global.fetch = originalFetch; } }); + + it('parses result cards through the public HTML parser', () => { + const results = parseBingHTML(` + Bing +
      +
    1. +

      Bing result

      +

      A & useful snippet.

      +
      2. +
    + + `, 5); + + expect(results).toEqual([{ + title: 'Bing result', + url: 'https://example.com/bing', + snippet: 'A & useful snippet.', + source: 'bing', + engines: ['bing'], + }]); + }); + + it('reports an unexpected 200 HTML shape instead of a successful empty result', async () => { + const originalFetch = global.fetch; + try { + global.fetch = async () => new Response( + 'Bingchanged', + { status: 200, headers: { 'Content-Type': 'text/html' } }, + ); + + await expect(searchBing('test query', 5, { throwOnError: true })) + .rejects.toMatchObject({ + failureType: 'parse_error', + retryable: false, + }); + } finally { + global.fetch = originalFetch; + } + }); + + it('keeps a valid zero-result search page as an empty success', async () => { + const originalFetch = global.fetch; + try { + global.fetch = async () => new Response( + 'Bing
      ', + { status: 200, headers: { 'Content-Type': 'text/html' } }, + ); + + await expect(searchBing('no matching result', 5, { throwOnError: true })) + .resolves.toEqual([]); + } finally { + global.fetch = originalFetch; + } + }); + + it('classifies a 200 anti-bot interstitial as a bounded challenge', async () => { + const originalFetch = global.fetch; + try { + global.fetch = async () => new Response( + 'Bing - Verify you are humancaptcha', + { status: 200, headers: { 'Content-Type': 'text/html' } }, + ); + + await expect(searchBing('test query', 5, { throwOnError: true })) + .rejects.toMatchObject({ + failureType: 'bot_challenge', + retryable: false, + cooldownMs: 3_600_000, + }); + } finally { + global.fetch = originalFetch; + } + }); }); diff --git a/tests/engines/yandex.test.ts b/tests/engines/yandex.test.ts index 374d113..ad854f6 100644 --- a/tests/engines/yandex.test.ts +++ b/tests/engines/yandex.test.ts @@ -1,5 +1,9 @@ import { describe, it, expect } from 'vitest'; -import { searchYandex, yandexProvider } from '../../src/engines/yandex.js'; +import { + parseYandexHTML, + searchYandex, + yandexProvider, +} from '../../src/engines/yandex.js'; describe('Yandex engine', () => { it('has correct provider metadata', () => { @@ -54,4 +58,77 @@ describe('Yandex engine', () => { global.fetch = originalFetch; } }); -}); \ No newline at end of file + + it('parses result cards through the public HTML parser', () => { + const results = parseYandexHTML(` + Yandex Search + + + `, 5); + + expect(results).toEqual([{ + title: 'Yandex result', + url: 'https://example.com/yandex', + snippet: 'A useful Yandex snippet.', + source: 'yandex', + engines: ['yandex'], + }]); + }); + + it('classifies a verification redirect URL as a bounded challenge', async () => { + const originalFetch = global.fetch; + try { + global.fetch = async () => ({ + ok: true, + url: 'https://yandex.com/showcaptchafast?retpath=search', + text: async () => 'Verification', + }) as Response; + + await expect(searchYandex('OpenAI', 5, { throwOnError: true })) + .rejects.toMatchObject({ + failureType: 'bot_challenge', + cooldownMs: 3_600_000, + }); + } finally { + global.fetch = originalFetch; + } + }); + + it('reports an unexpected 200 HTML shape instead of a successful empty result', async () => { + const originalFetch = global.fetch; + try { + global.fetch = async () => new Response( + 'Yandex Searchchanged', + { status: 200, headers: { 'Content-Type': 'text/html' } }, + ); + + await expect(searchYandex('test query', 5, { throwOnError: true })) + .rejects.toMatchObject({ + failureType: 'parse_error', + retryable: false, + }); + } finally { + global.fetch = originalFetch; + } + }); + + it('keeps a valid zero-result search page as an empty success', async () => { + const originalFetch = global.fetch; + try { + global.fetch = async () => new Response( + 'Yandex Search
        ', + { status: 200, headers: { 'Content-Type': 'text/html' } }, + ); + + await expect(searchYandex('no matching result', 5, { throwOnError: true })) + .resolves.toEqual([]); + } finally { + global.fetch = originalFetch; + } + }); +}); diff --git a/tests/infrastructure/doctor.test.ts b/tests/infrastructure/doctor.test.ts index 0720712..817ed43 100644 --- a/tests/infrastructure/doctor.test.ts +++ b/tests/infrastructure/doctor.test.ts @@ -64,6 +64,7 @@ describe('search configuration doctor', () => { TAVILY_API_KEY: ' ', DUCKDUCKGO_PROXY_URL: 'http://proxy-user:proxy-secret@proxy.example:8080', + BAIDU_PROXY_URL: 'http://baidu-proxy.example:8080', }; const report = createDoctorReport({ environment, @@ -88,6 +89,11 @@ describe('search configuration doctor', () => { status: 'present', provenance: ['environment:DUCKDUCKGO_PROXY_URL'], }); + expect(report.configuration.find(check => check.id === 'baidu-proxy')) + .toMatchObject({ + status: 'present', + provenance: ['environment:BAIDU_PROXY_URL'], + }); for (const secret of [ 'brave-secret-value', 'proxy-user', @@ -144,6 +150,21 @@ describe('search configuration doctor', () => { status: 'invalid', provenance: ['environment:USE_PROXY'], }), + expect.objectContaining({ + id: 'bing-proxy', + status: 'invalid', + provenance: ['environment:USE_PROXY'], + }), + expect.objectContaining({ + id: 'baidu-proxy', + status: 'invalid', + provenance: ['environment:USE_PROXY'], + }), + expect.objectContaining({ + id: 'yandex-proxy', + status: 'invalid', + provenance: ['environment:USE_PROXY'], + }), ]); expect(JSON.stringify(report)).not.toContain('ambient.example'); }); diff --git a/tests/infrastructure/engine-http.test.ts b/tests/infrastructure/engine-http.test.ts index 7ff06cf..b518bf3 100644 --- a/tests/infrastructure/engine-http.test.ts +++ b/tests/infrastructure/engine-http.test.ts @@ -66,6 +66,27 @@ describe('engine HTTP transport', () => { ); }); + it('routes Baidu through its explicit engine proxy without using ambient proxies', async () => { + let observedConnectTarget = ''; + const proxy = await listen(createTunnelProxy((request, socket) => { + observedConnectTarget = request.url ?? ''; + respondThroughTunnel(socket, 200, 'baidu-proxied'); + })); + servers.push(proxy.server); + vi.stubEnv('BAIDU_PROXY_URL', `http://127.0.0.1:${proxy.port}`); + vi.stubEnv('HTTPS_PROXY', 'http://ambient-proxy.invalid:8080'); + global.fetch = vi.fn(async () => new Response('ambient')) as typeof fetch; + + const response = await fetchForEngine( + 'baidu', + 'http://www.baidu.com/s?wd=test', + ); + + expect(await response.text()).toBe('baidu-proxied'); + expect(observedConnectTarget).toBe('www.baidu.com:80'); + expect(global.fetch).toHaveBeenCalledTimes(0); + }); + it('uses the existing USE_PROXY and PROXY_URL contract', async () => { const proxy = await listen(createTunnelProxy((_request, socket) => { respondThroughTunnel(socket, 204); From 867f779d51a2561a4b5c07e938fb9769e574730a Mon Sep 17 00:00:00 2001 From: lennney Date: Tue, 28 Jul 2026 16:21:19 +0800 Subject: [PATCH 2/5] fix: harden html adapters with live e2e coverage --- scripts/run-live-e2e.mjs | 14 ++++ src/engines/baidu.ts | 18 ++++- src/engines/bing.ts | 16 +++- src/engines/html-search.ts | 7 +- src/engines/yandex.ts | 15 +++- tests/e2e/basic-search.e2e.ts | 69 ++++++++++------- tests/e2e/live-probe-budget.test.ts | 22 ++++++ tests/e2e/live-probe-budget.ts | 27 +++++++ tests/e2e/live-probe-outcome.test.ts | 95 ++++++++++++++++++++++++ tests/e2e/live-probe-outcome.ts | 64 ++++++++++++++++ tests/e2e/live-runner-validation.test.ts | 37 +++++++++ tests/engines/baidu.test.ts | 46 ++++++++++++ tests/engines/bing.test.ts | 46 ++++++++++++ tests/engines/html-search.test.ts | 94 +++++++++++++++++++++++ tests/engines/yandex.test.ts | 46 ++++++++++++ tests/tools/free-search.test.ts | 56 ++++++++++++++ 16 files changed, 631 insertions(+), 41 deletions(-) create mode 100644 tests/e2e/live-probe-budget.test.ts create mode 100644 tests/e2e/live-probe-budget.ts create mode 100644 tests/e2e/live-probe-outcome.test.ts create mode 100644 tests/e2e/live-probe-outcome.ts create mode 100644 tests/e2e/live-runner-validation.test.ts create mode 100644 tests/engines/html-search.test.ts diff --git a/scripts/run-live-e2e.mjs b/scripts/run-live-e2e.mjs index 040b801..5f47d0e 100644 --- a/scripts/run-live-e2e.mjs +++ b/scripts/run-live-e2e.mjs @@ -15,6 +15,9 @@ if (process.env.LIVE_E2E !== 'true') { process.exit(2); } +const LIVE_ENGINE_NAMES = ['duckduckgo', 'bing', 'baidu', 'yandex']; +validateEngineSelection(process.env.LIVE_E2E_ENGINES); + const maxRequests = boundedInteger('LIVE_E2E_MAX_REQUESTS', 5, 1, 5); const minIntervalMs = boundedInteger( 'LIVE_E2E_MIN_INTERVAL_MS', @@ -46,6 +49,17 @@ function boundedInteger(name, fallback, minimum, maximum) { return parsed; } +function validateEngineSelection(rawValue) { + if (rawValue === undefined) return; + const engines = rawValue.split(',').map(engine => engine.trim()); + if (engines.length === 0 || engines.some(engine => !LIVE_ENGINE_NAMES.includes(engine))) { + console.error( + `LIVE_E2E_ENGINES must contain one or more supported engines: ${LIVE_ENGINE_NAMES.join(', ')}`, + ); + process.exit(2); + } +} + function run(command, args, environment = {}) { return new Promise((resolve, reject) => { const child = spawn(command, args, { diff --git a/src/engines/baidu.ts b/src/engines/baidu.ts index 3169acc..852c911 100644 --- a/src/engines/baidu.ts +++ b/src/engines/baidu.ts @@ -39,10 +39,11 @@ export async function searchBaidu( }, }); - if (!hasBaiduSearchSurface(html)) { + const results = parseBaiduHTML(html, limit); + if (!hasBaiduSearchSurface(html) || (hasBaiduResultCards(html) && results.length === 0)) { throw createHtmlParseError('baidu'); } - return parseBaiduHTML(html, limit); + return results; } catch (error) { options?.signal?.throwIfAborted(); if (options?.throwOnError) throw error; @@ -106,8 +107,17 @@ function extractBaiduSnippet(card: cheerio.Cheerio): string { function hasBaiduSearchSurface(html: string): boolean { const $ = cheerio.load(html); - return $('#content_left').length > 0 - || $('.c-container, .result').length > 0; + const resultContainer = $('#content_left').first(); + return hasBaiduResultCards(html) + || (resultContainer.length > 0 + && resultContainer.children().length === 0 + && normalizeHtmlText(resultContainer.text()) === ''); +} + +function hasBaiduResultCards(html: string): boolean { + return cheerio.load(html)( + '#content_left .c-container, #content_left .result, .c-container, .result', + ).length > 0; } function isExternalBaiduUrl(url: string): boolean { diff --git a/src/engines/bing.ts b/src/engines/bing.ts index df9cca5..1ce77d9 100644 --- a/src/engines/bing.ts +++ b/src/engines/bing.ts @@ -38,10 +38,11 @@ export async function searchBing( }, }); - if (!hasBingSearchSurface(html)) { + const results = parseBingHTML(html, limit); + if (!hasBingSearchSurface(html) || (hasBingResultCards(html) && results.length === 0)) { throw createHtmlParseError('bing'); } - return parseBingHTML(html, limit); + return results; } catch (error) { options?.signal?.throwIfAborted(); if (options?.throwOnError) throw error; @@ -85,8 +86,15 @@ export function parseBingHTML(html: string, limit: number = 10): SearchResult[] } function hasBingSearchSurface(html: string): boolean { const $ = cheerio.load(html); - return $('#b_results').length > 0 - || $('li.b_algo').length > 0; + const resultContainer = $('#b_results').first(); + return $('li.b_algo').length > 0 + || (resultContainer.length > 0 + && resultContainer.children().length === 0 + && normalizeHtmlText(resultContainer.text()) === ''); +} + +function hasBingResultCards(html: string): boolean { + return cheerio.load(html)('li.b_algo').length > 0; } function isExternalBingUrl(url: string): boolean { diff --git a/src/engines/html-search.ts b/src/engines/html-search.ts index 7ef41be..7ef83c5 100644 --- a/src/engines/html-search.ts +++ b/src/engines/html-search.ts @@ -35,12 +35,12 @@ export async function fetchSearchHtml( }); const html = await response.text(); - if (!response.ok) { - throw createHttpError(engine, response, html); - } if (looksLikeBotChallenge(html) || looksLikeBotChallengeUrl(response.url)) { throw createBotChallenge(engine); } + if (!response.ok) { + throw createHttpError(engine, response, html); + } return html; } @@ -101,6 +101,7 @@ function looksLikeBotChallenge(html: string): boolean { if (titleMarkers.some(marker => titleAndMeta.includes(marker))) return true; return [ + 'captcha', 'verify you are human', 'unusual traffic', 'robot check', diff --git a/src/engines/yandex.ts b/src/engines/yandex.ts index c6b1a26..9e171d2 100644 --- a/src/engines/yandex.ts +++ b/src/engines/yandex.ts @@ -37,10 +37,11 @@ export async function searchYandex( }, }); - if (!hasYandexSearchSurface(html)) { + const results = parseYandexHTML(html, limit); + if (!hasYandexSearchSurface(html) || (hasYandexResultCards(html) && results.length === 0)) { throw createHtmlParseError('yandex'); } - return parseYandexHTML(html, limit); + return results; } catch (error) { options?.signal?.throwIfAborted(); if (options?.throwOnError) throw error; @@ -90,7 +91,15 @@ export function parseYandexHTML(html: string, limit: number = 10): SearchResult[ function hasYandexSearchSurface(html: string): boolean { const $ = cheerio.load(html); - return $('.serp-list, li.serp-item, .serp-item, #search-result').length > 0; + const resultContainer = $('.serp-list, #search-result').first(); + return hasYandexResultCards(html) + || (resultContainer.length > 0 + && resultContainer.children().length === 0 + && normalizeHtmlText(resultContainer.text()) === ''); +} + +function hasYandexResultCards(html: string): boolean { + return cheerio.load(html)('li.serp-item, .serp-item').length > 0; } function isExternalYandexUrl(url: string): boolean { diff --git a/tests/e2e/basic-search.e2e.ts b/tests/e2e/basic-search.e2e.ts index 7821a17..d36b102 100644 --- a/tests/e2e/basic-search.e2e.ts +++ b/tests/e2e/basic-search.e2e.ts @@ -1,9 +1,14 @@ -import { describe, it, expect, afterEach, beforeAll } from 'vitest'; +import { describe, it, expect, afterEach, beforeAll, afterAll } from 'vitest'; import { spawn, ChildProcess } from 'child_process'; import { existsSync } from 'fs'; import { resolve, dirname } from 'path'; import { fileURLToPath } from 'url'; import { createInterface, Interface } from 'readline'; +import { createLiveProbeBudget } from './live-probe-budget.js'; +import { + hasSuccessfulProviderProbe, + shouldStopAfterLiveOutcome, +} from './live-probe-outcome.js'; const __dirname = dirname(fileURLToPath(import.meta.url)); const SERVER_PATH = process.env.E2E_SERVER_PATH @@ -40,20 +45,23 @@ const LIVE_E2E_INCLUDE_EXTRACT = process.env.LIVE_E2E_INCLUDE_EXTRACT !== 'false const liveNetworkIt = LIVE_NETWORK_E2E && LIVE_E2E_ENGINES.has('duckduckgo') ? it : it.skip; const liveExtractIt = LIVE_NETWORK_E2E && LIVE_E2E_INCLUDE_EXTRACT - && LIVE_REQUEST_LIMIT >= 2 + && LIVE_REQUEST_LIMIT > LIVE_E2E_ENGINES.size ? it : it.skip; const LIVE_MIN_INTERVAL_MS = Number.parseInt( process.env.LIVE_E2E_MIN_INTERVAL_MS || '10000', 10, ); -let liveRequests = 0; let lastLiveRequestAt = 0; let liveProbeStopReason: string | undefined; - -async function claimLiveRequest(): Promise { - if (liveRequests >= LIVE_REQUEST_LIMIT) { - throw new Error(`Live E2E request limit exceeded: ${LIVE_REQUEST_LIMIT}`); +let liveProviderSuccesses = 0; +const liveProbeBudget = createLiveProbeBudget(LIVE_REQUEST_LIMIT); + +async function claimLiveRequest(skip?: (reason?: string) => void): Promise { + if (!liveProbeBudget.claim()) { + const reason = `Live probe budget exhausted at ${LIVE_REQUEST_LIMIT} requests`; + if (skip) skip(reason); + return false; } const waitMs = Math.max( LIVE_MIN_INTERVAL_MS - (Date.now() - lastLiveRequestAt), @@ -62,8 +70,8 @@ async function claimLiveRequest(): Promise { if (lastLiveRequestAt > 0 && waitMs > 0) { await new Promise(resolve => setTimeout(resolve, waitMs)); } - liveRequests += 1; lastLiveRequestAt = Date.now(); + return true; } beforeAll(() => { @@ -224,6 +232,12 @@ function waitForStartup(ms: number = 500): Promise { } catch { /* ignore */ } }); + afterAll(() => { + if (LIVE_NETWORK_E2E && !E2E_SKIP && liveProviderSuccesses === 0) { + throw new Error('No successful live provider probe'); + } + }); + // ─── Helper: perform MCP handshake ─── async function initialize(p: ChildProcess, r: ReturnType) { // Send initialize @@ -258,7 +272,8 @@ function waitForStartup(ms: number = 500): Promise { function assertLiveEngineResults( response: JsonRpcResponse, engine: LiveSearchEngine, - ): void { + skip: (reason?: string) => void, + ): boolean { expect(response).toHaveProperty('result'); expect(response).not.toHaveProperty('error'); @@ -267,14 +282,18 @@ function waitForStartup(ms: number = 500): Promise { const failures = Array.isArray(structuredContent.partialFailures) ? structuredContent.partialFailures as Array> : []; - const challenge = failures.find(failure => failure.type === 'bot_challenge'); - if (challenge) { - liveProbeStopReason = `${engine} returned bot_challenge: ${String(challenge.message ?? 'challenge')}`; - throw new Error(`[E2E] Stopping live probes: ${liveProbeStopReason}`); + const providerOutcome = { ...structuredContent, engine, kind: 'provider_probe' }; + if (shouldStopAfterLiveOutcome(providerOutcome)) { + const challenge = failures.find(failure => failure.type === 'bot_challenge'); + liveProbeStopReason = `${engine} returned bot_challenge: ${String(challenge?.message ?? 'challenge')}`; + skip(`Live probes stopped after ${liveProbeStopReason}`); + return false; } expect(structuredContent.results).toEqual(expect.any(Array)); expect((structuredContent.results as unknown[]).length).toBeGreaterThan(0); + if (hasSuccessfulProviderProbe(providerOutcome)) liveProviderSuccesses += 1; + return true; } it('responds to initialize with server info', async () => { @@ -341,8 +360,8 @@ function waitForStartup(ms: number = 500): Promise { })); }, 20000); - liveNetworkIt('calls free_search and returns results', async () => { - await claimLiveRequest(); + liveNetworkIt('calls free_search and returns results', async ({ skip }) => { + if (skipIfLiveProbesStopped(skip) || !(await claimLiveRequest(skip))) return; proc = spawnServer(); // Each bounded live smoke selects one adapter explicitly and permits one // adapter attempt. It does not fan out across the free-provider set. @@ -366,7 +385,7 @@ function waitForStartup(ms: number = 500): Promise { }); const response = await reader.readMessage(); - assertLiveEngineResults(response as JsonRpcResponse, 'duckduckgo'); + if (!assertLiveEngineResults(response as JsonRpcResponse, 'duckduckgo', skip)) return; expect(response).toHaveProperty('jsonrpc', '2.0'); expect(response).toHaveProperty('id', 3); expect(response).toHaveProperty('result'); @@ -396,8 +415,7 @@ function waitForStartup(ms: number = 500): Promise { }, 60000); it.runIf(LIVE_NETWORK_E2E && LIVE_E2E_ENGINES.has('bing'))('calls free_search through Bing and returns results', async ({ skip }) => { - if (skipIfLiveProbesStopped(skip)) return; - await claimLiveRequest(); + if (skipIfLiveProbesStopped(skip) || !(await claimLiveRequest(skip))) return; proc = spawnServer(); reader = createMessageReader(proc, 50000); await waitForStartup(500); @@ -418,12 +436,11 @@ function waitForStartup(ms: number = 500): Promise { }, }); - assertLiveEngineResults(await reader.readMessage() as JsonRpcResponse, 'bing'); + if (!assertLiveEngineResults(await reader.readMessage() as JsonRpcResponse, 'bing', skip)) return; }, 60000); it.runIf(LIVE_NETWORK_E2E && LIVE_E2E_ENGINES.has('baidu'))('calls free_search through Baidu and returns results', async ({ skip }) => { - if (skipIfLiveProbesStopped(skip)) return; - await claimLiveRequest(); + if (skipIfLiveProbesStopped(skip) || !(await claimLiveRequest(skip))) return; proc = spawnServer(); reader = createMessageReader(proc, 50000); await waitForStartup(500); @@ -444,12 +461,11 @@ function waitForStartup(ms: number = 500): Promise { }, }); - assertLiveEngineResults(await reader.readMessage() as JsonRpcResponse, 'baidu'); + if (!assertLiveEngineResults(await reader.readMessage() as JsonRpcResponse, 'baidu', skip)) return; }, 60000); it.runIf(LIVE_NETWORK_E2E && LIVE_E2E_ENGINES.has('yandex'))('calls free_search through Yandex and returns results', async ({ skip }) => { - if (skipIfLiveProbesStopped(skip)) return; - await claimLiveRequest(); + if (skipIfLiveProbesStopped(skip) || !(await claimLiveRequest(skip))) return; proc = spawnServer(); reader = createMessageReader(proc, 50000); await waitForStartup(500); @@ -470,12 +486,11 @@ function waitForStartup(ms: number = 500): Promise { }, }); - assertLiveEngineResults(await reader.readMessage() as JsonRpcResponse, 'yandex'); + if (!assertLiveEngineResults(await reader.readMessage() as JsonRpcResponse, 'yandex', skip)) return; }, 60000); liveExtractIt('calls free_extract and returns content', async ({ skip }) => { - if (skipIfLiveProbesStopped(skip)) return; - await claimLiveRequest(); + if (skipIfLiveProbesStopped(skip) || !(await claimLiveRequest(skip))) return; proc = spawnServer(); reader = createMessageReader(proc, 20000); await waitForStartup(500); diff --git a/tests/e2e/live-probe-budget.test.ts b/tests/e2e/live-probe-budget.test.ts new file mode 100644 index 0000000..05ee3f7 --- /dev/null +++ b/tests/e2e/live-probe-budget.test.ts @@ -0,0 +1,22 @@ +import { describe, expect, it } from 'vitest'; + +import { createLiveProbeBudget } from './live-probe-budget.js'; + +describe('createLiveProbeBudget', () => { + it('allows claims up to the limit without throwing after exhaustion', () => { + const budget = createLiveProbeBudget(2); + + expect(budget.claim()).toBe(true); + expect(budget.claim()).toBe(true); + expect(budget.claim()).toBe(false); + expect(budget.used).toBe(2); + expect(budget.exhausted).toBe(true); + }); + + it.each([0, -1, 1.5, Number.NaN, Number.POSITIVE_INFINITY])( + 'rejects invalid limit %s with RangeError', + (limit) => { + expect(() => createLiveProbeBudget(limit)).toThrow(RangeError); + }, + ); +}); diff --git a/tests/e2e/live-probe-budget.ts b/tests/e2e/live-probe-budget.ts new file mode 100644 index 0000000..0691651 --- /dev/null +++ b/tests/e2e/live-probe-budget.ts @@ -0,0 +1,27 @@ +export interface LiveProbeBudget { + claim(): boolean; + readonly used: number; + readonly exhausted: boolean; +} + +/** Track bounded live requests without turning an expected skip into a failure. */ +export function createLiveProbeBudget(limit: number): LiveProbeBudget { + if (!Number.isInteger(limit) || limit <= 0) { + throw new RangeError('Live probe limit must be a positive integer'); + } + + let requestCount = 0; + return { + claim(): boolean { + if (requestCount >= limit) return false; + requestCount += 1; + return true; + }, + get used(): number { + return requestCount; + }, + get exhausted(): boolean { + return requestCount >= limit; + }, + }; +} diff --git a/tests/e2e/live-probe-outcome.test.ts b/tests/e2e/live-probe-outcome.test.ts new file mode 100644 index 0000000..a2c4bfe --- /dev/null +++ b/tests/e2e/live-probe-outcome.test.ts @@ -0,0 +1,95 @@ +import { describe, expect, it } from 'vitest'; + +import { + hasSuccessfulProviderProbe, + isBotChallengeFailure, + shouldStopAfterLiveOutcome, +} from './live-probe-outcome.js'; + +describe('isBotChallengeFailure', () => { + it('recognizes a direct partial failure', () => { + expect(isBotChallengeFailure({ + engine: 'bing', + type: 'bot_challenge', + message: 'verification required', + })).toBe(true); + }); + + it('recognizes a challenge nested in partialFailures', () => { + expect(isBotChallengeFailure({ + partialFailures: [ + { engine: 'bing', type: 'parse_error' }, + { engine: 'baidu', type: 'bot_challenge' }, + ], + })).toBe(true); + }); + + it('recognizes a challenge in a nested live outcome', () => { + expect(isBotChallengeFailure({ + result: { + partialFailures: [{ engine: 'yandex', type: 'bot_challenge' }], + }, + })).toBe(true); + }); + + it('does not classify ordinary parse errors or non-objects as challenges', () => { + expect(isBotChallengeFailure({ + engine: 'bing', + type: 'parse_error', + })).toBe(false); + expect(isBotChallengeFailure(null)).toBe(false); + expect(isBotChallengeFailure('bot_challenge')).toBe(false); + }); +}); + +describe('shouldStopAfterLiveOutcome', () => { + it('stops after a provider challenge', () => { + expect(shouldStopAfterLiveOutcome({ + partialFailures: [{ engine: 'bing', type: 'bot_challenge' }], + })).toBe(true); + }); + + it('does not stop after a successful result or parse error', () => { + expect(shouldStopAfterLiveOutcome({ + results: [{ title: 'Result', url: 'https://example.com' }], + })).toBe(false); + expect(shouldStopAfterLiveOutcome({ + partialFailures: [{ engine: 'bing', type: 'parse_error' }], + })).toBe(false); + }); +}); + +describe('hasSuccessfulProviderProbe', () => { + it('counts only a successful provider probe with non-empty results', () => { + expect(hasSuccessfulProviderProbe({ + kind: 'provider', + success: true, + results: [{ title: 'Result', url: 'https://example.com' }], + })).toBe(true); + }); + + it('does not count an empty provider result as coverage', () => { + expect(hasSuccessfulProviderProbe({ + kind: 'provider', + success: true, + results: [], + })).toBe(false); + }); + + it('does not count extract results as provider coverage', () => { + expect(hasSuccessfulProviderProbe({ + kind: 'extract', + success: true, + results: [{ title: 'Extracted page' }], + })).toBe(false); + }); + + it('does not count a challenged provider probe as coverage', () => { + expect(hasSuccessfulProviderProbe({ + kind: 'provider', + success: false, + results: [], + partialFailures: [{ engine: 'bing', type: 'bot_challenge' }], + })).toBe(false); + }); +}); diff --git a/tests/e2e/live-probe-outcome.ts b/tests/e2e/live-probe-outcome.ts new file mode 100644 index 0000000..79e4a3f --- /dev/null +++ b/tests/e2e/live-probe-outcome.ts @@ -0,0 +1,64 @@ +type RecordValue = Record; + +function asRecord(value: unknown): RecordValue | undefined { + return value !== null && typeof value === 'object' && !Array.isArray(value) + ? value as RecordValue + : undefined; +} + +function containsBotChallenge(value: unknown, seen: Set): boolean { + if (value === null || typeof value !== 'object') return false; + if (seen.has(value)) return false; + seen.add(value); + + if (Array.isArray(value)) { + return value.some(item => containsBotChallenge(item, seen)); + } + + const record = value as RecordValue; + if (record.type === 'bot_challenge' || record.failureType === 'bot_challenge') { + return true; + } + return ['partialFailures', 'failures', 'error', 'result', 'structuredContent'] + .some(key => containsBotChallenge(record[key], seen)); +} + +export function isBotChallengeFailure(value: unknown): boolean { + return containsBotChallenge(value, new Set()); +} + +export function shouldStopAfterLiveOutcome(value: unknown): boolean { + return isBotChallengeFailure(value); +} + +function unwrapOutcome(value: unknown): RecordValue | undefined { + const root = asRecord(value); + if (!root) return undefined; + return asRecord(root.structuredContent) + ?? asRecord(root.result) + ?? root; +} + +function isExtractOutcome(root: RecordValue): boolean { + return root.kind === 'extract' + || root.operation === 'free_extract' + || root.tool === 'free_extract'; +} + +function isProviderProbe(root: RecordValue): boolean { + return typeof root.engine === 'string' + || typeof root.provider === 'string' + || root.providerProbe === true + || root.kind === 'provider' + || root.kind === 'provider_probe' + || root.operation === 'free_search' + || root.tool === 'free_search'; +} + +export function hasSuccessfulProviderProbe(value: unknown): boolean { + const root = unwrapOutcome(value); + if (!root || isExtractOutcome(root) || isBotChallengeFailure(root)) return false; + if (root.success === false || root.status === 'failed' || root.error) return false; + const results = root.results; + return isProviderProbe(root) && Array.isArray(results) && results.length > 0; +} diff --git a/tests/e2e/live-runner-validation.test.ts b/tests/e2e/live-runner-validation.test.ts new file mode 100644 index 0000000..5a90460 --- /dev/null +++ b/tests/e2e/live-runner-validation.test.ts @@ -0,0 +1,37 @@ +import { describe, expect, it } from 'vitest'; +import { resolve } from 'node:path'; +import { spawnSync } from 'node:child_process'; + +const RUNNER_PATH = resolve( + import.meta.dirname, + '../../scripts/run-live-e2e.mjs', +); + +function runLiveRunner(engines: string) { + const result = spawnSync(process.execPath, [RUNNER_PATH], { + encoding: 'utf8', + env: { + ...process.env, + LIVE_E2E: 'true', + LIVE_E2E_ENGINES: engines, + }, + }); + + return { + code: result.status, + output: `${result.stdout ?? ''}${result.stderr ?? ''}`, + }; +} + +describe('live E2E runner validation', () => { + it.each(['', 'not-an-engine'])( + 'rejects LIVE_E2E_ENGINES=%j before build or network execution', + (engines) => { + const result = runLiveRunner(engines); + + expect(result.code).toBe(2); + expect(result.output).toContain('LIVE_E2E_ENGINES'); + expect(result.output).not.toContain('npm run build'); + }, + ); +}); diff --git a/tests/engines/baidu.test.ts b/tests/engines/baidu.test.ts index c36efcd..226dbb9 100644 --- a/tests/engines/baidu.test.ts +++ b/tests/engines/baidu.test.ts @@ -14,6 +14,7 @@ describe('Baidu engine', () => { try { global.fetch = async () => ({ ok: true, + url: 'https://www.baidu.com/s?wd=test+query', text: async () => 'test', }) as Response; @@ -44,6 +45,7 @@ describe('Baidu engine', () => { global.fetch = async () => ({ ok: false, status: 500, + url: 'https://www.baidu.com/s?wd=test+query', text: async () => 'Server Error', }) as Response; @@ -72,6 +74,50 @@ describe('Baidu engine', () => { } }); + it('reports known outer-container with unknown card drift as parse_error', async () => { + const originalFetch = global.fetch; + try { + global.fetch = async () => new Response( + ` +
        + +
        + `, + { status: 200, headers: { 'Content-Type': 'text/html' } }, + ); + + await expect(searchBaidu('test query', 5, { throwOnError: true })) + .rejects.toMatchObject({ + failureType: 'parse_error', + retryable: false, + }); + } finally { + global.fetch = originalFetch; + } + }); + + it('returns an empty array by default for known outer-container with unknown card drift', async () => { + const originalFetch = global.fetch; + try { + global.fetch = async () => new Response( + ` +
        + +
        + `, + { status: 200, headers: { 'Content-Type': 'text/html' } }, + ); + + await expect(searchBaidu('test query', 5)).resolves.toEqual([]); + } finally { + global.fetch = originalFetch; + } + }); + it('keeps a valid zero-result search page as an empty success', async () => { const originalFetch = global.fetch; try { diff --git a/tests/engines/bing.test.ts b/tests/engines/bing.test.ts index 5757d8e..13078a2 100644 --- a/tests/engines/bing.test.ts +++ b/tests/engines/bing.test.ts @@ -19,6 +19,7 @@ describe('Bing engine', () => { try { global.fetch = async () => ({ ok: true, + url: 'https://www.bing.com/search?q=test+query', text: async () => 'test', }) as Response; @@ -49,6 +50,7 @@ describe('Bing engine', () => { global.fetch = async () => ({ ok: false, status: 500, + url: 'https://www.bing.com/search?q=test+query', text: async () => 'Server Error', }) as Response; @@ -98,6 +100,50 @@ describe('Bing engine', () => { } }); + it('reports known outer-container with unknown card drift as parse_error', async () => { + const originalFetch = global.fetch; + try { + global.fetch = async () => new Response( + ` +
          +
        1. +

          Drifted result

          +
          2. +
        + `, + { status: 200, headers: { 'Content-Type': 'text/html' } }, + ); + + await expect(searchBing('test query', 5, { throwOnError: true })) + .rejects.toMatchObject({ + failureType: 'parse_error', + retryable: false, + }); + } finally { + global.fetch = originalFetch; + } + }); + + it('returns an empty array by default for known outer-container with unknown card drift', async () => { + const originalFetch = global.fetch; + try { + global.fetch = async () => new Response( + ` +
          +
        1. +

          Drifted result

          +
          2. +
        + `, + { status: 200, headers: { 'Content-Type': 'text/html' } }, + ); + + await expect(searchBing('test query', 5)).resolves.toEqual([]); + } finally { + global.fetch = originalFetch; + } + }); + it('keeps a valid zero-result search page as an empty success', async () => { const originalFetch = global.fetch; try { diff --git a/tests/engines/html-search.test.ts b/tests/engines/html-search.test.ts new file mode 100644 index 0000000..a61bc99 --- /dev/null +++ b/tests/engines/html-search.test.ts @@ -0,0 +1,94 @@ +import { afterEach, describe, expect, it, vi } from 'vitest'; + +import { fetchSearchHtml } from '../../src/engines/html-search.js'; + +function mockResponse( + body: string, + status: number, + url: string, + headers: Record = {}, +): Response { + return { + ok: status >= 200 && status < 300, + status, + url, + headers: new Headers(headers), + text: async () => body, + } as Response; +} + +afterEach(() => { + vi.unstubAllGlobals(); + vi.restoreAllMocks(); +}); + +describe('shared HTML search transport', () => { + it.each([ + [200, 'Verify you are human'], + [403, 'Access denied: verify you are human'], + [429, 'Too many requests. Please solve the captcha.'], + ])('classifies a body-only challenge for HTTP %s as bot_challenge', async (status, body) => { + vi.stubGlobal('fetch', vi.fn(async () => mockResponse( + body, + status, + 'https://www.bing.com/search?q=test', + ))); + + await expect(fetchSearchHtml('bing', 'https://www.bing.com/search?q=test')) + .rejects.toMatchObject({ + failureType: 'bot_challenge', + retryable: false, + cooldownMs: 3_600_000, + }); + }); + + it.each([ + 200, + 403, + 429, + ])('classifies a URL-only challenge for HTTP %s as bot_challenge', async (status) => { + vi.stubGlobal('fetch', vi.fn(async () => mockResponse( + 'ordinary provider response', + status, + 'https://www.bing.com/captcha/verify?return=search', + status === 429 ? { 'retry-after': '9' } : {}, + ))); + + await expect(fetchSearchHtml('bing', 'https://www.bing.com/search?q=test')) + .rejects.toMatchObject({ + failureType: 'bot_challenge', + retryable: false, + cooldownMs: 3_600_000, + }); + }); + + it('keeps an ordinary 403 as an upstream HTTP failure', async () => { + vi.stubGlobal('fetch', vi.fn(async () => mockResponse( + 'Forbidden', + 403, + 'https://www.bing.com/search?q=test', + ))); + + await expect(fetchSearchHtml('bing', 'https://www.bing.com/search?q=test')) + .rejects.toMatchObject({ + failureType: 'upstream_4xx', + retryable: false, + }); + }); + + it('keeps an ordinary 429 as a rate-limit failure', async () => { + vi.stubGlobal('fetch', vi.fn(async () => mockResponse( + 'Too many requests', + 429, + 'https://www.bing.com/search?q=test', + { 'retry-after': '7' }, + ))); + + await expect(fetchSearchHtml('bing', 'https://www.bing.com/search?q=test')) + .rejects.toMatchObject({ + failureType: 'rate_limited', + retryable: false, + cooldownMs: 7_000, + }); + }); +}); diff --git a/tests/engines/yandex.test.ts b/tests/engines/yandex.test.ts index ad854f6..a8bc915 100644 --- a/tests/engines/yandex.test.ts +++ b/tests/engines/yandex.test.ts @@ -19,6 +19,7 @@ describe('Yandex engine', () => { try { global.fetch = async () => ({ ok: true, + url: 'https://yandex.com/search/?text=test+query', text: async () => 'test', }) as Response; @@ -49,6 +50,7 @@ describe('Yandex engine', () => { global.fetch = async () => ({ ok: false, status: 500, + url: 'https://yandex.com/search/?text=test+query', text: async () => 'Server Error', }) as Response; @@ -117,6 +119,50 @@ describe('Yandex engine', () => { } }); + it('reports known outer-container with unknown card drift as parse_error', async () => { + const originalFetch = global.fetch; + try { + global.fetch = async () => new Response( + ` + + `, + { status: 200, headers: { 'Content-Type': 'text/html' } }, + ); + + await expect(searchYandex('test query', 5, { throwOnError: true })) + .rejects.toMatchObject({ + failureType: 'parse_error', + retryable: false, + }); + } finally { + global.fetch = originalFetch; + } + }); + + it('returns an empty array by default for known outer-container with unknown card drift', async () => { + const originalFetch = global.fetch; + try { + global.fetch = async () => new Response( + ` + + `, + { status: 200, headers: { 'Content-Type': 'text/html' } }, + ); + + await expect(searchYandex('test query', 5)).resolves.toEqual([]); + } finally { + global.fetch = originalFetch; + } + }); + it('keeps a valid zero-result search page as an empty success', async () => { const originalFetch = global.fetch; try { diff --git a/tests/tools/free-search.test.ts b/tests/tools/free-search.test.ts index f794fe3..43d0fd1 100644 --- a/tests/tools/free-search.test.ts +++ b/tests/tools/free-search.test.ts @@ -452,6 +452,28 @@ describe('searchWithFallback — parallel', () => { ])); }); + it('returns results and reports a Bing parser failure as a partial failure', async () => { + (searchBing as any).mockRejectedValue(new EngineAdapterError( + 'parse_error', + 'Bing result card markup changed', + { + retryable: false, + suggestion: 'Use another engine while the Bing parser is updated', + }, + )); + + const result = await searchWithFallback({ + query: 'bing parser drift', + count: 3, + engines: ['bing', 'sogou'], + }); + + expect(result.results.length).toBeGreaterThan(0); + expect(result.partialFailures).toEqual(expect.arrayContaining([ + expect.objectContaining({ engine: 'bing', type: 'parse_error' }), + ])); + }); + it('preserves a zero-key anti-bot challenge as its own failure type', async () => { (searchSogou as any).mockRejectedValue(new EngineAdapterError( 'bot_challenge', @@ -1025,6 +1047,40 @@ describe('searchWithFallback — waterfall', () => { } }); + it('returns results and reports a Bing parser failure in waterfall mode', async () => { + (searchBing as any).mockRejectedValue(new EngineAdapterError( + 'parse_error', + 'Bing result card markup changed', + { + retryable: false, + suggestion: 'Use another engine while the Bing parser is updated', + }, + )); + (checkConfidenceBasket as any).mockReturnValueOnce({ + sufficient: false, + basketConfidence: 0, + basketRelevance: 0, + relevantResultsCount: 0, + relevanceThreshold: 0.35, + providerFamilyCount: 1, + topResultsCount: 3, + analyzedCount: 3, + }); + + const result = await searchWithFallback({ + query: 'bing parser drift waterfall', + count: 3, + engines: ['sogou', 'bing'], + waterfall: true, + expandQueries: false, + }); + + expect(result.results.length).toBeGreaterThan(0); + expect(result.partialFailures).toEqual(expect.arrayContaining([ + expect.objectContaining({ engine: 'bing', type: 'parse_error' }), + ])); + }); + it('marks a waterfall early stop when a quality gate skips selected later phases', async () => { const result = await searchWithFallback({ query: 'waterfall-skips-later-phases', From d7cc350f2bda9c6d400f21a28efd8d267f744a5a Mon Sep 17 00:00:00 2001 From: lennney Date: Wed, 29 Jul 2026 11:30:11 +0800 Subject: [PATCH 3/5] test: cover HTML challenge detection false positives --- tests/engines/html-search.test.ts | 37 +++++++++++++++++++++++++++++++ 1 file changed, 37 insertions(+) diff --git a/tests/engines/html-search.test.ts b/tests/engines/html-search.test.ts index a61bc99..80f9105 100644 --- a/tests/engines/html-search.test.ts +++ b/tests/engines/html-search.test.ts @@ -23,6 +23,43 @@ afterEach(() => { }); describe('shared HTML search transport', () => { + it.each([ + 'captcha - Search', + 'access denied - Search', + ])('does not classify a query-derived title %j as bot_challenge', async (title) => { + vi.stubGlobal('fetch', vi.fn(async () => mockResponse( + `${title} +
        1. +

          Normal result

          +

          Normal result content.

          +
        + `, + 200, + 'https://www.bing.com/search?q=captcha', + ))); + + await expect(fetchSearchHtml('bing', 'https://www.bing.com/search?q=captcha')) + .resolves.toContain('Normal result'); + }); + + it('ignores challenge markers in scripts, styles, and ordinary result text', async () => { + vi.stubGlobal('fetch', vi.fn(async () => mockResponse( + `OpenAI - Search + + +
        1. +

          Normal result

          +

          This result explains captcha and access denied messages.

          +
        + `, + 200, + 'https://www.bing.com/search?q=OpenAI', + ))); + + await expect(fetchSearchHtml('bing', 'https://www.bing.com/search?q=OpenAI')) + .resolves.toContain('Normal result'); + }); + it.each([ [200, 'Verify you are human'], [403, 'Access denied: verify you are human'], From 304db85f39644769aa0dde364ab259cb86bd8902 Mon Sep 17 00:00:00 2001 From: lennney Date: Wed, 29 Jul 2026 11:34:44 +0800 Subject: [PATCH 4/5] fix: avoid false-positive HTML challenge detection --- src/engines/html-search.ts | 90 ++++++++++++++++++------------- tests/engines/html-search.test.ts | 5 +- 2 files changed, 58 insertions(+), 37 deletions(-) diff --git a/src/engines/html-search.ts b/src/engines/html-search.ts index 7ef83c5..fef258a 100644 --- a/src/engines/html-search.ts +++ b/src/engines/html-search.ts @@ -22,6 +22,43 @@ const DEFAULT_TIMEOUT_MS = 10_000; const CHALLENGE_COOLDOWN_MS = 60 * 60 * 1000; const DEFAULT_RATE_LIMIT_COOLDOWN_MS = 30_000; const MAX_RATE_LIMIT_COOLDOWN_MS = 5 * 60_000; +const TITLE_CHALLENGE_MARKERS = [ + 'verify you are human', + 'please confirm that you are not a robot', + 'are you not a robot', + 'robot check', +]; +const BODY_CHALLENGE_MARKERS = [ + 'verify you are human', + 'access denied: verify', + 'please solve the captcha', + 'please enter the captcha', + 'captcha challenge', + 'checking your browser before redirecting', + 'detected unusual traffic', + 'unusual traffic from', + 'security verification', + '安全验证', + '验证码', + '访问异常', + '请输入验证码', + '反爬', + 'я не робот', + 'проверка безопасности', +]; +const STANDALONE_CHALLENGE_MARKERS = [ + 'captcha', + 'access denied', + 'smartcaptcha', + 'security verification', + '安全验证', + '验证码', + '访问异常', + '请输入验证码', + '反爬', + 'я не робот', + 'проверка безопасности', +]; /** Fetch an HTML search page through the shared, explicitly configured transport. */ export async function fetchSearchHtml( @@ -79,46 +116,27 @@ function looksLikeBotChallenge(html: string): boolean { const metaDescription = normalizeHtmlText( $('meta[name="description"]').first().attr('content') ?? '', ); - const leadingText = normalizeHtmlText($('body').first().text()).slice(0, 800); + const body = $('body').first(); + body.find('script, style, noscript').remove(); + const visibleBodyText = normalizeHtmlText(body.text()).toLowerCase(); + const hasKnownSearchSurface = $('li.b_algo, #b_results, .c-container, .result, li.serp-item, .serp-item, .serp-list, #search-result').length > 0; const titleAndMeta = `${title} ${metaDescription}`.toLowerCase(); - const leadingChallengeText = leadingText.toLowerCase(); - const titleMarkers = [ - 'captcha', - 'verify you are human', - 'unusual traffic', - 'access denied', - 'robot check', - 'smartcaptcha', - 'security verification', - '安全验证', - '验证码', - '访问异常', - '请输入验证码', - '反爬', - 'я не робот', - 'проверка безопасности', - ]; - if (titleMarkers.some(marker => titleAndMeta.includes(marker))) return true; - - return [ - 'captcha', - 'verify you are human', - 'unusual traffic', - 'robot check', - 'smartcaptcha', - 'security verification', - 'checking your browser before redirecting', - '安全验证', - '验证码', - '访问异常', - '请输入验证码', - '反爬', - 'я не робот', - 'проверка безопасности', - ].some(marker => leadingChallengeText.includes(marker)); + if (!hasKnownSearchSurface + && TITLE_CHALLENGE_MARKERS.some(marker => titleAndMeta.includes(marker))) { + return true; + } + if (!hasKnownSearchSurface + && BODY_CHALLENGE_MARKERS.some(marker => visibleBodyText.includes(marker))) { + return true; + } + + return !hasKnownSearchSurface + && visibleBodyText.length <= 160 + && STANDALONE_CHALLENGE_MARKERS.some(marker => visibleBodyText === marker); } function looksLikeBotChallengeUrl(url: string): boolean { + if (!url) return false; const normalized = url.toLowerCase(); return normalized.includes('/showcaptcha') || normalized.includes('/captcha') diff --git a/tests/engines/html-search.test.ts b/tests/engines/html-search.test.ts index 80f9105..844c474 100644 --- a/tests/engines/html-search.test.ts +++ b/tests/engines/html-search.test.ts @@ -26,9 +26,12 @@ describe('shared HTML search transport', () => { it.each([ 'captcha - Search', 'access denied - Search', + 'unusual traffic - Search', ])('does not classify a query-derived title %j as bot_challenge', async (title) => { vi.stubGlobal('fetch', vi.fn(async () => mockResponse( - `${title} + `${title} + +

        1. Normal result

          Normal result content.

          From 6e14530b5efcf14f3f07dac5bc3feb5b6c0e7deb Mon Sep 17 00:00:00 2001 From: lennney Date: Wed, 29 Jul 2026 11:43:51 +0800 Subject: [PATCH 5/5] test: cover challenge marker false positives --- src/engines/html-search.ts | 32 ++++++++++--- tests/engines/html-search.test.ts | 75 ++++++++++++++++++++++++++++++- 2 files changed, 100 insertions(+), 7 deletions(-) diff --git a/src/engines/html-search.ts b/src/engines/html-search.ts index fef258a..e80cf9c 100644 --- a/src/engines/html-search.ts +++ b/src/engines/html-search.ts @@ -23,20 +23,35 @@ const CHALLENGE_COOLDOWN_MS = 60 * 60 * 1000; const DEFAULT_RATE_LIMIT_COOLDOWN_MS = 30_000; const MAX_RATE_LIMIT_COOLDOWN_MS = 5 * 60_000; const TITLE_CHALLENGE_MARKERS = [ + 'captcha', 'verify you are human', + 'unusual traffic', + 'access denied', 'please confirm that you are not a robot', 'are you not a robot', 'robot check', + 'smartcaptcha', + 'security verification', + '安全验证', + '验证码', + '访问异常', + '请输入验证码', + '反爬', + 'я не робот', + 'проверка безопасности', ]; const BODY_CHALLENGE_MARKERS = [ + 'captcha', 'verify you are human', + 'unusual traffic', 'access denied: verify', + 'access denied', 'please solve the captcha', 'please enter the captcha', 'captcha challenge', + 'robot check', + 'smartcaptcha', 'checking your browser before redirecting', - 'detected unusual traffic', - 'unusual traffic from', 'security verification', '安全验证', '验证码', @@ -59,6 +74,11 @@ const STANDALONE_CHALLENGE_MARKERS = [ 'я не робот', 'проверка безопасности', ]; +const SEARCH_SURFACE_SELECTORS: Record = { + bing: '#b_results, li.b_algo', + baidu: '#content_left, .c-container, .result', + yandex: '.serp-list, #search-result, .serp-item', +}; /** Fetch an HTML search page through the shared, explicitly configured transport. */ export async function fetchSearchHtml( @@ -72,7 +92,7 @@ export async function fetchSearchHtml( }); const html = await response.text(); - if (looksLikeBotChallenge(html) || looksLikeBotChallengeUrl(response.url)) { + if (looksLikeBotChallenge(engine, html) || looksLikeBotChallengeUrl(response.url)) { throw createBotChallenge(engine); } if (!response.ok) { @@ -110,7 +130,7 @@ export function resolveHtmlResultUrl(rawUrl: string, baseUrl: string | URL): str } /** Detect common anti-bot interstitials without treating result snippets as challenges. */ -function looksLikeBotChallenge(html: string): boolean { +function looksLikeBotChallenge(engine: HtmlSearchEngine, html: string): boolean { const $ = cheerio.load(html); const title = normalizeHtmlText($('title').first().text()); const metaDescription = normalizeHtmlText( @@ -119,7 +139,7 @@ function looksLikeBotChallenge(html: string): boolean { const body = $('body').first(); body.find('script, style, noscript').remove(); const visibleBodyText = normalizeHtmlText(body.text()).toLowerCase(); - const hasKnownSearchSurface = $('li.b_algo, #b_results, .c-container, .result, li.serp-item, .serp-item, .serp-list, #search-result').length > 0; + const hasKnownSearchSurface = $(SEARCH_SURFACE_SELECTORS[engine]).length > 0; const titleAndMeta = `${title} ${metaDescription}`.toLowerCase(); if (!hasKnownSearchSurface && TITLE_CHALLENGE_MARKERS.some(marker => titleAndMeta.includes(marker))) { @@ -160,7 +180,7 @@ function createHttpError( response: Response, html: string, ): EngineAdapterError { - if (looksLikeBotChallenge(html)) return createBotChallenge(engine); + if (looksLikeBotChallenge(engine, html)) return createBotChallenge(engine); if (response.status === 429) { return new EngineAdapterError( diff --git a/tests/engines/html-search.test.ts b/tests/engines/html-search.test.ts index 844c474..6239b54 100644 --- a/tests/engines/html-search.test.ts +++ b/tests/engines/html-search.test.ts @@ -45,11 +45,84 @@ describe('shared HTML search transport', () => { .resolves.toContain('Normal result'); }); - it('ignores challenge markers in scripts, styles, and ordinary result text', async () => { + it.each([ + 'SmartCaptcha', + 'Security verification', + 'Unusual traffic', + ])('preserves a title-only challenge marker %j', async (title) => { + vi.stubGlobal('fetch', vi.fn(async () => mockResponse( + `${title}`, + 200, + 'https://www.bing.com/search?q=test', + ))); + + await expect(fetchSearchHtml('bing', 'https://www.bing.com/search?q=test')) + .rejects.toMatchObject({ failureType: 'bot_challenge' }); + }); + + it.each([ + 'Robot check', + 'Unusual traffic', + 'Security verification', + ])('preserves a body-only challenge marker %j', async (body) => { + vi.stubGlobal('fetch', vi.fn(async () => mockResponse( + `${body}`, + 200, + 'https://www.bing.com/search?q=test', + ))); + + await expect(fetchSearchHtml('bing', 'https://www.bing.com/search?q=test')) + .rejects.toMatchObject({ failureType: 'bot_challenge' }); + }); + + it.each([ + 'verify you are human', + 'please confirm that you are not a robot', + 'are you not a robot', + 'robot check', + ])('does not classify a title-only query term %j as bot_challenge', async (marker) => { + vi.stubGlobal('fetch', vi.fn(async () => mockResponse( + `Search: ${marker} +
          1. +

            Normal result

            +
          + `, + 200, + 'https://www.bing.com/search?q=robot', + ))); + + await expect(fetchSearchHtml('bing', 'https://www.bing.com/search?q=robot')) + .resolves.toContain('Normal result'); + }); + + it.each([ + 'verify you are human', + 'please confirm that you are not a robot', + 'are you not a robot', + 'robot check', + ])('does not classify a meta-only query term %j as bot_challenge', async (marker) => { + vi.stubGlobal('fetch', vi.fn(async () => mockResponse( + `Search results + + +
          1. +

            Normal result

            +
          + `, + 200, + 'https://www.bing.com/search?q=robot', + ))); + + await expect(fetchSearchHtml('bing', 'https://www.bing.com/search?q=robot')) + .resolves.toContain('Normal result'); + }); + + it('ignores challenge markers in scripts, styles, noscript, and ordinary result text', async () => { vi.stubGlobal('fetch', vi.fn(async () => mockResponse( `OpenAI - Search +

          1. Normal result

            This result explains captcha and access denied messages.