feat: add fastCRW (crw) search + scrape provider#240
Conversation
|
Hi @danny-avila — this is the engine-side companion to the fastCRW PR you Codex-reviewed on LibreChat (danny-avila/LibreChat#13729). Codex correctly pointed out that selecting What it does
Verified locally (since this repo has no PR CI): How it lights up LibreChat (the full chain)
Happy to send a test key (generous credits) so you can try search+scrape once this lands, and glad to adjust anything to fit your conventions. Thanks again for the warm reception on the LibreChat side! 🙌 |
danny-avila
force-pushed
the
feat/add-fastcrw
branch
from
5ba060d to
8b0ae35
Compare
There was a problem hiding this comment.
Pull request overview
Adds fastCRW (crw) as an additional first-class provider in @librechat/agents’ search tooling, wiring it into the existing /v1/search + /v1/scrape tool pipeline alongside Serper/SearXNG/Tavily/Firecrawl. This expands the provider abstraction in src/tools/search/ while keeping the public configuration surface consistent with existing providers.
Changes:
- Extend search/scrape provider types and tool configuration to include
crw(API key/URL + options). - Add a new CRW search client (
createCrwAPI) and CRW scraper (CrwScraper) and wire both intocreateSearchAPI/createSearchTool. - Add Jest coverage for CRW search, scrape normalization, and end-to-end tool wiring (mocked axios).
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| src/tools/search/types.ts | Adds crw to provider unions and introduces CRW search/scrape config and response types. |
| src/tools/search/tool.ts | Wires crw scraper into tool creation and updates provider documentation + capability flags. |
| src/tools/search/search.ts | Adds crw dispatch branch in createSearchAPI and updates invalid-provider error message. |
| src/tools/search/crw-search.ts | Implements CRW /v1/search client and maps results into the library’s SearchResultData shape. |
| src/tools/search/crw-scraper.ts | Implements CRW /v1/scrape scraper with response normalization into the expected nested data shape. |
| src/tools/search/crw.test.ts | Adds unit tests for CRW search + scrape behavior and verifies tool wiring (search → scrape). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| const results: t.SearchResultData = { | ||
| organic: organicResults, | ||
| images: [], | ||
| topStories: [], | ||
| videos: [], | ||
| news: newsResults, | ||
| relatedSearches: [], | ||
| }; |
| * Search providers: Serper (Google results), SearXNG (self-hosted meta-search), Tavily (AI-optimized), fastCRW (Firecrawl-compatible, self-host or cloud). | ||
| * Scraper providers: Firecrawl (default, full-featured), Serper (lightweight), Tavily (batch extraction), fastCRW (Firecrawl-compatible, self-host or cloud). |
| // Self-host fastCRW may run without auth, so a missing key is only a | ||
| // warning — unlike Firecrawl/Tavily, scrapeUrl does NOT early-return on it. | ||
| if (!this.apiKey) { | ||
| this.logger.warn('CRW_API_KEY is not set. Scraping will not work.'); | ||
| } |
|
@codex review |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b3de90cd52
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| const data = body.data ?? {}; | ||
| const organicResults: t.OrganicResult[] = (data.web ?? []) |
There was a problem hiding this comment.
When /v1/search succeeds, CRW returns data as a flat array of result objects, not an object keyed by web, images, and news. With the current normalization, a real successful response makes data.web, data.images, and data.news all undefined, so CRW searches return empty organic/image/news lists even though the API returned hits.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Pushed back: live-tested 2026-07-02 with a real key, and /v1/search returns a source-keyed object, not a flat array. Both origins agree: POST https://api.fastcrw.com/v1/search and POST https://fastcrw.com/api/v1/search each return {"success":true,"data":{"web":[...]}} (and data.news / data.images for those sources). The source-keyed parsing here was written against those live responses and the provider was live-verified end-to-end (search -> scrape -> formatted output). The earlier flat-array shape this PR originally assumed is exactly what live testing disproved.
| const base = ( | ||
| apiUrl ?? | ||
| process.env.CRW_API_URL ?? | ||
| 'https://fastcrw.com/api' | ||
| ).replace(/\/+$/, ''); |
There was a problem hiding this comment.
The hosted CRW API origin is https://api.fastcrw.com, so the current default builds requests like https://fastcrw.com/api/v1/search instead of https://api.fastcrw.com/v1/search. Unless every cloud user overrides crwApiUrl/CRW_API_URL, the new provider sends both search and scraper defaults to the website path rather than the API host.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 4704904: both search and scrape defaults now point at https://api.fastcrw.com. Verified live that both endpoints work on that origin (for the record, https://fastcrw.com/api also serves the API and worked live, but the api subdomain is the canonical origin).
| const getSources = async ({ | ||
| query, | ||
| numResults = 8, | ||
| type, | ||
| }: t.GetSourcesParams): Promise<t.SearchResult> => { |
There was a problem hiding this comment.
When a caller supplies date through the search tool, this CRW branch drops it while building the request. CRW's /v1/search accepts the same qdr:h|d|w|m|y style filter via tbs, so date-constrained CRW searches currently return unrestricted results even though the tool schema exposes date and the Serper/Tavily providers honor it.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 4704904: the search date param is now forwarded as tbs: "qdr:<h|d|w|m|y>". Live-verified the filter is real: a news search for an evergreen query returned 3 results unfiltered and 0 with tbs: "qdr:h". Test added.
| proxy: options.proxy ?? this.proxy, | ||
| stealth: options.stealth ?? this.stealth, | ||
| jsonSchema: options.jsonSchema ?? this.jsonSchema, | ||
| timeout: payloadTimeout, |
There was a problem hiding this comment.
When users raise scraperTimeout or crwScraperOptions.timeout for slow JavaScript pages, this sends the value as timeout, but CRW's scrape request uses deadlineMs for the server-side render deadline. The Axios timeout is extended, yet the CRW server can still apply its default deadline and return before the requested scrape budget is used.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Pushed back: timeout is the honored request field, not deadlineMs. Discriminating live test on /v1/scrape: {timeout: 60000, deadlineMs: 50} succeeded (deadlineMs ignored), while {timeout: 50, deadlineMs: 60000} failed (timeout honored). A control request with a bogus unknown field also succeeded, confirming unknown fields are silently ignored. The related real issue (client HTTP timeout racing the server render budget) was already fixed in b3de90c by padding the axios timeout 5s over the payload budget.
| const data = raw.data ?? raw; | ||
| return { | ||
| success: true, | ||
| data: { |
There was a problem hiding this comment.
When a caller enables CRW structured extraction with formats: ['json'] and jsonSchema, successful scrape responses include the extracted data.json, but this normalization whitelists only text/link/metadata fields and drops it. The subsequent extractContent call then sees no usable content, so JSON-only CRW scrapes are reported as successful while producing an empty source.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Resolved differently in 4704904: /v1/scrape rejects formats: ["json"] outright — live response: formats:"json" is not available on /v1/scrape. Use /v1/extract for structured extraction. So there is no data.json to preserve on this endpoint; instead the misleading jsonSchema scrape option was removed entirely, since it could only ever produce an explicit server error. A /v1/extract integration would be a separate feature.
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4704904bb7
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| xpath: options.xpath ?? this.xpath, | ||
| proxy: options.proxy ?? this.proxy, | ||
| stealth: options.stealth ?? this.stealth, | ||
| timeout: payloadTimeout, |
There was a problem hiding this comment.
When callers raise scraperTimeout or CrwScrapeOptions.timeout for slow/JS-rendered pages, this payload still sends timeout, but CRW's /v1/scrape request uses deadlineMs/deadline_ms for the server-side budget, so the server ignores this field and keeps its default deadline while Axios merely waits longer. That makes configured longer scrape timeouts ineffective for CRW and causes long scrapes to fail early despite the caller opting into a larger timeout.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Pushed back again, now with both casings live-disproven on /v1/scrape: {timeout: 60000, deadlineMs: 50} and {timeout: 60000, deadline_ms: 50} both succeed (the deadline field is ignored — a control with a bogus unknown field also succeeds), while {timeout: 50} and {timeout: 1500, renderJs: true} on a slow page fail with a server-side error envelope, well within the client's HTTP window. So timeout IS the server-side budget field and configured longer timeouts are effective. If you have a spec source for deadlineMs/deadline_ms, please cite it — the live API contradicts it.
| // news: [...]}} (live-verified 2026-07-02). OrganicResult.link is a | ||
| // REQUIRED string (types.ts), so defend against null/empty urls to never | ||
| // feed a broken '' link into the scraper. | ||
| const data = body.data ?? {}; |
There was a problem hiding this comment.
When this is pointed at a current self-hosted CRW /v1/search instance, the success envelope's data is a wrapper with results (the grouped web/news/images payload) rather than the grouped payload itself. Because this line reads body.data directly and then looks for data.web, those searches return success with empty organic/image/news arrays, so searchProvider: 'crw' produces no sources even though CRW returned results.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Cloud is live-verified to return the groups directly under data ({"data":{"web":[...]}} on both api.fastcrw.com and fastcrw.com/api), and I could not verify the self-host data.results wrapper claim (no public spec at standard paths; note the round-1 review claimed data was a flat array, which also contradicts this). That said, a tolerant unwrap is cheap: 51eba56 reads data.results ?? data, with a test for the wrapped shape. Both shapes now work.
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 51eba5676f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| const container = body.data ?? {}; | ||
| const data = container.results ?? container; | ||
| const organicResults: t.OrganicResult[] = (data.web ?? []) |
There was a problem hiding this comment.
The current fastCRW /v1/search OpenAPI docs define SearchResponse.data as an array of result rows, not an object keyed by web/images/news (see https://docs.fastcrw.com/openapi.json). When the documented { success: true, data: [...] } response is returned, data.web is undefined here, so organicResults, imageResults, and newsResults all come back empty and createSearchTool({ searchProvider: 'crw' }) has no sources to scrape despite a successful search. Normalize flat array responses before reading source-specific buckets.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Handled in ad6a266. For the record, the live cloud API does NOT return the documented flat array — both origins return source-keyed groups ({"data":{"web":[...]}}, re-verified today with a real key) — so the docs' SearchResponse schema appears stale (the same spec's SearchRequest matches live behavior: sources/tbs/limit all verified). But since docs.fastcrw.com/openapi.json documents the flat SearchResult[] shape (with a category field), the normalizer now accepts all three shapes: keyed groups (cloud), flat array routed by category, and the results wrapper. Tests cover each.
| xpath: options.xpath ?? this.xpath, | ||
| proxy: options.proxy ?? this.proxy, | ||
| stealth: options.stealth ?? this.stealth, | ||
| timeout: payloadTimeout, |
There was a problem hiding this comment.
For CRW v1, the public OpenAPI schema exposes deadlineMs as the end-to-end scrape deadline and does not define timeout; in runs that pass scraperTimeout or crwScraperOptions.timeout to allow slow JS pages, this payload field is ignored by CRW while the server keeps its default deadline, so configured longer scrape budgets can still fail early. Map this value to deadlineMs (or include both only if a compatibility target requires it).
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Handled in ad6a266: the payload now sends BOTH timeout and deadlineMs (clamped to the documented 1..60000). Note the live cloud demonstrably honors timeout and ignores deadlineMs in both casings ({timeout:60000, deadlineMs:50} succeeds; {timeout:50} fails server-side), which contradicts the docs — so timeout stays, deadlineMs added for spec-conformant self-host builds. Live-verified the cloud accepts the dual-field payload.
| crwScraperOptions?.apiKey ?? crwApiKey ?? process.env.CRW_API_KEY, | ||
| apiUrl: crwScraperOptions?.apiUrl ?? crwApiUrl, | ||
| timeout: scraperTimeout ?? crwScraperOptions?.timeout, | ||
| formats: crwScraperOptions?.formats ?? ['markdown', 'rawHtml'], |
There was a problem hiding this comment.
With the default createSearchTool({ scraperProvider: 'crw' }) path, this asks CRW for markdown and rawHtml, but CrwScraper.extractContent only runs processContent when response.data.html is present. CRW's v1 schema separates html and rawHtml fields, so default CRW scrapes skip the reference-marker pass and highlights lose link/image/video Core References even though HTML was fetched; request html here or process rawHtml too.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Valid — fixed in ad6a266: default scrape formats are now ['markdown','html'] and extractContent's ref-marker pass falls back to rawHtml when html is absent, so Core References survive any format config. Live-verified: a default-config scrape now returns content WITH references (it previously skipped the ref pass).
|
@codex review |
|
Codex Review: Didn't find any major issues. Bravo. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
- scrape: read the nested data container (cloud nests like Firecrawl),
keep top-level fallback for self-host/legacy
- search: parse source-keyed data ({web, images, news}) instead of a
flat array, which threw on every live response
- use native news/images verticals and map their results
- pad the HTTP timeout over the render budget (cold scrapes exceed it)
All shapes live-verified against fastcrw.com.
- forward the search date param as a qdr tbs filter (live-verified to constrain results) - default both endpoints to https://api.fastcrw.com (canonical API origin; live-verified for search and scrape) - drop the jsonSchema scrape option: /v1/scrape rejects formats:json ('use /v1/extract'), so the option could only produce an error
Cloud returns the source groups directly under data (live-verified); unwrap a nested results container defensively for self-host variants.
- accept the documented flat SearchResult[] data shape, routed into source groups by category (cloud's keyed shape still primary) - send deadlineMs (documented, clamped 1..60000) alongside timeout (what the live cloud honors) - default scrape formats to markdown+html and let the ref-marker pass fall back to rawHtml, so default scrapes keep Core References
danny-avila
force-pushed
the
feat/add-fastcrw
branch
from
ad6a266 to
fa0ca9b
Compare
What
Adds fastCRW (
crw) as a first-class provider for both web search (/v1/search) and scraping (/v1/scrape), mirroring the existingtavily-searchandfirecrawlproviders.Why
This is the engine side that lets
@librechat/agentsconsumers (e.g. LibreChat) actually use fastCRW. Today, selecting acrwprovider downstream authenticates but throws here becausecreateSearchToolhas nocrwbranch — this PR adds it.Why fastCRW over Firecrawl?
The integration diff is intentionally small because fastCRW exposes a Firecrawl-compatible API —
crwclosely mirrors the existingfirecrawlprovider with a configurable base URL.Changes (additive only — no existing provider touched)
src/tools/search/crw-search.ts— search provider (mirrorstavily-search.ts).src/tools/search/crw-scraper.ts— scraper (mirrorsfirecrawl.ts).src/tools/search/types.ts—crwadded to the provider enums/types.src/tools/search/tool.ts,search.ts— wiredcrwintocreateSearchTooldispatch for search + scrape.src/tools/search/crw.test.ts— unit tests (mocked HTTP, no network) for both search and scrape paths.Self-host base URL is configurable; key via
CRW_API_KEY. Happy to adjust anything to fit your conventions — I maintain the integration and can provide free credits to evaluate.