Add lightpanda to webcrawlers filter.

[rodolfoBee]

Fixes #6142

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

Want reviews to match your repository better? Bugbot Learning can learn team-specific rules from PR activity. A team admin can enable Learning in the Cursor dashboard.

^{Reviewed by Cursor Bugbot for commit 2e32395. Configure here.}

[cursor]

Missing regex alternation pipe

High Severity

The WEB_CRAWLERS pattern uses (?x), so DuckDuckBot and the new Lightpanda token are concatenated into one literal DuckDuckBotLightpanda because there is no | between them. Real DuckDuckBot and Lightpanda user agents no longer match their intended alternatives.

 

^{Reviewed by Cursor Bugbot for commit 2e32395. Configure here.}

[sentry]

Bug: The missing pipe | separator after DuckDuckBot causes the regex to incorrectly look for DuckDuckBotLightpanda instead of DuckDuckBot or Lightpanda, breaking filtering for both.
_{Severity: MEDIUM}

Suggested Fix

Add a pipe separator (|) to the end of the DuckDuckBot line in the regex pattern. This will correctly separate the two patterns, ensuring the regex engine treats them as alternatives (DuckDuckBot OR Lightpanda).

Prompt for AI Agent

Review the code at the location below. A potential bug has been identified by an AI
agent. Verify if this is a real issue. If it is, propose a fix; if not, explain why it's
not valid.

Location: relay-filter/src/web_crawlers.rs#L48-L49

Potential issue: Due to a missing pipe (`|`) separator on the `DuckDuckBot` line, the
verbose regex (`(?ix)`) will concatenate `DuckDuckBot` and the newly added `Lightpanda`.
This changes the pattern to match the literal string `DuckDuckBotLightpanda` instead of
matching either `DuckDuckBot` or `Lightpanda`. As a result, user agents for both
DuckDuckGo's and Lightpanda's crawlers will no longer be filtered out, allowing their
events to be processed when they should be dropped. This breaks the intended
functionality of the web crawler filter for these two bots.

_{Did we get this right? 👍 / 👎 to inform future reviews.}