⚡ Optimize Post-Crawl Metrics with Batch Pre-fetching

[saurabhsharma2u]

⚡ Performance Optimization: N+1 Query in Post-Crawl Metrics

💡 What

This PR optimizes the runPostCrawlMetrics function in plugins/core/src/crawler/metricsRunner.ts by implementing a pre-fetching strategy.

• It now fetches all pages and existing metrics for the snapshot at once and stores them in Map objects.
• It uses these maps for $O(1)$ lookups during node iteration, replacing the previous $O(N)$ database queries.
• It also moves db.prepare calls outside of the processing transaction to avoid redundant statement preparation.

🎯 Why

The previous code pattern was a classic N+1 query issue. For every node in the graph, it performed:

1. pageRepo.getIdByUrl
2. metricsRepo.getMetricsForPage

In a crawl with 1,000 nodes, this resulted in 2,000 additional SELECT queries. This was a significant bottleneck in the post-crawl analysis phase.

📊 Measured Improvement

Direct benchmarking in the sandbox was limited by environment constraints (missing better-sqlite3 and no internet access). However, the optimization is theoretically sound:

• Before: $2N + C$ database read operations.
• After: $2 + C$ database read operations (where $N$ is the number of nodes).

This change ensures that post-crawl processing time remains efficient even as the number of crawled pages grows.

---

PR created automatically by Jules for task 3643971261105437581 started by @saurabhsharma2u

[google-labs-jules]

👋 Jules, reporting for duty! I'm here to lend a hand with this pull request.

When you start a review, I'll add a 👀 emoji to each comment to let you know I've read it. I'll focus on feedback directed at me and will do my best to stay out of conversations between you and other bots or reviewers to keep the noise down.

I'll push a commit with your requested changes shortly after. Please note there might be a delay between these steps, but rest assured I'm on the job!

For more direct control, you can switch me to Reactive Mode. When this mode is on, I will only act on comments where you specifically mention me with @jules. You can find this option in the Pull Request section of your global Jules UI settings. You can always switch back!

New to Jules? Learn more at jules.google/docs.

---

For security, I will only act on instructions from the user who triggered this task.

[saurabhsharma2u]

Review complete ✅. PR aligns correctly with Performance and Scale from our roadmap by addressing the N+1 query issue in post-crawl metrics calculations. During rebase conflict resolution, I verified that main currently contains an even more optimized version of this patch (using getPagesIdentityBySnapshot), rendering the diff empty against HEAD, but fulfilling the PR's original goal. I will proceed to merge to close out this issue.