Skip to content

⚡ Optimize duplicate detection with IntUnionFind and remove Set overhead#32

Merged
saurabhsharma2u merged 2 commits into
mainfrom
perf-optimize-duplicate-detection-11828779434289117858
Feb 25, 2026
Merged

⚡ Optimize duplicate detection with IntUnionFind and remove Set overhead#32
saurabhsharma2u merged 2 commits into
mainfrom
perf-optimize-duplicate-detection-11828779434289117858

Conversation

@saurabhsharma2u

Copy link
Copy Markdown
Contributor

Optimized findNearDuplicates in plugins/core/src/graph/duplicate.ts to improve performance and scalability.

Changes:

  • Replaced the generic UnionFind<string> class with a specialized, inline integer-based Union-Find using Uint32Array for parent and Uint8Array for rank.
  • Pre-calculated BigInt SimHashes into a BigUint64Array to avoid repeated BigInt() constructor calls and string parsing in the inner loop.
  • Removed the checkedPairs Set<string>, which was causing RangeError: Set maximum size exceeded with large datasets (e.g., 20,000 nodes).
  • Implemented an implicit connectivity check (find(u) === find(v)) to skip redundant Hamming distance calculations for nodes that are already part of the same connected component.

Performance Impact:

  • Stability: Fixed a crash on large inputs (20k nodes) due to Set size limits.
  • Speed: Reduced execution time for 20k nodes from infinite (crash) to ~6.3 seconds in benchmarks.
  • Memory: Significantly reduced memory usage by eliminating millions of string allocations for edge keys.

PR created automatically by Jules for task 11828779434289117858 started by @saurabhsharma2u

@google-labs-jules

Copy link
Copy Markdown

👋 Jules, reporting for duty! I'm here to lend a hand with this pull request.

When you start a review, I'll add a 👀 emoji to each comment to let you know I've read it. I'll focus on feedback directed at me and will do my best to stay out of conversations between you and other bots or reviewers to keep the noise down.

I'll push a commit with your requested changes shortly after. Please note there might be a delay between these steps, but rest assured I'm on the job!

For more direct control, you can switch me to Reactive Mode. When this mode is on, I will only act on comments where you specifically mention me with @jules. You can find this option in the Pull Request section of your global Jules UI settings. You can always switch back!

New to Jules? Learn more at jules.google/docs.


For security, I will only act on instructions from the user who triggered this task.

saurabhsharma2u and others added 2 commits February 25, 2026 11:06
… Set overhead

Replaced the object-based UnionFind and Set<string> tracking in findNearDuplicates
with an integer-based UnionFind (Uint32Array) and implicit connectivity checks.
This resolves a "RangeError: Set maximum size exceeded" on large datasets and
significantly improves performance by reducing memory allocation and redundant
SimHash distance calculations.

- Replaced `UnionFind<T>` class with inline integer array implementation.
- Used `BigUint64Array` for storing SimHashes to avoid BigInt parsing in loops.
- Replaced `checkedPairs` Set with `root1 === root2` check.
- Benchmarked: Crash -> ~6.3s on 20k nodes.
@saurabhsharma2u
saurabhsharma2u force-pushed the perf-optimize-duplicate-detection-11828779434289117858 branch from a88a483 to 9270241 Compare February 25, 2026 19:08
@saurabhsharma2u

Copy link
Copy Markdown
Contributor Author

Review complete ✅. Huge performance win! I manually rebased this branch to integrate the IntUnionFind optimization cleanly on top of the recent modular refactoring (PR 43). Addressed merge conflicts and ensured tests all pass. Great job reducing memory overhead drastically instead of crashing on large sizes.

@saurabhsharma2u
saurabhsharma2u marked this pull request as ready for review February 25, 2026 19:08
@saurabhsharma2u
saurabhsharma2u merged commit 3dc4390 into main Feb 25, 2026
6 checks passed
@saurabhsharma2u
saurabhsharma2u deleted the perf-optimize-duplicate-detection-11828779434289117858 branch February 25, 2026 19:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant