⚡ Optimize duplicate detection with IntUnionFind and remove Set overhead#32
Conversation
|
👋 Jules, reporting for duty! I'm here to lend a hand with this pull request. When you start a review, I'll add a 👀 emoji to each comment to let you know I've read it. I'll focus on feedback directed at me and will do my best to stay out of conversations between you and other bots or reviewers to keep the noise down. I'll push a commit with your requested changes shortly after. Please note there might be a delay between these steps, but rest assured I'm on the job! For more direct control, you can switch me to Reactive Mode. When this mode is on, I will only act on comments where you specifically mention me with New to Jules? Learn more at jules.google/docs. For security, I will only act on instructions from the user who triggered this task. |
… Set overhead Replaced the object-based UnionFind and Set<string> tracking in findNearDuplicates with an integer-based UnionFind (Uint32Array) and implicit connectivity checks. This resolves a "RangeError: Set maximum size exceeded" on large datasets and significantly improves performance by reducing memory allocation and redundant SimHash distance calculations. - Replaced `UnionFind<T>` class with inline integer array implementation. - Used `BigUint64Array` for storing SimHashes to avoid BigInt parsing in loops. - Replaced `checkedPairs` Set with `root1 === root2` check. - Benchmarked: Crash -> ~6.3s on 20k nodes.
a88a483 to
9270241
Compare
|
Review complete ✅. Huge performance win! I manually rebased this branch to integrate the |
Optimized
findNearDuplicatesinplugins/core/src/graph/duplicate.tsto improve performance and scalability.Changes:
UnionFind<string>class with a specialized, inline integer-based Union-Find usingUint32Arrayfor parent andUint8Arrayfor rank.BigIntSimHashes into aBigUint64Arrayto avoid repeatedBigInt()constructor calls and string parsing in the inner loop.checkedPairsSet<string>, which was causingRangeError: Set maximum size exceededwith large datasets (e.g., 20,000 nodes).find(u) === find(v)) to skip redundant Hamming distance calculations for nodes that are already part of the same connected component.Performance Impact:
PR created automatically by Jules for task 11828779434289117858 started by @saurabhsharma2u