Page: https://docs.nvidia.com/nemo/curator/v26.02/curate-text/process-data/deduplication/fuzzy ("Fuzzy Duplicate Removal", v26.02)
What the docs say
The similarity-threshold section currently reads:
More strict matching: Increase num_bands or decrease minhashes_per_band
Less strict matching: Decrease num_bands or increase minhashes_per_band
with defaults num_bands=20, minhashes_per_band=13, and example configurations of num_bands=25, minhashes_per_band=10 (labeled stricter) and num_bands=15, minhashes_per_band=15 (labeled looser).
Why this looks backwards
For standard MinHash-LSH banding with b bands of r minhashes each, where documents become a candidate pair if any band matches, two documents with Jaccard similarity s collide in one band with probability s^r, so:
P(candidate) = 1 − (1 − s^r)^b
The S-curve midpoint (the effective similarity threshold) is approximately:
Both knobs then work in the opposite direction from the current text:
- Increasing
b (num_bands) gives more chances for a collision → catches lower-similarity pairs → looser matching.
- Decreasing
r (minhashes_per_band) makes each band easier to match (s^r grows as r shrinks for s < 1) → also looser.
The page's own numbers corroborate this
| Config |
b |
r |
t ≈ (1/b)^(1/r) |
Current label |
| Default |
20 |
13 |
0.794 |
"balanced" — matches the documented ~0.8 threshold ✅ |
| Example A |
25 |
10 |
0.725 |
"more strict" — but the threshold dropped (looser) ❌ |
| Example B |
15 |
15 |
0.835 |
"less strict" — but the threshold rose (stricter) ❌ |
The default's ≈0.79 agreeing with the documented ~0.8 threshold confirms the formula's variable mapping is the right one for Curator's parameters; the two examples then land on the wrong sides of it.
Suggested fix
Swap the two directions (and the example labels):
More strict matching: decrease num_bands or increase minhashes_per_band
Less strict matching: increase num_bands or decrease minhashes_per_band
Optionally, adding the t ≈ (1/num_bands)^(1/minhashes_per_band) rule of thumb to the page would let users pick parameters for a target threshold directly.
If Curator's implementation defines these parameters differently from standard any-band-match banding, then the current text may be intentional — but in that case a short note explaining the non-standard semantics would help, since readers familiar with MinHash-LSH will expect the behavior above.
Thanks for maintaining Curator — happy to open a PR with the wording change if that's easier.
Page: https://docs.nvidia.com/nemo/curator/v26.02/curate-text/process-data/deduplication/fuzzy ("Fuzzy Duplicate Removal", v26.02)
What the docs say
The similarity-threshold section currently reads:
with defaults
num_bands=20,minhashes_per_band=13, and example configurations ofnum_bands=25, minhashes_per_band=10(labeled stricter) andnum_bands=15, minhashes_per_band=15(labeled looser).Why this looks backwards
For standard MinHash-LSH banding with
bbands ofrminhashes each, where documents become a candidate pair if any band matches, two documents with Jaccard similarityscollide in one band with probabilitys^r, so:The S-curve midpoint (the effective similarity threshold) is approximately:
Both knobs then work in the opposite direction from the current text:
b(num_bands) gives more chances for a collision → catches lower-similarity pairs → looser matching.r(minhashes_per_band) makes each band easier to match (s^rgrows asrshrinks fors < 1) → also looser.The page's own numbers corroborate this
The default's ≈0.79 agreeing with the documented ~0.8 threshold confirms the formula's variable mapping is the right one for Curator's parameters; the two examples then land on the wrong sides of it.
Suggested fix
Swap the two directions (and the example labels):
Optionally, adding the
t ≈ (1/num_bands)^(1/minhashes_per_band)rule of thumb to the page would let users pick parameters for a target threshold directly.If Curator's implementation defines these parameters differently from standard any-band-match banding, then the current text may be intentional — but in that case a short note explaining the non-standard semantics would help, since readers familiar with MinHash-LSH will expect the behavior above.
Thanks for maintaining Curator — happy to open a PR with the wording change if that's easier.