Add hadamard rotation in smelling - #586
Conversation
|
This PR has not had activity in the past 2 weeks, labeling it as stale. If the PR is waiting for review, notify the dev@lucene.apache.org list. Thank you for your contribution! |
mikemccand
left a comment
mikemccand
left a comment
There was a problem hiding this comment.
Thanks @shubhamvishu -- let's try to get these PRs merged soon so people can separately test pre-conditioning non-isotropic vector corpora.
| @@ -0,0 +1,236 @@ | |||
| #!/usr/bin/env python3 | |||
| """Hadamard rotation — Python implementation matching the structure of | |||
| org.apache.lucene.util.quantization.HadamardRotation. | |||
There was a problem hiding this comment.
This builds on the upstream Lucene PR right? (Not yet merged?).
Oh, not builds on, just refers to that PR, because this is pure Python (+ NumPy) impl, cool. So we can merge this before upstream PR and let us testing Hadamard separately/decoupled, yay!
There was a problem hiding this comment.
Thats correct!
| what OSQ wants. | ||
|
|
||
| This is a Python implementation of the same logic; it is NOT bit-exact with the | ||
| Java version because it uses NumPy's PRNG instead of Java's Random LCG. The |
There was a problem hiding this comment.
I'm confused -- is the rotation matrix a deterministic function of int dim? Or is there some randomness in its creation? If so, does it write the resulting matrix somewhere?
There was a problem hiding this comment.
The rotation or "hadamard" matrix (0s and 1s) is same. The seed is a deterministic based on the dim which is used for RNG (permutation and negation). We don't need to store the matrix anywhere since a hadamard matrix of dim 2 looks like [[1,1] [1,-1]] and the same could be generated for any dim of power 2 (so its fixed not random).
|
Thanks for taking a look @mikemccand. I fixed the conflicts now. This PR build on the other smelling PR, so maybe we could get that 1st or maybe we could merge this one (has the union of code changes). |
# Conflicts: # src/python/smell_vectors.py
mikemccand
left a comment
mikemccand
left a comment
There was a problem hiding this comment.
I left minor / non-blocking comments -- let's merge and address comments in follow-on -- thank you @shubhamvishu! Isotropic vectors here we go!
| ap.add_argument( | ||
| "--rotate", | ||
| action="store_true", | ||
| help="apply Lucene-faithful Hadamard rotation to every sampled vector in memory before analysis (no rotated .vec file is written)", |
There was a problem hiding this comment.
Looks like it's only for the ID ("intrinsic dimensions") smelling?
| print("Re-running smell with Hadamard rotation applied in memory (Lucene-faithful)...") | ||
| print("=" * 104) | ||
| print() | ||
| ROTATE = True |
There was a problem hiding this comment.
Kinda messy -- would be better to pass a parameter?
|
|
||
| def _maybe_rotate(samples, dim): | ||
| """If ROTATE is on, return a Hadamard-rotated copy of `samples` (rows are rotated independently).""" | ||
| if not ROTATE: |
There was a problem hiding this comment.
Better to just not call this method if not ROTATE from above?
2 participants
Notes :
--rotateto smelling so now it reports smells of a.vecfile and then rotates the vectors and analyse the rotated vectors and post both the results to indicate how rotation helped the situation. I ran it for Cohere etc datasets and it showed constant improvement in the isotropy diag (moving closer to 1)Cohere smelling
Common pattern: rotation drives
isotropy_diagto ≈1.0 and reshapes per-dim distributions toward Gaussian on both datasets, while leaving every distance/spectrum metric unchanged.