[WIP] JVector Benchmarking - #502
Conversation
Uh-oh -- let's figure out why this didn't work? I would think And maybe your dev box didn't already have |
| default: | ||
| throw new IllegalArgumentException("-indexType can be 'hnsw' or 'flat' only"); | ||
| } | ||
| indexType = switch (indexKind) { |
There was a problem hiding this comment.
Yay, I love this java improvement! Sheesh this happened in java 14, released at Covid start time (March 2020). Python expressly / intentionally chose not to allow match to be an expression to be consistent with other "decision making logic": https://peps.python.org/pep-0622/#make-it-an-expression
mikemccand
left a comment
mikemccand
left a comment
There was a problem hiding this comment.
Thanks @abernardi597! I left some initial comments ... haven't reviewed everything yet. I love the Cohere v3 improvements ....
| HnswGraph knnValues; | ||
| if (vectorsReader instanceof Lucene99HnswVectorsReader hnswVectorsReader) { | ||
| knnValues = hnswVectorsReader.getGraph(field); | ||
| final var knnValues = hnswVectorsReader.getGraph(KNN_FIELD); |
There was a problem hiding this comment.
Can we not insert final for local variables? This adds lots of noise in PRs and is rarely helpful.
Also, can we keep the static type instead of var type inference? It's easier for those times we have to read code outside of our IDE support.
|
|
||
| next_print_time_sec = start_time_sec | ||
| with open(csv_source_file, "w", newlines="") as meta_out, open(vec_source_file, "wb") as vec_out: | ||
| with open(csv_source_file, "w", newline="") as meta_out, open(vec_source_file, "wb") as vec_out: |
There was a problem hiding this comment.
Whoa! Why doesn't Python get angry when I was passing newlines!? Is it actually lenient to any random parameter coming in to the builtin open function!?
| # takes a long time! ~3.2 hours | ||
| # NOTE: train is the only split | ||
| docs = datasets.load_dataset(DATASET_NAME, LANG, split="train", streaming=True) | ||
| docs = docs.with_format("numpy") |
| emb = np.array(doc["emb"], dtype=np.float32) | ||
| emb = doc["emb"] | ||
|
|
||
| if emb.dtype != np.dtype('<f8'): |
There was a problem hiding this comment.
Hmm is f8 double (8 byte float) not float32 (4 byte float)? And then we are writing too many bytes to the output file?
28d1bd0 to
be5d2c3
Compare
2 participants
For visibility, here are the major changes I've made to support benchmarking the JVector codec I wrote for apache/lucene#15472.
I do have a few more changes for running on my test machines that are not included here.
Most notably:
BASE_DIR/dataoverSample,fanout, etc.DO_PSandDO_VMSTAT(needs more work to fix on my machines, looks like)