Add binary format for GeoNames indexing benchmark - #578
Conversation
Add a Python converter (buildBinaryGeoNames.py) that pre-converts the
GeoNames allCountries.txt to a chunked binary
format that IndexGeoNamesBinary.java reads.
This eliminates per-document overhead during indexing:
- No String.split("\t") on 19 fields per line
- No Integer.parseInt/Double.parseDouble/Long.parseLong from text
- No date string parsing
- StringField/KeywordField use BytesRef directly, avoiding the
UTF-8 -> UTF-16 -> UTF-8 round-trip for non-tokenized fields
|
I love this idea, sorry I haven't made time yet to review it! We absolutely try to drive to near 0 the overhead of running a benchmark -- reading docs, deserializing, creating |
|
Thanks Mike! I noticed the geo file loading optimization can benefit the stored fields benchmark https://benchmarks.mikemccandless.com/stored_fields_benchmarks.html, but that would require conveting I'm not familiar with how the dataset is prepared for the nightly benchs, any hints on where to look or how it's set up? And yes, once the one for the stored fields benchmark lands, I'll add to notation.py. Thanks for the reminder! |
|
This PR has not had activity in the past 2 weeks, labeling it as stale. If the PR is waiting for review, notify the dev@lucene.apache.org list. Thank you for your contribution! |
|
@mikemccand gentle nudge, would you mind taking a look of this PR? As for the potential benefit to stored_fields_benchmarks, since this requires a conversion over the geo files, would you guide me how to prepare the dataset for https://benchmarks.mikemccandless.com/stored_fields_benchmarks.html so that it can be used in the bench? |
|
This PR has not had activity in the past 2 weeks, labeling it as stale. If the PR is waiting for review, notify the dev@lucene.apache.org list. Thank you for your contribution! |
2 participants
The existing
src/extra/perf/IndexGeoNames.javareads the raw text geo namesallCountries.txtfile, every document construction has to pay the cost of string split, date format, int/long/double parsing and utf8->utf16-utf8 for string/keyword field. As a benchmarking tool that is supposed to measure indexing performance, it is a lot noise, for example, in apache/lucene#15990, Tim wants to validate the performance gains, but as Amdahl’s law plays here, the overall end-to-end improvement gets diminished.This PR adds a pre-conversion step (similar to how we already do
buildBinaryLineDocs.pyfor the wikipedia dataset) so that the indexing benchmark can read binary file instead of text file focusing more on what matters of indexing.Here I introduce:
buildBinaryGeoNames.py: converts geo namesallCountries.txtinto a chunked binary format. No String split, date format, int/long/double parsing are needed during runtime anymore.IndexGeoNamesBinary.java: reads the binary file into ByteBuffer chunks. Note that for non-tokenized fields (StringField/KeywordField), we pass BytesRef directly to avoid utf8->utf16-utf8 round-trip overhead.Benchmark
Ran
java perf.IndexGeoNamesBinary allCountries.bin /path/to/index 1with OpenJDK 25.0.2, 1 indexing threads, no merge policy, 128MB RAM buffer. I validated that the output segments file numbers and sizes for correctness.Mac (Apple M3 Pro, 12 cores, 36 GB)
Linux (m5.4xlarge, Intel Xeon 8175M, 16 vCPU, 64 GB)