[BUG] Crash when trying to make a Run comparable if numba JIT is disabled

[milyenpabo]

Describe the bug
Ranx is very very slow when I use it with many small Run and Qrels objects (e.g. to do question-by-question comparisons). This is likely because Ranx uses the Numba JIT compiler, which is optimized for few large objects with long compute times. (See previous issue: #23)

The solution could be to disable JIT compilation with NUMBA_DISABLE_JIT=1. However, the Ranx Run.make_comparable() function crashes then, because it contains illegal Python -- it removes items from the Run dict while iterating over it:

ranx/data_structures/run.py", line 150, in make_comparable
        for q_id in self.run:
                     ^^^^^^^^
RuntimeError: dictionary changed size during iteration

This is the offending line:

ranx/ranx/data_structures/run.py

Line 152 in 2cae52f

del self.run[q_id]

To Reproduce
Run an evaluation with NUMBA_DISABLE_JIT=1 and making the Run comparable.

Expected behavior
Things run smoothly.

Additional context
I have a local patch of Run.make_comparable() that first iterates over the elements and marks them for removal, and then in a second step removes the marked items. A similar solution could be merged into the official Ranx code to fix the bug.

[diegoceccarelli]

Thanks @milyenpabo I’ve also been thinking about supporting Ranx without Numba, by implementing the metrics using NumPy vectorized operations. I’d be curious to implement a parallel version of the metrics and then benchmark both Numba and non-Numba approaches across different collection sizes. My gut feeling is that on small datasets or some architectures, the non-Numba version might actually be faster.

@AmenRa what do you think? Would you be open to exploring this? I’d be happy to prototype it in a separate branch so we can compare before deciding anything.

[AmenRa]

Hello and sorry for the delay.

@milyenpabo

I will fix the issue with the referenced line.
However, I doubt disabling Numba will bring you improvements.
Numba code executed as Python is mostly for-loops instead of vectorized operations, which are much slower.

I tried on my very old laptop (Intel MacBook Pro from 2018) to compute NDCG for 1000 runs for 100 queries with 100 results each. It took ~7 seconds (135 runs/s).

Have you identified the bottleneck in your own code? Can you provide a working example with dummy Qrels and Runs to reproduce your situation? Maybe there is something in your specific data that is causing slow downs.
You can use the code below to generate dummy Qrels and Runs.

from random import randint, random

n_queries = 100
n_results = 100
n_judged = 100

# Build a dummy Qrels
qrels = Qrels(
    {str(i): {str(j): randint(1, 5) for j in range(n_judged)} for i in range(n_queries)}
)

# Build a dummy Run
run = Run({str(i): {str(j): random() for j in range(n_results)} for i in range(n_queries)})

@diegoceccarelli

As far as I remember, using Numba instead of Numpy was much faster when I evaluated both ranx and retriv efficiency. Happy to be proven wrong.

[milyenpabo]

However, I doubt disabling Numba will bring you improvements.

It did significantly speed things up for me. As I remember, Run/Qrels object creation was the bottleneck, it frequently took several seconds, even though I had only ~50 questions, and ~10 entries per question. Without Numba, all this got snappy.

Can you try your above test with creating 100 different Run objects, and evaluate each against your single Qrels object? What I'd expect that this would be really slow with Numba, but still fast without Numba. If you don't see that effect, then maybe it's something with my Numba config. But I did try to set the various Numba env variables to resolve the bottleneck, without success.

[AmenRa]

I think you may be using ranx in a cold start scenario (i.e., Numba functions have not been compiled yet) or somehow Numba is not caching the compiled functions.
On my end, instantiating 1000 unique runs for 100 queries with 100 results each takes ~13 seconds (81 runs/s).
Can you provide your code and data to reproduce the slowdowns you are experiencing?
Also, try to instantiate a single run first and then time the instantiation of the whole set of runs to check if there's any difference with what you are reporting.

[diegoceccarelli]

@AmenRa could it be that the hardware setup affects the performance you see with numba? E.g you have only one cpu or little ram etc?

[AmenRa]

Is the message for me or for @milyenpabo ?

My test setup has always been the same: MacBook Pro 2018, Intel Core I9 6 cores / 12 threads, 32 GB.
On my workstation with AMD 5950x (16 cores / 32 threads) and 64 GB, ranx works great.
Colab takes some time to compile the custom Numba functions, but after compilation I still get good results: instantiating 1000 unique runs for 100 queries with 100 results each takes ~15 seconds (68 runs/s).

I never tested ranx on Arm chips, as I do not own one.
Also, Numba automatic parallelization could impact performance on CPUs with very large core counts, but it is possible to control the number of threads used by Numba by setting threads > 0 when evaluating a run with evaluate.
There is no parallelization when Qrels and Run objects are created.

[milyenpabo]

I modified your test slightly to measure time:

from random import randint, random
from ranx import Qrels, Run
import time

n_queries = 100
n_results = 100
n_judged = 100

t0 = time.time()
# Build a dummy Qrels
qrels = Qrels(
    {str(i): {str(j): randint(1, 5) for j in range(n_judged)} for i in range(n_queries)}
)
t1 = time.time()

# Build a dummy Run
run = Run({str(i): {str(j): random() for j in range(n_results)} for i in range(n_queries)})

t2 = time.time()

qrels_time = t1 - t0
run_time = t2 - t1

print(f"Creating the Qrels took {qrels_time:.3f} sec.")
print(f"Creating the Run took {run_time:.3f} sec.")

This is what I've got:

Default config, first execution:
    Creating the Qrels took 5.732 sec.
    Creating the Run took 2.059 sec.

Default config, second and subsequent executions:
    Creating the Qrels took 0.401 sec.
    Creating the Run took 0.095 sec.

With NUMBA_DISABLE_JIT=1
    Creating the Qrels took 0.009 sec.
    Creating the Run took 0.006 sec.

From this, you can see that without Numba JIT it runs ~33x faster than with Numba when it utilizes the cache. I tried to use this in a UI application when I had to create 100s of small Run/Qrels objects upon a user action, and it was just too slow for an interactive experience. Without Numba, it works fine.

I work on an Apple M1 Pro with 16GB RAM.

[AmenRa]

I get very different results on my end:

Default config, first execution:
    Creating the Qrels took 0.845 sec.
    Creating the Run took 0.223 sec.

Default config, second and subsequent executions:
    Creating the Qrels took 0.029 sec.
    Creating the Run took 0.019 sec.

With NUMBA_DISABLE_JIT=1
    Creating the Qrels took 0.020 sec.
    Creating the Run took 0.012 sec.

Also, by disabling NUMBA everything degrades. Considering unique runs for 100 queries with 100 results each:

• with Numba enabled I can evaluate the NDCG of ~145 runs per second
• with Numba disabled I can evaluate the NDCG of ~4 runs per second

Any chance you can investigate whether your slowdowns are due to the M1 chip?

[milyenpabo]

Interesting. I can try to run this on a server with Intel CPUs, but I won't be near my computer in the coming 2 weeks.

[AmenRa]

No problem. Take your time!

These are the results I get from Colab:

Default config, first execution:
    Creating the Qrels took 2.233 sec.
    Creating the Run took 0.395 sec.

Default config, second and subsequent executions:
    Creating the Qrels took 0.025 sec.
    Creating the Run took 0.021 sec.

With NUMBA_DISABLE_JIT=1
    Creating the Qrels took 0.021 sec.
    Creating the Run took 0.017 sec.