OpenMP - Limit the number of threads spawned

[whatsthecraic]

Hi there,
I've deployed the library on a server with 20 cores and 40 physical threads in SMT, and interacting with the library with 40 logical threads. Due to the internal usage of OpenMP, this spawns 40 * 40 = 3600 logical threads in the machine, making the execution rather inefficient. Is there a way to constrain the number of threads used by stinger/openmp (e.g. worker pool), or am I better off using a single logical thread to operate with the library?

linux, gcc 9, libgomp

[ehein6]

Curious to hear more about your use case. Which library calls are you using most often? Are you running the stinger server as a separate process, or did you compile stinger into your application?

If stinger is in a separate process, you can use OMP_NUM_THREADS in your environment to limit the number of threads that stinger will use.
If stinger is in the same process, you might consider disabling nested parallelism in OpenMP.

[whatsthecraic]

Hi Eric,
I'm using Stinger as a library dynamically linked, OpenMP nested parallelism is already disabled, the most frequent calls are those to update the graph (insert/remove edges and vertices).

[ehein6]

Hmm, I don't know how you are getting more than P threads in a single OpenMP process with nested parallelism disabled. Which function calls are you using in particular, stinger_batch_insert_edges?

[whatsthecraic]

Mostly stinger_mapping_, stinger_insert_edge, stinger_insert_edge_pair, stinger_remove_edge, stinger_remove_edge_pair, stinger_vtype, stinger_remove_vertex, ...

To get oversubscription in my case it's just enough to use these functions from different (user) threads.

[ehein6]

The functions you mentioned are all serial (thread-safe) code. You can search through the source to confirm there are no OpenMP directives in these.

[whatsthecraic]

I suspect this happens when stinger is created, then all other user threads created afterwards clone the state of OpenMP? I'll try to confirm this tomorrow. There are still functions such as stinger_remove_vertex and max_vertex_id that do use OpenMP though.

[ehein6]

How are you creating threads within your application? Using OpenMP or something else?

[whatsthecraic]

C++ stdlib, but that's a wrapper of pthread afaik.

[ehein6]

So you're mixing pthreads and OpenMP, which is possible but not recommended. (see http://forum.openmp.org/forum/viewtopic.php?f=3&t=642 for some discussion). Hard to say how they will interact with each other, you're probably right about the OpenMP state being cloned.

You might try OMP_NUM_THREADS=1 to disable threading in stinger, but then you may miss out on performance in places that need it.

[whatsthecraic]

Hi Eric,
I've done a bit of investigation. So here are my findings:

• creating a new stinger instance with `stinger_new_full' already spawns 40 threads. There is a parallel region at the end of the function.
• my hypothesis about cloning the OpenMP state when creating new user threads turned out to be wrong, at least in my case. That is, when new user threads are created, it does not automatically instantiate a new OpenMP thread pool.
• what instantiates the thread pool is an invocation to an OpenMP parallel region. In my case this happens with stinger_insert_edge -> stinger_update_directed_edge -> new_eb -> get_from_ebpool. This is the first point where the openmp pool is created in my program, I don't know if there are other points.