How enable PSP_PRECON_TYPE as rrcom

[PaulinaEster]

Hi, I would like to know how I can enable the PSP_PRECON_TYPE equal to "rrcom". Is necessary compile the PSCOM with an specific flag?
I tring just export the env PSP_PRECON_TYPE=rrcom, but with the PSP_DEBUG_PRECON enabled I receive the error "Could not find provider with name 'rrcomm'. Using 'tcp' as the default." that its not possible to identify the provider rrcom and that will continue using
PSP_PRECON_TYPE=tcp.

Follow the pscom_info from configuration:
pscom_info.txt

[PaulinaEster]

@carsten-clauss can you help me please?

[carsten-clauss]

Hi @PaulinaEster, in order for rrcomm to be used as the precon provider (i.e., connection negotiation channel) for pscom, librrcomm.a must be found during the cmake step.

The librrcomm.a, in turn, is built as part of psmgmt, which means there is a necessary dependency: pscom with rrcomm can only be used if the ParaStation Daemon (psid) is used as the MPI process manager.

I don’t know what you know about rrcomm or why you want to use/try it, but the main advantage is better scalability and faster MPI startup on very large systems (i.e., toward exascale). For moderate clusters or even test systems, TCP is perfectly sufficient as a precon for pscom, while avoiding that dependency.

[PaulinaEster]

Hi @carsten-clauss, thank you for your reply.
I'm using TCP pre-communication and the UCX plugin with Infiniband for communication. But in scenarios with more than four nodes (with 16 processes on each node), my system's memory usage increases very quickly and the communication time becomes terrible. I suspect this is due to the type of pre-communication and I want to switch to RRCOM.

So, I need psmgmt for RRCOM, right? Is there a better tutorial for installing the psmgmt library than the one provided in the repository? And does this library work with Slurm jobs?

Note: I changed the UCX_TLS environment variable to 'tcp,self,shm' and the communication time improved a bit. I suspect this is because the processes communicate without TCP in intranode scenarios.

[carsten-clauss]

Hm... ParaStation MPI with pscom using just TCP as the precon (i.e., without rrcomm) has been running for years on systems with over 3,000 nodes (and up to 25,600 processes per job) -- I therefore doubt that rrcomm is the solution to your problem.

Are you sure your application doesn't have a memory leak and/or is using memory resources inefficiently?
What kind of application or communication scenario are we talking about?

However, if it really is a matter of running out of communication resources or respective memory, then switching to a different protocol in UCX could actually be a solution (i.e., switching from rc to dc or ud in UCX_TLS, depending on the features of your system).

As I said, I doubt that rrcomm will be of any help here -- especially since it's only used to establish the connections, while the resource usage of the established connections depends on the protocols used by UCX.

BTW: With pscom, intra-node communication is usually handled via its own shared-memory protocol and not via UCX, unless you disable this by setting PSP_SHM=0.

[PaulinaEster]

First of all, yes, I'm sure my application doesn't have memory leaks or inefficient resource usage, as the application was initially tested with the original version of MPICH (with the same configuration and communication protocol), but this memory usage and large increased communication time occurred when I tested with psmpi and pscom. I'm also trying different configurations by changing UCX_TLS.

The worst performance was in MPI_Alltoall calls, the others had expected performance.

I'm sure it's a configuration problem in my cluster; I'm just trying to get the configuration right.

And I will test with MPICH and pscom as the communication library; if this high communication time persists, I'll confirm that pscom is misconfigured. And that it's not something related to psmpi.