multiple nodes of a cluster?

[Sherry520]

I noticed that foreach could execute repeated operations on multiple nodes of a cluster. But so far I can only find parallelism in multiple threads on a single node. How to implement cross-node operation?

[HenrikBengtsson]

If you launch an HPC job, you can have the main use parallel or parallelly to launch a multi-computer cluster, e.g. via SSH or whatever the HPC scheduler allows you to access the other hosts allocated to your job. See https://parallelly.futureverse.org/articles/parallelly-17-hpc-workers.html for some illustrations.

Then, with that cluster set up, lets call it cl, you can then pass it to:

library(doParallel)
registerDoParallel(cl)

And, as the author of Futureverse, I recommend using doFuture, e.g.

library(doFuture)
plan(cluster, workers = cl)
registerDoFuture()

That way you get access to the more powerful features that Futureverse brings. See https://dofuture.futureverse.org/ for more info.

[HenrikBengtsson]

And, if your question does not involve an high-performance-compute (HPC) cluster, but you just meant a regular cluster of external computer, then see https://future.futureverse.org/reference/cluster.html. It's super simple as long as you have SSH access to each computer and it has R installed, e.g.

library(doFuture)
plan(cluster, workers = c("node1.example.org", "office2.company.org"))
registerDoFuture()

That's it.