Uses all cores instead of 6 cores

[LeaLe88]

Dear all,

Following my previous post in Seurat github satijalab/seurat#8315 I have run below code in order to use multiple cores for the findmarkers function using the future package.
I was thinking this question now is more appropriate to post here in future github:

For some reason I am using all 64 cores even though I specified to use only 6:

plan(multisession(workers = 6))   #seems not to be working correctly 
idents <- levels(Idents(kid.filtered_new))

# To avoid this error run:
#Error in getGlobalsAndPackages(expr, envir = envir, globals = globals) : 
#The total size of the 2 globals exported for future expression ('FUN()') is 1.27 GiB.. This exceeds the maximum allowed size of 500.00 MiB (option 'future.globals.maxSize'). 
options(future.globals.maxSize = 8000 * 1024^2)

#Create future
markers <- future_lapply(idents, function(x) {
  Clu_marker <- FindMarkers(object = kid.filtered_new, ident.1 = x, test.use="poisson",latent.vars = "orig.ident")
  Clu_marker$cluster <- x
},future.seed = TRUE)

BTW I had to add options(future.globals.maxSize = 8000 * 1024^2) in order to avoid the error shown above.

Do you know what could be the problem here?

Thank you!

[scottkosty]

It might be helpful if you give a minimal example. It's possible some of the functions you're using are doing parallelization themselves. Can you post a minimal example?

Additionally, if you use the sequential plan, can you confirm only one core is used?

[LeaLe88]

Thank you for your reply. I just tried the sequential plan and the CPU usage goes to 4000 on a 64core server...
I feel like the functions is doing parallelisation by itself as you suggested.

I found this discussion here, would it make sense to run it as below following that?
#343

I tried below however again it is using all cores:

plan(multisession, workers = 4)

#Try to run sequential
#plan(sequential)

idents <- levels(Idents(kid.filtered_new))

# To avoid this error run:
#Error in getGlobalsAndPackages(expr, envir = envir, globals = globals) : 
#The total size of the 2 globals exported for future expression ('FUN()') is 1.27 GiB.. This exceeds the maximum allowed size of 500.00 MiB (option 'future.globals.maxSize'). 
#There are two globals: 'kid.filtered_new' (1.27 GiB of class 'S4') and 'FUN' (5.11 KiB of class 'function')

options(future.globals.maxSize = 8000 * 1024^2)

availableCores()

#Create future

#Complete run with adjustment for batch

markers <- future_lapply(idents, function(x) {
  Clu_marker <- FindMarkers(object = kid.filtered_new, ident.1 = x, test.use="poisson",latent.vars = "orig.ident")
  Clu_marker$cluster <- x
  data.table::setDTthreads(1)   #limit number of threads used 
  },future.seed = TRUE)

I will create a minimal example.

[scottkosty]

The information about sequential is a helpful clue. Thanks for working on the minimal example.

[HenrikBengtsson]

Also, data.table::setDTthreads(1) has to go first in the function for it to apply to FindMarkers() and other calls.

[HenrikBengtsson]

@LeaLe88 , please see @scottkosty 's comment; that's the way forward to get to the bottom of your problem, which is not expected.

Unrelated to your problem, please be aware that with:

markers <- future_lapply(idents, function(x) {
  Clu_marker <- FindMarkers(object = kid.filtered_new, ident.1 = x, test.use="poisson",latent.vars = "orig.ident")
  Clu_marker$cluster <- x
},future.seed = TRUE)

then markers becomes a list populated with the x for each element, not Clu_marker. You probably meant:

markers <- future_lapply(idents, function(x) {
  Clu_marker <- FindMarkers(object = kid.filtered_new, ident.1 = x, test.use="poisson",latent.vars = "orig.ident")
  Clu_marker$cluster <- x
  Clu_marker
},future.seed = TRUE)

This is how R is designed, and has nothing to do with Futureverse or parallelization per se.