Post-mortem diagnostic: No process exists with this PID

[thesnakeguy]

Hi,

I am using future.lapply inside a custom function that prepares audio data to train a neural network. Essentially, the function is called 3 times (test, validation and training data). The code runs fine on the test and validation data, but gives me the following error when running on the (larger) training data:

Error in unserialize(node$con) : 
    Failed to retrieve the value of MultisessionFuture (future_lapply-1) from cluster RichSOCKnode #1 (PID 22400 on localhost 
    ‘localhost’). The reason reported was ‘error reading from connection’. Post-mortem diagnostic: No process exists with this 
    PID, i.e. the localhost worker is no longer alive.

When splitting the data in equally-sized partitions (folds), the script will run on the first partitions, but then give me that same error message after some runs.
The code:

I call the plan() arguments outside the function:
plan(multiprocess, workers = availableCores())

audioProcess <- function(files, limit = 10, ws = 10, stride = 5){
        windowSize <- seq(0, limit, by = stride)
        # iterate and parallelise
        batches <- future_lapply(windowSize, function(w){
                # execute
                melslice(files, from = w, to = w+ws)
        })
        # combine output into single array
        out <- abind(batches, along = 3)
        # reorder dimensions after adding single-channel as 4th
        dim(out) <- c(dim(out), 1)
        out <- aperm(out, c(3,1,2,4))
        return(out)

Could you help me understand what is causing the error and what might solve it?

R version 4.0.3 (2020-10-10)
future.apply_1.7.0
12 localhost cores available

Cheers,
Pablo