Post-mortem diagnostic: No process exists with this PID

[thesnakeguy]

Hi,

I am using future.lapply inside a custom function that prepares audio data to train a neural network. Essentially, the function is called 3 times (test, validation and training data). The code runs fine on the test and validation data, but gives me the following error when running on the (larger) training data:

Error in unserialize(node$con) : 
    Failed to retrieve the value of MultisessionFuture (future_lapply-1) from cluster RichSOCKnode #1 (PID 22400 on localhost 
    ‘localhost’). The reason reported was ‘error reading from connection’. Post-mortem diagnostic: No process exists with this 
    PID, i.e. the localhost worker is no longer alive.

When splitting the data in equally-sized partitions (folds), the script will run on the first partitions, but then give me that same error message after some runs.
The code:

I call the plan() arguments outside the function:
plan(multiprocess, workers = availableCores())

audioProcess <- function(files, limit = 10, ws = 10, stride = 5){
        windowSize <- seq(0, limit, by = stride)
        # iterate and parallelise
        batches <- future_lapply(windowSize, function(w){
                # execute
                melslice(files, from = w, to = w+ws)
        })
        # combine output into single array
        out <- abind(batches, along = 3)
        # reorder dimensions after adding single-channel as 4th
        dim(out) <- c(dim(out), 1)
        out <- aperm(out, c(3,1,2,4))
        return(out)

Could you help me understand what is causing the error and what might solve it?

R version 4.0.3 (2020-10-10)
future.apply_1.7.0
12 localhost cores available

Cheers,
Pablo

[HenrikBengtsson]

Hi, first, you're on MS Windows, correct? It's not the cause of the error, but it helps to rule out a few other things.

The error message indicates that your parallel R worker(s) has terminated. This means that there's something in the code crashing the worker. Sometimes that happens if you run out of memory but a more common reason is that there's a non-exportable globals/object that causes R to crash if received from another R process. It's hard to say more without knowing what melslice() is.

Per https://cran.r-project.org/web/packages/future/vignettes/future-4-non-exportable-objects.html, try to set:

options(future.globals.onReference = "error")

and rerun. See if you get another, more informative error from using that.

PS. Please know that multiprocess is deprecated and will eventually be removed; please use multisession instead.

[thesnakeguy]

Hi, Thanks for the suggestion! The error is caused by a non-exportable object indeed. FYI the other called functions are: melspec <- function(x, start, end){ mp3 <- readMP3(filename = x) %>% extractWave(xunit = "time", from = start, to = end) # return log-spectrogram with 256 Mel bands and compression sp <- melfcc(mp3, nbands = 256, usecmp = T, spec_out = T, hoptime = (end-start) / 256)$aspectrum # Median-based noise reduction noise <- apply(sp, 1, median) sp <- sweep(sp, 1, noise) sp[sp < 0] <- 0 # Normalize to max sp <- sp / max(sp) return(sp) } # iterate melspec over all samples, arrange output into array melslice <- function(x, from, to){ lapply(X = x, FUN = melspec, start = from, end = to) %>% simplify2array() } I will have a look into what is causing the error in the functions. Off course, if you would see the obvious mistake, help would be much appreciated. Kind regards, Pablo Op di 25 mei 2021 om 17:48 schreef Henrik Bengtsson < ***@***.***>:

…

Hi, first, you're on MS Windows, correct? It's not the cause of the error, but it helps to rule out a few other things. The error message indicates that your parallel R worker(s) has terminated. This means that there's something in the code crashing the worker. Sometimes that happens if you run out of memory but a more common reason is that there's a non-exportable globals/object that causes R to crash if received from another R process. It's hard to say more without knowing what melslice() is. Per https://cran.r-project.org/web/packages/future/vignettes/future-4-non-exportable-objects.html, try to set: options(future.globals.onReference = "error") and rerun. See if you get another, more informative error from using that. PS. Please know that multiprocess is deprecated and will eventually be removed; please use multisession instead. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#495 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFNAGCPGXAXYZ6G4UNRJLF3TPPBEVANCNFSM45PVFQGA> .

[HenrikBengtsson]

Good that you've identified the problem. FYI, in the next release of future, this type of troubleshooting will be somewhat automated.

The only solution to these types of problems is to make sure not to pass non-exportable objects back and forth between the main R session and the R parallel workers.

If you could add a reproducible example that I can cut'n'paste, I could at least add the problematic package (which I don't know what it is) to https://future.futureverse.org/articles/future-4-non-exportable-objects.html.

[thesnakeguy]

Hi Henrik, I got some time to make a reproducible example for you. The funs.R file contains the audioProcess function with future_lapply within it. Basically, the example script downloads some sound queries from a database (xeno canto) and then they are being processed in batch, but also each individual file is processed in pieces. I hope this is what you need. Otherwise, don't hesitate to contact me. Again, many thanks for digging further into this! Kind regards, Pablo Op wo 26 mei 2021 om 23:55 schreef Henrik Bengtsson < ***@***.***>:

…

Good that you've identified the problem. FYI, in the next release of *future*, this type of troubleshooting will be somewhat automated. The only solution to these types of problems is to make sure not to pass non-exportable objects back and forth between the main R session and the R parallel workers. If you could add a reproducible example that I can cut'n'paste, I could at least add the problematic package (which I don't know what it is) to https://future.futureverse.org/articles/future-4-non-exportable-objects.html . — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#495 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFNAGCPGRTMGJFO252G2V63TPVU43ANCNFSM45PVFQGA> .

[HenrikBengtsson]

I'll need a small reprex that i can cut'n'paste to do any investigation

[thesnakeguy]

Hi Henrik, Normally you can open the files, change the dir and run through the lines. Otherwise I would need to send you audio samples if it needs to be more concise. Cheers, Pablo Op vr 28 mei 2021 om 20:35 schreef Henrik Bengtsson < ***@***.***>:

…

I'll need a small reprex that i can cut'n'paste to do any investigation — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#495 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFNAGCPZ6TSXVB4G743MWP3TP7PADANCNFSM45PVFQGA> .

[thesnakeguy]

Hi Henrik, Do you need anything more from me to investigate this? Many thanks, Pablo Op vr 28 mei 2021 om 20:35 schreef Henrik Bengtsson < ***@***.***>:

…

I'll need a small reprex that i can cut'n'paste to do any investigation — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#495 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFNAGCPZ6TSXVB4G743MWP3TP7PADANCNFSM45PVFQGA> .

[HenrikBengtsson]

Do you need anything more from me to investigate this?

A reproducible example that I, and someone else, can cut'n'paste into R that runs from start to finish would be great. Right now we don't even know what readMP3(), etc. are. I assume they're from some R package. For an example MP3 file, maybe those packages come with example files, e.g. see their help()/example().

PS. Please use the online GitHub interface to comment, and please try to use Markdown formatting to make it easier/quicker to read. From the experience in other threads, when commenting by replying to their email notifications, it looks like it can break the comment flow, newlines are introduced in code, ...