Print messages to console as soon as they are generated

[vvandermeersch]

Hello,

I've been trying to print message to console as soon as they are generated.
I know this question has already been addressed (e.g. here), but I did not find a satisfactory answer.
I'm using future to run different parallel processes, and I would like to print a message to the console as soon as one of this process is finished.
Using future.conditions = "message", future.stdout = NA options, I was able to print some messages before the global process is over, but not as soon as the messages are generated.

For example, say I would like to print each process id when it is completed with something like:

future_apply(array(1:2), 1, function(i) {
    Sys.sleep(abs(i-2)*20)
    message(paste("Process",i,"completed !")) } )

Even if the process n°2 is completed before process n°1, the console do not print "Process 2 completed !" first.

Thank you for your help !

[HenrikBengtsson]

Hi. Let me start by explaining the current behavior. I'll end with a solution.

For example, say I would like to print each process id when it is completed with something like:

future_apply(array(1:2), 1, function(i) {
    Sys.sleep(abs(i-2)*20)
    message(paste("Process",i,"completed !")) } )

Even if the process n°2 is completed before process n°1, the console do not print "Process 2 completed !" first.

This is by design. The future.apply package replicates the behavior of the corresponding apply functions in base R. This way the threshold from going, say, apply() to future_apply() is kept at a minimum. More over, the behavior of the future_nnn() functions are designed to behave exactly the same regardless of sequential or parallel backend. So, the (relayed) output of future_apply() is the same as apply().

Using future.conditions = "message", future.stdout = NA options, I was able to print some messages before the global process is over, but not as soon as the messages are generated.

I want to say a few things here, and you might know this already, but I wanna make it clear for everyone else who reads this. There are two kinds of output in R: (i) directly to the standard output (stdout) and (ii) via R's condition system.

stdout: The traditional cat(), print(), str(), ... functions will output text by writing it to the so called standard output (stdout). When outputted on a parallel workers, this type of output often ends up in the void, i.e. it's not visible. This is true for all parallel frameworks in R. However, the future framework captures standard output produced on parallel workers and relay (re-output) it in the main R session when the future has been finished. This is what happens when you use the default future.stdout = TRUE. If you use future.stdout = FALSE, this output is dropped.

Now, depending on your operating system, your R environment (e.g. R in the terminal, and RStudio GUI), and parallel backend (e.g. forked/multicore, and PSOCK clusters), you, but not R, may be able to "see" the output in the terminal. This is what future.stdout = NA allows you play with, but it is not recommended and should never ever be used in an R package. I only provide future.stdout = NA people to play with very low-level behaviors, but I might decided deprecate it in a future release.

conditions: In constrast to above stdout functions, functions like message(), warning(), and stop() works by signalling a so called condition that contains a message/text. Unless captured by code along the way, these conditions are passed along by R itself and if they bubble up all the way up to the top, their messages are outputted to the standard error (stderr). Conditions signalled in parallel workers are typically not captured and end up outputting to the stderr on the parallel workers, which is also invisible to use. Now, the future framework captures these conditions by default and relays (re-signals) them in the main R session. If we don't want this to happen, we can use future.conditions = character(0).

In your example code, you are using message(). How these are handled are therefore controlled by future.conditions. The future.stdout argument has no affect on message conditions.

I'm using future to run different parallel processes, and I would like to print a message to the console as soon as one of this process is finished.

Looking at your example, it looks like you want to output messages that convey progress. If so, I highly recommend that you use the progressr package. It signals and relays progress information as soon as possible, independently of the order futures are finished. It supports parallel processing and there are examples how to use it with futures, e.g. https://progressr.futureverse.org/#parallel-processing-and-progress-updates.

library(future.apply)
plan(multisession)

library(progressr)
handlers(global = TRUE)
handlers("progress")

y <- local({
  p <- progressor(2)
  future_apply(array(1:2), 1, function(i) {
    Sys.sleep(abs(i-2)*20)
    p(paste("Process",i,"completed !"))
  })
})

will output

\ [==========>-----------]  50% Process 2 completed !

If you want to "output" progress messages that remain afterward, see https://progressr.futureverse.org/#sticky-messages.

See https://progressr.futureverse.org/#known-issues for the reason why we need to use local() above, which is not needed if you do the same within a function of yours.

Hope this helps

[vvandermeersch]

Thank you very much for your very comprehensive reply.
That really helps me. I will try to use progressr as soon as possible.

Thanks, and happy new year !