Geek Answers Handbook

How to log using a useless logger from a parallel method in R?

I use a useless logger in R for logging. I have a parallel algorithm implemented using snowfall in R. Each core of the parallel process registers an intermediate output in the registrar. But this result is not displayed in the log?

Can we record the use of a useless recorder from a parallel job using snowfall?

adding how it was done:

My particular case was a little different. I call the C function from R using the generic object that I created. This function is an iterative algorithm, and I need each file to be logged every few iterations. I was interested in registering from function C to a useless registrar. Why useless registrar? Since this is part of the web application, it makes sense to get all the output for the user session in a consistent format.

This is a general approach that I followed based on the accepted answer.

# init script
# iter logger namespace global variable
assign("MCMC_LOGGER_NAMESPACE", "iter.logger", envir = .GlobalEnv)  

loginit <- function(logfile) { 
  require('futile.logger')
  flog.layout(layout.simple, name = ITER_LOGGER_NAMESPACE)  
  flog.threshold(TRACE, name = ITER_LOGGER_NAMESPACE)
  flog.appender(appender.file(logfile), name = ITER_LOGGER_NAMESPACE)   
  NULL
}

parallel_funct_call_in_R <- function(required args) {    
require('snowfall')  
sfSetMaxCPUs() 
sfInit(parallel = TRUE, cpus = NUM_CPU) 
sfLibrary(required libs)
sfExport(required vars including logger namespace variable ITER_LOGGER_NAMESPACE)
iterLoggers = sprintf(file.path(myloggingdir, 'iterativeLogger_%02d.log', fsep = .Platform$file.sep), seq_len(NUM_CPU))
sfClusterApply(iterLoggers, loginit)  
sfSource(required files)
estimates <- sfLapply(list_to_apply_over, func_callling_C_from_R, required args)
sfStop()  
return(estimates)
}

iterTrackNumFromC <- function(numvec){
# convert numvec to json and log using flog.info
# the logger namespace has already been registered in the individual cores
flog.info("%s", toJSON(numvec), name = ITER_LOGGER_NAMESPACE) 
}

func_callling_C_from_R <- function(args){
 load shared obh using dyn.load
estimates = .C("C_func", args, list(iterTrackNumFromC)) # can use .Call also I guess
return(estimates)
}

Function C now

void C_func(other args, char **R_loggerfunc){ // R_loggerfunc is passed iterTrackNumFromC    
// do stuff
// call function that logs numeric values to futile.logger
logNumericVecInR();
}

void logNumericVecInR (char *Rfunc_logger, double *NumVec, int len_NumVec){        
    long nargs = 1;        
    void *arguments[1];
    arguments[0] = (double*)NumVec;    
    char *modes[1];
    modes[0] = "double";        
    long lengths[1];
    lengths[0] = len_NumVec;        
    char *results[1];
    //    void call_R(char *func, long nargs, void **arguments, char **modes, long *lengths, char **names, long nres, char **results)    
    call_R(Rfunc_logger, nargs, arguments, modes, lengths, (char**)0, (long)1, results);
}

Hope this helps. If for R and C there is an easier way to share a common registrar, let me know.

+4
source share
1

futile.logger sfInit slaveOutfile='', .

library(snowfall)
sfInit(parallel=TRUE, cpus=3, slaveOutfile='')
sfLibrary(futile.logger)
work <- function(i) {
  flog.info('Got task %d', i)
  i
}
sfLapply(1:10, work)
sfStop()

< <23 > outfile=''. GUI, Rgui, , , Windows Rterm.exe.

, . :

library(snowfall)
nworkers <- 3
sfInit(parallel=TRUE, cpus=nworkers)

loginit <- function(logfile) {
  library(futile.logger)
  flog.appender(appender.file(logfile))
  NULL
}
sfClusterApply(sprintf('out_%02d.log', seq_len(nworkers)), loginit)

work <- function(i) {
  flog.info('Got task %d', i)
  i
}
sfLapply(1:10, work)
sfStop()

, , , .

+5

More articles:


All Articles