[R] Rsge: recursive parallelization

Peter Danenberg pcd at roxygen.org
Fri Apr 9 17:52:21 CEST 2010


In principle, I'd like to be able to do something like this:

  sge.parLapply(seq(10), function(x) parLapply(seq(x), function(x) x^2))

In practice, however, I have to resort to acrobatics like this:

  sge.options(sge.remove.files=FALSE)
  sge.options(sge.qsub.options='-cwd -V')
  sge.parLapply(seq(10),
                function(x) {
                  sge.options(sge.save.global=TRUE)
                  sge.options(sge.remove.files=FALSE)
                  sge.parLapply(seq(x),
                                function(x) x^2,
                                cluster=TRUE,
                                debug=FALSE,
                                trace=FALSE,
                                file.prefix='Rsge_data',
                                global.savelist=NULL,
                                packages=NULL)
                },
                function.savelist=c('sge.parLapply', 'sge.parParApply',
                  'sge.options', 'sge.taskPrep'),
                global.savelist=c('sge.parParApply', 'sge.globalPrep',
                  'global.savelist', 'sge.taskPrep', 'sge.checkNotNow',
                  'sge.get.jobid', 'sge.get.result', 'docall',
                  'enquote'),
                packages=NULL)

and I still get bizarre behavior: half of the results will be NULL,
for instance; the other half, incomplete.

Would non-trivial changes to Rsge be required to make something like
this possible?



More information about the R-help mailing list