[R] error with more 100 forked processes

[Jiefei Wang]

Dear Guido,

R does not support that many connections so this is the reason you get
the error. Unfortunately, it is a hard limit in R so there is no way
to bypass the issue.

However, what you can do is to use a different parallelization
backend. In Bioconductor, we use BiocParallel to do the
parallelization, and we have a specific backend called RedisParam,
which uses the in-memory database Redis to support parallel computing.
The idea is that the Redis server will connect with both master and
workers, so the master does not have to create a connection for each
worker. Therefore, it is able to support unlimited number of workers.
You can find RedisParam here: https://github.com/mtmorgan/RedisParam

Another choice is to use foreach package with doRedis backend. The
idea is the same. Unfortunately, both ways require you to learn how to
run a Redis server. This is the cost we must pay to solve this
problem:(

Best,
Jiefei



On Fri, Apr 8, 2022 at 4:03 PM Guido Kraemer via R-help
<r-help using r-project.org> wrote:
>
> I am trying to run a parallel job on a computer with many CPUs and get
> the following error:
>
>  > library(parallel)
>  > cl <- makeForkCluster(128)
> Error in UseMethod("sendData") :
>    no applicable method for 'sendData' applied to an object of class "NULL"
>
> If I scale down to 100 CPUs it doesn't produce an error. I can reproduce
> this with a self compiled R 4.1.3 on Ubuntu 20.04 and Manjaro, as well
> as the R binaries that come with both distributions.
>
>
> --
> Guido Kraemer
> Max Planck Institute for Biogeochemistry Jena
> Department for Biogeochemical Integration
> Hans-Knöll-Str. 10
> 07745 Jena
> Germany
>
> phone: +49 3641 576293
> e-mail: gkraemer using bgc-jena.mpg.de
>
> ______________________________________________
> R-help using r-project.org mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.