[R] Warning messages while parallel computing

[Bill Dunlap]

The warnings come from the garbage collector, which may be called from
almost anywhere.  It is possible that the file that is sourced causes
the problem, but if you don't call parallel::stopCluster before
removing the cluster object you will get those warnings.

> cl <- parallel::makeCluster(3, type="PSOCK")
> invisible(gc())
> rm(cl)
> invisible(gc())
Warning messages:
1: In .Internal(gc(verbose, reset, full)) :
  closing unused connection 6 (<-Bill-T490:11216)
2: In .Internal(gc(verbose, reset, full)) :
  closing unused connection 5 (<-Bill-T490:11216)
3: In .Internal(gc(verbose, reset, full)) :
  closing unused connection 4 (<-Bill-T490:11216)
>
>
> cl <- parallel::makeCluster(3, type="PSOCK")
> invisible(gc())
> parallel::stopCluster(cl)
> invisible(gc())
> rm(cl)
> invisible(gc())
>

The fact that he got 8 warnings when playing a cluster of size 8 makes
me suspect that omitting stopCluster is the problem.

-Bill

On Thu, Mar 4, 2021 at 10:12 AM Henrik Bengtsson
<henrik.bengtsson using gmail.com> wrote:
>
> I don't think 'parallel' is to blame in this case. Those warnings:
>
> Warning in for (i in seq_len(Ne + echo)) { :
>   closing unused connection 19
>
> come from base::source()
> [https://github.com/wch/r-source/blob/9caddc1eaad1f480283f1e98af34a328699d1869/src/library/base/R/source.R#L166-L244].
>
> Unless there's a bug in source() that leaves connections open, which
> is unlikely, I think there's something in the 'xx.R' script that opens
> a connection but doesn't close it.  Possibly multiple times.  A good
> check is to see if the same warnings are produced when calling
> source("xx.R") sequentially in a for() loop or an lapply() call.
>
> Hope this helps,
>
> Henrik
>
> On Thu, Mar 4, 2021 at 9:58 AM Bill Dunlap <williamwdunlap using gmail.com> wrote:
> >
> > To avoid the warnings from gc(), call parallel::stopCluster(cl) before
> > removing or overwriting cl.
> >
> > -Bill
> >
> > On Thu, Mar 4, 2021 at 1:52 AM Shah Alam <dr.alamsolangi using gmail.com> wrote:
> > >
> > > Hello everyone,
> > >
> > > I am using the "parallel" R package for parallel computation.
> > >
> > > Code:
> > >
> > >    # set number of cores
> > >     cl <- makeCluster(8, type = "PSOCK")  # Mac/Linux need to set as "FORK"
> > >
> > >       # pass functions and objects to the cluster environment and set seed
> > >       # all the items exported need to stay in the global environment!!
> > >       clusterCall(cl, function() { source("xx.R" )})
> > >       clusterExport(cl, list("a", "b", "c", "d",
> > >                              "5"))
> > >       clusterSetRNGStream(cl, 1)
> > >
> > > While parallel processing, I receive the following warning signs.  Do I
> > > need to ignore these signs or do they potentially slow the whole process?
> > >
> > >  *  Warning signs:*
> > > Warning in for (i in seq_len(Ne + echo)) { :
> > >   closing unused connection 19
> > > Warning in for (i in seq_len(Ne + echo)) { :
> > >   closing unused connection 18
> > > Warning in for (i in seq_len(Ne + echo)) { :
> > >   closing unused connection 17
> > > Warning in for (i in seq_len(Ne + echo)) { :
> > >   closing unused connection 16
> > > Warning in for (i in seq_len(Ne + echo)) { :
> > >   closing unused connection 15
> > > Warning in for (i in seq_len(Ne + echo)) { :
> > >   closing unused connection 14
> > > Warning in for (i in seq_len(Ne + echo)) { :
> > >   closing unused connection 13
> > > Warning in for (i in seq_len(Ne + echo)) { :
> > >   closing unused connection 12
> > >
> > > Best regards,
> > > Shah Alam
> > >
> > >         [[alternative HTML version deleted]]
> > >
> > > ______________________________________________
> > > R-help using r-project.org mailing list -- To UNSUBSCRIBE and more, see
> > > https://stat.ethz.ch/mailman/listinfo/r-help
> > > PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> > > and provide commented, minimal, self-contained, reproducible code.
> >
> > ______________________________________________
> > R-help using r-project.org mailing list -- To UNSUBSCRIBE and more, see
> > https://stat.ethz.ch/mailman/listinfo/r-help
> > PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> > and provide commented, minimal, self-contained, reproducible code.