[R-pkg-devel] R package submission - too many threads error

[Ivan Krylov]

Dear Stephen Abrams,

Welcome to R-package-devel!

В Thu, 13 Feb 2025 22:20:50 -0500
Stephen Abrams <stephen.abrams using gmail.com> пишет:

> A secondary worry is that even if I resolve this, there might be
> something else causing threads to spin up.

Instead of using detectCores() [*] and creating cluster objects
yourself, how about letting the user provide a cluster object for you
as a function argument? Yes, it takes slightly more typing for the user,
but on the other hand it lets the user:

 - choose the number of cores for themselves (currently the code seems
   to be ignoring the 'cores' argument)
 - distribute the computation over the network by connecting to the
   machines they know about
 - provide a completely custom, non-PSOCK cluster object that
   'parallel' will nevertheless will work with

Since you're already using doParallel, maybe the right choice is to let
the user call registerDoParallel()?

Determining the right amount of parallelism in your code is a
surprisingly hard problem. Especially on shared computers, a program
naively deciding to use all (or 3/4 of all, or 1/2 of all) processors
may end up working much worse than a purely sequential one [**].

While rendering the vignette in a CRAN package, create a two-process
cluster or set use_parallel = FALSE: CRAN needs the rest of the
processors to check other packages in parallel with yours [***].

Good luck!

-- 
Best regards,
Ivan

[*]
https://github.com/dawdawdo/spect/blob/d48b002332f1a1c2d302afb28e08e7998f416200/R/spect.R#L388

[**]
https://mastodon.social/@henrikbengtsson/113835651303559942

[***]
http://contributor.r-project.org/cran-cookbook/code_issues.html#using-more-than-2-cores