[R] Number of Cores limited to two in CRAN

[Ravi Varadhan]

Dear Henrik,
Thank you!  This is quite helpful, especially your longer blog post.

Best regards,
Ravi
________________________________
From: Henrik Bengtsson <henrik.bengtsson using gmail.com>
Sent: Sunday, July 2, 2023 4:33 AM
To: Ravi Varadhan <ravi.varadhan using jhu.edu>
Cc: R-Help <r-help using r-project.org>
Subject: Re: [R] Number of Cores limited to two in CRAN


      External Email - Use Caution



Short answer: You don't want to override that limit in your R package.
Don't do it.

Long answer: You'll find the reason for this in the 'CRAN Repository
Policy' (https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcran.r-project.org%2Fweb%2Fpackages%2Fpolicies.html&data=05%7C01%7Cravi.varadhan%40jhu.edu%7Cbf56c90173504853907408db7ad70f08%7C9fa4f438b1e6473b803f86f8aedf0dec%7C0%7C0%7C638238836311718317%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=KDB6%2BqMgn4KrKtVEgWezfRY9isT926wLwZGg5yuDKI4%3D&reserved=0<https://cran.r-project.org/web/packages/policies.html>).
Specifically, the following passage:

"Checking the package should take as little CPU time as possible, as
the CRAN check farm is a very limited resource and there are thousands
of packages. Long-running tests and vignette code can be made optional
for checking, but do ensure that the checks that are left do exercise
all the features of the package.

**If running a package uses multiple threads/cores it must never use
more than two simultaneously: the check farm is a shared resource and
will typically be running many checks simultaneously.**

Examples should run for no more than a few seconds each: they are
intended to exemplify to the would-be user how to use the functions in
the package."

Basically, you can use two cores to demonstrate or validate (e.g. in
package tests) that your code *can* run in parallel, but you must not
use more than that to demonstrate that your code can "run super fast".

Even-longer answer: See my blog post 'Please Avoid detectCores() in
your R Packages' (https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.jottr.org%2F2022%2F12%2F05%2Favoid-detectcores%2F&data=05%7C01%7Cravi.varadhan%40jhu.edu%7Cbf56c90173504853907408db7ad70f08%7C9fa4f438b1e6473b803f86f8aedf0dec%7C0%7C0%7C638238836311718317%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=xPkA6JCccHZcST76lWa6xFYWWKBOZYQCOezF5t4YVPM%3D&reserved=0<https://www.jottr.org/2022/12/05/avoid-detectcores/>)
from 2022-12-05 for even more reasons.

/Henrik

On Sun, Jul 2, 2023 at 9:55 AM Ravi Varadhan via R-help
<r-help using r-project.org> wrote:
>
> This is the specific error messsage from R CMD check --as-cran
>
>
> Error in .check_ncores(length(names)) : 16 simultaneous processes spawned
>   Calls: prepost -> makeCluster -> makePSOCKcluster -> .check_ncores
>   Execution halted
>
>
> Thanks,
> Ravi
>
> ________________________________
> From: Ravi Varadhan
> Sent: Saturday, July 1, 2023 1:15 PM
> To: R-Help <r-help using r-project.org>
> Subject: Number of Cores limited to two in CRAN
>
> Hi,
> I am developing a package where I would like to utilize multiple cores for parallel computing.  However, I get an error message when I run R CMD check --as-cran.
>
> I read that CRAN limits the number of cores to 2.  Is this correct? Is there any way to overcome this limitation?
>
> Thank you,
> Ravi
>
>         [[alternative HTML version deleted]]
>
> ______________________________________________
> R-help using r-project.org mailing list -- To UNSUBSCRIBE and more, see
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fstat.ethz.ch%2Fmailman%2Flistinfo%2Fr-help&data=05%7C01%7Cravi.varadhan%40jhu.edu%7Cbf56c90173504853907408db7ad70f08%7C9fa4f438b1e6473b803f86f8aedf0dec%7C0%7C0%7C638238836311873942%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=KTNwXurNvbEhviVf15RF0la%2FvUVGBW60yuyVVy9Umxc%3D&reserved=0<https://stat.ethz.ch/mailman/listinfo/r-help>
> PLEASE do read the posting guide https://nam02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.r-project.org%2Fposting-guide.html&data=05%7C01%7Cravi.varadhan%40jhu.edu%7Cbf56c90173504853907408db7ad70f08%7C9fa4f438b1e6473b803f86f8aedf0dec%7C0%7C0%7C638238836311873942%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=HvGb20UiUey9O3plynLiRmHQtqpjfrndk2iPfvleT3Q%3D&reserved=0<http://www.r-project.org/posting-guide.html>
> and provide commented, minimal, self-contained, reproducible code.

	[[alternative HTML version deleted]]