[R] loading libraries on MPI cluster

[Dirk Eddelbuettel]

On 23 November 2006 at 16:08, Michela Cameletti wrote:
| Dear R-users,
| we are using library(snow) for computation on a linux cluster with RMPI.
| We have a problem with clusterEvalQ: after launching clusterEvalQ it seems
| loading the required library on each node but if we type a function
| belonging to the loaded package R doesn't find it.
| 
| > library(snow)
| 
| # making cluster with 3 nodes
| > cl <- makeCluster(3, type = "MPI")
| Loading required package: Rmpi
|         3 slaves are spawned successfully. 0 failed.
| 
| #loading library abind on each node
| > clusterEvalQ(cl,library(abind))
| [[1]]
|  [1] "abind"     "EMsspir"   "mvtnorm"   "MASS"      "snow"      "Rmpi"
|  [7] "methods"   "stats"     "graphics"  "grDevices" "utils"     "datasets"
| [13] "base"
| 
| [[2]]
|  [1] "abind"     "EMsspir"   "mvtnorm"   "MASS"      "snow"      "Rmpi"
|  [7] "methods"   "stats"     "graphics"  "grDevices" "utils"     "datasets"
| [13] "base"
| 
| [[3]]
|  [1] "abind"     "EMsspir"   "mvtnorm"   "MASS"      "snow"      "Rmpi"
|  [7] "methods"   "stats"     "graphics"  "grDevices" "utils"     "datasets"
| [13] "base"
| 
| #searching for function "abind" belonging to library "abind"
| > abind
| Error: object "abind" not found
| 
| Is it possible to know which libraries each node has loaded?

Two quick hunches:

i)  do you have the same directory layout on all nodes, i.e. is it say
    /usr/local/lib/R/site-packages/abind everywhere ?  I seem to recall that
    this matters, but it may only have been relevant for binaries and the path.

ii) did you try manually on each machine if abind can in fact be loaded, eg
    $ echo "stopifnot(require(abind,quiet=TRUE)); cat('Ok\n')" | R --slave

I tend to just keep one installation of /usr/local/lib/R/site-packages and
share that via NFS to the nodes in my MPI/PVM/snow cluster as this helps to
keep tabs on i) and ii).

Hope this helps, Dirk

-- 
Hell, there are no rules here - we're trying to accomplish something. 
                                                  -- Thomas A. Edison