[R] sparse matrix, rnorm, malloc
g l
glarowe at indiana.edu
Sat Jun 10 19:53:29 CEST 2006
Hi,
I'm Sorry for any cross-posting. I've reviewed the archives and could
not find an exact answer to my question below.
I'm trying to generate very large sparse matrices (< 1% non-zero
entries per row). I have a sparse matrix function below which works
well until the row/col count exceeds 10,000. This is being run on a
machine with 32G memory:
sparse_matrix <- function(dims,rnd,p) {
ptm <- proc.time()
x <- round(rnorm(dims*dims),rnd)
x[((abs(x) - p) < 0)] <- 0
y <- matrix(x,nrow=dims,ncol=dims)
proc.time() - ptm
}
When trying to generate the matrix around 20,000 rows/cols on a
machine with 32G of memory, the error message I receive is:
R(335) malloc: *** vm_allocate(size=3200004096) failed (error code=3)
R(335) malloc: *** error: can't allocate region
R(335) malloc: *** set a breakpoint in szone_error to debug
R(335) malloc: *** vm_allocate(size=3200004096) failed (error code=3)
R(335) malloc: *** error: can't allocate region
R(335) malloc: *** set a breakpoint in szone_error to debug
Error: cannot allocate vector of size 3125000 Kb
Error in round(rnorm(dims * dims), rnd) : unable to find the argument
'x' in selecting a method for function 'round'
* Last error line is obvious. Question: on machine w/32G memory, why
can't it allocate a vector of size 3125000 Kb?
When trying to generate the matrix around 30,000 rows/cols, the error
message I receive is:
Error in rnorm(dims * dims) : cannot allocate vector of length 900000000
Error in round(rnorm(dims * dims), rnd) : unable to find the argument
'x' in selecting a method for function 'round'
* Last error line is obvious. Question: is this 900000000 bytes?
kilobytes? This error seems to be specific now to rnorm, but it
doesn't indicate the length metric (b/Kb/Mb) as it did for 20,000
rows/cols. Even if this Mb, why can't this be allocated on a machine
with 32G free memory?
When trying to generate the matrix with over 50,000 rows/cols, the
error message I receive is:
Error in rnorm(n, mean, sd) : invalid arguments
In addition: Warning message:
NAs introduced by coercion
Error in round(rnorm(dims * dims), rnd) : unable to find the argument
'x' in selecting a method for function 'round'
* Same.
Why would it generate different errors in each case? Code fixes? Any
simple ways to generate sparse matrices which would avoid above
problems?
Thanks in advance,
Gavin
More information about the R-help
mailing list