[R] Coarsening the Resolution of a Dataset

[jim holtman]

In the first case, 'crops' is an object that is 142MB in size, and
that alone will take up almost 50% of your memory.  You are also
probably generating another object of like size in doing the
operation.  Can you cut down the size of the objects that you need.
What operating system do you have and how much real memory do you
have.  When doing the 'cut' you are also probably creating some
objects of equivalent size.  I would suggest that you work with
smaller objects, or maybe you can put them in a database and do some
manipulation there.

On Thu, Jul 31, 2008 at 4:52 PM, Steve Murray <smurray444 at hotmail.com> wrote:
>
> Please find below my command inputs, subsequent outputs and errors that I've been receiving.
>
>> crops <- read.table("crop2000AD.asc", colClasses = "numeric", na="-9999")
>> str(crops[1:10])
> 'data.frame':   2160 obs. of  10 variables:
>  $ V1 : num  NA NA NA NA NA NA NA NA NA NA ...
>  $ V2 : num  NA NA NA NA NA NA NA NA NA NA ...
>  $ V3 : num  NA NA NA NA NA NA NA NA NA NA ...
>  $ V4 : num  NA NA NA NA NA NA NA NA NA NA ...
>  $ V5 : num  NA NA NA NA NA NA NA NA NA NA ...
>  $ V6 : num  NA NA NA NA NA NA NA NA NA NA ...
>  $ V7 : num  NA NA NA NA NA NA NA NA NA NA ...
>  $ V8 : num  NA NA NA NA NA NA NA NA NA NA ...
>  $ V9 : num  NA NA NA NA NA NA NA NA NA NA ...
>  $ V10: num  NA NA NA NA NA NA NA NA NA NA ...
>
> Don't worry about all the NAs - this is because there is no data available at the poles of the Earth (at the top and bottom of the dataset).
>
>> min.5 <- 5/60
>> dim(crops)
> [1] 2160 4320
>> n <- 2160*4320
>
>> memory.limit()
> [1] 382.9844
>> crops <- cbind(interval=seq(0, by=min.5, length=n), value=runif(n))
> Error: cannot allocate vector of size 142.4 Mb
> In addition: Warning messages:
> 1: In cbind(interval = seq(0, by = min.5, length = n), value = runif(n)) :
>  Reached total allocation of 382Mb: see help(memory.size)
> 2: In cbind(interval = seq(0, by = min.5, length = n), value = runif(n)) :
>  Reached total allocation of 382Mb: see help(memory.size)
> 3: In cbind(interval = seq(0, by = min.5, length = n), value = runif(n)) :
>  Reached total allocation of 382Mb: see help(memory.size)
> 4: In cbind(interval = seq(0, by = min.5, length = n), value = runif(n)) :
>  Reached total allocation of 382Mb: see help(memory.size)
>
> But seems to run when 'value = runif(n)' is excluded....
>
>> crops <- cbind(interval=seq(0, by=min.5, length=n))
>> head(crops)
>       interval
> [1,] 0.00000000
> [2,] 0.08333333
> [3,] 0.16666667
> [4,] 0.25000000
> [5,] 0.33333333
> [6,] 0.41666667
>> str(crops[1:10])
>  num [1:10] 0.0000 0.0833 0.1667 0.2500 0.3333 ...
>
>> breaks <- c(seq(min(crops[,'interval']), max(crops[, 'interval']), by=0.5), Inf)
>> head(breaks)
> [1] 0.0 0.5 1.0 1.5 2.0 2.5
>> str(breaks)
>  num [1:1555201] 0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 ...
>
>> myCuts <- cut(crops[, 'interval'], breaks, include.lowest=TRUE)
> Error: evaluation nested too deeply: infinite recursion / options(expressions=)?
> In addition: Warning messages:
> 1: In formatC(breaks, digits = dig, width = 1) :
>  Reached total allocation of 382Mb: see help(memory.size)
> 2: In formatC(breaks, digits = dig, width = 1) :
>  Reached total allocation of 382Mb: see help(memory.size)
>>
>
> This is as far as I've got because of the above errors I encounter. Any pointers and advice, or if I'm doing something obviously wrong, then please let me know.
>
> Thanks again for your help.
>
> Steve
>
> _________________________________________________________________
> 100's of Nikon cameras to be won with Live Search
> http://clk.atdmt.com/UKM/go/101719808/direct/01/



-- 
Jim Holtman
Cincinnati, OH
+1 513 646 9390

What is the problem you are trying to solve?