[R] cannot allocate vector of size... restructuring suggestion please...

[Prof Brian Ripley]

On Mon, 15 Dec 2008, tsunhin wong wrote:

> Dear R Users,
>
> I was running some data analysis scripts and ran into this error:
>
> Error: cannot allocate vector of size 27.6 Mb
>
> Doing a "memory.size(max=TRUE)" will give me:
> [1] 1506.812
>
> The current situation is:
> I'm working on a Windows Vista 32bit laptop with 4GB RAM (effectively
> 3GB I assume...)

Why do you assume?  Reading the rw-FAQ should make clear that a 2GB 
address limit is a lot more plausible given the data you presented.

> I have a data file of 450Mb loaded into R and have around 1500
> data.frames floating in the global space as my data source.
> The way I run this analysis:
> I call a patch processing & procedure script

'batch'?

>>> it retrieves 4 lists of info (each around 400x100) from an index data.frame, and then it calls another script to retrieve info from the corresponding data.frames on the 4 lists in the global space
>>> through calling another script, about 1000x3 will be retrieved by another script
>>> the 1000x3 will be passed to a third script expanded to 20001x3, and only 20001x1 will be used
>>> 20001x1 will accumulate into a matrix of up to 20001x1500 (number of data frames / trials), say I have to divide the trials into 2 groups and do a comparison, then that's processing of 2 matrices of size 20001x750
> But the allocation error stopped the script after script has processed
> around 280 data frames, i.e. made the first matrix up to 20001x280...
>
> I know running the analysis should possibly be achieved by
> restructuring my script a little bit, but I have no idea where to
> start with to try...

Run the script on each 100 data frames in turn.  You can even use R to set 
up scripts for Rscript to run.

> Also, I have no idea about Garbage Collection ability or memory
> recycle / reuse ability in R and I think some memory may have been
> lost in the middle of the process, and it may be possible to put them
> back to the system for R to make use of...

Well, do read the help page ?"Memory-limits' as well as the appropriate 
FAQ.  (And also the parts of the posting guide about doing your homework 
before posting.  Since this list is a public record, we do need to correct 
false claims for the sake of future enquirers doing their homework before 
posting.)

> Please advise me to let me to find out the most efficient way of
> eliminating the error...
> Thanks so much!
>
> Regards,
>
>
>      John
>
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>

-- 
Brian D. Ripley,                  ripley at stats.ox.ac.uk
Professor of Applied Statistics,  http://www.stats.ox.ac.uk/~ripley/
University of Oxford,             Tel:  +44 1865 272861 (self)
1 South Parks Road,                     +44 1865 272866 (PA)
Oxford OX1 3TG, UK                Fax:  +44 1865 272595