[R] analysis of large data set

[Prof Brian Ripley]

On Fri, 16 Nov 2007, sj wrote:

> All,
>
> I am working with a large data set (~ 450,000 rows by 34 columns) I am
> trying to fit a regression model (I have tried to use several procedures psm
> (Design package) lm, glm). However whenever I try to fit the model I get the
> following error:
>
>
> Error: cannot allocate vector of size 1.1 Gb
>
> Here are the specs of the machine and version of R I am using
>
> Windows Server 2003 R2 Enterprise x64 Service Pack 2
>
> Intel Pentium D 3.00 Ghz
> 3.93 GB Ram
>
> R 2.6.0
>
> when I type the command
>
> memory.limit()
> I get:
> 3583.875
>
> I assume that means that I have about 3.5 GB at my disposal  so I am
> confused why I can't allocate a vector of 1.1 GB. Any suggestions on what to
> do.

See ?"Memory-limits", which explains this in detail.

What is strange is that no one has ever thanked us for finding out 
(despite most Microsoft documentation) that you can get up to 3.5Gb in a 
32-bit process on certain 64-bit versions of Windows and enabling you to 
use it in recent versions of R.

-- 
Brian D. Ripley,                  ripley at stats.ox.ac.uk
Professor of Applied Statistics,  http://www.stats.ox.ac.uk/~ripley/
University of Oxford,             Tel:  +44 1865 272861 (self)
1 South Parks Road,                     +44 1865 272866 (PA)
Oxford OX1 3TG, UK                Fax:  +44 1865 272595