[R] Memory limit on Linux?

[Kevin E. Thorpe]

On 08/12/2013 10:18 AM, Stackpole, Chris wrote:
> Greetings, I have a user who is running an R program on two different
> Linux systems. For the most part, they are very similar in terms of
> hardware and 64bit OS. However, they perform significantly different.
> Under one box the program uses upwards of 20GB of ram but fluctuates
> around 15GB of ram and the job runs for a few hours. The second box
> has even more memory available to it, however, the exact same program
> with the exact same data set peaks at 7GB of ram but runs around 5GB
> of ram and takes 3x longer to run the job!
>
> I did some research, and from what I can tell R should just use as
> much memory as it needs on Linux. So a lot of the "help" I found
> online has been windows related information (eg: --max-mem-size ) and
> not very useful to me. I looked at the ulimits and everything looks
> like it should be correct (or at least it is comparable to the
> ulimits on the system that is working correctly). I have also checked
> other tidbits here and there but nothing seems to be of use. I also
> checked that a single user can allocate large quantities of memory
> (eg: Matlab and SAS both were able to allocate 20GB+ of memory) so I
> don't think it is a user-restriction placed by the OS.
>
> The only differences I have found between the two boxes that really
> stands out is that the system that works runs RHEL proper and has R
> compiled but the one that doesn't allocate all of the memory was
> installed via EPEL RPM on CentOS. Compiling R on the CentOS system is
> on the try-this list, but before I spend that time trying to compile
> I thought I would ask a few questions.
>
> 1) Anyone know why I might be seeing this strange behavior? 5-7GB of
> ram is clearly over any 32bit limitation so I don't think it has
> anything to do with that. It could be a RHEL vs CentOS thing, but
> that seems very strange to me.
>
> 2) When I compile from source to test this, is there a specific
> option I should pass to ensure max usage?
>
> Thank you.
>
> Chris Stackpole
>

What does "ulimit -a" report on both of these machines?

-- 
Kevin E. Thorpe
Head of Biostatistics,  Applied Health Research Centre (AHRC)
Li Ka Shing Knowledge Institute of St. Michael's
Assistant Professor, Dalla Lana School of Public Health
University of Toronto
email: kevin.thorpe at utoronto.ca  Tel: 416.864.5776  Fax: 416.864.3016