[R] Reading large files

[Matthew Dowle]

I agree with Jim.  The term "do analysis" is almost meaningless, the posting 
guide makes reference to statements such as that. At least he tried to 
define large, but inconsistenly (first of all 850MB, then changed to 
10-20-15GB).

> Satish wrote: "at one time I will need to load say 15GB into R"

Assuming the user is always right then, here is some information :

R has been 64bit on unix for a very long time (over a decade).  64bit R is 
also available for Win64.
It uses as much RAM you install on the box, e.g. 64GB.
Yes R users do that,  and they've been doing that for years and years.
The data.table package was mainly designed for 64bit,  although its a point 
of consternation when people think thats all its useful for.
If you don't have the hardware, then you can rent the time on EC2. There are 
tools and packages to make that easy e.g. pre-built images you can just use. 
Look at the HPC task view. Search the archives. Don't miss Biocep at 
http://biocep-distrib.r-forge.r-project.org/doc.html.

Albert Einstein said "A clever person solves a problem. A wise person avoids 
it.".   So an option for you is to be wise and move to 64bit.


"jim holtman" <jholtman at gmail.com> wrote in message 
news:644e1f321002050513y242304der84b5674930b54698 at mail.gmail.com...
> Where should be shine it?  No information provided on operating
> system, version, memory, size of files, what you want to do with them,
> etc.  Lot of options: put it in a database, read partial file (lines
> and/or columns), preprocess, etc.  Your option.
>
> On Fri, Feb 5, 2010 at 8:03 AM, Satish Vadlamani
> <SATISH.VADLAMANI at fritolay.com> wrote:
>>
>> Folks:
>> Can anyone throw some light on this? Thanks.
>> Satish
>>
>>
>> -----
>> Satish Vadlamani
>> --
>> View this message in context: 
>> http://n4.nabble.com/Reading-large-files-tp1469691p1470169.html
>> Sent from the R help mailing list archive at Nabble.com.
>>
>> ______________________________________________
>> R-help at r-project.org mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-help
>> PLEASE do read the posting guide 
>> http://www.R-project.org/posting-guide.html
>> and provide commented, minimal, self-contained, reproducible code.
>>
>
>
>
> -- 
> Jim Holtman
> Cincinnati, OH
> +1 513 646 9390
>
> What is the problem that you are trying to solve?
>