[R] Parsing large amounts of csv data with limited RAM

[Jeff Newmiller]

You seem to want your cake and eat it too. Not unexpected, but you may have your work cut out to learn about the price of having it all.

Plotting: pretty silly to stick with gigabytes of data in your plots. Some kind of aggregation seems required here, with the raw data being a stepping stone to that goal.

Loading: if you don't have RAM, buy more or use one of the disk-based solutions. There are proprietary solutions for a fee, and there are packages like ff. When I have dealt with large data sets I have used sqldf or RODBC (which I think works best for read-only access), so I cannot advise you on ff.
---------------------------------------------------------------------------
Jeff Newmiller                        The     .....       .....  Go Live...
DCN:<jdnewmil at dcn.davis.ca.us>        Basics: ##.#.       ##.#.  Live Go...
                                      Live:   OO#.. Dead: OO#..  Playing
Research Engineer (Solar/Batteries            O.O#.       #.O#.  with
/Software/Embedded Controllers)               .OO#.       .OO#.  rocks...1k
--------------------------------------------------------------------------- 
Sent from my phone. Please excuse my brevity.

On July 14, 2015 3:21:42 PM PDT, "Dupuis, Robert" <dupuis at beaconpower.com> wrote:
>I'm relatively new to using R, and I am trying to find a decent
>solution for my current dilemma.
>
>Right now, I am currently trying to parse second data from a 7 months
>of CSV data. This is over 10GB of data, and I've run into some "memory
>issues" loading them all into a single dataset to be plotted. If
>possible, I'd really like to keep both the one second resolution, and
>all 100 or so columns intact to make things easier on myself.
>
>The problem I have is that the machine that is running this script only
>has 8GB of RAM. I've had issues parsing files with lapply, and some
>sort of csv reader. So far I've tried read.csv, readr.read_table, and
>data.table.fread with only fread having any sort of memory management
>(fread seems to crash on me however). The basic approach I am using is
>as follows:
>
># Get the data
>files = list.files(pattern="*.csv")
>set <- lapply(files, function(x) fread(x, header = T, sep = ','))
>#replace fread with something that can parse csv data
>
># Handle the data (Do my plotting down here)
>...
>
>These processes work with smaller data sets, but I would like to in a
>worse case scenario be able to parse through 1 year data which would be
>around 20GB.
>
>Thank you for your time,
>Robert Dupuis
>
>______________________________________________
>R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see
>https://stat.ethz.ch/mailman/listinfo/r-help
>PLEASE do read the posting guide
>http://www.R-project.org/posting-guide.html
>and provide commented, minimal, self-contained, reproducible code.