[R] Reasons to Use R (no memory limitations :-))

[charles loboz]

This thread discussed R memory limitations, compared handling with S and SAS. Since I routinely use R to process multi-gigababyte sets on computers with sometimes 256mb of memory - here are some comments on that. 

Most memory limitations vanish if R is used with any relational database. [My personal preference is SQLite (RSQLite packaga)  because of speed and no-admin (used in embedded mode)]. The comments below apply to any relational database, unless otherwise stated.

Most people appear to think about database tables as dataframes - that is to store and load the _whole_ dataframe in one go - probably because appropriate function names are suggesting this approach. Also, it is a natural mapping. This is convenient if the data set can fit fully in memory - but limits the size of the data set the same way as without using the database.

However, using SQL language directly one can expand the size of the data set R is capable of operating on - we just have to stop treating database tables as 'atomic'. For example, assume we have a set of several million patients and want to analyze some specific subset - the following SQL statement 
  SELECT * FROM patients WHERE gender='M" AND AGE BETWEEN 30 AND 35
will result in bringing to R much smaller dataframe than selection of the whole table. [Also, such subset selection may take _less_time_ then selecting from the total dataframe - assuming the table is properly indexed]. 
Also, direct SQL statements can be used to pre-compute some characteristics internally in the database and bring only the summaries to R:
 SELECT AVG(age) FROM patients GROUP BY gender
will bring a data frame of two rows only.

Admittedly, if the data set is really large and we cannot operate on its subsets, the above does not help. Though I do not believe that this would the the majority of the situations. 

Naturally, going for a 64bit system with enough memory will solve some problems without using the database -  but not all of them. Relational databases can be very efficient at selecting subsets as they do not have to do linear scans [when the tables are indexed] - while R has to do a linear scan every time(??? I did not look up the source code of R - please correct me if I am wrong). Two other areas where a database is better than R, especially for large data sets:
 - verification of data correctness for individual points [a frequent problem with large data sets]
 - combining data from several different types of tables into one dataframe

In summary: using SQL from R allows to process extremely large data sets in a limited memory, sometimes even faster then if we had a large memory and kept our data set fully in it. Relational database perfectly complements R capabilities.