[R] Running out of memory when using lapply

[Seth Falcon]

Hi Kamila,

"Kamila Naxerova" <kamila.naxerova at childrens.harvard.edu> writes:

> Hi all!
>
> I'm afraid I programmed something totally non-sensical and inefficient, 
> but I can't figure out how to do it better.
>
> I have a list of ~ 40 000 characters. I want to take each element at a 
> time, map it to a large data frame with 
> hit=which(data.frame$column==elementFromList), then compute some 
> statistic on data.frame[hit,] and return a result that consists of 
> either 1) a list of integers or 2) a character.
>
> res=lapply(listof40000,myfunction,dataframeToSearchIn)
>
> On a small scale, this works and returns something like
>
> str(res)
> [[1]]
> [1] "UNIQUE"
> [[2]]
> [1]   405   406   407 16351
> [[3]]
> [1] "REMOVE"
> [[4]]
> [1] "REMOVE"
>
> If I try this with the entire 40 000 character list, though, I get the 
> "Reached total allocation of 1022Mb: see help(memory.size)" error message.
>
> Can someone please give me a hint how to solve this problem correctly? 
> THANKS!

One thing you might try is not running the entire 40K list at once.
Perhaps try breaking it into 4 10K lists, running each, and combining
the results.  This may get you around the allocation problem.

Another thing would be to find a system with more RAM (also read the
FAQ regarding ways to make the most amount of RAM available to R on
Windows, is that where you are?).

+ seth