[R] Memory filling up while looping

[Duncan Murdoch]

On 12-12-20 6:26 PM, Peter Meissner wrote:
> Hey,
>
> I have an double loop like this:
>
>
> chunk <- list(1:10, 11:20, 21:30)
> for(k in 1:length(chunk)){
> 	print(chunk[k])
> 	DummyCatcher <- NULL
> 	for(i in chunk[k]){
> 		print("i load something")
> 		dummy <- 1
> 		print("i do something")
> 		dummy <- dummy + 1
> 		print("i do put it together")
> 		DummyCatcher = rbind(DummyCatcher, dummy)
> 	}
> 	print("i save a chunk and restart with another chunk of data")
> }
>
> The problem now is that with each 'chunk'-cycle the memory used by R
> becomes bigger and bigger until it exceeds my RAM but the RAM it needs
> for any of the chunk-cycles alone is only a 1/5th of what I have overall.
>
> Does somebody have an idea why this behaviour might occur? Note that all
> the objects (like 'DummyCatcher') are reused every cycle so that I would
> assume that the RAM used should stay about the same after the first
> 'chunk' cycle.

You should pre-allocate your result matrix.  By growing it a few rows at 
a time, R needs to do this:

allocate it
allocate a bigger one, copy the old one in
delete the old one, leaving a small hole in memory
allocate a bigger one, copy the old one in
delete the old one, leaving a bigger hold in memory, but still too small 
to use...

etc.

If you are lucky, R might be able to combine some of those small holes 
into a bigger one and use that, but chances are other variables will 
have been created there in the meantime, so the holes will go mostly 
unused.  R never moves an object during garbage collection, so if you 
have fragmented memory, it's mostly wasted.

If you don't know how big the final result will be, then allocate large, 
and when you run out, allocate bigger.  Not as good as one allocation, 
but better than hundreds.

Duncan Murdoch

>
>
> Best, Peter
>
>
> SystemInfo:
>
> R version 2.15.2 (2012-10-26)
> Platform: x86_64-w64-mingw32/x64 (64-bit)
> Win7 Enterprise, 8 GB RAM
>
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>