[R] memory error

[Matthew Dowle]

> someone else on this list may be able to give you a ballpark estimate
> of how much RAM this merge would require.

I don't have an absolute estimate, but try data.table::merge, as it needs 
less
working memory than base::merge.

20 million rows of 5 columns isn't beyond 32bit :
   (1*4 + 4*8)*19758564/1024^3 = 0.662GB

Also try sqldf to do the join.

Matthew


"Sharpie" <chuck at sharpsteen.net> wrote in message
news:1270102758449-1747733.post at n4.nabble.com...
>
>
> Janet Choate-2 wrote:
>>
>> Thanx for clarification on stating my problem, Charlie.
>>
>> I am attempting to merge to files, i.e.:
>> hi39 = merge(comb[,c("hillID","geo")], hi.h39, by=c("hillID"))
>>
>> if this is relevant or helps to explain:
>> the file 'comb' is 3 columns and 1127 rows
>> the file 'hi.h39' is 5 columns and 19758564 rows
>>
>> i started a new clean R session in which i was able to read those 2 files
>> in, but get the following error when i try to merge them:
>>
>> R(2175) malloc: *** mmap(size=79036416) failed (error code=12)
>> *** error: can't allocate region
>> *** set a breakpoint in malloc_error_break to debug
>> R(2175) malloc: *** mmap(size=79036416) failed (error code=12)
>> *** error: can't allocate region
>> *** set a breakpoint in malloc_error_break to debug
>> R(2175) malloc: *** mmap(size=158068736) failed (error code=12)
>> *** error: can't allocate region
>> *** set a breakpoint in malloc_error_break to debug
>> R(2175) malloc: *** mmap(size=158068736) failed (error code=12)
>> *** error: can't allocate region
>> *** set a breakpoint in malloc_error_break to debug
>> R(2175) malloc: *** mmap(size=158068736) failed (error code=12)
>> *** error: can't allocate region
>> *** set a breakpoint in malloc_error_break to debug
>> Error: cannot allocate vector of size 150.7 Mb
>>
>> so the final error is "Cannot allocate vector of size 150.7 Mb", as
>> suggested when R runs out of memory.
>>
>> i am running R version 2.9.2, on mac os X 10.5 - leopard.
>>
>> any suggestion on how to increase R's memory on a mac?
>> thanx for any much needed help!
>> Janet
>>
>
> Ah, so it is indeed a shortage of memory problem.  With R 2.9.2, you are
> likely running a 32 bit version of R which will be limited to accessing at
> most 4 GB of RAM. You may want to try the newest version of R, 2.10.1, as
> it includes a 64 bit version that will allow you to access significantly
> more memory- provided you have the RAM installed on your system.
>
> I'm not too hot on memory usage calculation, but someone else on this list
> may be able to give you a ballpark estimate of how much RAM this merge
> would
> require.  If it turns out to be a ridiculous amount, you will need to
> consider breaking the merge up into chunks or finding an out-of-core (i.e.
> not dependent on RAM for storage) merge tool.
>
> Hope this helps!
>
> -Charlie
>
> -----
> Charlie Sharpsteen
> Undergraduate-- Environmental Resources Engineering
> Humboldt State University
> -- 
> View this message in context:
> http://n4.nabble.com/memory-error-tp1747357p1747733.html
> Sent from the R help mailing list archive at Nabble.com.
>