[R] error loading huge .RData

[Liaw, Andy]

Patrick,

I appreciate your comments, and practice everything that you preach.
However, that workspace image contains only 2~3 R objects: the input and
output of a single R command.  I knew there could be problems, so I've
stripped it down to the bare minimum.  Yes, I also kept the commands in a
script.  That single command (in case you want to know: a random forest run
with 4000 rows and nearly 7000 variables) took over 3 days to run.  There's
not a whole lot I can do here when the data is so large.

Andy

> -----Original Message-----
> From: Patrick Connolly [mailto:P.Connolly at hortresearch.co.nz]
> Sent: Tuesday, April 23, 2002 7:18 PM
> To: andy_liaw at merck.com
> Subject: Re: [R] error loading huge .RData
> 
> 
> According to Liaw, Andy:
> |> 
> |> Dear R-help,
> |> 
> |> I've run into a problem loading .RData:  I was running a 
> large computation,
> |> which supposedly produce a large R object.  At the end of 
> the session, I did
> |> a save.image() and then quit.  The .RData has size 
> 613,249,399 bytes.  Now I
> |> can't get R to load this .RData file.  Whenever I tried, I 
> get "Error:
> |> vector memory exhausted (limit reached)".  I tried adding
> |> "--min-vsize=1000M", but that didn't help.  I also tried R 
>  --vanilla and
> |> then attach(".RData"), same error.
> |> 
> |> From what I can see, the file is not corrupted.  How can I 
> get R to load it?
> 
> What would you look for to see if it's corrupted?  That's a big file
> to find it in.
> 
> I've never used a .RData file greater than 100 Mb, so my experience is
> not directly comparable, but I feel uneasy having too much in one
> file.  It's a safer practice to make sure you keep a file of the
> commands you used to create your R objects so that it's easy (even if
> not quick) to reproduce them.  It's also good to keep not too many
> objects in one Data file.  I had only one occasion when I had a
> corrupt .RData file, but it was easy to recover.  The bigger the file,
> the more opportunity for corruption to happen.  How many objects would
> there have been in your .RData file?
> 
> 
> 
> Might not be much help for now, but it might help in the future.
> 
> 
> best
> 
> -- 
> *************************************************************
>    ___      Patrick Connolly
>  {~._.~}    HortResearch             Great minds discuss ideas;
>  _( Y )_    Mt Albert                Average minds discuss events; 
> (:_~*~_:)   Auckland                 Small minds discuss people.
>  (_)-(_)    New Zealand                                    .... Anon
>             Ph: +64-9 815 4200 x 7188
> *************************************************************
> 
> 
> ______________________________________________________
> The contents of this e-mail are privileged and/or confidential to the
> named recipient and are not to be used by any other person and/or
> organisation. If you have received this e-mail in error, 
> please notify 
> the sender and delete all material pertaining to this e-mail.
> ______________________________________________________
> 


------------------------------------------------------------------------------
Notice:  This e-mail message, together with any attachments, contains information of Merck & Co., Inc. (Whitehouse Station, New Jersey, USA) that may be confidential, proprietary copyrighted and/or legally privileged, and is intended solely for the use of the individual or entity named in this message.  If you are not the intended recipient, and have received this message in error, please immediately return this by e-mail and then delete it.

==============================================================================

-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-
r-help mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html
Send "info", "help", or "[un]subscribe"
(in the "body", not the subject !)  To: r-help-request at stat.math.ethz.ch
_._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._