[R] Comparative evaluation studies of R packages and functions. Was: Re: a proposal regarding documentation

[Ravi Varadhan]

Hi Spencer,

You have brought up an important topic.  I agree with you that a comparative evaluation of R packages providing similar capabilities is a worthwhile task.  Your example of "gam" is a very good one.  Let me give you a couple of other examples of comparative evaluation that I am working on: (1) optimization - John Nash and I are currently doing this for basic "optimization" packages. By basic optimization, I am referring to the optimization of smooth functions, with box-constarints or no constraints, (2) smoothing or non-parametric regression. I am currently comparing some smoothers for their ability to accurately estimate functions and their first 2 derivatives from discretely sampled data (in terms of mean integrated squared error).  

Comparative studies, if conducted rigorously and objectively by independent folks not associated with packages, are likely to be very useful to R users. The new R Journal would be a natural avenue to disseminate the results of comparative studies.  I can think of some criteria for comparing packages or functions (in different packages) that are meant for essentially similar tasks (e.g. the 2 gam functions):

1.  Ease of use and misuse
2.  Documenation and examples
3.  Transparency of coding (this would give more weight to "all-R" functions and packages as opposed to C/Fortran interfaces)
4.  Theoretical foundation
5.  Performance on simulated examples designed to highlight potential trouble spots
6.  Actual performance on real problems.
7.  Identification of "niche" areas for each package, i.e. things that no other package can do or do as well.

Best,
Ravi. 
____________________________________________________________________

Ravi Varadhan, Ph.D.
Assistant Professor,
Division of Geriatric Medicine and Gerontology
School of Medicine
Johns Hopkins University

Ph. (410) 502-2619
email: rvaradhan at jhmi.edu


----- Original Message -----
From: spencerg <spencer.graves at prodsyse.com>
Date: Sunday, June 14, 2009 6:08 pm
Subject: Re: [R] a proposal regarding documentation
To: Peter Flom <peterflomconsulting at mindspring.com>
Cc: r-help at stat.math.ethz.ch, Philippe Grosjean <phgrosjean at sciviews.org>, John Sorkin <jsorkin at grecc.umaryland.edu>, Patrick Burns <pburns at pburns.seanet.com>


>       I agree that the documentation is a primary source, made now 
> more 
>  accessible with the availability of the RSiteSearch package that 
> allows 
>  more structures searches of the help pages in contributed packages 
> than 
>  previously available.  On the other hand, it can sometimes be 
> difficult 
>  to get the attention and support of someone who can make necessary 
>  improvements to the help pages. 
>  
>  
>        Moreover, the R Wiki should be the primary place for comparing 
> 
>  similar capabilities in different packages.  For example, a function 
> 
>  named "gam" appears in two different packages, with similar 
>  functionality but subtle incompatibilities.  The R Wiki provides a 
> forum 
>  for an open-ended debate about which is best for which purpose.  The 
> 
>  Wiki model has proven its worth by allowing anyone to write anything 
> 
>  they feel moved to write (in the language of their choice).  If 
>  something is disputed, there are referees to lock particular Wiki 
> pages, 
>  after which point the dispute can continue but without burdening the 
> 
>  casual reader with the details.  The success of Wikipedia shows a 
>  context in which anarchy is almost uniformly superior to prior 
>  censorship.  The current "task views" provide a very limited overview 
> of 
>  contributed capabilities but the way those are currently managed make 
> 
>  them entirely too static and unsuited to this kind of thing.  I plan 
> 
>  eventually to start contributing to the R Wiki, making comparisons of 
> 
>  similar capabilities in different packages and inviting the package 
>  maintainers to correct or expand on anything I write.  However, I 
> have 
>  other tasks I need to complete before I can start working on that. 
>  
>  
>        Best Wishes,
>        Spencer Graves
>  
>  
>  Peter Flom wrote:
>  > I certainly don't have anything against the WIKI, but I think that 
> the documentation
>  > is where the action is, especially for newbies.  It's the  natural 
> first step
>  > when you want to learn about a function or when you  get an error 
> message you 
>  > don't understand.
>  >
>  > Peter
>  >
>  > Peter L. Flom, PhD
>  > Statistical Consultant
>  > www DOT peterflomconsulting DOT com
>  >
>  > ______________________________________________
>  > R-help at r-project.org mailing list
>  > 
>  > PLEASE do read the posting guide 
>  > and provide commented, minimal, self-contained, reproducible code.
>  >
>  >
>  
>  ______________________________________________
>  R-help at r-project.org mailing list
>  
>  PLEASE do read the posting guide 
>  and provide commented, minimal, self-contained, reproducible code.