[R] model comparison and Wald-tests (e.g. in lmer)

[Douglas Bates]

On 9/5/05, Thomas Petzoldt <thpe at hhbio.wasser.tu-dresden.de> wrote:
> Dear expeRts,
> 
> there is obviously a general trend to use model comparisons, LRT and AIC
> instead of Wald-test-based significance, at least in the R community.
> I personally like this approach. And, when using LME's, it seems to be
> the preferred way (concluded from postings of Brian Ripley and Douglas
> Bates' article in R-News 5(2005)1), esp. because of problems with the
> d.f. approximation.
> 
> But, on the other hand I found that not all colleagues are happy with the
> resulting AIC/LRT tables and the comparison of multiple models.
> 
> As a compromise, and after a suggestion in Crawley's "Statistical
> computing" one may consider to supply "traditional" ANOVA tables as an
> additional explanation for the reader (e.g. field biologists).
> 
> An example:
> 
> one has fitted 5 models m1..m5 and after:
> 
>  >anova(m1,m2,m3,m4,m5) # giving AIC and LRT-tests
> 
> he selects m3 as the most parsimonious model and calls anova with the
> best model (Wald-test):
> 
>  >anova(m3)             # the additional explanatory table

Whether or not this is a good idea will depend on what the differences
in the models are.  Two mixed-effects models for the same data set can
differ in their random effects specification or in the fixed-effects
specification or both.  The anova() function applied to a single lmer
model produces a sequential anova table for the fixed-effects terms.

If the models differ in the random effects specification - say the
full model has random effects for slope and intercept but the
restricted model has a random effect for the intercept only - then a
Wald test is not appropriate (and it is not provided).  In those cases
I would use a likelihood ratio test and, if necessary, approximate the
p-value by simulating from the null hypothesis rather than assuming a
chi-squared distribution of the test statistic.

Recent versions of the mlmRev package have a vignette with extensive
analysis of the Exam data, including MCMC samples from the posterior
distribution of the parameters.  The marginal posterior distribution
of the variance components are quite clearly skewed (not surprisingly,
they look like scaled chi-squared distributions).  Testing whether
such a parameter could be zero by creating a z-statistic from the
estimate and its standard error is inappropriate.

Changing both the fixed-effects and the random-effects specification
is tricky.  I would try to do such changes in stages, settling on the
fixed-effects terms first then checking the random-effects
specification.
> 
> My questions:
> 
> * Do people outside the S-PLUS/R world still understand us?
> 
> * Is it wise to add such an explanatory table (in particular when the
> results are the same) to make the results more transparent to the reader?
> 
> * Are such additional ANOVA tables *really helpful* or are they (in
> combination with a model comparison) just another source of confusion?
> 
> 
> Thank you!
> 
> Thomas P.
> 
> ______________________________________________
> R-help at stat.math.ethz.ch mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
>