[R] Conservative "ANOVA tables" in lmer

[Greg Snow]

[snip]

 Douglas Bates wrote:
> Hmm - I'm not sure what confidence interval and what number of levels
> you mean there so I can't comment on that method.
> 
> Suppose we go back to Spencer's example and consider if there is a
> signficant effect for the Nozzle factor.  That is equivalent to the
> hypothesis H_0: beta_2 = beta_3 = 0 versus the general alternative.  A
> "p-value" could be formulated from an MCMC sample if we assume that
> the marginal distribution of the parameter estimates for beta_2 and
> beta_3 has roughly elliptical contours and you can evaluate that by,
> say, examining a hexbin plot of the values in the MCMC sample. One
> could take the ellipses as defined by the standard errors and
> estimated correlation or, probably better, by the observed standard
> deviations and correlations in the MCMC sample.  Then determine the
> proportion of (beta_2, beta_3) pairs in the sample that fall outside
> the ellipse centered at the estimates and with that eccentricity and
> scaling factors that passes through (0,0).  That would be an empirical
> p-value for the test.
> 
> I would recommend calculating this for a couple of samples to check on
> the reproducibility.

Here is another thought for an empirical p-value that may be easier to
compute and would require fewer assumptions:

Take the proportion of MCMC samples that fall into each quadrant (++,
+-, -+, --) and use the smallest of these proportions as the p-value
(or the smallest out of a subset of the quadrants for a one-sided
style test).

Think of it this way, if the smallest proportion is greater than
alpha, then any closed curve (ellipse, polygon, even concave polygons)
that includes 1-alpha proportion of the points would need to include
points from all 4 quadrants and therefore any convex curve would have
to include (0,0) which is consistent with the null hypothesis.

On the other hand if there is a quadrant that contains fewer than
alpha percent of the points then there exists at least one confidence
region (possibly concave) that contains 1-alpha proportion of the
points and excludes (0,0) and that entire quadrant, which is
consistent with the alternative that at least one of the betas differs
from 0.

A more conservative p-value would be to take the minimum proportion
and muliply it by 4 (or 2^p for p simultaneous tests) which is the
same idea as multipying by 2 for a 2 sided univariate test and assumes
that the confidence regions would exclude similar proportions of
points in each  direction (central confidence regions rather than
minimum length or other confidence regions).  This seems to me that it
would be over conservative in some cases (since all the proportions
must sum to 1, we don't really have 4 degrees of freedom and a smaller
adjustment factor may still be correct and less conservative).

Some simulations would be a good idea to see if the plain minimum is
to liberal and how conservative the other approach is for common
situations.

This is just my first thoughts on the matter, I have not tested
anything, so any comments or other discussion of this idea is welcome.


-- 
Gregory (Greg) L. Snow Ph.D.
Statistical Data Center
Intermountain Healthcare
greg.snow at intermountainmail.org
(801) 408-8111