[R] which alternative tests instead of AIC/BIC for choosing models

[Ben Bolker]

> > Dear R Users,
> >
> > I am looking for an alternative to AIC or BIC to choose model parameters.
> > This is somewhat of a general statistics question, but I ask it in this
> > forum as I am looking for a R solution.
> >
> > Suppose I have one dependent variable, y, and two independent variables,
> > x1 an x2.
> >
> > I can perform three regressions:
> > reg1: y~x1
> > reg2: y~x2
> > reg3: y~x1+x2
> >
> > The AIC of reg1 is 2000, reg2 is 1000 and reg3 is 950. One would,
> > presumably, conclude that one should use both x1 and x2.  However, the
> > R^2's are quite different: R^2 of reg1 is 0.5%, reg2 is 95% and reg3 is
> > 95.25%. Knowing that, I would actually conclude that x1 adds litte and
> > should probably not be used.
> >
> > There is the overall question of what potentially explains this outcome,
> > i.e. the reduction in AIC in going from reg2 to reg3 even though R^2 does
> > not materially improve
> > with the addition of x1 to reg 2 (to get to reg3). But that is more of a
> > generic statistics issue and not my question here.
> >

  I know you didn't ask the "generic statistics question", but
I think it's fairly important.  I suspect the reason that
you're getting (what you consider to be) a "spurious" result
that includes x1, or equivalently that your delta-AICs are
so big, is that you have a huge data set.  Lindsey (p. 15)
talks a bit about calibration that changes with the size of 
the data set.

  Model 3 will very probably give you better predictive power
than model 2.  If you want to select on the basis of improvement
in R^2, why not just do that?

  Ben Bolker

Lindsey, J. K. 1999. Some Statistical Heresies. The Statistician 48, no. 1: 1-40.