[R] model selection vs. H0 based testing

[Ruben Roa]

> -----Original Message-----
> From: r-help-bounces at stat.math.ethz.ch
> [mailto:r-help-bounces at stat.math.ethz.ch]On Behalf Of Thomas Petzoldt
> Sent: 06 September 2005 06:34
> Cc: petzoldt at rcs.urz.tu-dresden.de; R-Help
> Subject: Re: [R] model selection vs. H0 based testing
> 
> 
> Hello,
> 
> I wish to thank Douglas Bates very much for clarification and 
> pointing me to the MCMC simulation method to get p values even for cases where 
> Wald tests are inappropriate.
> 
> One question however remains when publishing statistical 
> results: does it help readers if we combine both,
> 
> - AIC based model selection
> *and*
> - null hypothesis based tests statistics
> 
> or should we focus on model selection only and try to reduce 
> the amount of tables provided?

IMHO the AIC is sufficient and the null hypothesis test is
not well suited to the problem. As stated by Akaike (1974,
A new look at the statistical model identification, IEEE
Transactions on Automatic Control 19:716-723):"As was noticed by
Lehman [this is the classic book on the Neyman-Pearson theory of
hypothesis testing], hypothesis testing procedures are traditionally
applied to the situation where actually multiple decision procedures
are required. If the statistical identification procedure is considered
as a decision procedure the very basic problem is the appropriate choice
of the loss function. In the Neyman-Pearson theory of statistical
hypothesis testing only the probabilities of rejecting and accepting
the correct and incorrect hypothesis, respectively, are considered
to define the loss caused by the decision. In practical situations
the assumed null hypotheses are only approximations and they
are almost always different from the reality. Thus the choice of the
loss function in the test theory makes its practical application
logically contradictory. The recongnition of this point that the
hypothesis testing procedure is not adequately formulated as a 
procedure of approximation is very important for the development
of practically useful identification procedures."
Note that Akaike speaks of 'model identification' whereas now this
subject are is usually referred to as 'model selection'.
Ruben