[R] bootstrap

[Prof Brian Ripley]

On Mon, 12 Nov 2007, Stefano Ghirlanda wrote:

> i am using the boot package for some bootstrap calculations in place
> of anovas. one reason is that my dependent variable is distributed
> bimodally, but i would also like to learn about bootstrapping in
> general (i have ordered books but they have not yet arrived).
>
> i get the general idea of bootstrapping but sometimes i do not know
> how to define suitable statistics to test specific hypotheses.

That's a basic issue in statistics.  Bootstrapping is only another way to 
assess the variability of a pre-determined statistic (and incidentally is 
not much used for testing, more often for confidence intervals).

> two examples follow.
>
> 1) comparing the means of more than two groups. a suitable statistics
>   could be the sum of squared deviations of group means from the
>   grand mean. does this sound reasonable?

No.  That means nothing by itself, but needs to be compared to the 
residual variation (e.g. by an F statistic).

> 2) testing for interactions. e.g., i want to see whether an
>   independent variable has the same effect in two different
>   samples. in an anova this would be expressed as the significance,
>   or lack thereof, of the interaction between a "sample" factor and
>   another factor for the independent variable. how would i do this
>   with a bootstrap calculation?
>
> my problem with 2) is that when one fits a linear model to the data,
> from which sums of squares for the anova are calculated, the
> interaction between the two factors corresponds to many regression
> coefficients in the linear model (e.g., i actually have three samples
> and an independent variable with four levels). i do not know how to
> summarize these in a single statistics.

Any good book on statistics with R (e.g. MASS) would point you at the 
anova() function to compare models.

> i have seen somewhere that some people calculate F ratios
> nevertheless, but then test them against a bootstrapped distribution
> rather than against the F distribution. is this a sensible approach?
> could one also use sums of squares directly as the bootstrapped
> statistics?

It is not totally off the wall.  The problems are

- How you bootstrap, given that you don't have a single homogeneous group.
   You seem to want to test, so you need to emulate the null-hypothesis
   distribution.  The most common way to do that is to fit a model, find
   some sort of residuals, bootstrap those and use them as errors to
   simulate from the null hypothesis.  At that point you will have to work
   hard to convince many statisticians that you have improved over the
   standard theory or a simulation from a long-tailed error distribution.

- That if you don't believe in the normal distribution of your errors
   (and not response), you probably should not be using least-squares
   based statistical methodology.  And remember that the classical ANOVA
   tests are supported by permutation arguments which are very similar to
   the bootstrap (just sampling without replacement instead of with).

These points are discussed with examples in the linear models chapter of 
MASS (the book) and also in the Davison-Hinkley book which the 'boot' 
package supports.

[Shame about the broken shift key, although it seems to work with F: 
keyboards are really cheap to replace these days.]

-- 
Brian D. Ripley,                  ripley at stats.ox.ac.uk
Professor of Applied Statistics,  http://www.stats.ox.ac.uk/~ripley/
University of Oxford,             Tel:  +44 1865 272861 (self)
1 South Parks Road,                     +44 1865 272866 (PA)
Oxford OX1 3TG, UK                Fax:  +44 1865 272595