[R] t-test behavior given that the null hypothesis is true

[(Ted Harding)]

Ah! You have aqssigned a parameter "equal.var=TRUE", and "equal.var"
is not a listed paramater for t.test() -- see ?t.test :

  t.test(x, y = NULL,
    alternative = c("two.sided", "less", "greater"),
    mu = 0, paired = FALSE, var.equal = FALSE,
    conf.level = 0.95, ...)

Try it instead with "var.equal=TRUE", i.e. in your code:
  for(i in 1:k){
    rv.t.pvalues[i] <- t.test(rv[i, 1:(c/2)], rv[i, (c/2+1):c],
  ##equal.var=TRUE, alternative="two.sided")$p.value
    var.equal=TRUE, alternative="two.sided")$p.value
  }

When I run your code with "equal.var", I indeed repeatedly see
the deficient bin for the lowest P-values that you observed.
When I run your code with "var.equal" I do not see it.

The explanation is that, since "equal.var" is not a recognised
parameter for t.test(), it has assumed the default value FALSE
for var.equal, and has therefore (since it is a 2-sample test)
adopted the Welch/Satterthwaite procedure:

  var.equal: a logical variable indicating whether to treat
    the two variances as being equal. If 'TRUE' then the
    pooled variance is used to estimate the variance
    otherwise the Welch (or Satterthwaite) approximation
    to the degrees of freedom is used.

This has the effect of somewhat adapting the test procedure to
the data, so that extreme (i.e. small) values of P are even
rarer than they should be.

With best wishes,
Ted.

On 09-Jan-2013 13:24:59 Pavlos Pavlidis wrote:
> Hi Ted,
> thanks for the reply. I use a similar code which you can see below:
> 
> k <- 10000
> c <- 6
> rv <- array(NA, dim=c(k, c) )
> for(i in 1:k){
>   rv[i,] <- rnorm(c, mean=0, sd=1)
> }
> 
> rv.t.pvalues <- array(NA, k)
> 
> for(i in 1:k){
>   rv.t.pvalues[i] <- t.test(rv[i, 1:(c/2)], rv[i, (c/2+1):c],
> equal.var=TRUE, alternative="two.sided")$p.value
> }
> 
> hist(rv.t.pvalues)
> 
> The histogram is this one:
> *http://tinyurl.com/histogram-rt-pvalues-pdf
> 
> *
> *all the best
> idaios
> *
> 
> 
> On Wed, Jan 9, 2013 at 12:29 PM, Ted Harding <Ted.Harding at wlandres.net>wrote:
> 
>> On 09-Jan-2013 08:50:46 Pavlos Pavlidis wrote:
>> > Dear all,
>> > I observer a strange behavior of the pvalues of the t-test under
>> > the null hypothesis. Specifically, I obtain 2 samples of 3
>> > individuals each from a normal distribution of mean 0 and variance 1.
>> > Then, I calculate the pvalue using the t-test (var.equal=TRUE,
>> > samples are independent). When I make a histogram of pvalues
>> > I see that consistently the bin of the smallest pvalues has a
>> > lower frequency. Is this a known behavior of the t-test or it's
>> > a kind of bug/random number generation problem?
>> >
>> > kind regards,
>> > idaios
>>
>> Using the following code, I did not observe the behavious you describe.
>> The histograms are consistent with a uniform distribution of the
>> P-values, and the lowest bin for the P-values (when the code is
>> run repeatedly) is not consistently lower (or higher, or anything
>> else) than the other bins.
>>
>> ## My code:
>> N <- 10000
>> Ps <- numeric(N)
>> for(i in (1:N)){
>>   X1 <- rnorm(3,0,1) ; X2 <- rnorm(3,0,1)
>>   Ps[i] <- t.test(X1,X2,var.equal=TRUE)$p.value
>> }
>> hist(Ps)
>> ################################################
>>
>> If you would post the code you used, the reason why you are observing
>> this may become more evident!
>>
>> Hoping this helps,
>> Ted.
>>
>> -------------------------------------------------
>> E-Mail: (Ted Harding) <Ted.Harding at wlandres.net>
>> Date: 09-Jan-2013  Time: 10:29:21
>> This message was sent by XFMail
>> -------------------------------------------------
>>
> 
>       [[alternative HTML version deleted]]
> 
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.

-------------------------------------------------
E-Mail: (Ted Harding) <Ted.Harding at wlandres.net>
Date: 09-Jan-2013  Time: 14:51:04
This message was sent by XFMail