[R] [EXT] Re: A very small p-value

[Robert Knight]

I have not reviewed the formulas presented, but you err in assertion of
wrong information om that site.  The Central Limit Theorem is about the raw
data and the law of large numbers applies to averages obtained from it.

T if n<30 was the metric I was trained to use.  If I recall correctly, z is
more reliable below 30 than t is above it.

Unless the math formulas are wrong, that site seems useful.  I wonder how
far t diverges from z, but do not have time to compare them exactly.




On Wed, Nov 5, 2025, 9:04 AM Viechtbauer, Wolfgang (NP) via R-help <
r-help using r-project.org> wrote:

> Eik, thanks for posting this. I thought that the page was making the usual
> (just somewhat flawed) argument that once the dfs are sufficiently large,
> whether one does pnorm(...) or pt(..., df=<>) makes little difference
> (although far out in the tails it still does).
>
> Your post made me look at the page and I hope nobody takes anything
> written there serious. The argument is so utterly wrong. I am absolutely
> flabbergasted how somebody could write so many pages of text based on such
> a flawed understanding of basic statistical concepts.
>
> Just to give some examples:
>
> "The next issue I have is that I can't see the underlying data. So I don't
> know what the actual shape of the distribution is, but it's probably fair
> to say it's normally distributed (assuming the Central Limit Theorem
> applies)." The CLT says nothing about the distribution of the raw data.
>
> "As the sample size increases, samples will begin to operate and appear
> more and more like the population they are drawn from. This is the Law of
> Large Numbers." The law of large numbers has nothing to do with this.
>
> And as Eik already pointed out, the 'z-test' the author is describing is
> not a test at all, but essentially just calculates the standardized mean
> difference (and computing a p-value from it makes no sense).
>
> Best,
> Wolfgang
>
> > -----Original Message-----
> > From: R-help <r-help-bounces using r-project.org> On Behalf Of Eik Vettorazzi
> via R-
> > help
> > Sent: Tuesday, November 4, 2025 20:44
> > To: Petr Pikal <peprcon.asc using centrum.cz>; Christophe Dutang <
> dutangc using gmail.com>
> > Cc: r-help using r-project.org
> > Subject: Re: [R] [EXT] Re: A very small p-value
> >
> > Hi,
> > Stepping briefly outside the R context, I noticed a statistical point in
> > the text you linked that, in my opinion, isn't quite right. I believe
> > there's a key misunderstanding here: The statement that the z-test does
> > not depend on the number of cases is incorrect. The p-value of the
> > z-test is —just like other tests— very much dependent on the sample
> > size, assuming the same mean difference and standard deviation.
> > The text you linked is actually calculating an Effect Size, which is
> > (largely) independent of the sample size. Effect Size answers the
> > question of how "relevant" or "large" the difference between groups is.
> > This is fundamentally different from testing for "significant"
> differences.
> > Specifically, the crucial 1/\sqrt{n} term, which is necessary for
> > calculating the standard error of the mean difference, seems to be
> > missing from the presented formula for the z-score. I just wanted to
> > quickly point this out.
> >
> > Best regards
> >
> > Am 27.10.2025 um 14:12 schrieb Petr Pikal:
> > > Hallo
> > >
> > > The t test is probably not the best option in your case. With 95
> > > observations your data behave more like a population and you  may get
> > > better insight using z-test. See
> > >
> https://toxictruthblog.com/avoiding-little-known-problems-with-the-t-test/
> > >
> > > Best regards.
> > > Petr
> > >
> > > so 25. 10. 2025 v 11:46 odesílatel Christophe Dutang <
> dutangc using gmail.com>
> > > napsal:
> > >
> > >> Dear list,
> > >>
> > >> I'm computing a p-value for the Student test and discover some
> > >> inconsistencies with the cdf pt().
> > >>
> > >> The observed statistic is 11.23995 for 95 observations, so the
> p-value is
> > >> very small
> > >>
> > >>> t_score <- 11.23995
> > >>> n <- 95
> > >>> print(pt(t_score, df = n-2, lower=FALSE), digits=22)
> > >> [1] 2.539746620181247991746e-19
> > >>> print(integrate(dt, lower=t_score, upper=Inf, df=n-2)$value, digits
> = 22)
> > >> [1] 2.539746631161970791961e-19
> > >>
> > >> But if I compute with pt(lower=TRUE), I got 0
> > >>
> > >>> print(1-pt(t_score, df = n-2, lower=TRUE), digits=22)
> > >> [1] 0
> > >>
> > >> Indeed, the p-value is lower than the epsilon machine
> > >>
> > >>> pt(t_score, df = n-2, lower=FALSE) < .Machine$double.eps
> > >> [1] TRUE
> > >>
> > >> Using the square of t statistic which follows a Fisher distribution,
> I got
> > >> the same issue:
> > >>
> > >>> print(pf(z, 1, n-2, lower=FALSE), digits=22)
> > >> [1] 5.079493240362495983491e-19
> > >>> print(integrate(df, lower=z, upper=Inf, df1=1, df2=n-2)$value,
> digits =
> > >> 22)
> > >> [1] 5.079015231299358486828e-19
> > >>> print(1-pf(z, 1, n-2, lower=TRUE), digits=22)
> > >> [1] 0
> > >>
> > >> When using the t.test() function, the p-value is naturally printed :
> > >> p-value < 2.2e-16.
> > >>
> > >> Any comment is welcome.
> > >>
> > >> Christophe
> ______________________________________________
> R-help using r-project.org mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
> https://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>

	[[alternative HTML version deleted]]