Limit of p value from multcompare?

(I'm having exactly the same question as this one on Stack Exchange, so I'm quoting it verbatim as follows)

It seems to me that MATLAB multcompare cannot output any p value (for pairwise comparison) smaller than 9.56061674273201e-10.

To confirm this, I did

y = [ones(100,1) 2*ones(100,1) 3*ones(100,1)];
[p,tbl,stats] = anova1(y);
[c,~,~,gnames] = multcompare(stats);
p = c(:,6)

and got p values for the three pairwise comparisons

p =

        9.56061674273201e-10
        9.56061674273201e-10
        9.56061674273201e-10

Why is there this limit in MATLAB? How should I report my p values then? It looks strange to report exactly same values for several different tests.

[Brendan Hamm]

I believe this has to do with the tests being performed. ANOVA is returning a p-statistic from an F-test and multcompare is using the Tukey's honest significance test to correct for the Type I error rate. In doing so MATLAB is actually using the relationship with the t-distribution rather than a table of Tukey's values. It seems likely that this is just roundoff error from the extra computations to get there, but I would feel safe reporting that the p-value is zero in your above case. If you really want throw in a footnote and give the precision.

[the cyclist]

P-values should never be reported as 0. That is equivalent to saying the observed result is impossible, given the null hypothesis.

What should be reported in cases like this is somewhat dependent on one's field (because there are different conventions). It's common in medical research, for example, to never report P-values less than 0.001, reporting "P < 0.001" instead.

There are theoretical underpinnings to this, related to the precision of the P-value itself.

[Eric Zhang]

@the cyclist

Thanks a lot for your reply! I don't think it's common in my field to just report "p < 0.001" -- I think I still need to understand why there is this limit in multcompare. :/ Any ideas why? Or any pointers to a paper/book that might explain the theoretical underpinnings? Thanks a lot!

[Eric Zhang]

@Brendan Hamm

Many thanks for the reply! Very interested in how exactly this rounding error occurs (or this limit is reached). Greatly appreciate it if you could elaborate more? Thanks a lot!

[the cyclist]

I've attached a couple references that I've found useful.