multcompare: 6th column of 'c'?

Hi

1. Using multcompare, the doc describes c as a x 6 matrix with the last row indicating the p-value of each pairwise comparison. However, following the examples and on my own data, my c-matrix has only 5 columns (the first five columns according to the doc). I am running 2012b version. Was this change possibly added after my version? That would be a bit disapointing since I would really like a matrix with p-values.

2. Comparing many pairs within a dataset after anova, the alpha-error-propability increases. Hence, you could do a bonferroni-correction with multcompare(stats, 'CType', 'bonferroni');. In General, by bonferroni-correction, alpha is divided by the number of comparisons (a = a/n). So given that I have one factor with 4 steps (hence 1vs2, 1vs3, 1vs4, 2vs3, 2vs4, 3vs4) resulting in 6 comparisons, is Alpha the same if I write this:

multcompare(stats, 'CType', 'bonferroni'); OR multcompare(stats, 'Alpha', '0.05/6');?

3. As Bonferroni-Correction depends on the number of comparison, that is, making more comparisons lowers the possibility for a single comparison to reach significance, is it possible to limit the number of comparisons (e.g., only compare conditions 1 to 4 and not 5 to 10)?

Thanks!

Answer 1

[Aditya]

Hi Powel,

1)You are correct that in recent MATLAB versions, the c-matrix output from multcompare includes a sixth column containing the p-value for each pairwise comparison. However, in MATLAB R2012b and earlier, the c-matrix only has five columns: group indices, group means difference, lower and upper confidence intervals, and the test statistic. The p-value column was added in later releases (around R2014b). Unfortunately, in your version, you cannot obtain p-values directly from the c-matrix output of multcompare. If you need p-values, you will need to compute them manually for each pairwise comparison, as shown in previous answers.

2)When you use multcompare(stats, 'CType', 'bonferroni'), MATLAB automatically applies the Bonferroni correction for you, adjusting the confidence intervals and significance tests accordingly. The Bonferroni method divides your chosen alpha-level by the number of pairwise comparisons. For example, if you have 4 groups (so 6 pairwise comparisons), the Bonferroni-corrected alpha for each test is 0.05/6 ≈ 0.0083. If you instead specify multcompare(stats, 'Alpha', 0.05/6), you are manually setting the per-comparison alpha to the Bonferroni-adjusted value. Both approaches are statistically equivalent in terms of significance thresholding, but using 'CType', 'bonferroni' is preferable because it also adjusts the confidence intervals appropriately, not just the alpha threshold.

3)Yes, you can limit the number of comparisons in post-hoc analysis. This is often desirable if you have specific hypotheses or only care about certain group differences. In MATLAB, you can specify which group pairs you want to compare using the 'ComparisonType' option (in newer versions) or by manually extracting and comparing only the desired pairs. For your version, you can subset the c-matrix after running multcompare to focus on the comparisons of interest, or you can manually perform t-tests (or other appropriate tests) only for the pairs you care about, applying the Bonferroni correction based on the number of those comparisons.

% Suppose you have run anova and have the stats structure
c = multcompare(stats, 'CType', 'bonferroni'); % c is N x 5 in R2012b

% To extract only comparisons where group 1 is compared to groups 2, 3, 4:
idx = (c(:,1) == 1) & ismember(c(:,2), [2 3 4]);
c_limited = c(idx, :);
disp(c_limited)