Irreproducibility of anovan on Mac vs. PC implementations of MATLAB

We are experiencing issues with a basic implementation of anovan yielding different F statistics when the same code is run on different computers. The biggest difference is between Mac and PCs, but there are also differences among multiple laptops of the same type. If we load the following dataset (Testdata.mat) and then attempt to call anovan as follows, we get a different answer on a Mac:

compared to what we obtain on a PC (below). The problem persists when we remove nesting, change computers, etc. Note that the sum-of-squares and mean-squared estimates are identical, but the F statistics diverge indicating differences in how the degrees of freedom are calculated. We appreciate any guidance on this matter.

load Testdata
anovan(data,grouplabels,'continuous',1,'random',3, ...
    'varnames',{'Day' 'Volume' 'Replicate'},'model','interaction', ...
    'nested',[0 0 0; 0 0 1; 0 0 0]);

[dpb]

load Testdata
anovan(data,grouplabels,'continuous',1,'random',3, ...
    'varnames',{'Day' 'Volume' 'Replicate'},'model','interaction', ...
    'nested',[0 0 0; 0 0 1; 0 0 0]);

The difference is in the error term used; the Day factor in the second uses the Day:Replicate error term,

8.34345/0.07099

ans = 117.5299

whereas the other case uses the overall model error

8.34345/0.29538

ans = 28.2465

Would need to see the actual input instruction that generated the other alternative...either something is different in the specified model or there is an actual difference in the ANOVA function between the two implementations; I've not dug into the model enough to try to reproduce externally as to what it really should be...let's see what actually generated the other result first.

[Kevin Janes]

Thank you—it is the exact same input instruction and source file used on both computers, which is why it has been so difficult to troubleshoot. Both computers are running R2023a, and when we do a document compare on our respective anovan.m files they are also identical. We ran the debugger through anovan, and everthing is identical through the first factor ("Day"), but it diverges with the second factor right at about the first usage of gettermconstr within anovan. The qr decompositions within this local function give slightly different answers (all within ±eps), which appear to lead to qualitative changes in how the factors are handled after that. At first, we thought it was a problem related to the nesting, but we can both turn the nesting off and we again get divergent answers. If we each run the carbig example provided in the anovan documentation, we get identical results.

[dpb]

Realistically, that probably will take a TMW bug/support request to get to the bottom of, then.

There is at least one other regular here who has Mac who may see this and may be inclined to delve deeper, but I don't know that he has the wherewithal to generate/compare the result on the Windows side.

I simply don't have time to dig much deeper at the moment, sorry...

[Kevin Janes]

Bug support request just submitted, thanks.

Answer 1

[Shivansh]

Hi Kevin!

I understand that you are getting different results for F statistics when the same code is run on different computers. The difference is maximum when calculated on “Mac” than other PCs.

I have faced a similar issue and found a possible workaround to resolve it. Please follow the below the steps which involves editing the “anovan.m” file in Matlab to resolve the issue,

• Open MATLAB and run the following command in the command window.

edit anovan.m

• Copy the entire code content into a new file "anovan_temp.m"
• Modify line 680 as shown below

tol = 100 * max(size(R)) * eps(R(1,1)); % before
tol = 100 * max(size(adjx1)) * eps(R(1,1)); % after

• Save the file in any directory.
• Navigate to the path of the original "anovan.m" file in File Explorer
• E.g 'C:\Program Files\MATLAB\R2023a\toolbox\stats\stats'
• Rename the "anovan.m" file to "anovan_old.m"
• Paste the "anovan_temp.m" file in the same directory as "anovan_old.m"
• Rename "anovan_temp.m" to "anovan.m".
• Restart MATLAB and run the command for "anovan".

You can refer to the following link to get more information on “anovan”: https://in.mathworks.com/help/stats/anovan.html.

Hope it helps!

[Kevin Janes]

This fixed it for us, thank you!