How to calculate for significant difference between Cohen's Kappa values?
31 Ansichten (letzte 30 Tage)
Ältere Kommentare anzeigen
I have calculated the Cohen's Kappa value determining agreement between Test A and Test B, as well as Cohen's Kappa for agreement between Test A and Test C. What method would I use to calculate for a significant difference in Kappa values between agreement for A-B compard to A-C? Are there any existing scripts/functions available for this?
2 Kommentare
Jeff Miller
am 8 Sep. 2021
Is there a single sample for which you have classifications on all 3 tests, or do you have tests A & B on one sample and tests A & C on a different sample? I think these two cases would have to be treated differently...
Antworten (3)
Jeff Miller
am 14 Sep. 2021
As I understand it, the fundamental question is whether tests A & B agree better than tests A & C, beyond a minor improvement that could just be due to chance (or agree worse, depending on how the tests B and C are labelled). The null hypothesis is that the agreement between A & B is equal to the agreement between A & C.
The most straightforward test for this case is the chi-square test for independence. Imagine the data summarized in a 2x2 table like this:
% Tests agree Tests disagree
% A & B group: 57 17
% A & C group: 35 8
with total N's of 74 in the first group and 43 in the second group. MATLAB's 'crosstab' command will compute that chi-square test for you. See this answer for an explanation of how to format the data and run the test.
Cohen's Kappa is a useful numerical measure of the extent of agreement, but it isn't really optimal for deciding whether the levels of agreement are different for the two pairs of tests.
1 Kommentar
Peter H Charlton
am 22 Aug. 2022
Bearbeitet: Peter H Charlton
am 22 Aug. 2022
In case it's helpful, here is some example code for formatting data and running the test (adapted from the code here):
% input data (from above):
tbl = [57,17;35,8];
% format as two input vectors:
x1 = [repmat(1,[tbl(1,1),1]); repmat(2, [tbl(2,1),1]); repmat(1, [tbl(1,2),1]); repmat(2,[tbl(2,2),1])]; x2 = [repmat(1,[tbl(1,1),1]); repmat(1, [tbl(2,1),1]); repmat(2, [tbl(1,2),1]); repmat(2,[tbl(2,2),1])];
% run the test:
[tbl_new,chi2stat,pval] = crosstab(x1,x2);
% check:
if isequal(tbl,tbl_new)
fprintf('The cross-tabulation table was correctly generated')
end
And I think the following code is generalisable to an mxn table (using data from here as an example):
% input data (from above link):
tbl = [90,60,104,95;30,50,51,20;30,40,45,35];
% format as two input vectors
[x1,x2] = deal([]);
for row_no = 1 : height(tbl)
for col_no = 1 : width(tbl)
x1 = [x1; repmat(row_no, [tbl(row_no,col_no),1])];
x2 = [x2; repmat(col_no, [tbl(row_no,col_no),1])];
end
end
% run the test:
[tbl_new,chi2stat,pval] = crosstab(x1,x2);
% check:
if isequal(tbl,tbl_new)
fprintf('The cross-tabulation table was correctly generated')
end
Star Strider
am 6 Sep. 2021
Bearbeitet: Star Strider
am 13 Sep. 2021
I used Cohen’s κ many years ago. From my understanding, from reading Fliess’s book (and correspoinding with him), Cohen’s κ is normally distributed. An excellent (in my opinion) and free resource is: Interrater reliability: the kappa statistic . There are others, although not all are free.
EDIT — (13 Sep 2021 at 10:58)
To get p-values and related statistics for normally-distributed variables, the ztest function would likely be appropriate.
.
0 Kommentare
Siehe auch
Kategorien
Mehr zu Hypothesis Tests finden Sie in Help Center und File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!