Weighted Kappa for Interobserver Agreement and Missing Data
The weighted kappa coefficient is commonly used for assessing agreement between two raters on an ordinal scale. This study is the first to assess the impact of missing data on the value of weighted kappa. We compared three methods for handling missing data in a simulation study: predictive mean matc...
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2025-02-01
|
| Series: | Machine Learning and Knowledge Extraction |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2504-4990/7/1/18 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850204494417100800 |
|---|---|
| author | Matthijs J. Warrens Alexandra de Raadt Roel J. Bosker Henk A. L. Kiers |
| author_facet | Matthijs J. Warrens Alexandra de Raadt Roel J. Bosker Henk A. L. Kiers |
| author_sort | Matthijs J. Warrens |
| collection | DOAJ |
| description | The weighted kappa coefficient is commonly used for assessing agreement between two raters on an ordinal scale. This study is the first to assess the impact of missing data on the value of weighted kappa. We compared three methods for handling missing data in a simulation study: predictive mean matching, listwise deletion and a weighted version of Gwet’s kappa. We compared their performances under three missing data mechanisms, using agreement tables with various numbers of categories and different values of weighted kappa. Predictive mean matching outperformed the other two methods in most simulated cases in terms of root mean squared error and in all cases in terms of bias. |
| format | Article |
| id | doaj-art-0bb90c6b45c14d119825d48fcede0a4e |
| institution | OA Journals |
| issn | 2504-4990 |
| language | English |
| publishDate | 2025-02-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Machine Learning and Knowledge Extraction |
| spelling | doaj-art-0bb90c6b45c14d119825d48fcede0a4e2025-08-20T02:11:17ZengMDPI AGMachine Learning and Knowledge Extraction2504-49902025-02-01711810.3390/make7010018Weighted Kappa for Interobserver Agreement and Missing DataMatthijs J. Warrens0Alexandra de Raadt1Roel J. Bosker2Henk A. L. Kiers3GION Education/Research, Faculty of Behavioural and Social Sciences, University of Groningen, Grote Rozenstraat 3, 9712 TG Groningen, The NetherlandsGION Education/Research, Faculty of Behavioural and Social Sciences, University of Groningen, Grote Rozenstraat 3, 9712 TG Groningen, The NetherlandsGION Education/Research, Faculty of Behavioural and Social Sciences, University of Groningen, Grote Rozenstraat 3, 9712 TG Groningen, The NetherlandsHeymans Institute for Psychological Research, Faculty of Behavioural and Social Sciences, University of Groningen, Grote Kruisstraat 2/1, 9712 TS Groningen, The NetherlandsThe weighted kappa coefficient is commonly used for assessing agreement between two raters on an ordinal scale. This study is the first to assess the impact of missing data on the value of weighted kappa. We compared three methods for handling missing data in a simulation study: predictive mean matching, listwise deletion and a weighted version of Gwet’s kappa. We compared their performances under three missing data mechanisms, using agreement tables with various numbers of categories and different values of weighted kappa. Predictive mean matching outperformed the other two methods in most simulated cases in terms of root mean squared error and in all cases in terms of bias.https://www.mdpi.com/2504-4990/7/1/18ordinal ratingsinter-rater reliabilityweighted kappamissing datalistwise deletionpredictive mean matching |
| spellingShingle | Matthijs J. Warrens Alexandra de Raadt Roel J. Bosker Henk A. L. Kiers Weighted Kappa for Interobserver Agreement and Missing Data Machine Learning and Knowledge Extraction ordinal ratings inter-rater reliability weighted kappa missing data listwise deletion predictive mean matching |
| title | Weighted Kappa for Interobserver Agreement and Missing Data |
| title_full | Weighted Kappa for Interobserver Agreement and Missing Data |
| title_fullStr | Weighted Kappa for Interobserver Agreement and Missing Data |
| title_full_unstemmed | Weighted Kappa for Interobserver Agreement and Missing Data |
| title_short | Weighted Kappa for Interobserver Agreement and Missing Data |
| title_sort | weighted kappa for interobserver agreement and missing data |
| topic | ordinal ratings inter-rater reliability weighted kappa missing data listwise deletion predictive mean matching |
| url | https://www.mdpi.com/2504-4990/7/1/18 |
| work_keys_str_mv | AT matthijsjwarrens weightedkappaforinterobserveragreementandmissingdata AT alexandraderaadt weightedkappaforinterobserveragreementandmissingdata AT roeljbosker weightedkappaforinterobserveragreementandmissingdata AT henkalkiers weightedkappaforinterobserveragreementandmissingdata |