Skip to content
DSML.KZ
Publications

Article · Bulletin of the National Engineering Academy of the Republic of Kazakhstan · September 15, 2023

Features of Сohen’s kappa coefficient as a measure of expert opinion agreement

Galiya Ybytayeva, Orken J. Mamyrbayev, Nina Khairova, Kuralay Mukhsina, Багашар Жумажанов

Orken Mamyrbayev
Open via DOI

Abstract

Modern metrics for evaluating agreement coefficients between the experimental results and expert opinion are compared, and the possibility of using these metrics in experimental research in automatic text processing by machine learning methods is assessed. The choice of Cohen’s kappa coefficient as a measure of expert opinion agreement in the NLP and Text Mining problems is justified. An example of using Cohen’s kappa coefficient for evaluating the level of agreement between the opinion of an expert and the results of ML classification and the measure of agreement of expert opinions in the alignment of sentences of the Kazakh-Russian parallel corpus is given. Based on this analysis, it is proved that Cohen’s kappa coefficient is one of the best statistical methods for determining the level of agreement in experimental studies due to its ease of use, computing simplicity, and high accuracy of the results.

Reliability and Agreement in MeasurementComputational and Text Analysis MethodsNatural Language Processing Techniquesinterobserver agreementexpert opinionagreement measures

More by Orken Mamyrbayev