Article,
A better confidence interval for Kappa on measuring agreement between two raters with binary outcomes
Journal of Computational and Graphical Statistics, 3 (3): 301--321 (1994)
Abstract
Although the kappa statistic is widely used in measuring interrater agreement, it is known that the standard confidence interval estimation behaves poorly in small samples and for nonzero kappas. Efforts have been made to improve the estimation through transformation and Edgeworth expansion (Flack 1987). The results remain unsatisfactory when kappa is far from 0, however, even with the sample size as large as 100. In this article we reparameterize the kappa statistic to reveal its relationship with the marginal probability of agreement. The reparameterization not only gives a more meaningful interpretation of kappa but also clearly demonstrates that the range of kappa depends on the marginal probabilities. Various two- and three-dimensional plots are shown to illustrate the relationship among these parameters. The immediate application is to construct a new confidence interval based on the profile variance and reparameterization. Extensive simulation studies show that the new confidence interval performs very well in almost all parameter settings even when other methods fail.
