Application and evaluation of the Kappa statistic in the design and interpretation of chiropractic clinical research.

Lantz CA. Application and evaluation of the kappa statistic in the design and interpretation of chiropractic clinical research. J Manipulative Physiol Ther 1997 Oct;20(8):521-8

Abstract: The Kappa statistic (K) is becoming more widely used in chiropractic clinical research. Although the calculation of K is straightforward, assumptions made regarding the acquisition and interpretation of data require serious consideration before conclusions regarding the final outcome can be made. Of particular interest is the application of K for the interpretation of validity and reliability. In this regard, I argue against the use of K as a summary statistic.

I present here a description of K as a function of observer agreement (P(o)), without which K has little meaning, and show how a knowledge of this function should direct the design and, therefore, the interpretation of chiropractic clinical research. Only when all concerns regarding experimental design are either addressed directly or compensated for in the analysis can interpretations be made regarding validity or reliability.

Lacking such attention, K becomes an indicator of experimental design rather than an indicator of validity or reliability.