An Investigation of Different Methods of Analyzing Rater Disagreement and its Consequences

Abstract

The current study investigated how analyzing rater disagreement in the same data with different statistical approaches – difference scores, Pearson correlations, intraclass correlations, and tri-factor models – led to different insights. These analyses were conducted across several predictors of rater disagreement, including symptom types, rater pairs, children’s ages, and child-reported insecure attachment to parents. Each statistical approach offered different insights into rater disagreement and its predictors and the extent of these differences varied across predictors. Specifically, minimal differences were observed between statistical approaches when evaluating rater disagreement across symptom types and rater pairs. However, greater divergence emerged when evaluating rater disagreement across levels of insecure attachment and children’s age. These findings indicate that the choice of statistical approach has a greater impact on the insights gained for some predictors of rater disagreement than others, likely because the features of these predictors are more effectively captured by certain statistical approaches due to their underlying mathematical properties. The study therefore highlights the importance of carefully selecting statistical analyses for studies of rater disagreement, as well as exercising caution when interpreting and comparing results across studies using different statistical methods.