An Analysis of Variance Model for Assessing Reliability of Naturalistic Observations

This paper presents a subjects × raters partially nested factorial analysis of variance model for estimating coefficients of interrater reliability. Procedures and formulae are described for computing unbiased estimates using the between-subjects and error mean squares from the model. Inclusion or exclusion of rater variance in the estimates is also discussed. Nine specific advantages for using the analysis of variance approach over existing methods are listed. A direction for future research on reliability is suggested.

Download Full-text

A Taxonomy for Psycho-Oncological Intervention Techniques in an Acute Care Hospital in Germany

Oncology Research and Treatment ◽

10.1159/000517532 ◽

2021 ◽

pp. 1-7

Author(s):

Martina Madl ◽

Marietta Lieb ◽

Katharina Schieber ◽

Tobias Hepp ◽

Yesim Erim

Keyword(s):

Acute Care ◽

Interrater Reliability ◽

Acute Care Hospital ◽

Future Research ◽

Systematic Research ◽

Care Hospital ◽

Certification System ◽

Stepwise Procedure ◽

Further Development

Background: Due to the establishment of a nationwide certification system for cancer centers in Germany, the availability of psycho-oncological services for cancer patients has increased substantially. However, little is known about the specific intervention techniques that are applied during sessions in an acute care hospital, since a standardized taxonomy is lacking. With this study, we aimed at the investigation of psycho-oncological intervention techniques and the development of a comprehensive and structured taxonomy thereof. Methods: In a stepwise procedure, a team of psycho-oncologists generated a data pool of interventions and definitions that were tested in clinical practice during a pilot phase. After an adaptation of intervention techniques, interrater reliability (IRR) was attained by rating 10 previously recorded psycho-oncological sessions. A classification of interventions into superordinate categories was performed, supported by cluster analysis. Results: Between April and June 2017, 980 psycho-oncological sessions took place. The experts agreed on a total number of 22 intervention techniques. An IRR of 89% for 2 independent psycho-oncological raters was reached. The 22 techniques were classified into 5 superordinate categories. Discussion/Conclusion: We developed a comprehensive and structured taxonomy of psycho-oncological intervention techniques in an acute care hospital that provides a standardized basis for systematic research and applied care. We expect our work to be continuously subjected to further development: future research should evaluate and expand our taxonomy to other contexts and care settings.

Download Full-text

How the one-way analysis of variance model is affected by the degree of precision of the data

Journal of Applied Statistics ◽

10.1080/02664769200000005 ◽

1992 ◽

Vol 19 (1) ◽

pp. 61-73 ◽

Cited By ~ 1

Author(s):

A. R. Tricker

Keyword(s):

Analysis Of Variance ◽

Variance Model ◽

The One

Download Full-text

Speech-Language Pathologists' Ratings of Speech Accuracy in Children With Speech Sound Disorders

American Journal of Speech-Language Pathology ◽

10.1044/2021_ajslp-20-00381 ◽

2021 ◽

pp. 1-12

Author(s):

Linye Jing ◽

Maria I. Grigos

Keyword(s):

Interrater Reliability ◽

Rating Scales ◽

Rating Scale ◽

Speech Sound ◽

Future Research ◽

Training Procedure ◽

Speech Sound Disorders ◽

Speech Language Pathologists ◽

Point Rating Scale ◽

Whole Word

Purpose: Forming accurate and consistent speech judgments can be challenging when working with children with speech sound disorders who produce a large number and varied types of error patterns. Rating scales offer a systematic approach to assessing the whole word rather than individual sounds. Thus, these scales can be an efficient way for speech-language pathologists (SLPs) to monitor treatment progress. This study evaluated the interrater reliability of an existing 3-point rating scale using a large group of SLPs as raters. Method: Utilizing an online platform, 30 SLPs completed a brief training and then rated single words produced by children with typical speech patterns and children with speech sound disorders. Words were closely balanced across the three rating categories of the scale. The interrater reliability of the SLPs ratings to a consensus judgment was examined. Results: The majority of SLPs (87%) reached substantial interrater reliability to a consensus judgment using the 3-point rating scale. Correct productions had the highest interrater reliability. Productions with extensive errors had higher agreement than those with minor errors. Certain error types, such as vowel distortions, were especially challenging for SLPs to judge. Conclusions: This study demonstrated substantial interrater reliability to a consensus judgment among a large majority of 30 SLPs using a 3-point rating. The clinical implications of the findings are discussed along with proposed modifications to the training procedure to guide future research.

Download Full-text