The influence of rater training on inter-rater reliability when using the rat grimace scale

Background. Rodent grimace scales facilitate evaluation of the affective component of pain and can identify a range of acute pain levels. Reported rater training in the use of these scales varies considerably and may contribute to observed variability in inter-rater reliability. This study evaluated the effect of training on inter-rater reliability with the Rat Grimace Scale (RGS). Methods. Two training sets, of 42 and 150 images, were prepared from several acute pain models. Four trainee raters, with no previous experience with the RGS, progressed through 2 rounds of training, first scoring 42 images (S1) followed by 150 images (S2a). After each round, trainees reviewed the RGS and any problematic images with an experienced rater. The 150 images were re-scored in a final round (S2b). Inter-rater reliability was evaluated using the intra-class correlation coefficient (ICC) and ICCs compared with a Feldt test. Results. Inter-rater reliability increased from moderate (ICC 0.58 [95%CI: 0.43-0.72]) to very good (ICC 0.85 [0.81-0.88]) between S1 and S2b (p < 0.01) with a significant increase also observed between S2a and S2b (p < 0.01). The ICCs for individual action units orbital tightening, ears and nose/cheek also improved from S1 to S2b (p < 0.01). The action units with the highest and lowest ICCs at S2b were orbital tightening (0.84 [0.80-0.87]) and whiskers (0.63 [0.57-0.70]), respectively. In comparison to an experienced rater the ICCs for all trainees improved, ranging from 0.88 to 0.91 at S2b. Discussion. Training improves inter-rater reliability between trainees, with an associated reduction in 95%CI. Additionally, training resulted in improved inter-rater reliability alongside an experienced rater. Training improves the scoring of individual action units though scoring of whiskers is more difficult that other sites. Conclusion. The beneficial effects of training potentially reduce data variability and improve experimental animal welfare.

Download Full-text

The influence of rater training on inter-and intra-rater reliability when using the rat grimace scale

10.7287/peerj.preprints.26721v2 ◽

2018 ◽

Author(s):

Emily Zhang ◽

Vivian Leung ◽

Daniel SJ Pang

Keyword(s):

Acute Pain ◽

Spontaneous Pain ◽

Rater Training ◽

Rater Reliability ◽

Intra Class Correlation ◽

Beneficial Effects ◽

Data Variability ◽

Pain Models ◽

Rater Performance ◽

Intra Class Correlation Coefficient

Rodent grimace scales facilitate assessment of spontaneous pain and can identify a range of acute pain levels. Reported rater training in using these scales varies considerably and may contribute to observed variability in inter-rater reliability. This study evaluated the effect of training on inter-rater reliability with the Rat Grimace Scale (RGS). Two training sets, of 42 and 150 images, were prepared from several acute pain models. Four trainee raters progressed through 2 rounds of training, first scoring 42 images (S1) followed by 150 images (S2a). After each round, trainees reviewed the RGS and any problematic images with an experienced rater. The 150 images were then re-scored (S2b). Four years after training, all trainees re-scored the 150 images (S2c). Inter- and intra-rater reliability was evaluated using the intra-class correlation coefficient (ICC) and ICCs compared with a Feldt test. Inter-rater reliability increased from moderate (0.58 [95%CI: 0.43-0.72]) to very good (0.85 [0.81-0.88]) between S1 and S2b (p < 0.01) and also increased between S2a and S2b (p < 0.01). The action units with the highest and lowest ICCs at S2b were orbital tightening (0.84 [0.80-0.87]) and whiskers (0.63 [0.57-0.70]), respectively. In comparison to an experienced rater the ICCs for all trainees improved, ranging from 0.88 to 0.91 at S2b. Four years later, very good inter-rater reliability was retained (0.82 [0.76-0.84]) and intra-rater reliability was good or very good (0.78-0.87). Training improves inter-rater reliability between trainees, with an associated reduction in 95%CI. Additionally, training resulted in improved inter-rater reliability alongside an experienced rater. Performance was retained after several years. The beneficial effects of training potentially reduce data variability and improve experimental animal welfare.

Download Full-text

The influence of rater training on inter-and intra-rater reliability when using the rat grimace scale

10.7287/peerj.preprints.26721 ◽

2018 ◽

Author(s):

Emily Zhang ◽

Vivian Leung ◽

Daniel SJ Pang

Keyword(s):

Acute Pain ◽

Spontaneous Pain ◽

Rater Training ◽

Rater Reliability ◽

Intra Class Correlation ◽

Beneficial Effects ◽

Data Variability ◽

Pain Models ◽

Rater Performance ◽

Intra Class Correlation Coefficient

Download Full-text

Influence of Rater Training on Inter- and Intrarater Reliability When Using the Rat Grimace Scale

Journal of the American Association for Laboratory Animal Science ◽

10.30802/aalas-jaalas-18-000044 ◽

2019 ◽

Vol 58 (2) ◽

pp. 178-183 ◽

Cited By ~ 8

Author(s):

Emily Q Zhang ◽

Vivian SY Leung ◽

Daniel SJ Pang

Keyword(s):

Acute Pain ◽

Interrater Reliability ◽

Intraclass Correlation ◽

Training Group ◽

Intrarater Reliability ◽

Rater Training ◽

Trainee Group ◽

Pain Models ◽

Ongoing Pain ◽

And Performance

Rodent grimace scales facilitate assessment of ongoing pain. Reported rater training using these scales varies considerably and may contribute to the observed variability in interrater reliability. This study evaluated the effect of training on interrater reliability with the Rat Grimace Scale (RGS). Two training sets (42 and 150 images) were prepared from acute pain models. Four trainee raters progressed through 2 rounds of training, scoring 42 images (set 1) followed by 150 images (set 2a). After each round, trainees reviewed the RGS and any problematic images with an experienced rater. The 150 images were then rescored (set 2b). Four years later, trainees rescored the 150 images (set 2c). A second group of raters (no-training group) scored the same image sets without review with the experienced rater. Inter- and intrarater reliability were evaluated by using the intraclass correlation coefficient (ICC), and ICC values were compared by using the Feldt test. In the trainee group, interrater reliability increased from moderate to very good between sets 1 and 2b and increased between sets 2a and 2b. Action units with the highest and lowest ICC at set 2b were orbital tightening and whiskers, respectively. In comparison to an experienced rater, the ICC for all trainees improved, ranging from 0.88 to 0.91 at set 2b. Four years later, very good interrater reliability was retained, and intrarater reliability was good or very good). The interrater reliability of the no-training group was moderate and did not improve from set 1 to set 2b. Training improved interrater reliability, with an associated reduction in 95%CI. In addition, training improved interrater reliability with an experienced rater, and performance was retained.

Download Full-text

Development and Validation of an Instrument to Assess Horseback Mobility in Hippotherapy

Perceptual and Motor Skills ◽

10.1177/00315125211036578 ◽

2021 ◽

pp. 003151252110365

Author(s):

Alessandra V. Prieto ◽

Kênnea Martins Almeida Ayupe ◽

Ana C. A. Abreu ◽

Paulo J. B. Gutierres Filho

Keyword(s):

Correlation Coefficient ◽

Factor Structure ◽

Internal Consistency ◽

Rater Reliability ◽

Intra Class Correlation ◽

Exploratory Factor Analyses ◽

Excellent Internal Consistency ◽

Development And Validation ◽

Functional Gain ◽

Intra Class Correlation Coefficient

Improvement in rider mobility represents an important functional gain for people with disabilities undergoing hippotherapy. However, there is no validated measuring instrument to track and document the rider's progress in riding activities. In this study, we aimed to develop and establish validity evidence for an instrument to assess hippotherapy participants’ mobility on horseback. We report on this development through the stages of: (a) content validation, (b) construct validation, (c) inter- and intra-rater reliability and (d) internal consistency analysis. We evaluated its factor structure with exploratory factor analyses, calculated values for inter- and intra-rater reliability using the intra-class correlation coefficient, and calculated its internal consistency using Cronbach's alpha. We followed recommendations by the Guidelines for Reporting Reliability and Agreement Studies. We found good inter-rater reliability (intra-class correlation coefficient – ICC = 0.991–0.999) and good intra-rater reliability (ICC = 0.997–1.0), and there was excellent internal consistency (Cronbach's α = 0.937–0.999). The instrument’s factor structure grouped its three domains into one factor. As this instrument is theoretically consistent and has been found to be appropriate and reliable for its intended use, it is now available for the measurement of horseback mobility among hippotherapy riders.

Download Full-text

Facial expressions of pain in cats: the development and validation of a Feline Grimace Scale

Scientific Reports ◽

10.1038/s41598-019-55693-8 ◽

2019 ◽

Vol 9 (1) ◽

Cited By ~ 13

Author(s):

Marina C. Evangelista ◽

Ryota Watanabe ◽

Vivian S. Y. Leung ◽

Beatriz P. Monteiro ◽

Elizabeth O’Toole ◽

...

Keyword(s):

Acute Pain ◽

Pain Assessment ◽

Rater Reliability ◽

Analgesic Treatment ◽

Naturally Occurring ◽

Total Pain ◽

Excellent Internal Consistency ◽

Development And Validation ◽

Validated Instrument ◽

Control Study

AbstractGrimace scales have been used for pain assessment in different species. This study aimed to develop and validate the Feline Grimace Scale (FGS) to detect naturally-occurring acute pain. Thirty-five client-owned and twenty control cats were video-recorded undisturbed in their cages in a prospective, case-control study. Painful cats received analgesic treatment and videos were repeated one hour later. Five action units (AU) were identified: ear position, orbital tightening, muzzle tension, whiskers change and head position. Four observers independently scored (0–2 for each AU) 110 images of control and painful cats. The FGS scores were higher in painful than in control cats; a very strong correlation with another validated instrument for pain assessment in cats was observed (rho = 0.86, p < 0.001) as well as good overall inter-rater reliability [ICC = 0.89 (95% CI: 0.85–0.92)], excellent intra-rater reliability (ICC > 0.91), and excellent internal consistency (Cronbach’s alpha = 0.89). The FGS detected response to analgesic treatment (scores after analgesia were lower than before) and a cut-off score was determined (total pain score > 0.39 out of 1.0). The FGS is a valid and reliable tool for acute pain assessment in cats.

Download Full-text

A video anchored rating scale leads to high inter-rater reliability of inexperienced and expert raters in the absence of rater training

The American Journal of Surgery ◽

10.1016/j.amjsurg.2019.12.026 ◽

2020 ◽

Vol 219 (2) ◽

pp. 221-226 ◽

Cited By ~ 1

Author(s):

Ronit Patnaik ◽

Nicholas E. Anton ◽

Dimitrios Stefanidis

Keyword(s):

Rating Scale ◽

Rater Training ◽

Rater Reliability

Download Full-text

Oligoanalgesia in the Emergency Department: Short-Term Beneficial Effects of an Education Program on Acute Pain

Annals of Emergency Medicine ◽

10.1016/j.annemergmed.2007.01.019 ◽

2007 ◽

Vol 50 (4) ◽

pp. 462-471 ◽

Cited By ~ 74

Author(s):

Isabelle Decosterd ◽

Olivier Hugli ◽

Emmanuel Tamchès ◽

Catherine Blanc ◽

Elyazid Mouhsine ◽

...

Keyword(s):

Emergency Department ◽

Acute Pain ◽

Education Program ◽

Short Term ◽

Beneficial Effects

Download Full-text

Preliminary appraisal of the reliability and validity of the Colorado State University Feline Acute Pain Scale

Journal of Feline Medicine and Surgery ◽

10.1177/1098612x18777506 ◽

2018 ◽

Vol 21 (4) ◽

pp. 335-339 ◽

Cited By ~ 6

Author(s):

Hilary Shipley ◽

Alonso Guedes ◽

Lynelle Graham ◽

Elizabeth Goudie-DeAngelis ◽

Erin Wendt-Hornickle

Keyword(s):

Acute Pain ◽

Convergent Validity ◽

Correlation Coefficients ◽

Pain Scale ◽

State University ◽

Good Reliability ◽

Rater Reliability ◽

Spearman Rank ◽

Colorado State University ◽

Cohen's Kappa

Objectives The objective of this study was to determine the inter-rater reliability and convergent validity of the Colorado State University Feline Acute Pain Scale (CSU-FAPS) in a preliminary appraisal of its performance in a clinical teaching setting. Methods Sixty-eight female cats were assessed for pain after ovariohysterectomy. A cohort of 21 cats was examined independently by four raters (two board-certified anesthesiologists and two anesthesia residents) with the CSU-FAPS, and intra-class correlation coefficient (ICC) was used to determine inter-rater reliability. Weighted Cohen’s kappa was used to determine inter-rater reliability centered on the ‘need to reassess analgesic plan’ (dichotomous scale). A separate cohort of 47 cats was evaluated independently by two raters (one board-certified anesthesiologist and one veterinary small animal rotating intern) using the CSU-FAPS and the Glasgow Composite Measure Pain Scale (CMPS-Feline), and Spearman rank-order correlation was determined to assess convergent validity. Reliability was interpreted using Altman’s classification as very good, good, moderate, fair and poor. Validity was considered adequate if correlation coefficients were between 0.4 and 0.8. Results The ICC was 0.61 for anesthesiologists and 0.67 for residents, indicating good reliability. Weighted Cohen’s kappa was 0.79 for anesthesiologists and 0.44 for residents, indicating moderate to good reliability. The Spearman rank correlation indicated a statistically significant ( P = 0.0003) positive correlation (0.31; 95% confidence interval 0.14–0.46) between the CSU-FAPS and the CMPS-Feline. Conclusions and relevance The CSU-FAPS showed moderate-to-good inter-rater reliability when used by veterinarians to assess pain level or need to reassess analgesic plan after ovariohysterectomy in cats. The validity fell short of current guidelines for correlation coefficients and further refinement and testing are warranted to improve its performance.

Download Full-text

Qualitative Evaluation of the Project P.A.T.H.S. Based on the Perceptions of the Program Participants

The Scientific World JOURNAL ◽

10.1100/tsw.2006.354 ◽

2006 ◽

Vol 6 ◽

pp. 2254-2263 ◽

Cited By ~ 6

Author(s):

Daniel T.L. Shek ◽

Tak Yan Lee ◽

Andrew Siu ◽

Ching Man Lam

Keyword(s):

Hong Kong ◽

Focus Groups ◽

Qualitative Data ◽

Qualitative Evaluation ◽

Chinese Adolescents ◽

Rater Reliability ◽

Holistic Development ◽

Beneficial Effects ◽

Specific Focus ◽

Tier 1

Qualitative evaluation was carried out to understand the perceptions of the students participating in the Tier 1 Program of the P.A.T.H.S. Project. Five focus groups based on 43 students recruited from four schools were conducted to generate qualitative data to evaluate the program. With specific focus on how the informants described the program, results showed that the descriptors used were mainly positive in nature. When the informants were invited to name three metaphors that could stand for the program, the related metaphors were basically positive in nature. Finally, the program participants perceived many beneficial effects of the program in different psychosocial domains. Intra- and inter-rater reliability analyses revealed that the coding of the positive or negative nature of the responses was reliable. The present study provides qualitative support for the effectiveness of the Tier 1 Program of the Project P.A.T.H.S. in promoting holistic development in Chinese adolescents in Hong Kong.

Download Full-text