The Motivation Assessment Scale: inter-rater reliability and internal consistency in a British sample

Improvement in rider mobility represents an important functional gain for people with disabilities undergoing hippotherapy. However, there is no validated measuring instrument to track and document the rider's progress in riding activities. In this study, we aimed to develop and establish validity evidence for an instrument to assess hippotherapy participants’ mobility on horseback. We report on this development through the stages of: (a) content validation, (b) construct validation, (c) inter- and intra-rater reliability and (d) internal consistency analysis. We evaluated its factor structure with exploratory factor analyses, calculated values for inter- and intra-rater reliability using the intra-class correlation coefficient, and calculated its internal consistency using Cronbach's alpha. We followed recommendations by the Guidelines for Reporting Reliability and Agreement Studies. We found good inter-rater reliability (intra-class correlation coefficient – ICC = 0.991–0.999) and good intra-rater reliability (ICC = 0.997–1.0), and there was excellent internal consistency (Cronbach's α = 0.937–0.999). The instrument’s factor structure grouped its three domains into one factor. As this instrument is theoretically consistent and has been found to be appropriate and reliable for its intended use, it is now available for the measurement of horseback mobility among hippotherapy riders.

Download Full-text

Deterioration of housing environment and mental health of asylum seekers – a multi-level analysis

European Journal of Public Health ◽

10.1093/eurpub/ckz185.323 ◽

2019 ◽

Vol 29 (Supplement_4) ◽

Author(s):

A Mohsenpour ◽

L Biddle ◽

K Bozorgmehr

Keyword(s):

Mental Health ◽

Internal Consistency ◽

Odds Ratio ◽

Asylum Seekers ◽

Rapid Assessment ◽

Living Environment ◽

Generalized Anxiety ◽

Rater Reliability ◽

General Anxiety ◽

Housing Environment

Abstract Background Asylum seekers arriving in Germany are assigned to a local district and an accommodation centre wherein. We developed and validated a six-item questionnaire for rapid assessment of housing deterioration and investigated its association with inhabitant mental health. Methods Using cross-sectional data from a state-wide survey in Germany, we applied a random-effects modelling approach to estimate the exposure effect of housing deterioration on depression and general anxiety among a random sample of asylum seekers, using validated instruments (GAD2/PHQ2) for outcome variables. Housing deterioration was assessed on six items (windows/glass, walls/roof, garbage, graffiti, outside spaces, overall living environment) resulting in a deterioration score as exposure of interest. Additionally, we assessed the instrument’s intra- and inter-rater reliability and internal consistency. Results Of the 412 asylum seekers living in 58 accommodation centres, 45.7% reported symptoms of depression and 45.0% suffered general anxiety. Most centres (76.7%) were based in urban municipalities and 45.7% of inhabitants were living in an accommodation centre hosting ≥51 inhabitants. Preliminary adjusted odds ratio for accommodation centres with highest deterioration (Q 4) was 2.07 (0.67-6.40) for generalized anxiety, compared to 1.17 (0.45-3.08) for centres with lowest deterioration (Q 1). For depression, preliminary odds ratio was 1.92 (0.87-4.27) compared to 1.26 (0.63-2.50). The validation study confirmed inter-/intra-rater reliability (Brennan-Prediger coefficient: 0.81 and 0.92, respectively) and internal consistency (Crohnbach’s α: 0.80). Conclusions There are higher odds ratios for generalized anxiety disorder and depression among asylum seekers based on higher deterioration of housing environment. A questionnaire for rapid deterioration assessment and identification of accommodations needing further evaluation has been developed and successfully validated. Key messages Deterioration of small-scale housing environment is associated with poorer mental health for asylum seekers living in accommodation centres. A highly reliable new tool has been developed for rapid assessment of deterioration status of accommodation centres and identification of those needing further evaluation.

Download Full-text

Reliability of three linguistically and culturally validated pain assessment tools for sedated ICU patients by ICU nurses in Finland

Scandinavian Journal of Pain ◽

10.1515/sjpain-2017-0139 ◽

2018 ◽

Vol 18 (2) ◽

pp. 165-173 ◽

Cited By ~ 1

Author(s):

Sanna-Mari Pudas-Tähkä ◽

Sanna Salanterä

Keyword(s):

Intensive Care ◽

Internal Consistency ◽

Pain Assessment ◽

Assessment Tools ◽

Endotracheal Suctioning ◽

Icu Patients ◽

Rater Reliability ◽

Intensive Care Patients ◽

Video Recordings ◽

Test Retest Reliability

Abstract Background and aims: Pain assessment in intensive care is challenging, especially when the patients are sedated. Sedated patients who cannot communicate verbally are at risk of suffering from pain that remains unnoticed without careful pain assessment. Some tools have been developed for use with sedated patients. The Behavioral Pain Scale (BPS), the Critical-Care Pain Observation Tool (CPOT) and the Nonverbal Adult Pain Assessment Scale (NVPS) have shown promising psychometric qualities. We translated and culturally adapted these three tools for the Finnish intensive care environment. The objective of this feasibility study was to test the reliability of the three pain assessment tools translated into Finnish for use with sedated intensive care patients. Methods: Six sedated intensive care patients were videorecorded while they underwent two procedures: an endotracheal suctioning was the nociceptive procedure, and the non-nociceptive treatment was creaming of the feet. Eight experts assessed the patients’ pain by observing video recordings. They assessed the pain using four instruments: the BPS, the CPOT and the NVPS, and the Numeric Rating Scale (NRS) served as a control instrument. Each expert assessed the patients’ pain at five measurement points: (1) right before the procedure, (2) during the endotracheal suctioning, (3) during rest (4) during the creaming of the feet, and (5) after 20 min of rest. Internal consistency and inter-rater reliability of the tools were evaluated. After 6 months, the video recordings were evaluated for testing the test-retest reliability. Results: Using the BPS, the CPOT, the NVPS and the NRS, 960 assessments were obtained. Internal consistency with Cronbach’s alpha coefficient varied greatly with all the instruments. The lowest values were seen at those measurement points where the pain scores were 0. The highest scores were achieved after the endotracheal suctioning at rest: for the BPS, the score was 0.86; for the CPOT, 0.96; and for the NVPS, 0.90. The inter-rater reliability using the Shrout-Fleiss intraclass correlation coefficient (ICC) tests showed the best results after the painful procedure and during the creaming. The scores were slightly lower for the BPS compared to the CPOT and the NVPS. The test-retest results using the Bland-Altman plots show that all instruments gave similar results. Conclusions: To our knowledge, this is the first time all three behavioral pain assessment tools have been evaluated in the same study in a language other than English or French. All three tools had good internal consistency, but it was better for the CPOT and the NVPS compared to the BPS. The inter-rater reliability was best for the NVPS. The test-retest reliability was strongest for the CPOT. The three tools proved to be reliable for further testing in clinical use. Implications: There is a need for feasible, valid and reliable pain assessment tools for pain assessment of sedated ICU patients in Finland. This was the first time the psychometric properties of these tools were tested in Finnish use. Based on the results, all three instruments could be tested further in clinical use for sedated ICU patients in Finland.

Download Full-text

Reliability of the Indonesian version of the School-Years Screening Test for Evaluation of Mental Status-Revised as a cognitive screening tool for children

Paediatrica Indonesiana ◽

10.14238/pi56.3.2016.149-54 ◽

2016 ◽

Vol 56 (3) ◽

pp. 149

Author(s):

Ferriandis Harsono ◽

Purboyo Solek ◽

Kusnandi Rusmil

Keyword(s):

Mental Status ◽

Internal Consistency ◽

Behavioral Problems ◽

Screening Test ◽

Screening Tool ◽

The United States ◽

Cognitive Screening ◽

Cross Sectional ◽

Rater Reliability ◽

Language Version

Background Developmental and behavioral problems are among the most common conditions of childhood. These problems affect 12−16% of children in the United States and 13−18% of children in Indonesia. Early detection of developmental deficits among children requires clinicians to screen with accurate tools. Cognitive function screening in children has been increasingly used in many clinical and educational settings. The School-Years Screening Test for Evaluation of Mental Status-Revised (SYSTEMS-R) is becoming a widely-used, reliable, and valid cognitive screening tool for children aged 4−15 years. Prior to our study, there was no Indonesian language version of the SYSTEMS-R.Objective To determine the reliability of the Indonesian translation of the SYSTEMS-R.Methods A cross-sectional study was conducted in children aged 4−15 years who understood the Indonesian language and did not have neurologic or communication impairments. Data were analysed to determine reliability (internal consistency and inter-rater reliability) of the Indonesian version of SYSTEMS-R. Internal consistency was determined using Cronbach’s alpha formula. Internal consistency is a reflection of inter-item correlation and item-to-total correlation. Inter-rater reliability was determined using the Bland-Altman method.Results This study was conducted on 133 children aged 4−15 years in a kindergarten, elementary, junior high, and senior high school in Bandung. The Indonesian version of SYSTEMS-R had significant internal consistency (Cron bach’s alpha 0.936−0.941), and the scores obtained by two raters had good agreement (difference within mean + 1.96 SD).Conclusion The Indonesian version of SYSTEMS-R is reliable for use as a cognitive screening tool for Indonesian children. [Paediatr Indones. 2016;56:149-54.].

Download Full-text

Inter-rater reliability in Learner Corpus Research

International Journal of Learner Corpus Research ◽

10.1075/ijlcr.20001.lar ◽

2020 ◽

Vol 6 (2) ◽

pp. 237-251

Author(s):

Tove Larsson ◽

Magali Paquot ◽

Luke Plonsky

Keyword(s):

Internal Consistency ◽

Collaborative Research ◽

Instrument Design ◽

Common Source ◽

Study Quality ◽

Linguistic Features ◽

Rater Reliability ◽

Reporting Practices ◽

Learner Corpus ◽

Reliability Coefficients

Abstract In Learner Corpus Research (LCR), a common source of errors stems from manual coding and annotation of linguistic features. To estimate the amount of error present in a coded dataset, coefficients of inter-rater reliability are used. However, despite the importance of reliability and internal consistency for validity and, by extension, study quality, interpretability and generalizability, it is surprisingly uncommon for studies in the field of LCR to report on such reliability coefficients. In this Methods Report, we use a recent collaborative research project to illustrate the pertinence of considering inter-rater reliability. In doing so, we hope to initiate methodological discussion on instrument design, piloting and evaluation. We also suggest some ways forward to encourage increased transparency in reporting practices.

Download Full-text