The INTERMED Japanese version: Inter-rater reliability and internal consistency

Improvement in rider mobility represents an important functional gain for people with disabilities undergoing hippotherapy. However, there is no validated measuring instrument to track and document the rider's progress in riding activities. In this study, we aimed to develop and establish validity evidence for an instrument to assess hippotherapy participants’ mobility on horseback. We report on this development through the stages of: (a) content validation, (b) construct validation, (c) inter- and intra-rater reliability and (d) internal consistency analysis. We evaluated its factor structure with exploratory factor analyses, calculated values for inter- and intra-rater reliability using the intra-class correlation coefficient, and calculated its internal consistency using Cronbach's alpha. We followed recommendations by the Guidelines for Reporting Reliability and Agreement Studies. We found good inter-rater reliability (intra-class correlation coefficient – ICC = 0.991–0.999) and good intra-rater reliability (ICC = 0.997–1.0), and there was excellent internal consistency (Cronbach's α = 0.937–0.999). The instrument’s factor structure grouped its three domains into one factor. As this instrument is theoretically consistent and has been found to be appropriate and reliable for its intended use, it is now available for the measurement of horseback mobility among hippotherapy riders.

Download Full-text

Deterioration of housing environment and mental health of asylum seekers – a multi-level analysis

European Journal of Public Health ◽

10.1093/eurpub/ckz185.323 ◽

2019 ◽

Vol 29 (Supplement_4) ◽

Author(s):

A Mohsenpour ◽

L Biddle ◽

K Bozorgmehr

Keyword(s):

Mental Health ◽

Internal Consistency ◽

Odds Ratio ◽

Asylum Seekers ◽

Rapid Assessment ◽

Living Environment ◽

Generalized Anxiety ◽

Rater Reliability ◽

General Anxiety ◽

Housing Environment

Abstract Background Asylum seekers arriving in Germany are assigned to a local district and an accommodation centre wherein. We developed and validated a six-item questionnaire for rapid assessment of housing deterioration and investigated its association with inhabitant mental health. Methods Using cross-sectional data from a state-wide survey in Germany, we applied a random-effects modelling approach to estimate the exposure effect of housing deterioration on depression and general anxiety among a random sample of asylum seekers, using validated instruments (GAD2/PHQ2) for outcome variables. Housing deterioration was assessed on six items (windows/glass, walls/roof, garbage, graffiti, outside spaces, overall living environment) resulting in a deterioration score as exposure of interest. Additionally, we assessed the instrument’s intra- and inter-rater reliability and internal consistency. Results Of the 412 asylum seekers living in 58 accommodation centres, 45.7% reported symptoms of depression and 45.0% suffered general anxiety. Most centres (76.7%) were based in urban municipalities and 45.7% of inhabitants were living in an accommodation centre hosting ≥51 inhabitants. Preliminary adjusted odds ratio for accommodation centres with highest deterioration (Q 4) was 2.07 (0.67-6.40) for generalized anxiety, compared to 1.17 (0.45-3.08) for centres with lowest deterioration (Q 1). For depression, preliminary odds ratio was 1.92 (0.87-4.27) compared to 1.26 (0.63-2.50). The validation study confirmed inter-/intra-rater reliability (Brennan-Prediger coefficient: 0.81 and 0.92, respectively) and internal consistency (Crohnbach’s α: 0.80). Conclusions There are higher odds ratios for generalized anxiety disorder and depression among asylum seekers based on higher deterioration of housing environment. A questionnaire for rapid deterioration assessment and identification of accommodations needing further evaluation has been developed and successfully validated. Key messages Deterioration of small-scale housing environment is associated with poorer mental health for asylum seekers living in accommodation centres. A highly reliable new tool has been developed for rapid assessment of deterioration status of accommodation centres and identification of those needing further evaluation.

Download Full-text

Reliability and validity of the Japanese version of the Ocular Surface Disease Index for dry eye disease

BMJ Open ◽

10.1136/bmjopen-2019-033940 ◽

2019 ◽

Vol 9 (11) ◽

pp. e033940 ◽

Cited By ~ 7

Author(s):

Akie Midorikawa-Inomata ◽

Takenori Inomata ◽

Shuko Nojiri ◽

Masahiro Nakamura ◽

Masao Iwagami ◽

...

Keyword(s):

Internal Consistency ◽

Dry Eye ◽

Ocular Surface ◽

Discriminant Validity ◽

Reliability And Validity ◽

Japanese Version ◽

Disease Index ◽

Eye Disease ◽

Ocular Surface Disease Index ◽

Test Retest Reliability

ObjectivesThe Ocular Surface Disease Index (OSDI) questionnaire is widely used to evaluate subjective symptoms of dry eye disease (DED) as a primary diagnostic criterion. This study aimed to develop a Japanese version of the OSDI (J-OSDI) and assess its reliability and validity.Design and settingHospital-based cross-sectional observational study.ParticipantsA total of 209 patients recruited from the Department of Ophthalmology at Juntendo University Hospital.MethodsWe translated and culturally adapted the OSDI into Japanese. The J-OSDI was then assessed for internal consistency, reliability and validity. We also evaluated the optimal cut-off value to suspect DED using an area under the receiver operating characteristic curve (AUC) analysis.Primary outcome measuresInternal consistency, test–retest reliability and discriminant validity of the J-OSDI as well as the optimal cut-off value to suspect DED.ResultsOf the participants, 152 had DED and 57 did not. The J-OSDI total score showed good internal consistency (Cronbach's alpha=0.884), test–retest reliability (interclass correlation coefficient=0.910) and discriminant validity by known-group comparisons (non-DED, 19.4±16.0; DED, 37.7±22.2; p<0.001). Factor validity was used to confirm three subscales within the J-OSDI according to the original version of the questionnaire. Concurrent validity was assessed by Pearson correlation analysis, and the J-OSDI total score showed a strong positive correlation with the Dry Eye-Related Quality-of-Life Score (γ=0.829). The optimal cut-off value of the J-OSDI total score was 36.3 (AUC=0.744).ConclusionsThe J-OSDI was developed and validated in terms of reliability and validity as an effective tool for DED assessment and monitoring in the Japanese population.

Download Full-text

Reliability of three linguistically and culturally validated pain assessment tools for sedated ICU patients by ICU nurses in Finland

Scandinavian Journal of Pain ◽

10.1515/sjpain-2017-0139 ◽

2018 ◽

Vol 18 (2) ◽

pp. 165-173 ◽

Cited By ~ 1

Author(s):

Sanna-Mari Pudas-Tähkä ◽

Sanna Salanterä

Keyword(s):

Intensive Care ◽

Internal Consistency ◽

Pain Assessment ◽

Assessment Tools ◽

Endotracheal Suctioning ◽

Icu Patients ◽

Rater Reliability ◽

Intensive Care Patients ◽

Video Recordings ◽

Test Retest Reliability

Abstract Background and aims: Pain assessment in intensive care is challenging, especially when the patients are sedated. Sedated patients who cannot communicate verbally are at risk of suffering from pain that remains unnoticed without careful pain assessment. Some tools have been developed for use with sedated patients. The Behavioral Pain Scale (BPS), the Critical-Care Pain Observation Tool (CPOT) and the Nonverbal Adult Pain Assessment Scale (NVPS) have shown promising psychometric qualities. We translated and culturally adapted these three tools for the Finnish intensive care environment. The objective of this feasibility study was to test the reliability of the three pain assessment tools translated into Finnish for use with sedated intensive care patients. Methods: Six sedated intensive care patients were videorecorded while they underwent two procedures: an endotracheal suctioning was the nociceptive procedure, and the non-nociceptive treatment was creaming of the feet. Eight experts assessed the patients’ pain by observing video recordings. They assessed the pain using four instruments: the BPS, the CPOT and the NVPS, and the Numeric Rating Scale (NRS) served as a control instrument. Each expert assessed the patients’ pain at five measurement points: (1) right before the procedure, (2) during the endotracheal suctioning, (3) during rest (4) during the creaming of the feet, and (5) after 20 min of rest. Internal consistency and inter-rater reliability of the tools were evaluated. After 6 months, the video recordings were evaluated for testing the test-retest reliability. Results: Using the BPS, the CPOT, the NVPS and the NRS, 960 assessments were obtained. Internal consistency with Cronbach’s alpha coefficient varied greatly with all the instruments. The lowest values were seen at those measurement points where the pain scores were 0. The highest scores were achieved after the endotracheal suctioning at rest: for the BPS, the score was 0.86; for the CPOT, 0.96; and for the NVPS, 0.90. The inter-rater reliability using the Shrout-Fleiss intraclass correlation coefficient (ICC) tests showed the best results after the painful procedure and during the creaming. The scores were slightly lower for the BPS compared to the CPOT and the NVPS. The test-retest results using the Bland-Altman plots show that all instruments gave similar results. Conclusions: To our knowledge, this is the first time all three behavioral pain assessment tools have been evaluated in the same study in a language other than English or French. All three tools had good internal consistency, but it was better for the CPOT and the NVPS compared to the BPS. The inter-rater reliability was best for the NVPS. The test-retest reliability was strongest for the CPOT. The three tools proved to be reliable for further testing in clinical use. Implications: There is a need for feasible, valid and reliable pain assessment tools for pain assessment of sedated ICU patients in Finland. This was the first time the psychometric properties of these tools were tested in Finnish use. Based on the results, all three instruments could be tested further in clinical use for sedated ICU patients in Finland.

Download Full-text

The Japanese version of the questionnaire about the process of recovery: development and validity and reliability testing

BMC Psychiatry ◽

10.1186/s12888-017-1520-y ◽

2017 ◽

Vol 17 (1) ◽

Cited By ~ 6

Author(s):

Akiko Kanehara ◽

Risa Kotake ◽

Yuki Miyamoto ◽

Yousuke Kumakura ◽

Kentaro Morita ◽

...

Keyword(s):

Mental Health ◽

Factor Analysis ◽

Mental Health Services ◽

Health Services ◽

Internal Consistency ◽

Convergent Validity ◽

Short Form ◽

Japanese Version ◽

Internal Consistency Reliability ◽

Validity And Reliability

Abstract Background Personal recovery is increasingly recognised as an important outcome measure in mental health services. This study aimed to develop a Japanese version of the Questionnaire about the Process of Recovery (QPR-J) and test its validity and reliability. Methods The study comprised two stages that employed the cross-sectional and prospective cohort designs, respectively. We translated the questionnaire using a standard translation/back-translation method. Convergent validity was examined by calculating Pearson’s correlation coefficients with scores on the Recovery Assessment Scale (RAS) and the Short-Form-8 Health Survey (SF-8). An exploratory factor analysis (EFA) was conducted to examine factorial validity. We used intraclass correlation and Cronbach’s alpha to examine the test-retest and internal consistency reliability of the QPR-J’s 22-item full scale, 17-item intrapersonal and 5-item interpersonal subscales. We conducted an EFA along with a confirmatory factor analysis (CFA). Results Data were obtained from 197 users of mental health services (mean age: 42.0 years; 61.9% female; 49.2% diagnosed with schizophrenia). The QPR-J showed adequate convergent validity, exhibiting significant, positive correlations with the RAS and SF-8 scores. The QPR-J’s full version, subscales, showed excellent test-retest and internal consistency reliability, with the exception of acceptable but relatively low internal consistency reliability for the interpersonal subscale. Based on the results of the CFA and EFA, we adopted the factor structure extracted from the original 2-factor model based on the present CFA. Conclusion The QPR-J is an adequately valid and reliable measure of the process of recovery among Japanese users with mental health services.

Download Full-text

Reliability of the Indonesian version of the School-Years Screening Test for Evaluation of Mental Status-Revised as a cognitive screening tool for children

Paediatrica Indonesiana ◽

10.14238/pi56.3.2016.149-54 ◽

2016 ◽

Vol 56 (3) ◽

pp. 149

Author(s):

Ferriandis Harsono ◽

Purboyo Solek ◽

Kusnandi Rusmil

Keyword(s):

Mental Status ◽

Internal Consistency ◽

Behavioral Problems ◽

Screening Test ◽

Screening Tool ◽

The United States ◽

Cognitive Screening ◽

Cross Sectional ◽

Rater Reliability ◽

Language Version

Background Developmental and behavioral problems are among the most common conditions of childhood. These problems affect 12−16% of children in the United States and 13−18% of children in Indonesia. Early detection of developmental deficits among children requires clinicians to screen with accurate tools. Cognitive function screening in children has been increasingly used in many clinical and educational settings. The School-Years Screening Test for Evaluation of Mental Status-Revised (SYSTEMS-R) is becoming a widely-used, reliable, and valid cognitive screening tool for children aged 4−15 years. Prior to our study, there was no Indonesian language version of the SYSTEMS-R.Objective To determine the reliability of the Indonesian translation of the SYSTEMS-R.Methods A cross-sectional study was conducted in children aged 4−15 years who understood the Indonesian language and did not have neurologic or communication impairments. Data were analysed to determine reliability (internal consistency and inter-rater reliability) of the Indonesian version of SYSTEMS-R. Internal consistency was determined using Cronbach’s alpha formula. Internal consistency is a reflection of inter-item correlation and item-to-total correlation. Inter-rater reliability was determined using the Bland-Altman method.Results This study was conducted on 133 children aged 4−15 years in a kindergarten, elementary, junior high, and senior high school in Bandung. The Indonesian version of SYSTEMS-R had significant internal consistency (Cron bach’s alpha 0.936−0.941), and the scores obtained by two raters had good agreement (difference within mean + 1.96 SD).Conclusion The Indonesian version of SYSTEMS-R is reliable for use as a cognitive screening tool for Indonesian children. [Paediatr Indones. 2016;56:149-54.].

Download Full-text

Inter-rater reliability in Learner Corpus Research

International Journal of Learner Corpus Research ◽

10.1075/ijlcr.20001.lar ◽

2020 ◽

Vol 6 (2) ◽

pp. 237-251

Author(s):

Tove Larsson ◽

Magali Paquot ◽

Luke Plonsky

Keyword(s):

Internal Consistency ◽

Collaborative Research ◽

Instrument Design ◽

Common Source ◽

Study Quality ◽

Linguistic Features ◽

Rater Reliability ◽

Reporting Practices ◽

Learner Corpus ◽

Reliability Coefficients

Abstract In Learner Corpus Research (LCR), a common source of errors stems from manual coding and annotation of linguistic features. To estimate the amount of error present in a coded dataset, coefficients of inter-rater reliability are used. However, despite the importance of reliability and internal consistency for validity and, by extension, study quality, interpretability and generalizability, it is surprisingly uncommon for studies in the field of LCR to report on such reliability coefficients. In this Methods Report, we use a recent collaborative research project to illustrate the pertinence of considering inter-rater reliability. In doing so, we hope to initiate methodological discussion on instrument design, piloting and evaluation. We also suggest some ways forward to encourage increased transparency in reporting practices.

Download Full-text

The Development and Psychometric Validation of the Japanese Version of the Home Healthcare Nurses’ Job Satisfaction: A Cross-Sectional Study Among Japanese Home Healthcare Nurses

10.21203/rs.3.rs-841900/v1 ◽

2021 ◽

Author(s):

Yoko MORI ◽

Miki SASAKI ◽

Yasuko OGATA ◽

Taisuke TOGARI

Keyword(s):

Job Satisfaction ◽

Clinical Practice ◽

Factor Structure ◽

Internal Consistency ◽

Goodness Of Fit ◽

Japanese Version ◽

Home Healthcare ◽

Psychometric Validation ◽

Satisfaction Scale ◽

Cross Sectional

Abstract Background: High-quality and attractive work environments and nurses’ intention to stay are highly related to the job satisfaction of home healthcare nurses. Accordingly, a reliable and valid tool is required to assess home healthcare nurses’ job satisfaction for evaluating and improving the work environment and clinical practice of home healthcare. This study aimed to develop and examine the Japanese version of the Home Healthcare Nurses’ Job Satisfaction Scale (HHNJS-J).Methods: The Home Healthcare Nurses’ Job Satisfaction Scale (HHNJS) was translated into Japanese, and a backward-translation was performed until equivalence between the original and backward-translated the HHNJS was confirmed. Subsequently, a mail survey was conducted among 409 home healthcare nurses from 154 home healthcare agencies in Japan. We evaluated construct validity through confirmatory factor analysis (CFA), and criterion-related validity and internal consistency were also tested.Results: The CFA revealed, a second-order seven-factor structure and adequate internal consistency, although, the fit of the data to the factor structure was moderate. As per the goodness-of-fit indices of the final model of the CFA, the comparative fit index was 0.89 and root mean square error of approximation was 0.06. This newly translated scale can be used to assess the job satisfaction of home healthcare nurses within Japan.Conclusions: The HHNJS-J evaluated acceptable reliability and validity among Japanese home healthcare nurses and had application in clinical practice in Japan.

Download Full-text