Validation of the Simplified Modified Rankin Scale Questionnaire

2020 ◽  
Vol 83 (5) ◽  
pp. 493-499
Author(s):  
Eva Isaksson ◽  
Per Wester ◽  
Ann Charlotte Laska ◽  
Per Näsman ◽  
Erik Lundström
Keyword(s):  
Assessment Tool ◽  
Rank Correlation ◽  
Weighted Kappa ◽  
Modified Rankin Scale ◽  
Next Of Kin ◽  
Cohen’S Kappa ◽  
Time Saving ◽  
Face To Face ◽  
Cohen's Kappa ◽  
Number Of Patients

<b><i>Introduction:</i></b> The modified Rankin scale (mRS) is the most common assessment tool for measuring overall functional outcome in stroke studies. The traditional way of using mRS face-to-face is time- and cost-consuming. The aim of this study was to test the validity of the Swedish translation of the simplified modified Rankin scale questionnaire (smRSq) as compared with the mRS assessed face-to-face 6 months after a stroke. <b><i>Methods:</i></b> Within the ongoing EFFECTS trial, smRSq was sent out to 108 consecutive stroke patients 6 months after a stroke. The majority, 90% (97/108), of the patients answered the questionnaire; for the remaining 10%, it was answered by the next of kin. The patients were assessed by face-to-face mRS by 7 certified healthcare professionals at 4 Swedish stroke centres. The primary outcome was assessed by Cohen’s kappa and weighted kappa. <b><i>Results:</i></b> There was good agreement between postal smRSq, answered by the patients, and the mRS face-to-face; Cohen’s kappa was 0.43 (CI 95% 0.31–0.55), weighted kappa was 0.64 (CI 95% 0.55–0.73), and Spearman rank correlation was 0.82 (<i>p</i> &#x3c; 0.0001). In 55% (59/108), there was full agreement, and of the 49 patients not showing exact agreement, 44 patients differed by 1 grade and 5 patients had a difference of 2 grades. <b><i>Discussion/Conclusion:</i></b> Our results show good validity of the postal smRSq, answered by the patients, compared with the mRS carried out face-to-face at 6 months after a stroke. This result could help trialists in the future simplify study design and make multicentre trials and quality registers with a large number of patients more feasible and time-saving.

scholarly journals New Interpretations of Cohen’s Kappa

2014 ◽  
Vol 2014 ◽  
pp. 1-9 ◽  
Author(s):  
Matthijs J. Warrens
Keyword(s):  
Weighted Kappa ◽  
Interrater Agreement ◽  
Cohen’S Kappa ◽  
Association Coefficient ◽  
Nominal Scale ◽  
Weighted Averages ◽  
Cohen's Kappa ◽  
Single Number ◽  
Research Studies

Cohen’s kappa is a widely used association coefficient for summarizing interrater agreement on a nominal scale. Kappa reduces the ratings of the two observers to a single number. With three or more categories it is more informative to summarize the ratings by category coefficients that describe the information for each category separately. Examples of category coefficients are the sensitivity or specificity of a category or the Bloch-Kraemer weighted kappa. However, in many research studies one is often only interested in a single overall number that roughly summarizes the agreement. It is shown that both the overall observed agreement and Cohen’s kappa are weighted averages of various category coefficients and thus can be used to summarize these category coefficients.

scholarly journals Reliability of the modified Rankin Scale applied by telephone

2013 ◽  
Vol 5 (1) ◽  
pp. 2 ◽  
Author(s):  
Katia Savio ◽  
Gian Luca Della Pietra ◽  
Elodie Oddone ◽  
Monica Reggiani ◽  
Maurizio A. Leone
Keyword(s):  
Confidence Interval ◽  
Acute Stroke ◽  
Scale Score ◽  
Weighted Kappa ◽  
Hospitalized Patients ◽  
The Other ◽  
Modified Rankin Scale ◽  
Face To Face ◽  
The Face ◽  
Stroke Disability

We aimed to evaluate the reliability of the modified Rankin Scale applied telephonically compared with face-to-face assessment in clinically stable hospitalized patients with acute stroke. One hundred and thirty-one patients were interviewed twice by 2 certified nurses (unstructured interview). Half of the patients were randomized to be interviewed by telephone followed by the face-to-face assessment, and half in the reverse order. The median value of the modified Rankin Scale score was 4 (first to third interquartile range 3-5) by telephone as well as by face-to-face assessment (P=0.8). The weighted kappa between the two methods was 0.82 (95% confidence interval: 0.77-0.88). Sensitivity of the telephone assessment was lower for scores 2 and 3 (17% and 46%, respectively) than for the other scores (range 67-90%). Telephone assessment of stroke disability with the modified Rankin Scale is reliable in comparison to direct face- to-face assessment.

Learning how to Differ: Agreement and Reliability Statistics in Psychiatry

1995 ◽  
Vol 40 (2) ◽  
pp. 60-66 ◽  
Author(s):  
L. Streiner David
Keyword(s):  
Correlation Coefficient ◽  
Intraclass Correlation Coefficient ◽  
Intraclass Correlation ◽  
Weighted Kappa ◽  
The Other ◽  
Cohen’S Kappa ◽  
Cohen's Kappa ◽  
The Subject

Whenever two or more raters evaluate a patient or student, it may be necessary to determine the degree to which they assign the same label or rating to the subject. The major problem in deciding which statistic to use is the plethora of different techniques which are available. This paper reviews some of the more commonly used techniques, such as Raw Agreement, Cohen's kappa and weighted kappa, and shows that, in most circumstances, they can all be replaced by the intraclass correlation coefficient (ICC). This paper also shows how the ICC can be used in situations where the other statistics cannot be used and how to select the best subset of raters.

scholarly journals Reproducibility and relative validity of a newly developed web-based food-frequency questionnaire for assessment of preconception diet

BMC Nutrition ◽  
2019 ◽  
Vol 5 (1) ◽  
Author(s):  
L. Salvesen ◽  
E. R. Hillesund ◽  
F. N. Vik ◽  
A. L. Brantsæter ◽  
N. C. Øverby
Keyword(s):  
Correlation Coefficient ◽  
Rank Correlation ◽  
Food Groups ◽  
Cohen’S Kappa ◽  
Food Frequency ◽  
Relative Validity ◽  
Spearman’S Rank Correlation ◽  
Cohen's Kappa ◽  
Spearman’S Rank Correlation Coefficient ◽  
Median Correlation

Abstract Background The importance of diet and nutrition during preconception age is a window of opportunity to promote future parental and transgenerational health. As a sub-study to a large Norwegian study, ‘Diet today – health of tomorrow’, a food-frequency questionnaire (FFQ) was developed to assess diet during the preconception phase in young adults aged 20 – 30 years and in this paper we report the reproducibility and relative validity of this questionnaire. Methods The FFQ was developed from an existing FFQ validated in adolescents. Participants were recruited on social media and at a university. Reproducibility was assessed by comparing the test and retest of the FFQ. Relative validity was assessed by comparing intake measured by the FFQ with a 7-day weighed food record. Energy, nutrients and food intake were used to assess the reproducibility and relative validity of the FFQ. The study applied the Spearman’s rank correlation coefficient, percentage of agreement and Cohen’s Kappa to assess reproducibility and validity. Results There were 32 participants recruited to the study, of which 21 participants completed both the test-retest reproducibility and the relative validation. The test-retest reproducibility had a median correlation coefficient of 0.85 for energy and nutrients, a median Spearman’s rank correlation coefficient of 0.75 and a median Cohen’s Kappa of 0.51 for food groups. The relative validity of the FFQ had a median correlation coefficient of 0.59 for energy and nutrients, a median Spearman’s rank correlation coefficient of 0.54 and a median Cohen’s Kappa of 0.28 for food groups. Conclusion This newly developed FFQ for preconception diet in young adults had a satisfactory test-retest reproducibility and fair relative validity.

scholarly journals Application of the intervention Complexity Assessment Tool for Systematic Reviews within a Cochrane review: an illustrative case study

2020 ◽  
Vol 3 ◽  
pp. 31
Author(s):  
Cathal A. Cadogan ◽  
Audrey Rankin ◽  
Simon Lewin ◽  
Carmel M. Hughes
Keyword(s):  
Systematic Reviews ◽  
Assessment Tool ◽  
Cochrane Review ◽  
Guidance Document ◽  
Cohen’S Kappa ◽  
Rater Reliability ◽  
Cohen's Kappa ◽  
Holistic Understanding ◽  
Complexity Assessment

Background: The intervention Complexity Assessment Tool for Systematic Reviews (iCAT_SR) has been developed to facilitate detailed assessments of intervention complexity in systematic reviews. Worked examples of the tool’s application are needed to promote its use and refinement. The aim of this case study was to apply the iCAT_SR to a subset of 20 studies included in a Cochrane review of interventions aimed at improving appropriate polypharmacy in older people. Methods: Interventions were assessed independently by two authors using the six core iCAT_SR dimensions: (1) ‘Target organisational levels/categories’; (2) ‘Target behaviour/actions’; (3) ‘Active intervention components’; (4) ‘Degree of tailoring’; (5) ‘Level of skill required by intervention deliverers’; (6) ‘Level of skill required by intervention recipients’. Attempts were made to apply four optional dimensions: ‘Interaction between intervention components’; ‘Context/setting’; ‘Recipient/provider factors’; ‘Nature of causal pathway’. Inter-rater reliability was assessed using Cohen’s Kappa coefficient. Disagreements were resolved by consensus discussion. The findings are presented narratively. Results: Assessments involving the core iCAT_SR dimensions showed limited consistency in intervention complexity across included studies, even when categorised according to clinical setting. Interventions were delivered across various organisational levels and categories (i.e. healthcare professionals and patients) and typically comprised multiple components. Intermediate skill levels were required by those delivering and receiving the interventions across all studies. A lack of detail in study reports precluded application of the iCAT_SR’s optional dimensions. The inter-rater reliability was substantial (Cohen's Kappa = 0.75) Conclusions: This study describes the application of the iCAT_SR to studies included in a Cochrane systematic review. Future intervention studies need to ensure more detailed reporting of interventions, context and the causal pathways underlying intervention effects to allow a more holistic understanding of intervention complexity and facilitate replication in other settings. The experience gained has helped to refine the original guidance document relating to the application of iCAT_SR.

scholarly journals Fixed-Effects Modeling of Cohen's Weighted Kappa for Bivariate Multinomial Data: A Perspective of Generalized Inverse

2011 ◽  
Vol 2011 ◽  
pp. 1-14
Author(s):  
Jingyun Yang ◽  
Vernon M. Chinchilli
Keyword(s):  
Fixed Effects ◽  
Generalized Inverse ◽  
Weighted Kappa ◽  
Small Sample ◽  
Kappa Statistics ◽  
Cohen’S Kappa ◽  
Multinomial Data ◽  
Cohen's Kappa ◽  
Bootstrap Simulation ◽  
Measuring Agreement

Cohen's kappa and weighted kappa statistics are the conventional methods used frequently in measuring agreement for categorical responses. In this paper, through the perspective of a generalized inverse, we propose an alternative general framework of the fixed-effects modeling of Cohen's weighted kappa, proposed by Yang and Chinchilli (2011). Properties of the proposed method are provided. Small sample performance is investigated through bootstrap simulation studies, which demonstrate good performance of the proposed method. When there are only two categories, the proposed method reduces to Cohen's kappa.

scholarly journals Validation of Prototype Research Software– Infant Prakriti Assessment (PRS-IPA): Inferences from a Cross Sectional Pattern

2020 ◽  
Vol 9 (4) ◽  
pp. 391-395
Keyword(s):  
Correlation Coefficient ◽  
Rank Correlation ◽  
Kappa Coefficient ◽  
Cohen’S Kappa ◽  
Cross Sectional ◽  
Research Software ◽  
Spearman’S Rank Correlation ◽  
Cohen's Kappa ◽  
Spearman’S Rank Correlation Coefficient ◽  
Assessment Questionnaire

Introduction: Prakriti determination in an infant helps to live more healthy and prosperous life by changing their rearing plan accordingly to prevent onset of related disorders in future. Validation under this study was planned to evaluate the agreement between Infant’s Prakriti Assessment questionnaire (IPAQ) and PRS-IPA (Prototype Research Software-Infant Prakriti Assessment), a software for Prakriti assessment in infants. Materials and methods: A cross sectional study was carried out in Sir Sunderlal Hospital BHU and data were analyzed through two step Prakriti determination process i.e. first, by Infant’s Prakriti Assessment questionnaire (IPAQ) and second by PRS-IPA. Validation was done by Spearman’s Rank Correlation Coefficient and Cohen's kappa coefficient. Observations and Result: In this study, Cohen's kappa coefficient was found 0.955 which is highly significant. Prakriti assessed by PRS-IPA software is valid. Validation by Spearman’s Rank Correlation coefficient was also found highly significant (p=0.000). Discussion & Conclusion: The data, derived from spearman’s and kappa suggest its validity and PRS-IPA can be used for the Prakriti assessment with accuracy in infants

scholarly journals Empirical evaluation of SUCRA-based treatment ranks in network meta-analysis: quantifying robustness using Cohen’s kappa

BMJ Open ◽  
2019 ◽  
Vol 9 (9) ◽  
pp. e024625 ◽  
Author(s):  
Caitlin H Daly ◽  
Binod Neupane ◽  
Joseph Beyene ◽  
Lehana Thabane ◽  
Sharon E Straus ◽  
...  
Keyword(s):  
Meta Analysis ◽  
Weighted Kappa ◽  
Data Sets ◽  
Cohen’S Kappa ◽  
Data Set ◽  
Small Shift ◽  
Factors Associated ◽  
Treatment Comparison ◽  
Cohen's Kappa ◽  
Potential Factors

ObjectiveTo provide a framework for quantifying the robustness of treatment ranks based on Surface Under the Cumulative RAnking curve (SUCRA) in network meta-analysis (NMA) and investigating potential factors associated with lack of robustness.MethodsWe propose the use of Cohen’s kappa to quantify the agreement between SUCRA-based treatment ranks estimated through NMA of a complete data set and a subset of it. We illustrate our approach using five published NMA data sets, where robustness was assessed by removing studies one at a time.ResultsOverall, SUCRA-based treatment ranks were robust to individual studies in the five data sets we considered. We observed more incidences of disagreement between ranks in the networks with larger numbers of treatments. Most treatments moved only one or two ranks up or down. The lowest quadratic weighted kappa estimate observed across all networks was in the network with the smallest number of treatments (4), where weighted kappa=40%. In the network with the largest number of treatments (12), the lowest observed quadratic weighted kappa=89%, reflecting a small shift in this network's treatment ranks overall. Preliminary observations suggest that a study’s size, the number of studies making a treatment comparison, and the agreement of a study’s estimated treatment effect(s) with those estimated by other studies making the same comparison(s) may explain the overall robustness of treatment ranks to studies.ConclusionsInvestigating robustness or sensitivity in an NMA may reveal outlying rank changes that are clinically or policy-relevant. Cohen’s kappa is a useful measure that permits investigation into study characteristics that may explain varying sensitivity to individual studies. However, this study presents a framework as a proof of concept and further investigation is required to identify potential factors associated with the robustness of treatment ranks using more extensive empirical evaluations.

Abstract WP276: Simplification of a Prehospital Short NIHSS Scale Does not Increase Interrater Agreement Between Emergency Medical Services and Stroke Specialists

Stroke ◽  
2017 ◽  
Vol 48 (suppl_1) ◽  
Author(s):  
Jelle Demeestere ◽  
Carlos Garcia-Esperon ◽  
Longting Lin ◽  
Allan Loudfoot ◽  
Andrew Bivard ◽  
...  
Keyword(s):  
Emergency Medical Services ◽  
Weighted Kappa ◽  
Interrater Agreement ◽  
Medical Services ◽  
Kappa Statistics ◽  
Single Centre ◽  
Cohen’S Kappa ◽  
Cohen's Kappa ◽  
Emergency Medical ◽  
Patient Arrival

Objective: To assess if simplifying a prehospital 8-item NIHSS scale (NIHSS-8, fig 1) to a 0 (symptom absent) – 1 (symptom present) scoring system increases interrater agreement between emergency medical services (EMS) and stroke specialists. Methods: We analysed interrater agreement between EMS and stroke specialists of a single centre on a prospectively collected cohort of 64 suspected acute ischemic stroke patients. EMS performed NIHSS-8 score upon patient arrival at the emergency department. The stroke specialist scored the full 15-item NIHSS blind to the EMS scores and within 5 minutes of patient arrival. Linear-weighted Cohen’s kappa statistics was used to assess agreement between EMS and stroke specialist on the total NIHSS-8 score and each NIHSS-8 scale item. We then simplified each item to a 0-1 score and reassessed interrater agreement for the overall NIHSS-8 scale using linear-weighted Cohen’s kappa statistics and for each NIHSS-8 item using Cohen’s kappa statistics. We used Cohen’s kappa statistics to assess agreement for original and simplified NIHSS-8 cut-off scores. Results: EMS and stroke specialist reached substantial agreement on overall NIHSS-8 scoring (linear-weighted kappa 0.69). Optimum agreement was reached for right arm weakness (linear-weighted kappa 0.79; Table 1) and a cut-off score of 2 and 5 (Cohen’s kappa 0.78; Table 2). When the score was simplified to a 0-1, overall agreement between EMS and stroke specialists was substantial (linear-weighted kappa 0.65). Optimum agreement was seen for LOC questions (Cohen’s kappa 0.78; Table 1) and a cut-off score of 2 (Cohen’s kappa 0.77; Table 2). Conclusion: Simplifying an 8-item prehospital NIHSS stroke scale does not increase interrater agreement between emergency medical services and stroke specialists.

Sign in / Sign up

Export Citation Format

Share Document