Development and reliability of the AOSpine CROST (Clinician Reported Outcome Spine Trauma): a tool to evaluate and predict outcomes from clinician’s perspective

Abstract Purpose To report on the development of AOSpine CROST (Clinician Reported Outcome Spine Trauma) and results of an initial reliability study. Methods The AOSpine CROST was developed using an iterative approach of multiple cycles of development, review, and revision including an expert clinician panel. Subsequently, a reliability study was performed among an expert panel who were provided with 20 spine trauma cases, administered twice with 4-week interval. The results of the developmental process were analyzed using descriptive statistics, the reliability per parameter using Kappa statistics, inter-rater rater agreement using intraclass correlation coefficient (ICC), and internal consistency using Cronbach’s α. Results The AOSpine CROST was developed and consisted of 10 parameters, 2 of which are only applicable for surgically treated patents (‘Wound healing’ and ‘Implants’). A dichotomous scoring system (‘yes’ or ‘no’ response) was incorporated to express expected problems for the short term and long term. In the reliability study, 16 (84.2%) participated in the first round and 14 (73.7%) in the second. Intra-rater reliability was fair to good for both time points (κ = 0.40–0.80 and κ = 0.31–0.67). Results of inter-rater reliability were lower (κ = 0.18–0.60 and κ = 0.16–0.46). Inter-rater agreement for total scores showed moderate results (ICC = 0.52–0.60), and the internal consistency was acceptable (α = 0.76–0.82). Conclusions The AOSpine CROST, an outcome tool for the surgeons, was developed using an iterative process. An initial reliability analysis showed fair to moderate results and acceptable internal consistency. Further clinical validation studies will be performed to further validate the tool.

Download Full-text

Development and assessment of the inter-rater and intra-rater reproducibility of a self-administration version of the ALSFRS-R

Journal of Neurology Neurosurgery & Psychiatry ◽

10.1136/jnnp-2019-321138 ◽

2019 ◽

Vol 91 (1) ◽

pp. 75-81 ◽

Cited By ~ 7

Author(s):

Leonhard A Bakker ◽

Carin D Schröder ◽

Harold H G Tan ◽

Simone M A G Vugts ◽

Ruben P A van Eijk ◽

...

Keyword(s):

Rating Scale ◽

Clinical Care ◽

Intraclass Correlation ◽

Correlation Coefficients ◽

The Self ◽

Coefficient Alpha ◽

Rater Agreement ◽

Self Administration ◽

Limits Of Agreement ◽

Rater Reliability

ObjectiveThe Amyotrophic Lateral Sclerosis Functional Rating Scale-Revised (ALSFRS-R) is widely applied to assess disease severity and progression in patients with motor neuron disease (MND). The objective of the study is to assess the inter-rater and intra-rater reproducibility, i.e., the inter-rater and intra-rater reliability and agreement, of a self-administration version of the ALSFRS-R for use in apps, online platforms, clinical care and trials.MethodsThe self-administration version of the ALSFRS-R was developed based on both patient and expert feedback. To assess the inter-rater reproducibility, 59 patients with MND filled out the ALSFRS-R online and were subsequently assessed on the ALSFRS-R by three raters. To assess the intra-rater reproducibility, patients were invited on two occasions to complete the ALSFRS-R online. Reliability was assessed with intraclass correlation coefficients, agreement was assessed with Bland-Altman plots and paired samples t-tests, and internal consistency was examined with Cronbach’s coefficient alpha.ResultsThe self-administration version of the ALSFRS-R demonstrated excellent inter-rater and intra-rater reliability. The assessment of inter-rater agreement demonstrated small systematic differences between patients and raters and acceptable limits of agreement. The assessment of intra-rater agreement demonstrated no systematic changes between time points; limits of agreement were 4.3 points for the total score and ranged from 1.6 to 2.4 points for the domain scores. Coefficient alpha values were acceptable.DiscussionThe self-administration version of the ALSFRS-R demonstrates high reproducibility and can be used in apps and online portals for both individual comparisons, facilitating the management of clinical care and group comparisons in clinical trials.

Download Full-text

Comparison of Kristjansson Respiratory Score and Wang Respiratory Score in infants with bronchiolitis in a hospital emergency department

Hong Kong Physiotherapy Journal ◽

10.1142/s1013702520500146 ◽

2020 ◽

Vol 40 (02) ◽

pp. 145-153

Author(s):

Frederico Ramos Pinto ◽

Liane Correia-Costa ◽

Inês Azevedo

Keyword(s):

Emergency Department ◽

Emergency Room ◽

Internal Consistency ◽

Health Professionals ◽

Prospective Observational Study ◽

Intraclass Correlation ◽

Weighted Kappa ◽

Hospital Emergency ◽

Rater Reliability ◽

Hospital Emergency Room

Objective: Several respiratory scores have been created to evaluate bronchiolitis’ severity level, but it is still not clear which is the best score. The aim of this study is to compare the Wang Respiratory Score (WRS) and the Kristjansson Respiratory Score (KRS) in the setting of an emergency room. Methods: We performed a prospective observational study with 60 infants with bronchiolitis admitted to a paediatric emergency department. For both scores, we assessed inter-rater reliability between two different health professionals (physician and physiotherapist), internal consistency, and correlation with SpO2 testing the intraclass-correlation coefficient (ICC), weighted kappa, Cronbach [Formula: see text] coefficient and Spearman tests, respectively. Results: The inter-rater reliability was higher in KRS (ICC 0.79) and the Cronbach [Formula: see text] and weighted kappa had similar values in KRS versus WRS. The correlation between the KRS/WRS and SpO2 was poor/moderate upon admission and discharge for the first observer and the second observer. Conclusions: While the internal consistency was similar in both scores, inter-rater reliability of KRS was higher than WRS, which allows us to conclude that it would have more consistent results when used to assess bronchiolitis’ level of severity by health personnel in a busy hospital emergency room.

Download Full-text

Mansoura Fiberoptic Endoscopic Evaluation of Swallowing Residue Rating Scale (MFRRS): An Anatomically Based Tool – A Preliminary Study

Folia Phoniatrica et Logopaedica ◽

10.1159/000512158 ◽

2020 ◽

pp. 1-13

Author(s):

Aliaa Sabry ◽

James L. Coyle ◽

Tamer Abou-Elsaad

Keyword(s):

Internal Consistency ◽

Rating Scale ◽

Pyriform Sinus ◽

Hard Copy ◽

Kappa Statistics ◽

Endoscopic Evaluation ◽

Rater Reliability ◽

Functional Problem ◽

Excellent Internal Consistency ◽

Residue Levels

Objective: The aim of this work was to design an anatomically based scale for judging post-swallow residue in the pharyngeal cavities, for use during the fiberoptic endoscopic evaluation of swallowing (FEES) in patients with dysphagia, and to assess its feasibility. Methods: Two 7-point ordinal scales (one for vallecular residue and one for pyriform sinus residue), were developed using detailed anatomic landmarks to denote residue levels. Hard copy color images of a specified frame, from 210 videos of 30 adult FEES evaluations demonstrating the range of all possible residue patterns, were selected (n = 56 valleculae, 62 pyriform sinuses). Half of these images were used to train 4 raters. The remaining half of the images were randomly ordered and rated by the trained raters. Two weeks later the same images were randomized again, and each rater re-analyzed them. The inter- and intra-rater reliability and criterion validity were determined using the kappa statistics and their standard errors. The internal consistency of the items in MFRRS was examined. Results: MFRRS showed strong inter-rater reliability (valleculae, κ = 0.832 ± 0.038; pyriform sinus, κ = 0.855 ± 0.034), almost perfect intra-rater reliability (valleculae, κ = 0.964 ± 0.018; pyriform sinus, κ = 0.962 ± 0.02), almost perfect concurrent validity (valleculae, κ = 0.968 ± 0.020; pyriform sinus, κ = 0.0971 ± 0.017), and excellent internal consistency (valleculae, Cronbach’s α = 0.990; pyriform sinus, Cronbach’s α = 0.985). Conclusion: MFRRS is a feasible and reliable, anatomically based tool that can provide more accurate pharyngeal residue judgments. The optimized description of residue accumulation patterns can contribute to a better overall description of the functional problem and future description of dysphagia phenotypes.

Download Full-text

Development and reliability of the Korean version of the Feeding Abilities Assessment

Hong Kong Journal of Occupational Therapy ◽

10.1177/1569186119850694 ◽

2019 ◽

Vol 32 (1) ◽

pp. 69-74

Author(s):

Seul Gi Koo ◽

Hae Yean Park ◽

Jongbae Kim ◽

Areum Han

Keyword(s):

Correlation Coefficient ◽

Internal Consistency ◽

Content Validity ◽

Assessment Tool ◽

High Reliability ◽

Intraclass Correlation ◽

Rater Reliability ◽

Retest Reliability ◽

Korean Version ◽

Test Retest Reliability

Objective The purpose of this study is to introduce a standardised assessment tool by verifying the reliability of the translated Korean version of the Feeding Abilities Assessment (K-FAA), which was developed to suit Korean culture. Methods The research subjects were 65 patients with dementia living in nursing homes. The K-FAA was completed by verifying the suitability of translation and reverse translation. The validity of the K-FAA was established through content validity, while its reliability was analysed based on internal consistency reliability for the items, test–retest reliability and inter-rater reliability. Results The content validity index determined, based on the assessment of professors, occupational therapists, and nurses, was more than .70. Cronbach’s α was more than .929, showing good internal consistency. A test–retest reliability of .884 was derived using Pearson’s correlation coefficient (p < .01), and an inter-rater reliability of .800 was derived using the kappa coefficients; intraclass correlation coefficient was .897, which also indicated good reliability. Conclusion The K-FAA was modified to fit the Korean domestic situation, and this assessment had high reliability. Therefore, K-FAA can evaluate the feeding ability of patients with dementia. Future studies should focus on providing evidence-based data to maintain or supplement the feeding ability of patients with dementia in Korea.

Download Full-text

Validity and Reliability of the Comfort Behavior Scale in Children Undergoing Wound Dressing Replacement in Vietnam

MedPharmRes ◽

10.32895/ump.mpr.4.3.6 ◽

2020 ◽

Vol 4 (3) ◽

pp. 24-31

Author(s):

Huong Lan Thi Tran ◽

Katrina Einhellig ◽

Linh Thuy Khanh Tran ◽

Oanh Hoang Thi Pham ◽

Cuc Thu Thi Tran ◽

...

Keyword(s):

Wound Dressing ◽

Intraclass Correlation ◽

Correlation Coefficients ◽

World Health ◽

Kappa Statistics ◽

Validity And Reliability ◽

Rater Reliability ◽

Behavior Scale ◽

Health Organization ◽

Children's Pain

Background: This study translated and culturally adapted the Comfort Behavior Scale (Comfort-B) into Vietnamese using a standard protocol guided by the World Health Organization. Methods: The Comfort-B was translated into Vietnamese and then English back-translated by independent translators. These versions were reviewed and assessed by a Vietnamese expert’s panel and an English expert’s panel. Thirty-four nurses of the Nhi Dong 1 Hospital were invited to use the Vietnamese Comfort-B to assess pain while watching five videos recorded before, during and after wound dressing replacement. The eight characteristics of the Vietnamese Comfort-B were assessed by 34 nurses. Fifteen nurses agreed to do the second assessment two weeks from the first assessment. The content validity index was used to assess the relevance and clarity of all items and the whole scale. Agreements between raters were explored using Kappa statistics. Intraclass correlation coefficients (ICC) were used to assess intra-rater and inter-rater reliability. Multi-level linear regression was used to assess changes in the Vietnamese Comfort-B before, during and after wound dressing replacement between two assessments. Results: The Vietnamese Comfort-B was accredited by the Vietnamese expert’s panel. The English-back translated version was approved by the English expert’s panel. The nurses agreed that the Vietnamese Comfort-B can be used in clinical practice and research. Kappas of all items were ≥0.96 indicating excellent agreement between raters. Alpha coefficients of two assessments were ≥0.97 indicating excellent internal consistency. All ICCs ≥ 0.79 indicated good intra-rater and inter-rater reliability. Conclusions: The study suggested that the Vietnamese Comfort-B can be used for future studies assessing children’s pain in the local hospital context.

Download Full-text

Assessing Electronic Note Quality Using the Physician Documentation Quality Instrument (PDQI-9)

Applied Clinical Informatics ◽

10.4338/aci-2011-11-ra-0070 ◽

2012 ◽

Vol 03 (02) ◽

pp. 164-174 ◽

Cited By ~ 39

Author(s):

S. Bakken ◽

J.O. Wrenn ◽

E.L. Siegler ◽

P.D. Stetson

Keyword(s):

Internal Consistency ◽

Discriminant Validity ◽

Intraclass Correlation ◽

Internal Consistency Reliability ◽

Standard Group ◽

Validity And Reliability ◽

Rater Reliability ◽

Documentation Quality ◽

General Impression ◽

Quality Instrument

SummaryObjective: To refine the Physician Documentation Quality Instrument (PDQI) and test the validity and reliability of the 9-item version (PDQI-9).Methods: Three sets each of admission notes, progress notes and discharge summaries were evaluated by two groups of physicians using the PDQI-9 and an overall general assessment: one gold standard group consisting of program or assistant program directors (n = 7), and the other of attending physicians or chief residents (n = 24). The main measures were criterion-related validity (correlation coefficients between Total PDQI-9 scores and 1-item General Impression scores for each note), discriminant validity (comparison of PDQI-9 scores on notes rated as best and worst using 1-item General Impression score), internal consistency reliability (Cronbach’s alpha), and inter-rater reliability (intraclass correlation coefficient (ICC)).Results: The results were criterion-related validity (r = –0.678 to 0.856), discriminant validity (best versus worst note, t = 9.3, p = 0.003), internal consistency reliability (Cronbach’s alphas = 0.87–0.94), and inter-rater reliability (ICC = 0.83, CI = 0.72–0.91).Conclusion: The results support the criterion-related and discriminant validity, internal consistency reliability, and inter-rater reliability of the PDQI-9 for rating the quality of electronic physician notes. Tools for assessing note redundancy are required to complement use of PDQI-9. Trials of the PDQI-9 at other institutions, of different size, using different EHRs, and incorporating additional physician specialties and notes of other healthcare providers are needed to confirm its generaliz-ability.

Download Full-text

Validation and Inter-rater Reliability of the Modified Videofluoroscopic Dysphagia Scale (mVDS) in Dysphagic Patients with Multiple Etiologies

Journal of Clinical Medicine ◽

10.3390/jcm10132990 ◽

2021 ◽

Vol 10 (13) ◽

pp. 2990

Author(s):

Min Cheol Chang ◽

Changbae Lee ◽

Donghwi Park

Keyword(s):

Aspiration Pneumonia ◽

Intraclass Correlation ◽

Clinical Settings ◽

Rater Agreement ◽

Rater Reliability ◽

Feeding Method ◽

Pharyngeal Swallow ◽

Long Term Prognosis ◽

Dysphagic Patients

Background: the Videofluoroscopic Dysphagia Scale (VDS) is used to interpret and predict the long-term prognosis of patients with dysphagia. However, the inter-rater agreement of the VDS was shown to be lower in a previous study. To overcome the mentioned limitation of the VDS, a modified version (mVDS) was created and applied clinically. We aimed to validate its usefulness in determining the appropriate feeding method and predicting the prognosis of dysphagia. Methods: the videofluroscopic swallowing study (VFSS) data of 50 patients with dysphagia were collected retrospectively. The VFSS data were evaluated using the mVDS, and the inter-rater reliability was calculated. We also evaluated the association between the mVDS and type of feeding method selected, and between the mVDS and presence of aspiration pneumonia in patients with dysphagia. Results: among the different parameters of mVDS, “aspiration” showed the highest reliability (k = 0.767), followed by “mastication” and “lip closure” (k = 0.648 and k = 0.634, respectively). Conversely, “triggering pharyngeal swallow” and “pyriformis residue” demonstrated the lowest reliabilities (k = 0.312 and k = 0.324, respectively). The intraclass correlation coefficient (ICC), which is used as a measure of the reliability of the total mVDS score, was 0.876. In all patients with dysphagia, the mVDS score correlated significantly with the type of feeding method selected (p < 0.05), and the presence of aspiration pneumonia (p < 0.05). Conclusion: the ICC of the total mVDS score was 0.876. Therefore, the mVDS could be a useful tool for quantifying the severity of dysphagia. It could be helpful in the analysis of the VFSS findings among patients with dysphagia in clinical settings and research.

Download Full-text

Assessment of Psycho-Education of Carers Questionnaire: APEC-U” translation and cross cultural adaptation of an Urdu Version

Pakistan Journal of Medical Sciences ◽

10.12669/pjms.35.4.661 ◽

2019 ◽

Vol 35 (4) ◽

Author(s):

Imran Ijaz Haider ◽

Farah Tiwana ◽

Noor Zohra ◽

Khaleeq Ur Rehman

Keyword(s):

Internal Consistency ◽

Cultural Adaptation ◽

Intraclass Correlation ◽

Correlation Coefficients ◽

Internal Consistency Reliability ◽

Cross Cultural ◽

Kappa Statistics ◽

Cross Cultural Adaptation ◽

Creative Commons ◽

Good Mental Health

Background and Objective: Psycho-education of carers is a part of good mental health practice. Our objective was to translate and validate the English questionnaire “Assessment of Psycho-Education of Carers” (APEC) into Urdu (APEC-U), for use in Pakistan. Methods: Following development and validation of APEC, it was translated into Urdu after consultation with experts and translators. After pretesting, one hundred and twenty bilingual male 67(55.8%) and female 53(44.2%) primary carers, who could understand both Urdu and English, and were carers for more than three months, were asked to fill in the self reporting Urdu questionnaire at the Fatima Memorial Hospital Psychiatry Out-Patient Department. The data were collected over a period of three months from September, 2018 to November, 2018. Main outcome measures: Responses were analyzed for internal consistency, reliability, Intraclass correlation coefficients and kappa statistics. Results: APEC-U was understandable and capable of assessing psycho-education in Urdu. High internal consistency was demonstrated on the full scale as 0.859. Degree of agreement (<0.001) between the Urdu and the originally developed English version was evaluated by Cohen’s Kappa, and a high degree of agreement was demonstrated. Conclusion: The Urdu questionnaire can adequately assess psycho-education of carers in psychiatric settings. doi: https://doi.org/10.12669/pjms.35.4.661 How to cite this:Haider II, Tiwana F, Zohra N, Khaleeq Ur Rehman. Assessment of Psycho-Education of Carers Questionnaire: APEC-U” translation and cross cultural adaptation of an Urdu Version. Pak J Med Sci. 2019;35(4):---------. doi: https://doi.org/10.12669/pjms.35.4.661 This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Download Full-text

Reliability and Validity of the UMove Mobility Screen

Journal of Nursing Measurement ◽

10.1891/jnm-d-21-00001 ◽

2021 ◽

pp. JNM-D-21-00001

Author(s):

Chris Wells ◽

Julie Pittas ◽

Cynthia Roman ◽

Krystal Lighty ◽

Barbara Resnick

Keyword(s):

Construct Validity ◽

Internal Consistency ◽

Intraclass Correlation ◽

Reliability And Validity ◽

Functional Mobility ◽

Alpha Coefficient ◽

Rater Reliability ◽

Interclass Correlation ◽

The Mean ◽

Future Work

Background and PurposeThe UMove Mobility Screen (UMove) was developed to help bedside nurses accurately assess patient’s mobility. Objectives: The purpose of this study was to assess reliability and validity of the UMove.MethodsInterclass correlation coefficient (ICC) and alpha coefficient was completed to assess was based on internal consistency and inter-rater reliability. Construct validity was determined by ICC using two-way random model.ResultsAmong the 176 participants the mean age was 57 years (SD = 15), and 60% were men (N = 105). Internal consistency and inter-rater reliability were acceptable (alpha coefficient of .94 and an intraclass correlation of .98). There was evidence of construct validity with an intraclass correlation of .95 between the UMove and the standard therapists' evaluation of patient functional mobility.ConclusionsThere was evidence for reliability and validity of UMove. Future work should focus on the effectiveness of UMove on clinical outcomes.

Download Full-text

Histologic processing artifacts and inter-pathologist variation in measurement of inked margins of canine mast cell tumors

Journal of Veterinary Diagnostic Investigation ◽

10.1177/1040638718757582 ◽

2018 ◽

Vol 30 (3) ◽

pp. 377-385 ◽

Cited By ~ 4

Author(s):

Patti K. Kiser ◽

Christiane V. Löhr ◽

Danielle Meritet ◽

Sean T. Spagnoli ◽

Milan Milovancev ◽

...

Keyword(s):

Mast Cell ◽

Intraclass Correlation ◽

Rank Correlation ◽

Correlation Coefficients ◽

Kappa Statistics ◽

Spearman Rank Correlation ◽

Rater Reliability ◽

Intraclass Correlation Coefficients ◽

Mast Cell Tumors ◽

Poor Understanding

Although quantitative assessment of margins is recommended for describing excision of cutaneous malignancies, there is poor understanding of limitations associated with this technique. We described and quantified histologic artifacts in inked margins and determined the association between artifacts and variance in histologic tumor-free margin (HTFM) measurements based on a novel grading scheme applied to 50 sections of normal canine skin and 56 radial margins taken from 15 different canine mast cell tumors (MCTs). Three broad categories of artifact were 1) tissue deformation at inked edges, 2) ink-associated artifacts, and 3) sectioning-associated artifacts. The most common artifacts in MCT margins were ink-associated artifacts, specifically ink absent from an edge (mean prevalence: 50%) and inappropriate ink coloring (mean: 45%). The prevalence of other artifacts in MCT skin was 4–50%. In MCT margins, frequency-adjusted kappa statistics found fair or better inter-rater reliability for 9 of 10 artifacts; intra-rater reliability was moderate or better in 9 of 10 artifacts. Digital HTFM measurements by 5 blinded pathologists had a median standard deviation (SD) of 1.9 mm (interquartile range: 0.8–3.6 mm; range: 0–6.2 mm). Intraclass correlation coefficients demonstrated good inter-pathologist reliability in HTFM measurement (κ = 0.81). Spearman rank correlation coefficients found negligible correlation between artifacts and HTFM SDs ( r ≤ 0.3). These data confirm that although histologic artifacts commonly occur in inked margin specimens, artifacts are not meaningfully associated with variation in HTFM measurements. Investigators can use the grading scheme presented herein to identify artifacts associated with tissue processing.

Download Full-text