scholarly journals Inter-rater reliability of paediatric emergency assessment: physiological and clinical features

2020 ◽  
pp. archdischild-2019-318664
Author(s):  
Calvin Heal ◽  
Sarah Cotterill ◽  
Andrew Graeme Rowland ◽  
Natalie Garratt ◽  
Tony Long ◽  
...  
Keyword(s):  
Clinical Features ◽  
Emergency Departments ◽  
Assessment Tool ◽  
Vital Signs ◽  
Scoring Systems ◽  
Weighted Kappa ◽  
Urgent Care ◽  
Convenience Sample ◽  
Rater Reliability ◽  
North West

ObjectiveThe Paediatric Admission Guidance in the Emergency Department (PAGE) score is an assessment tool currently in development that helps predict hospital admission using components including patient characteristics, vital signs (heart rate, temperature, respiratory rate and oxygen saturation) and clinical features (eg, breathing, behaviour and nurse judgement). It aims to assist in safe admission and discharge decision making in environments such as emergency departments and urgent care centres. Determining the inter-rater reliability of scoring tools such as PAGE can be difficult. The aim of this study was to determine the inter-rater reliability of seven clinical components of the PAGE Score.DesignInter-rater reliability was measured by each patient having their clinical components recorded by two separate raters in succession. The first rater was the assessing nurse, and the second rater was a research nurse.SettingTwo emergency departments and one urgent care centre in the North West of England. Measurements were recorded over 1 week; data were collected for half a day at each of the three sites.PatientsA convenience sample of 90 paediatric attendees (aged 0–16 years), 30 from each of the three sites.Main outcome measuresTwo independent measures for each child were compared using kappa or prevalence-adjusted bias-adjusted kappa (PABAK). Bland-Altman plots were also constructed for continuous measurements.ResultsInter-rater reliability ranged from moderate (0.62 (95% CI 0.48 to 0.74) weighted kappa) to very good (0.98 (95% CI 95 to 0.99) weighted kappa) for all measurements except ‘nurse judgement’ for which agreement was fair (0.30, 95% CI 0.09 to 0.50 PABAK). Complete information from both raters on all the clinical components of the PAGE score were available for 73 children (81%). These total scores showed good’ inter-rater reliability (0.64 (95% CI 0.53 to 0.74) weighted kappa).ConclusionsOur findings suggest different nurses would demonstrate good inter-rater reliability when collecting acute assessments needed for the PAGE score, reinforcing the applicability of the tool. The importance of determining reliability in scoring systems is highlighted and a suitable methodology was presented.

scholarly journals Development of the Huddle Observation Tool for structured case management discussions to improve situation awareness on inpatient clinical wards

2017 ◽  
Vol 27 (5) ◽  
pp. 365-372 ◽  
Author(s):  
Julian Edbrooke-Childs ◽  
Jacqueline Hayes ◽  
Evelyn Sharples ◽  
Dawid Gondek ◽  
Emily Stapley ◽  
...  
Keyword(s):  
Case Management ◽  
Correlation Coefficient ◽  
Intraclass Correlation Coefficient ◽  
Situation Awareness ◽  
Assessment Tool ◽  
Intraclass Correlation ◽  
Weighted Kappa ◽  
Observational Assessment ◽  
Rater Reliability ◽  
Team Processes

Background‘Situation Awareness For Everyone’ (SAFE) was a 3-year project which aimed to improve situation awareness in clinical teams in order to detect potential deterioration and other potential risks to children on hospital wards. The key intervention was the ‘huddle’, a structured case management discussion which is central to facilitating situation awareness. This study aimed to develop an observational assessment tool to assess the team processes occurring during huddles, including the effectiveness of the huddle.MethodsA cross-sectional observational design was used to psychometrically develop the ‘Huddle Observation Tool’ (HOT) over three phases using standardised psychometric methodology. Huddles were observed across four NHS paediatric wards participating in SAFE by five researchers; two wards within specialist children hospitals and two within district general hospitals, with location, number of beds and length of stay considered to make the sample as heterogeneous as possible. Inter-rater reliability was calculated using the weighted kappa and intraclass correlation coefficient.ResultsInter-rater reliability was acceptable for the collaborative culture (weighted kappa=0.32, 95% CI 0.17 to 0.42), environment items (weighted kappa=0.78, 95% CI 0.52 to 1) and total score (intraclass correlation coefficient=0.87, 95% CI 0.68 to 0.95). It was lower for the structure and risk management items, suggesting that these were more variable in how observers rated them. However, agreement on the global score for huddles was acceptable.ConclusionWe developed an observational assessment tool to assess the team processes occurring during huddles, including the effectiveness of the huddle. Future research should examine whether observational evaluations of huddles are associated with other indicators of safety on clinical wards (eg, safety climate and incidents of patient harm), and whether scores on the HOT are associated with improved situation awareness and reductions in deterioration and adverse events in clinical settings, such as inpatient wards.

scholarly journals P-OGC50 An Objective Evaluation of Youtube Videos on Oesophageal Cancer using the PEMAT Score

2021 ◽  
Vol 108 (Supplement_9) ◽  
Author(s):  
Aya Musbahi ◽  
Arul Immanuel
Keyword(s):  
Oesophageal Cancer ◽  
Assessment Tool ◽  
Scoring Systems ◽  
Objective Evaluation ◽  
Case Scenario ◽  
Rater Reliability ◽  
Education Material ◽  
Search Terms ◽  
Youtube Videos

Abstract Background Studies in patient literature particularly regarding online video literature in all fields are few. Scoring systems for video materials such as the validated PEMAT(Patient Education Material Assessment Tool)  have been used before to look at video patient literature. The aim of this study is to use the PEMAT tool to evaluate the quality of Youtube patient literature on oesophageal cancer and look at the inter rater reliability between lay and medical scorers. Methods A Youtube search was performed in April 2021 using the search terms “oesophageal cancer”, “esophageal cancer” “gullet cancer”. Characteristic data collected included language, ratings (thumbs up), type of video, country of origin and presence of advertising as well as intended audience. A PEMAT tool which is validated instrument to rate patient video material was used. A score of 70% is acceptable in the actionability and understandability domains. Cohen’s kappa coefficient was used to test inter-rater reliability between two lay person raters; and two medical raters. Results Seven sites were rated as understandable by the medical raters average and 13 were rated understandable by the lay raters average. Only two videos achieved best case scenario where both medical raters rated as understandable, rather than the average of both. Twelve videos were rated by both lay raters as understandable. Actionability rated poorer with only two videos rated as actionable on average by the medical raters and seven rated actionable by the lay raters on average. Conclusions Youtube videos on Oesophageal cancer score poorly in terms of actionability and understandability.

scholarly journals The utility of the Brief Edinburgh Depression Scale (BEDS) in assessing severity of depression in advanced cancer patients

BJPsych Open ◽  
2021 ◽  
Vol 7 (S1) ◽  
pp. S42-S42
Author(s):  
Zeryab Meyer ◽  
Christopher Shiels ◽  
Christopher Dowrick ◽  
Mari Lloyd-Williams
Keyword(s):  
Cancer Patients ◽  
Advanced Cancer ◽  
Assessment Tool ◽  
Depression Scale ◽  
Severe Depression ◽  
Weighted Kappa ◽  
Advanced Cancer Patients ◽  
Mild Depression ◽  
North West ◽  
Severity Of Depression

AimsWhen using an assessment tool, brevity and validity are essential. Although brief depression inventories exist, they rely heavily on the inclusion of somatic symptoms. This can be problematic in advanced cancer populations; weight loss and sleep disturbance are for the most part ubiquitous in these patients and may not necessarily be indicative of depression.The Brief Edinburgh Depression Scale (BEDS) is a 6-item shortened version of the Edinburgh Depression Scale which has been validated for use in patients with advanced cancer and is used internationally. The BEDS cut off threshold of 6/18 indicates that depression may be present. However, the BEDS currently provides no information regarding severity. The aim of this study is to establish severity thresholds for the BEDS by comparing it to another depression scale: the commonly used, rigorously validated, Patient Health Questionnaire (PHQ-9).Method284 advanced cancer patients attending hospice day services in the North West of England completed both the PHQ-9 and the BEDS. Mean participant age was 66.7 (Standard Deviation = 13.2) and the sample contained both males (n = 102, 36%) and females (n = 182, 64%). BEDS severity thresholds with the highest Sensitivity (Sn) and Specificity (Sp) were selected based on their ability to predict PHQ-9 categories.ResultA BEDS score of 4 to 6 was selected to indicate ‘mild depression’ (Sn = 81.7, Sp = 65); 7 to 8 ‘moderate depression’ (Sn = 74.8, Sp = 78.7); 9 to 11 ‘moderately severe depression’ (Sn = 82, Sp = 82.9) and 12 or more ‘severe depression’ (Sn = 63.2, Sp = 92.8). A linearly weighted kappa (with s weighting) showed a moderate level of agreement (0.47, 95% Confidence Interval: 0.40-0.54).ConclusionThe BEDS is a simple and brief tool used to screen for depression in advanced cancer patients. It is administered throughout the UK and multiple translation studies have enabled its global a (including in resource poor countries). The severity thresholds calculated here are derived from a large sample of patients with advanced cancer attending hospice services and demonstrate acceptable sensitivity and specificity in relation to the PHQ-9, a thoroughly validated reference standard. We conclude that the generated BEDS thresholds support use of the BEDS in determining the presence and severity of depression in advanced cancer populations.

Psychometric properties of the Cumulated Ambulation Score French translation

2020 ◽  
pp. 026921552098432
Author(s):  
Valcarenghi Jérôme ◽  
Chahidi Esfandiar ◽  
Kristensen Morten Tange ◽  
Ledoux Amandine ◽  
Jennart Harold ◽  
...  
Keyword(s):  
Hip Fracture ◽  
Effect Size ◽  
Convergent Validity ◽  
Assessment Tool ◽  
Weighted Kappa ◽  
Measurement Properties ◽  
Strong Positive Correlation ◽  
French Translation ◽  
Rater Reliability ◽  
Motor Test

Objective: To examine measurement properties (inter-rater reliability, agreement, validity, and responsiveness) of the Cumulated Ambulation Score French translation in patients with hip fracture. Design: Methodological study. Setting: A 20-bed orthopedic unit and 20-bed geriatric unit. Subjects: About 140 consecutive patients with a mean (SD) age of 83 (12) years. Interventions: The English version of the Cumulated Ambulation Score used to evaluate basic mobility was translated into French following international guidelines. Two raters independently assessed all patients on postoperative days one, two, three, and 30 after a hip fracture surgery. Relative and absolute inter-rater reliability and responsiveness (effect size for improvement from postoperative day two to 30) were evaluated. Convergent validity was analyzed by Spearman’s correlation coefficient comparing the Cumulated Ambulation Score with two other measures on postoperative day two and 30. Main measure: Cumulated Ambulation Score. Results: The weighted Kappa value ranged from 0.89 to 1.0. The standard error of measurement and the smallest real difference of the Cumulated Ambulation Score ranged, respectively, from 0.12 to 0.23 and from 0.32 to 0.6 points, while the effect size reached 1.03 (95% CI 0.87–1.26). There was a strong positive correlation comparing the Cumulated Ambulation Score with the French Tinetti Assessment Tool ( r ⩾ 0.83) and the French Mini Motor Test ( r ⩾ 0.79). Conclusions: Our findings indicate that the Cumulated Ambulation Score’s French version is a reliable and valid tool to assess patients’ basic mobility with hip fractures.

Inter-rater Reliability of Preceptors on Clinical Pharmacy Competency Evaluation

2020 ◽  
Vol 15 ◽  
Author(s):  
Dixon Thomas ◽  
Sherief Khalifa ◽  
Jayadevan Sreedharan ◽  
Rucha Bond
Keyword(s):  
United Arab Emirates ◽  
Clinical Pharmacy ◽  
Clinical Competence ◽  
Assessment Tool ◽  
Clinical Skills ◽  
Group Discussion ◽  
Pharmacy Students ◽  
Rater Reliability ◽  
Quasi Experimental ◽  
Competency Evaluation

Background:: Clinical competence of pharmacy students is better evaluated at their practice sites. compared to the classroom. A clinical pharmacy competency evaluation rubric like that of the American College of Clinical Pharmacy (ACCP)is an effective assessment tool for clinical skills and can be used to show item reliability. The preceptors should be trained on how to use the rubrics as many inherent factors could influence inter-rater reliability. Objective:: To evaluate inter-rater reliability among preceptors on evaluating clinical competence of pharmacy students, before and after a group discussion intervention. Methods:: In this quasi experimental study in a United Arab Emirates teaching hospital, Seven clinical pharmacy preceptors rated clinical pharmacy competencies of ten recent PharmD graduates referring to their portfolios and preceptorship. Clinical pharmacy competencies were adopted from ACCP and mildly modified to be relevant for the local settings. Results:: Inter-rater reliability (Cronbach's Alpha) among preceptors was reasonable being practitioners at a single site for 2-4 years. At domain level, inter-rater reliability ranged from 0.79 - 0.93 before intervention and 0.94 - 0.99 after intervention. No inter-rater reliability was observed in relation to certain competency elements ranging from 0.31 – 0.61 before intervention, but improved to 0.79 – 0.97 after intervention. Intra-class correlation coefficient improved among all individual preceptors being reliable with each other after group discussion though some had no reliability with each other before group discussion. Conclusion:: Group discussion among preceptors at the training site was found to be effective in improving inter-rater reliability on all elements of the clinical pharmacy competency evaluation. Removing a preceptor from analysis did not affect inter-rater reliability after group discussion.

Finding the path of least resistance: Locally adapting the MITIGATE toolkit in emergency departments and urgent care centers

2021 ◽  
pp. 1-3
Author(s):  
Joanne Huang ◽  
Zahra Kassamali Escobar ◽  
Todd S. Bouchard ◽  
Jose Mari G. Lansang ◽  
Rupali Jain ◽  
...  
Keyword(s):  
Primary Care ◽  
Emergency Department ◽  
Antimicrobial Stewardship ◽  
Emergency Departments ◽  
Urgent Care ◽  
Complex Data ◽  
University Of Washington ◽  
Primary Care Clinics ◽  
Urgent Care Centers ◽  
The University

Abstract The MITIGATE toolkit was developed to assist urgent care and emergency departments in the development of antimicrobial stewardship programs. At the University of Washington, we adopted the MITIGATE toolkit in 10 urgent care centers, 9 primary care clinics, and 1 emergency department. We encountered and overcame challenges: a complex data build, choosing feasible outcomes to measure, issues with accurate coding, and maintaining positive stewardship relationships. Herein, we discuss solutions to challenges we encountered to provide guidance for those considering using this toolkit.

Are the Naranjo Criteria Reliable and Valid for Determination of Adverse Drug Reactions in the Intensive Care Unit?

2005 ◽  
Vol 39 (11) ◽  
pp. 1823-1827 ◽  
Author(s):  
Sandra L Kane-Gill ◽  
Levent Kirisci ◽  
Dev S Pathak
Keyword(s):  
Intensive Care Unit ◽  
Intensive Care ◽  
Adverse Drug Reactions ◽  
Reliability And Validity ◽  
Kappa Statistic ◽  
Weighted Kappa ◽  
Drug Reactions ◽  
Rater Reliability ◽  
Cronbach Alpha

BACKGROUND The Naranjo criteria are frequently used for determination of causality for suspected adverse drug reactions (ADRs); however, the psychometric properties have not been studied in the critically ill. OBJECTIVE To evaluate the reliability and validity of the Naranjo criteria for ADR determination in the intensive care unit (ICU). METHODS All patients admitted to a surgical ICU during a 3-month period were enrolled. Four raters independently reviewed 142 suspected ADRs using the Naranjo criteria (review 1). Raters evaluated the 142 suspected ADRs 3–4 weeks later, again using the Naranjo criteria (review 2). Inter-rater reliability was tested using the kappa statistic. The weighted kappa statistic was calculated between reviews 1 and 2 for the intra-rater reliability of each rater. Cronbach alpha was computed to assess the inter-item consistency correlation. The Naranjo criteria were compared with expert opinion for criterion validity for each rater and reported as a Spearman rank (rs) coefficient. RESULTS The kappa statistic ranged from 0.14 to 0.33, reflecting poor inter-rater agreement. The weighted kappa within raters was 0.5402–0.9371. The Cronbach alpha ranged from 0.443 to 0.660, which is considered moderate to good. The rs coefficient range was 0.385–0.545; all rs coefficients were statistically significant (p < 0.05). CONCLUSIONS Inter-rater reliability is marginal; however, within-rater evaluation appears to be consistent. The inter-item correlation is expected to be higher since all questions pertain to ADRs. Overall, the Naranjo criteria need modification for use in the ICU to improve reliability, validity, and clinical usefulness.

scholarly journals Effect of Head Impact Exposures on Changes in Cognitive Testing

2018 ◽  
Vol 6 (3) ◽  
pp. 232596711876103 ◽  
Author(s):  
Eleni Diakogeorgiou ◽  
Theresa L. Miyashita
Keyword(s):  
Reaction Time ◽  
National Collegiate Athletic Association ◽  
Assessment Tool ◽  
Vital Signs ◽  
Head Impact ◽  
Cognitive Testing ◽  
Repeated Impact ◽  
Level Of Evidence ◽  
Head Impacts ◽  
Significant Difference

Background: Gaining a better understanding of head impact exposures may lead to better comprehension of the possible effects of repeated impact exposures not associated with clinical concussion. Purpose: To assess the correlation between head impacts and any differences associated with cognitive testing measurements pre- and postseason. Study Design: Case-control study; Level of evidence, 3. Methods: A total of 34 National Collegiate Athletic Association Division I men’s lacrosse players wore lacrosse helmets instrumented with an accelerometer during the 2014 competitive season and were tested pre- and postseason with the Sport Concussion Assessment Tool (SCAT 3) and Concussion Vital Signs (CVS) computer-based neurocognitive tests. The number of head impacts >20 g and results from the 2 cognitive tests were analyzed for differences and correlation. Results: There was no significant difference between pre- and postseason SCAT 3 scores, although a significant correlation between pre- and postseason cognitive scores on the SCAT 3 and total number of impacts sustained was noted ( r = –0.362, P = .035). Statistically significant improvements on half of the CVS testing components included visual reaction time ( P = .037, d = 0.37), reaction time ( P = .001, d = 0.65), and simple reaction time ( P = .043, d = 0.37), but no correlation with head impacts was noted. Conclusion: This study did not find declines in SCAT 3 or CVS scores over the course of a season among athletes who sustained multiple head impacts but no clinical concussion. Thus, it could not be determined whether there was no cognitive decline among these athletes or whether there may have been subtle declines that could not be measured by the SCAT 3 or CVS.

Needs and Concerns of Lewy Body Disease Family Caregivers: A Qualitative Study

2021 ◽  
pp. 019394592110509
Author(s):  
Kelly E. Stacy ◽  
Joseph Perazzo ◽  
Rhonna Shatz ◽  
Tamilyn Bakas
Keyword(s):  
Content Analysis ◽  
Qualitative Study ◽  
Family Caregivers ◽  
Lewy Body ◽  
Assessment Tool ◽  
Lewy Body Disease ◽  
Analysis Approach ◽  
Convenience Sample ◽  
Caregiver Interventions ◽  
Disease Family

Lewy body disease (LBD) is a devastating condition with cognitive and physical deficits that pose a challenge to family caregivers. The purpose of this study was to identify the needs and concerns of family caregivers of persons with LBD. A convenience sample of LBD caregivers were interviewed regarding their caregiving needs, concerns, strategies, and advice. A content analysis approach was used to organize data into themes from an existing needs and concerns framework. Findings included the need for more information about the disease, strategies for managing LBD-related emotions and behaviors, support and assistance with physical and instrumental care, and strategies for managing one’s own personal responses to caregiving. Findings highlight the need for a Lewy body specific caregiver assessment tool and future caregiver interventions.

Abstract WP55: Through Thick and Thin: Improved Aspects Grading and Dense Vessel Detection Using Simple Ncct Post-processing

Stroke ◽  
2017 ◽  
Vol 48 (suppl_1) ◽  
Author(s):  
James W Evans ◽  
Sadanand Dey ◽  
Muneer Eesa ◽  
Prasanna Eswaradass ◽  
Ronda Lun ◽  
...  
Keyword(s):  
Maximum Intensity Projection ◽  
Improve Patient Care ◽  
Weighted Kappa ◽  
Slice Thickness ◽  
Post Processing ◽  
Rater Reliability ◽  
Thin Slice ◽  
Vessel Detection ◽  
Contrast Ct ◽  
Processing Techniques

Introduction: Assessing Alberta Stroke Program Early CT Score (ASPECTS) and identifying hyperdense arteries on non-contrast CT (NCCT) are important components of decision-making in acute stroke. Conventional practice uses 5mm averaged slice thickness NCCT for interpretation of these features. We have systematically evaluated several post processing techniques on NCCT to determine if there is improved reliability in identification of ASPECTS and hyperdense artery. Methodology: We assessed four post-processing techniques on NCCT namely (1) 5mm averaged thickness (2) Minimum Intensity Projection (mIP) - 5mm thickness (3) thin slices (0.625mm) and (4) Maximum Intensity Projection (MIP) - 5mm thickness (Figure 1). Three raters (student, fellow and expert) independently assessed 100 NCCT scans from the PRoveIT database. All scans were read at four different times 10-14 days apart. At each time-point the post processing modality was changed and the patient order randomized. Information on side of suspected infarction was provided. Raters were asked to score ASPECTS and identify presence of hyperdense artery at each reading. Inter-rater reliability was assessed using Intra-cluster correlation (ICC) for ASPECTS and weighted kappa (wKap) for hyperdense artery. Results: The highest inter-rater reliability was found with the MIP technique (ICC 0.42; p<0.001), followed by 5 mm average, mIP and thin slice respectively (ICC 0.33, 0.32, 0.20; all p<0.01). Highest agreement for hyperdense vessel detection was noted with thin slice (wKap 0.30; p<0.001) followed by Average, MIPs and mIPs respectively (wKap 0.25, 0.18, 0.13; all p <0.05). Conclusion: The use of MIP images for ASPECTS grading and thin images for hyperdense vessel detection improves reliability on NCCT. These simple processing steps are easily available on any modern scanner and may help improve patient care.

Sign in / Sign up

Export Citation Format

Share Document