Rater Reliability of Assessing Driving Errors Among Returning Combat Veterans

2021 ◽  
Vol 75 (Supplement_2) ◽  
pp. 7512500019p1-7512500019p1
Author(s):  
Mary Jeghers ◽  
James Wersal ◽  
Sandra Winter ◽  
Sherrilene Classen
Keyword(s):  
Interrater Reliability ◽  
Combat Veterans ◽  
High Fidelity ◽  
Training Process ◽  
Rater Reliability ◽  
Primary Author

Abstract Date Presented 04/13/21 As part of a larger study to determine the effectiveness of an OT driving intervention to improve returning combat veterans' driver fitness, researchers measured interrater reliability among three driver rehabilitation specialists. We present the training process, results, and strategy to achieve strong rater reliability for the assessment of driving errors on a DriveSafety 250 high-fidelity simulator. Primary Author and Speaker: Mary Jeghers Contributing Authors: Amber L. Stober, Elisabeth Popoviciu, Gabriella Nieves, and Magaret Norton

The Head Control Scale: Interrater Reliability Among Therapy Students

2021 ◽  
Vol 75 (Supplement_2) ◽  
pp. 7512500012p1-7512500012p1
Author(s):  
Amy Armstrong-Heimsoth ◽  
Rachel Reed ◽  
Samantha Grant ◽  
Jodi Thomas ◽  
Roy St. Laurent
Keyword(s):  
Physical Therapy ◽  
Interrater Reliability ◽  
Kappa Coefficient ◽  
Control Scale ◽  
Head Control ◽  
Primary Author ◽  
Strong Agreement

Abstract Date Presented 04/13/21 This study assesses reliability and accuracy of the Head Control Scale (HCS) when used by inexperienced raters. Physical therapy and OT students used the HCS to rate five videotaped pediatric subjects. The kappa coefficient for interrater reliability among students was "almost perfect" (>.80). In one subscale, when comparing student raters with clinicians, there was strong agreement in grading between each group. The HCS may be consistently used by both new and experienced raters. Primary Author and Speaker: Amy Armstrong-Heimsoth Additional Authors and Speakers: Emily Mei Chun, Elizabeth Diane Hesse, Kelsey E. Ranneklev, and Camila E. Sanchez

Interrater Reliability of a Novel Goniometric Technique to Measure Scapular Protraction and Retraction

2021 ◽  
Vol 75 (Supplement_2) ◽  
pp. 7512500032p1-7512500032p1
Author(s):  
Nathan Short ◽  
Michelle Mays ◽  
Abigail Baist ◽  
Anthony Clifton ◽  
Adam Horty ◽  
...  
Keyword(s):  
Standard Error ◽  
Interrater Reliability ◽  
Reliable Method ◽  
Occupational Performance ◽  
Motor Component ◽  
Healthy Sample ◽  
Primary Author

Abstract Date Presented Accepted for AOTA INSPIRE 2021 but unable to be presented due to online event limitations. This study describes the interrater reliability of a novel goniometric technique to measure scapular protraction and retraction. Measurements obtained by two experienced OTs demonstrated a standard error of measure of less than 4 degrees using this technique on a healthy sample of adults (N = 80). This technique may offer a more reliable method to assess scapular mobility as an essential motor component of occupational performance. Primary Author and Speaker: Nathan Short Additional Authors and Speakers: Michelle Mays, Abigail Baist, Anthony Clifton, Adam Horty, Micaela Kosty, Courtney Olson, and Riddhi Patel Contributing Authors: Thomas Almonreoder

Scoring Second Signed Language Assessment

2021 ◽  
pp. 315-328
Author(s):  
Tobias Haug ◽  
Eveline Boers-Visker ◽  
Wolfgang Mann ◽  
Geoffrey Poor ◽  
Beppie Van den Bogaerde
Keyword(s):  
Language Production ◽  
Interrater Reliability ◽  
Language Assessment ◽  
Rating System ◽  
Assessment Instruments ◽  
Validity And Reliability ◽  
Rater Reliability ◽  
Signed Language ◽  
Assessment Research

There exists a scarcity in signed language assessment research, especially on scoring issues and interrater reliability. This chapter describes two related assessment instruments, the SLPI and the NFA, which offer scoring criteria. Raters are provided with scales for evaluating the different components of the language production of the candidate. Through its use, the rating system has been proved successful; there is, however, hardly any data on interrater reliability. In this chapter, the authors describe reliability issues with attention to raters’ training and score resolution techniques and discuss how to identify and increase rater reliability. The dearth of knowledge on signed language assessment, and in particular its validity and reliability, indicates an urgent need for more research in this area.

scholarly journals Interrater Reliability of the Berg Balance Scale When Used by Clinicians of Various Experience Levels to Assess People With Lower Limb Amputations

2014 ◽  
Vol 94 (3) ◽  
pp. 371-378 ◽  
Author(s):  
Christopher K. Wong
Keyword(s):  
Lower Limb ◽  
Interrater Reliability ◽  
Clinical Training ◽  
Intraclass Correlation ◽  
Berg Balance Scale ◽  
Intrarater Reliability ◽  
Rater Reliability ◽  
Study Objective ◽  
Balance Scale ◽  
Scale Scores

Background People with lower limb amputations frequently have impaired balance ability. The Berg Balance Scale (BBS) has excellent psychometric properties for people with neurologic disorders and elderly people dwelling in the community. A Rasch analysis demonstrated the validity of the BBS for people with lower limb amputations of all ability strata, but rater reliability has not been tested. Objective The study objective was to determine the interrater reliability and intrarater reliability of BBS scores and the differences in scores assigned by testers with various levels of experience when assessing people with lower limb amputations. Design This reliability study of video-recorded single-session BBS assessments had a cross-sectional design. Methods From a larger study of people with lower limb amputations, 5 consecutively recruited participants using prostheses were video recorded during an in-person BBS assessment. Sixteen testers independently rated the video-recorded assessments. Testers were 3 physical therapists, 1 occupational therapist, 3 third-year and 4 second-year doctor of physical therapy (DPT) students, and 5 first-year DPT students without clinical training. Rater reliability was calculated using intraclass correlation coefficients (ICC [2,k]). Differences in scores assigned by testers with various levels of experience were determined by use of an analysis of variance with Tukey post hoc tests. Results The average age of the participants was 53.0 years (SD=15.7). Amputations had occurred at the ankle disarticulation, transtibial, and transfemoral levels because of vascular, trauma, and medical etiologies an average of 8.2 years earlier (SD=7.9). Berg Balance Scale scores spanned all ability strata. Interrater reliability (ICC [2,k]=.99) and intrarater reliability of scores determined in person and through video-recorded assessments by the same testers (ICC [2,k]=.99) were excellent. For participants with the lowest levels of ability, licensed professionals assigned lower scores than did DPT students without clinical training. Limitations Intrarater reliability calculations were based on 2 testers. Conclusions Berg Balance Scale scores assigned to people using prostheses by testers with various levels of clinical experience had excellent interrater reliability and intrarater reliability.

Meta-Analysis of Interrater Reliability of Job Performance Ratings in Validity Studies of Personnel Selection

1996 ◽  
Vol 83 (3_suppl) ◽  
pp. 1195-1201 ◽  
Author(s):  
Jesús F. Salgado ◽  
Silvia Moscoso
Keyword(s):  
Job Performance ◽  
Personnel Selection ◽  
Interrater Reliability ◽  
Rating Scales ◽  
Meta Analysis ◽  
Performance Ratings ◽  
Rater Reliability ◽  
Validity Studies ◽  
Meta Analyses

Rating scales are most frequently used to assess the criterion in studies of validity in personnel selection. However, only a few articles report the interrater reliability for these scales. This paper presents four meta-analyses in which the inter-rater reliability was estimated for Civil Composite Criterion, Military Composite Criterion, Total Composite Criterion, and Single Civil Criterion. Mean reliabilities were .64, .53, .62, and .40, respectively. The implications of these findings for single and meta-analytic studies are discussed.

scholarly journals Competency-based simulation assessment of resuscitation skills in emergency medicine postgraduate trainees – a Canadian multi-centred study

2016 ◽  
Vol 7 (1) ◽  
pp. e57-e67 ◽  
Author(s):  
J. Damon Dagnone ◽  
Andrew K. Hall ◽  
Stefanie Sebok-Syer ◽  
Don Klinger ◽  
Karen Woolfrey ◽  
...  
Keyword(s):  
Emergency Medicine ◽  
Assessment Tool ◽  
Postgraduate Medical Education ◽  
Intraclass Correlation ◽  
Correlation Coefficients ◽  
High Fidelity ◽  
Rater Reliability ◽  
Simulation Based ◽  
Competency Based ◽  
Generalizability Study

Background: The use of high-fidelity simulation is emerging as a desirable method for competency-based assessment in postgraduate medical education. We aimed to demonstrate the feasibility and validity of a multi-centre simulation-based Objective Structured Clinical Examination (OSCE) of resuscitation competence with Canadian Emergency Medicine (EM) trainees.Method: EM postgraduate trainees (n=98) from five Canadian academic centres participated in a high fidelity, 3-station simulation-based OSCE.  Expert panels of three emergency physicians evaluated trainee performances at each centre using the Queen’s Simulation Assessment Tool (QSAT).  Intraclass correlation coefficients were used to measure the inter-rater reliability, and analysis of variance was used to measure the discriminatory validity of each scenario.  A fully crossed generalizability study was also conducted for each examination centre.   Results: Inter-rater reliability in four of the five centres was strong with a median absolute intraclass correlation coefficient (ICC) across centres and scenarios of 0.89 [0.65-0.97]. Discriminatory validity was also strong (p < 0.001 for scenarios 1 and 3; p < 0.05 for scenario 2). Generalizability studies found significant variations at two of the study centres.Conclusions: This study demonstrates the successful pilot administration of a multi-centre, 3-station simulation-based OSCE for the assessment of resuscitation competence in post-graduate Emergency Medicine trainees.

scholarly journals Predicting violence in veterans with posttraumatic stress disorder

2009 ◽  
Vol 66 (1) ◽  
pp. 13-21 ◽  
Author(s):  
Aleksandar Jovanovic ◽  
Dusica Lecic-Tosevski ◽  
Maja Ivkovic ◽  
Aleksandar Damjanovic ◽  
Miroslava Jasovic-Gasic
Keyword(s):  
Posttraumatic Stress Disorder ◽  
Posttraumatic Stress ◽  
Interrater Reliability ◽  
Combat Veterans ◽  
Stress Disorder ◽  
Predictive Accuracy ◽  
Physical Violence ◽  
Psychiatric Patients ◽  
Violent Behavior

Background/Aim. Frequent expression of negative affects, hostility and violent behavior in individuals suffering from posttraumatic stress disorder (PTSD) were recognized long ago, and have been retrospectively well documented in war veterans with PTSD who were shown to have an elevated risk for violent behavior when compared to both veterans without PTSD and other psychiatric patients. The aim of this study was to evaluate the accuracy of clinical prediction of violence in combat veterans suffering from PTSD. Methods. The subjects of this study, 104 male combat veterans with PTSD were assessed with the Historical, Clinical and Risk Management 20 (HCR-20), a 20-item clinicianrated instrument for assessing the risks for violence, and their acts of violence during one-year follow-up period were registered based on bimonthly check-up interviews. Results. Our findings showed that the HCR-20, as an actuarial measure, had good internal consistency reliability (? = 0.82), excellent interrater reliability (Interaclass Correlation ICC = 0.85), as well as excellent predictive validity for acts of any violence, non-physical violence or physical violence in the follow-up period (AUC = 0.82-0.86). The HCR-20 also had good interrater reliability (Cohen's kappa = 0.74), and acceptable predictive accuracy for each outcome criterion (AUC = 0.73-0.79). Conclusion. The results of this research confirm that the HCR-20 may also be applied in prediction of violent behavior in the population of patients suffering from PTSD with reliability and validity comparable with the results of previous studies where this instrument was administered to other populations of psychiatric patients.

Interrater Reliability and Internal Consistency of the Just Write! Functional Handwriting Assessment

2021 ◽  
Vol 75 (Supplement_2) ◽  
pp. 7512500059p1-7512500059p1
Author(s):  
Tiffany Bolton ◽  
Brittney Stevenson ◽  
William Janes
Keyword(s):  
Internal Consistency ◽  
Interrater Reliability ◽  
Quantitative Measure ◽  
Preschool Age ◽  
Age Group ◽  
General Lack ◽  
Primary Author

Abstract Date Presented Accepted for AOTA INSPIRE 2021 but unable to be presented due to online event limitations. The requirements for handwriting performance continue to increase for children entering kindergarten. There is a general lack of consensus on a quantitative measure of handwriting specifically in the preschool age group. The Just Write! is a tool written by the researchers to measure functional handwriting and shows promising results in the initial standardization phase. Primary Author and Speaker: Tiffany Bolton Additional Authors and Speakers: Brittney Stevenson, William Janes

scholarly journals LATERAL TROCHLEAR INCLINATION IN CHILDREN AND ADOLESCENTS: MODIFIED MEASUREMENT TECHNIQUE TO CHARACTERIZE PATELLAR INSTABILITY

2019 ◽  
Vol 7 (3_suppl) ◽  
pp. 2325967119S0014
Author(s):  
Sheeba M. Joseph ◽  
Christopher Cheng ◽  
Matthew J. Solomito ◽  
J. Lee Pace
Keyword(s):  
Regression Analysis ◽  
Children And Adolescents ◽  
Reliability Analysis ◽  
Interrater Reliability ◽  
Patellar Instability ◽  
Trochlear Dysplasia ◽  
Perfect Agreement ◽  
Rater Reliability ◽  
Entire Cohort ◽  
Dejour Classification

Background: Patellar instability (PI) is relatively rare but occurs most often in younger patients with underlying pathoanatomy. Trochlear dysplasia (TD) is one of many identified PI risk factors, but consensus is lacking on ideal radiographic measurements. The Dejour classification of TD on lateral radiographs is widely accepted but has suboptimal intra and interrater reliability and does not allow quantification of TD. Lateral trochlear inclination (LTI) measured on the most proximal axial magnetic resonance image (MRI) of the trochlear chondral surface is another described measurement of TD. LTI has historically been described with reference to the posterior aspect of the femur at the same axial level at which the proximal trochlea is measured. However, given the transitional anatomy of the distal femur, the LTI may be better represented by referencing the axis of the fully formed posterior femoral condyles. The posterior condyles represent a true axis of rotation that serves as an internal reference for knee motion and is clearly visible on MRI. We hypothesized that modified LTI measurements (LTI) referencing the posterior condylar axis would differ from the apparent LTI (ALTI) in a pediatric and adolescent population. We also hypothesized that the LTI would have stronger intra and inter reliability than the ALTI measurement and Dejour classification. Lastly, we hypothesized that the most proximal level of the trochlea would adequately represent overall proximal trochlear morphology. This is clinically relevant because dysplasia is most severe on the proximal trochlea and normalizes distally towards the intercondylar notch. Methods: Patients aged 9 to 18 years treated for PI at our tertiary referral center between January 2014 and August 2017 were identified. The Dejour classification was determined on lateral knee radiographs. The ALTI was measured as previously described on axial MRI images (Figure 1A). The LTI (also referred to as LTI #1) was measured on the same MRI image with respect to the angle of the posterior condyles (Figure 1B-C). The LTI was measured again in this fashion at the three subsequent, consecutive axial levels (LTI#2, LTI#3, LTI#4) to capture the first 12 mm of the proximal trochlea. The average of these measurements (LTI-avg) was calculated for each patient. All measurements were performed by two independent observers. A cohort of 30 patients were randomly selected for reliability analysis which was performed twice by three independent observers at least two weeks apart. Inter- and intra-rater correlation coefficients were calculated on this subgroup. Regression analysis was performed on the entire cohort. Results: Sixty-five patients met inclusion criteria for this study, and thirty patients were randomly selected for reliability analysis. Inter- and intra-rater reliability for ALTI showed good agreement but trended towards more variability than the inter- and intra-rater reliability for LTI#1 which had near perfect agreement (Table 1). Inter- and intra-rater reliability for all subsequent LTI measurements and LTI-avg had high or near perfect agreement (Table 2). The Dejour classification had poor to moderate inter-rater and good to near perfect intra-rater reliability. The crossing sign was the most reliable radiographic feature (Table 3). In the entire cohort of 65 patients, the average ALTI (9.2+/-12.6 degrees) was 7.0+/-3.4 degrees greater (less dysplastic) than the average LTI #1 (4.2+/-11.9 degrees) (p = 0.013). Referencing the 11 degrees LTI threshold value for trochlear dysplasia reported in the literature, the ALTI was below 11 degrees in 60% of our PI patients indicating dysplasia, while the LTI was less than 11 degrees in 71% of our PI patients. Regression analysis demonstrated statistically significant positive correlation between LTI#1 and LTI#2 (r=0.88, beta=0.81, p<0.0001), LTI#1 and LTI#3 (r=0.67, beta=0.54, p<0.0001), LTI#1 and LTI#4 (r=0.65, beta=0.43, p<0.001), and LTI#1 and LTI-avg (r=0.91, beta=0.70, p<0.0001). Conclusion: LTI has higher intra and interrater reliability when performed with reference to the posterior condyles compared to the historical measurement (ALTI) and the Dejour classification. The significant and strong correlation between LTI#1 and subsequent LTI measures as well as LTI-avg shows that 90% of TD is represented on the first, most proximal axial image and thus provides an appropriate, reliable and quantifiable measurement of TD in children and adolescents with PI. The significant difference found between LTI and ALTI shows that the historical measurement appears to underestimate dysplasia. Thus, previously described threshold values should be re-examined using this new technique to appropriately characterize trochlear dysplasia in patients with patellar instability as this can have implications for treatment algorithms for these patients. [Table: see text][Table: see text][Table: see text][Figure: see text]

Sign in / Sign up

Export Citation Format

Share Document