scholarly journals Development of a Musculoskeletal Ultrasound Protocol to Examine Upper Extremity Rehabilitation Outcomes in Systemic Sclerosis

2020 ◽  
Vol 37 (1) ◽  
pp. 13-23
Author(s):  
Susan L. Murphy ◽  
Donnamarie Krause ◽  
Shawn C. Roll ◽  
Girish Gandikota ◽  
Mary Barber ◽  
...  
Keyword(s):  
Systemic Sclerosis ◽  
Interrater Reliability ◽  
Outcome Measurement ◽  
Intraclass Correlation ◽  
Correlation Coefficients ◽  
Thickness Measurement ◽  
Musculoskeletal Ultrasound ◽  
Gray Scale ◽  
Strain Elastography ◽  
Rehabilitation Outcomes

Objectives: This study developed a musculoskeletal ultrasound (MSUS) protocol to evaluate rehabilitation outcomes in systemic sclerosis. Materials and Methods: Three MSUS methods (gray-scale, Doppler, strain elastography) and two acquisition techniques (long- vs short-axis; transducer on skin vs floating on gel) were examined in the forearm before and after rehabilitation treatment. For gray-scale, tissue thickness measures and intrarater and interrater reliability were calculated (intraclass correlation coefficients [ICCs]), and paired t tests examined differences among techniques. Results: Five people with diffuse cutaneous systemic sclerosis participated. The most valid and reliable gray-scale technique was with the transducer in long-axis, floating on gel. Doppler and strain elastography did not detect changes. Both dermal and subcutaneous thickness measurement error was small; intrarater and interrater reliability was good to excellent. Preliminary data indicate that treatment may lead to dermal thinning. Conclusion: A replicable protocol was established and may be an adjunct to rehabilitation outcome measurement in systemic sclerosis.

Interobserver Reliability Using the Phonetic Level Evaluation With Severely and Profoundly Hearing-Impaired Children

1991 ◽  
Vol 34 (5) ◽  
pp. 989-999 ◽  
Author(s):  
Stephanie Shaw ◽  
Truman E. Coggins
Keyword(s):  
Interrater Reliability ◽  
Interobserver Reliability ◽  
Intraclass Correlation ◽  
Correlation Coefficients ◽  
Hearing Impaired ◽  
Intraclass Correlation Coefficients ◽  
Assessment Measure ◽  
Impaired Children ◽  
Speech Assessment ◽  
Hearing Impaired Children

This study examines whether observers reliably categorize selected speech production behaviors in hearing-impaired children. A group of experienced speech-language pathologists was trained to score the elicited imitations of 5 profoundly and 5 severely hearing-impaired subjects using the Phonetic Level Evaluation (Ling, 1976). Interrater reliability was calculated using intraclass correlation coefficients. Overall, the magnitude of the coefficients was found to be considerably below what would be accepted in published behavioral research. Failure to obtain acceptably high levels of reliability suggests that the Phonetic Level Evaluation may not yet be an accurate and objective speech assessment measure for hearing-impaired children.

Development and Initial Validation of a Project-Based Rubric to Assess the Systems-Based Practice Competency of Residents in the Clinical Chemistry Rotation of a Pathology Residency

2014 ◽  
Vol 138 (6) ◽  
pp. 809-813
Author(s):  
Carolyn R. Vitek ◽  
Jane C. Dale ◽  
Henry A. Homburger ◽  
Sandra C. Bryant ◽  
Amy K. Saenger ◽  
...  
Keyword(s):  
Critical Thinking ◽  
Interrater Reliability ◽  
Clinical Chemistry ◽  
Core Competencies ◽  
Intraclass Correlation ◽  
Reliability And Validity ◽  
Correlation Coefficients ◽  
Thinking Skills ◽  
Project Evaluation ◽  
Critical Thinking Skills

Context.— Systems-based practice (SBP) is 1 of 6 core competencies required in all resident training programs accredited by the Accreditation Council for Graduate Medical Education. Reliable methods of assessing resident competency in SBP have not been described in the medical literature. Objective.— To develop and validate an analytic grading rubric to assess pathology residents' analyses of SBP problems in clinical chemistry. Design.— Residents were assigned an SBP project based upon unmet clinical needs in the clinical chemistry laboratories. Using an iterative method, we created an analytic grading rubric based on critical thinking principles. Four faculty raters used the SBP project evaluation rubric to independently grade 11 residents' projects during their clinical chemistry rotations. Interrater reliability and Cronbach α were calculated to determine the reliability and validity of the rubric. Project mean scores and range were also assessed to determine whether the rubric differentiated resident critical thinking skills related to the SBP projects. Results.— Overall project scores ranged from 6.56 to 16.50 out of a possible 20 points. Cronbach α ranged from 0.91 to 0.96, indicating that the 4 rubric categories were internally consistent without significant overlap. Intraclass correlation coefficients ranged from 0.63 to 0.81, indicating moderate to strong interrater reliability. Conclusions.— We report development and statistical analysis of a novel SBP project evaluation rubric. The results indicate the rubric can be used to reliably assess pathology residents' critical thinking skills in SBP.

Development of a Model for the Acquisition and Assessment of Advanced Laparoscopic Suturing Skills Using an Automated Device

2018 ◽  
Vol 25 (3) ◽  
pp. 286-290 ◽  
Author(s):  
Elif Bilgic ◽  
Madoka Takao ◽  
Pepa Kaneva ◽  
Satoshi Endo ◽  
Toshitatsu Takao ◽  
...  
Keyword(s):  
Laparoscopic Surgery ◽  
Interrater Reliability ◽  
Intraclass Correlation ◽  
Correlation Coefficients ◽  
Instructional Video ◽  
Validity Evidence ◽  
Laparoscopic Suturing ◽  
Intraclass Correlation Coefficients ◽  
Operative Assessment ◽  
Suturing Skills

Background. Needs assessment identified a gap regarding laparoscopic suturing skills targeted in simulation. This study collected validity evidence for an advanced laparoscopic suturing task using an Endo StitchTM device. Methods. Experienced (ES) and novice surgeons (NS) performed continuous suturing after watching an instructional video. Scores were based on time and accuracy, and Global Operative Assessment of Laparoscopic Surgery. Data are shown as medians [25th-75th percentiles] (ES vs NS). Interrater reliability was calculated using intraclass correlation coefficients (confidence interval). Results. Seventeen participants were enrolled. Experienced surgeons had significantly greater task (980 [964-999] vs 666 [391-711], P = .0035) and Global Operative Assessment of Laparoscopic Surgery scores (25 [24-25] vs 14 [12-17], P = .0029). Interrater reliability for time and accuracy were 1.0 and 0.9 (0.74-0.96), respectively. All experienced surgeons agreed that the task was relevant to practice. Conclusion. This study provides validity evidence for the task as a measure of laparoscopic suturing skill using an automated suturing device. It could help trainees acquire the skills they need to better prepare for clinical learning.

scholarly journals AB0798 IS THERE A REPRODUCIBILITY OF THE HISTOGRAM AND GRAY SCALE IN ECOEXTRUCTURAL DAMAGES IN RHEUMATOTOLOGICAL DISEASES?

2021 ◽  
Vol 80 (Suppl 1) ◽  
pp. 1423.2-1424
Author(s):  
J. A. Mendonça ◽  
I. Siste de Almeida Aoki ◽  
C. C. Cavuto ◽  
V. A. Leandro-Merhi ◽  
J. L. B. D. Aquino
Keyword(s):  
Correlation Coefficient ◽  
Intraclass Correlation Coefficient ◽  
Rheumatic Diseases ◽  
Femoral Condyle ◽  
Intraclass Correlation ◽  
Correlation Coefficients ◽  
Concordance Correlation Coefficient ◽  
Image Method ◽  
Concordance Correlation ◽  
Gray Scale

Background:The gray scale (GS) in high resolution ultrasound is already well validated for use in rheumatological diseases, but the color map or the histogram, can be considered a new proposal, to better define and complement the echotextural damages detection1.Objectives:To calculate the lesions area measures reproducibility index in arthropathies, between 3 blind evaluators and correlate these measures using the GS and the histogram.Methods:Observational and retrospective study approved by the ethics committee of the Pontifical Catholic University of Campinas, with the opinion number: 1.526.307. A total of 29 patients have been assessed (31% males and 69% females) on period 2014 to 2019 in Rheumatology service. A MyLab 50 -Esaote equipment was used with frequency transducer that ranged between 6.0 and 18.0 MHz, 10 different area measures were performed from each recorded images previously, by the GS and the histogram. Statistical analysis: Spearman’s correlation coefficients, Lin’s concordance coefficient (CCC) and the intraclass correlation coefficient (ICC) and their respective 95% confidence intervals, with the SPSS software package for Windows v. 17.0 (SPSS Inc., Chicago, IL, USA).Results:Average age 43.5 ± 21.5 years of age; with disease duration that varied between ≤ 1 month (48.3%) and ≥36 months (24.1%); with the following diseases: juvenile idiopathic arthritis (17.24%); osteoarthritis (13.79%); psoriatic arthritis (13.79%); undifferentiated spondyloarthritis (3.44%); gout (20.68%); rheumatoid arthritis (27.58%) and reactive arthritis (3.44%). A total of 840 measures of exudative (27.58%), proliferative (27.58%) and snowstorm appearance (6.89%) synovitis were performed; femoral-condyle cartilage (3.44%); synovial cyst (3.44%); paratendinitis (6.89%); calcification (3.44%); nail enthesitis (3.44%); tenosynovitis (6.89%) and tophi (10.34%) (Figure 1). The concordance correlation coefficient showed values closer to 1; p <0.001, the intraclass correlation coefficients with excellent reproducibility (ICC ≥ 0.75); p <0.001, always in relation to the three evaluators (Table 1) and the Spearman correlation between the GS and the histogram ranged from rs = 0.665 to rs = 1,000; p <0.001.Conclusion:The histogram can be considered an image method to better identify echotextural damages.References:[1]Mendonça J, Provenza J, Guissa V, et al AB1059 2D Histogram Ultrasound and 3D Ultrasound Correlation in Rheumatic Diseases Annals of the Rheumatic Diseases 2015; 74:1253-1254.Table 1.Concordance Correlation Coefficient (CCC) and Intraclass Correlation Coefficient (ICC):EvaluatorsGS - CCC(IC 95%)p-valorHistogram (IC 95%)p-valor1 e 20,998(0,994-0,999)<0,0010,999(0,995-1,000)<0,0011 e 30,998(0,995-0,999)<0,0010,999(0,995-1,000)<0,0012 e 30,992(0,980-0,997)<0,010,996(0,979-0,999)<0,01Standard by US 2DICCp-valorGS0,997(0,992-0,999)<0,001Histogram0,998(0,992-0,999)<0,001Legends: Gray Scale (GS).Figure 1.Patient with gout: A and B: Tophi area measures (star) in right metatarsos and efusion (arrow) by GS (45 mm2) and histogram (39 mm2), respectively.Disclosure of Interests:José Alexandre Mendonça Speakers bureau: Novartis, Janssen, Bristol, UCB, Isabella Siste de Almeida Aoki: None declared, Caique Chagas Cavuto: None declared, Vânia Aparecida Leandro-Merhi: None declared, José Luis Braga de Aquino: None declared

Interrater Reliability of the Handwriting Speed Test

1997 ◽  
Vol 17 (4) ◽  
pp. 280-287 ◽  
Author(s):  
Margaret Wallen ◽  
Mary-Ann Bonney ◽  
Lyn Lennox
Keyword(s):  
Interrater Reliability ◽  
Test Development ◽  
Intraclass Correlation ◽  
Correlation Coefficients ◽  
Objective Evaluation ◽  
Research Tool ◽  
School Students ◽  
Speed Test ◽  
Intraclass Correlation Coefficients ◽  
Handwriting Speed

The Handwriting Speed Test (HST), a standardized, norm-referenced test, was developed to provide an objective evaluation of the handwriting speed of school students from approximately 8 to 18 years of age. Part of the test development involved an examination of interrater reliability. Two raters scored 165 (13%) of the total 1292 handwriting samples. Using intraclass correlation coefficients, the interrater reliability was found to be excellent (ICC=1.00, P<0.0001). The process of examining interrater reliability resulted in modification to the scoring criteria of the test. Excellent interrater reliability provides support for the HST as a valuable clinical and research tool.

scholarly journals Validation of an Evidence-Based Medicine Critically Appraised Topic Presentation Evaluation Tool (EBM C-PET)

2013 ◽  
Vol 5 (2) ◽  
pp. 252-256 ◽  
Author(s):  
Hans B. Kersten ◽  
John G. Frohna ◽  
Erin L. Giudice
Keyword(s):  
Internal Consistency ◽  
Evidence Based Medicine ◽  
Interrater Reliability ◽  
Intraclass Correlation ◽  
Correlation Coefficients ◽  
Evidence Based ◽  
Evaluation Tool ◽  
Intraclass Correlation Coefficients ◽  
Excellent Internal Consistency ◽  
Based Medicine

Abstract Background Competence in evidence-based medicine (EBM) is an important clinical skill. Pediatrics residents are expected to acquire competence in EBM during their education, yet few validated tools exist to assess residents' EBM skills. Objective We sought to develop a reliable tool to evaluate residents' EBM skills in the critical appraisal of a research article, the development of a written EBM critically appraised topic (CAT) synopsis, and a presentation of the findings to colleagues. Methods Instrument development used a modified Delphi technique. We defined the skills to be assessed while reviewing (1) a written CAT synopsis and (2) a resident's EBM presentation. We defined skill levels for each item using the Dreyfus and Dreyfus model of skill development and created behavioral anchors using a frame-of-reference training technique to describe performance for each skill level. We evaluated the assessment instrument's psychometric properties, including internal consistency and interrater reliability. Results The EBM Critically Appraised Topic Presentation Evaluation Tool (EBM C-PET) is composed of 14 items that assess residents' EBM and global presentation skills. Resident presentations (N  =  27) and the corresponding written CAT synopses were evaluated using the EBM C-PET. The EBM C-PET had excellent internal consistency (Cronbach α  =  0.94). Intraclass correlation coefficients were used to assess interrater reliability. Intraclass correlation coefficients for individual items ranged from 0.31 to 0.74; the average intraclass correlation coefficients for the 14 items was 0.67. Conclusions We identified essential components of an assessment tool for an EBM CAT synopsis and presentation with excellent internal consistency and a good level of interrater reliability across 3 different institutions. The EBM C-PET is a reliable tool to document resident competence in higher-level EBM skills.

scholarly journals Reliability of Safe Maximum Lifting Determinations of a Functional Capacity Evaluation

2002 ◽  
Vol 82 (4) ◽  
pp. 364-371 ◽  
Author(s):  
Douglas P Gross ◽  
Michele C Battié
Keyword(s):  
Functional Capacity ◽  
Repeated Measures ◽  
Interrater Reliability ◽  
Intraclass Correlation ◽  
Correlation Coefficients ◽  
Functional Capacity Evaluation ◽  
Measurement Variability ◽  
Retest Reliability ◽  
Repeated Measures Design ◽  
Test Retest Reliability

Abstract Background and Purpose. Functional capacity evaluations (FCEs) are measurement tools used in predicting readiness to return to work following injury. The interrater and test-retest reliability of determinations of maximal safe lifting during kinesiophysical FCEs were examined in a sample of people who were off work and receiving workers' compensation. Subjects. Twenty-eight subjects with low back pain who had plateaued with treatment were enrolled. Five occupational therapists, trained and experienced in kinesiophysical methods, conducted testing. Methods. A repeated-measures design was used, with raters testing subjects simultaneously, yet independently. Subjects were rated on 2 occasions, separated by 2 to 4 days. Analyses included intraclass correlation coefficients (ICCs) and 95% confidence intervals. Results. The ICC values for interrater reliability ranged from .95 to .98. Test-retest values ranged from .78 to .94. Discussion and Conclusion. Inconsistencies in subjects' performance across sessions were the greatest source of FCE measurement variability. Overall, however, test-retest reliability was good and interrater reliability was excellent.

Developing and validating a methodology for crowdsourcing L2 speech ratings in Amazon Mechanical Turk

2019 ◽  
Vol 5 (2) ◽  
pp. 294-323 ◽  
Author(s):  
Charles Nagle
Keyword(s):  
Interrater Reliability ◽  
Intraclass Correlation ◽  
Correlation Coefficients ◽  
Spanish Speakers ◽  
Mechanical Turk ◽  
Amazon Mechanical Turk ◽  
Native Spanish Speakers ◽  
Intraclass Correlation Coefficients ◽  
Future Data ◽  
Rater Severity

Abstract Researchers have increasingly turned to Amazon Mechanical Turk (AMT) to crowdsource speech data, predominantly in English. Although AMT and similar platforms are well positioned to enhance the state of the art in L2 research, it is unclear if crowdsourced L2 speech ratings are reliable, particularly in languages other than English. The present study describes the development and deployment of an AMT task to crowdsource comprehensibility, fluency, and accentedness ratings for L2 Spanish speech samples. Fifty-four AMT workers who were native Spanish speakers from 11 countries participated in the ratings. Intraclass correlation coefficients were used to estimate group-level interrater reliability, and Rasch analyses were undertaken to examine individual differences in rater severity and fit. Excellent reliability was observed for the comprehensibility and fluency ratings, but indices were slightly lower for accentedness, leading to recommendations to improve the task for future data collection.

Assessment of the Intrarater and Interrater Reliability of an Established Clinical Task Analysis Methodology

2002 ◽  
Vol 96 (5) ◽  
pp. 1129-1139 ◽  
Author(s):  
Jason Slagle ◽  
Matthew B. Weinger ◽  
My-Than T. Dinh ◽  
Vanessa V. Brumer ◽  
Kevin Williams
Keyword(s):  
Real Time ◽  
Task Analysis ◽  
Interrater Reliability ◽  
Intraclass Correlation ◽  
Correlation Coefficients ◽  
Intrarater Reliability ◽  
Intraclass Correlation Coefficients ◽  
Percent Time ◽  
Analysis Methodology ◽  
And Task

Background Task analysis may be useful for assessing how anesthesiologists alter their behavior in response to different clinical situations. In this study, the authors examined the intraobserver and interobserver reliability of an established task analysis methodology. Methods During 20 routine anesthetic procedures, a trained observer sat in the operating room and categorized in real-time the anesthetist's activities into 38 task categories. Two weeks later, the same observer performed task analysis from videotapes obtained intraoperatively. A different observer performed task analysis from the videotapes on two separate occasions. Data were analyzed for percent of time spent on each task category, average task duration, and number of task occurrences. Rater reliability and agreement were assessed using intraclass correlation coefficients. Results Intrarater reliability was generally good for categorization of percent time on task and task occurrence (mean intraclass correlation coefficients of 0.84-0.97). There was a comparably high concordance between real-time and video analyses. Interrater reliability was generally good for percent time and task occurrence measurements. However, the interrater reliability of the task duration metric was unsatisfactory, primarily because of the technique used to capture multitasking. Conclusions A task analysis technique used in anesthesia research for several decades showed good intrarater reliability. Off-line analysis of videotapes is a viable alternative to real-time data collection. Acceptable interrater reliability requires the use of strict task definitions, sophisticated software, and rigorous observer training. New techniques must be developed to more accurately capture multitasking. Substantial effort is required to conduct task analyses that will have sufficient reliability for purposes of research or clinical evaluation.

scholarly journals Alberta Infant Motor Scale: Reliability and Validity When Used on Preterm Infants in Taiwan

2000 ◽  
Vol 80 (2) ◽  
pp. 168-178 ◽  
Author(s):  
Suh-Fang Jeng ◽  
Kuo-Inn Tsou Yau ◽  
Li-Chiou Chen ◽  
Shu-Fang Hsiao
Keyword(s):  
Preterm Infants ◽  
Interrater Reliability ◽  
Physical Therapist ◽  
Intraclass Correlation ◽  
Reliability And Validity ◽  
Correlation Coefficients ◽  
Intraclass Correlation Coefficients ◽  
Scale Reliability ◽  
Scale Scores ◽  
Acceptable Reliability

Abstract Background and Purpose. The goal of this study was to examine the reliability and validity of measurements obtained with the Alberta Infant Motor Scale (AIMS) for evaluation of preterm infants in Taiwan. Subjects. Two independent groups of preterm infants were used to investigate the reliability (n=45) and validity (n=41) for the AIMS. Methods. In the reliability study, the AIMS was administered to the infants by a physical therapist, and infant performance was videotaped. The performance was then rescored by the same therapist and by 2 other therapists to examine the intrarater and interrater reliability. In the validity study, the AIMS and the Bayley Motor Scale were administered to the infants at 6 and 12 months of age to examine criterion-related validity. Results. Intraclass correlation coefficients (ICCs) for intrarater and interrater reliability of measurements obtained with the AIMS were high (ICC=.97–.99). The AIMS scores correlated with the Bayley Motor Scale scores at 6 and 12 months (r=.78 and .90), although the AIMS scores at 6 months were only moderately predictive of the motor function at 12 months (r=.56). Conclusion and Discussion. The results suggest that measurements obtained with the AIMS have acceptable reliability and concurrent validity but limited predictive value for evaluating preterm Taiwanese infants.

Sign in / Sign up

Export Citation Format

Share Document