Assessing the Reliability and Validity of Technical Skill Assessment Tools and the Impact of Rater Training

2017 ◽  
Vol 225 (4) ◽  
pp. S172
Author(s):  
Reagan Maniar ◽  
Ashley Vergis ◽  
Lawrence Gillman ◽  
Jason Park
Keyword(s):  
Reliability And Validity ◽  
Technical Skill ◽  
Assessment Tools ◽  
Skill Assessment ◽  
Rater Training ◽  
The Impact

scholarly journals Patient Simulation: A Literary Synthesis of Assessment Tools in Anesthesiology

2009 ◽  
Vol 6 ◽  
pp. 3 ◽  
Author(s):  
Alice A. Edler ◽  
Ruth G. Fanning ◽  
Michael. I. Chen ◽  
Rebecca Claure ◽  
Dondee Almazan ◽  
...  
Keyword(s):  
Assessment Tool ◽  
Generalizability Theory ◽  
Patient Simulation ◽  
Reliability And Validity ◽  
Test Construction ◽  
Test Theory ◽  
Assessment Tools ◽  
Rater Training ◽  
Test Items ◽  
Construction Methods

High-fidelity patient simulation (HFPS) has been hypothesized as a modality for assessing competency of knowledge and skill in patient simulation, but uniform methods for HFPS performance assessment (PA) have not yet been completely achieved. Anesthesiology as a field founded the HFPS discipline and also leads in its PA. This project reviews the types, quality, and designated purpose of HFPS PA tools in anesthesiology. We used the systematic review method and systematically reviewed anesthesiology literature referenced in PubMed to assess the quality and reliability of available PA tools in HFPS. Of 412 articles identified, 50 met our inclusion criteria. Seventy seven percent of studies have been published since 2000; more recent studies demonstrated higher quality. Investigators reported a variety of test construction and validation methods. The most commonly reported test construction methods included ?占퐉odified Delphi Techniques??for item selection, reliability measurement using inter-rater agreement, and intra-class correlations between test items or subtests. Modern test theory, in particular generalizability theory, was used in nine (18%) of studies. Test score validity has been addressed in multiple investigations and shown a significant improvement in reporting accuracy. However the assessment of predicative has been low across the majority of studies. Usability and practicality of testing occasions and tools was only anecdotally reported. To more completely comply with the gold standards for PA design, both shared experience of experts and recognition of test construction standards, including reliability and validity measurements, instrument piloting, rater training, and explicit identification of the purpose and proposed use of the assessment tool, are required.

scholarly journals Objective Assessment of Microsurgery Competency—In Search of a Validated Tool

2019 ◽  
Vol 52 (02) ◽  
pp. 216-221
Author(s):  
Sheeja Rajan ◽  
Ranjith Sathyan ◽  
L. S. Sreelesh ◽  
Anu Anto Kallerey ◽  
Aarathy Antharjanam ◽  
...  
Keyword(s):  
Skill Acquisition ◽  
Convergent Validity ◽  
Assessment Tool ◽  
Objective Assessment ◽  
Reliability And Validity ◽  
Assessment Tools ◽  
Skill Assessment ◽  
P Value ◽  
Global Rating ◽  
Skills Acquisition

AbstractMicrosurgical skill acquisition is an integral component of training in plastic surgery. Current microsurgical training is based on the subjective Halstedian model. An ideal microsurgery assessment tool should be able to deconstruct all the subskills of microsurgery and assess them objectively and reliably. For our study, to analyze the feasibility, reliability, and validity of microsurgery skill assessment, a video-based objective structured assessment of technical skill tool was chosen. Two blinded experts evaluated 40 videos of six residents performing microsurgical anastomosis for arteriovenous fistula surgery. The generic Reznick's global rating score (GRS) and University of Western Ontario microsurgical skills acquisition/assessment (UWOMSA) instrument were used as checklists. Correlation coefficients of 0.75 to 0.80 (UWOMSA) and 0.71 to 0.77 (GRS) for interrater and intrarater reliability showed that the assessment tools were reliable. Convergent validity of the UWOMSA tool with the prevalidated GRS tool showed good agreement. The mean improvement of scores with years of residency was measured with analysis of variance. Both UWOMSA (p-value: 0.034) and GRS (p-value: 0.037) demonstrated significant improvement in scores from postgraduate year 1 (PGY1) to PGY2 and a less marked improvement from PGY2 to PGY3. We conclude that objective assessment of microsurgical skills in an actual clinical setting is feasible. Tools like UWOMSA are valid and reliable for microsurgery assessment and provide feedback to chart progression of learning. Acceptance and validation of such objective assessments will help to improve training and bring uniformity to microsurgery education.

scholarly journals Non-technical Skill Assessment Tools for Surgical Team Members in the Operating Room: A Critical Review

2020 ◽  
Vol In Press (In Press) ◽  
Author(s):  
Peigham Heidarpoor ◽  
Shahram Yazdani ◽  
Leila Sadati
Keyword(s):  
Operating Room ◽  
Critical Review ◽  
Technical Skills ◽  
Technical Skill ◽  
Assessment Tools ◽  
Skill Assessment ◽  
Surgical Team ◽  
Team Members ◽  
Technical Performance ◽  
Surgical Teams

Context: The operating room is a unique environment in which learning happens through interactions with others and facing complex situations. The results of studies show that non-technical skills play a key role in successful technical performance in critical situations. Therefore, the identification and assessment of them are very valuable. This review study was performed to identify and critique non-technical skill assessment tools for surgical team members. Evidence Acquisition: The Carnwell and Daly critical review method was used for the study design. Articles and documents were searched in databases of PubMed, Google Scholar, Scopus, and ScienceDirect from 1999 to 2019. Results: This literature review yielded 13 assessment tools of non-technical skills in the operating room, including ANTS, AS-NTS, N-ANTS, ANTS-AP for the anesthesia team, OTAS, Oxford NOTEHS, Oxford NOTECHS II, and Revised NOTECHS for the surgical team, SPINTS for scrub nurses, and OSANTS, NOTSS, SDM-RS, and SLI for surgeons. Conclusions: Since the development of the tools depends entirely on the context and task analysis of any member of the surgical team, all countries must customize the available tools and develop similar tools for other members of surgical teams.

scholarly journals How can surgical skills in laparoscopic colon surgery be objectively assessed?—a scoping review

2021 ◽  
Author(s):  
Tora Rydtun Haug ◽  
Mai-Britt Worm Ørntoft ◽  
Danilo Miskovic ◽  
Lene Hjerrild Iversen ◽  
Søren Paaske Johnsen ◽  
...  
Keyword(s):  
Technical Skills ◽  
Colon Surgery ◽  
Assessment Tools ◽  
Skill Assessment ◽  
Validity Evidence ◽  
Rater Training ◽  
Surgical Skill Assessment ◽  
Laparoscopic Colon Surgery ◽  
Technical Skills Assessment ◽  
Laparoscopic Colon

Abstract Background In laparoscopic colorectal surgery, higher technical skills have been associated with improved patient outcome. With the growing interest in laparoscopic techniques, pressure on surgeons and certifying bodies is mounting to ensure that operative procedures are performed safely and efficiently. The aim of the present review was to comprehensively identify tools for skill assessment in laparoscopic colon surgery and to assess their validity as reported in the literature. Methods A systematic search was conducted in EMBASE and PubMed/MEDLINE in May 2021 to identify studies examining technical skills assessment tools in laparoscopic colon surgery. Available information on validity evidence (content, response process, internal structure, relation to other variables, and consequences) was evaluated for all included tools. Results Fourteen assessment tools were identified, of which most were procedure-specific and video-based. Most tools reported moderate validity evidence. Commonly not reported were rater training, assessment correlation with variables other than training level, and validity reproducibility and reliability in external educational settings. Conclusion The results of this review show that several tools are available for evaluation of laparoscopic colon cancer surgery, but few authors present substantial validity for tool development and use. As we move towards the implementation of new techniques in laparoscopic colon surgery, it is imperative to establish validity before surgical skill assessment tools can be applied to new procedures and settings. Therefore, future studies ought to examine different aspects of tool validity, especially correlation with other variables, such as patient morbidity and pathological reports, which impact patient survival.

Towards AI-powered Language Assessment Tools

2021 ◽  
Author(s):  
Mah Parsa ◽  
Muhammad Raisul Alam ◽  
Alex Mihailidis
Keyword(s):  
Language Impairment ◽  
Assessment Tool ◽  
Reliability And Validity ◽  
High Sensitivity ◽  
Language Assessment ◽  
Assessment Tools ◽  
Acoustic Features ◽  
Linguistic Markers ◽  
Binary Classifiers ◽  
The Impact

Abstract Objectives: The main objective of this paper is to propose an approach for developing an Artificial Intelligence (AI)-powered Language Assessment (LA) tool. Such tools can be used to assess language impairments associated with dementia in older adults. The Machine Learning (ML) classifiers are the main parts of our proposed approach, therefore to develop an accurate tool with high sensitivity and specificity, we consider different binary classifiers and evaluate their performances. We also assess the reliability and validity of our approach by comparing the impact of different types of language tasks, features, and recording media on the performance of ML classifiers. Approach: Our approach includes the following steps: 1) Collecting language datasets or getting access to available language datasets; 2) Extracting linguistic and acoustic features from subjects' speeches which have been collected from subjects with dementia (N=9) and subjects without dementia (N=13); 3) Selecting most informative features and using them to train ML classifiers; and 4) Evaluating the performance of classifiers on distinguishing subjects with dementia from subjects without dementia and select the most accurate classier to be the basis of the AI tool. Results: Our results indicate that 1) we can nd more predictive linguistic markers to distinguish language impairment associated with dementia from participants' speech produced during the Picture Description (PD) language task than the Story Recall (SR) task; and 2) phone-based recording interfaces provide more high-quality language datasets than the web-based recording systems Conclusion: Our results verify that the tree-based classifiers, which have been trained using the linguistic and acoustic features extracted from interviews' transcript and audio, can be used to develop an AI-powered language assessment tool for detecting language impairment associated with dementia.

scholarly journals Investigation of the Motor Skills Assessments of Typically Developing Preschool Children in China

2021 ◽  
Author(s):  
Huiqi Song ◽  
Jing Jing Wang ◽  
Patrick WC Lau
Keyword(s):  
Motor Skills ◽  
Motor Development ◽  
Intellectual Development ◽  
Assessment Tool ◽  
Reliability And Validity ◽  
Assessment Tools ◽  
Skill Assessment ◽  
Skills Assessment ◽  
Validity And Reliability ◽  
Development Assessment

Abstract Background: The assessment of perschoolers’ motor skills is essential to know young children’s motor development and evaluate the intervention effects of promotion in children’s sport activities. The purpose of this study was to review the motor skills assessment tools in Chinese preschool-aged children, compare them in the international context, and provide guidelines to find appropriate motor skill assessment tool in China. Methods: The comprehensive literature search was carried out in WANGFAGN, CNKI, VIP, ERIC, EMBASE, MEDLINE, Ovid PsycINFO, SPORTDiscus and BIOSIS previews databases. Relevant articles published between January 2000 and May 2020 were retrieved. Studies that described the discriminative and evaluative measures of motor skills among the population aged 3-6 years in China were included. Results: A total of 17 studies were included in this review describing 7 tools including 4 self-developed tools and 3 international tools used in China. TGMD-2 appeared in a large proportion of studies, international tools used in China were incomplete in terms of translation, verification of reliability and validity, item selection and the implementation. Regarding the self-constructed tools, the CDCC was the most utilized self-developed tool, but it was mainly applied in intellectual development assessment. Through the comparison between Chinese self-constructed and international tools, the construction of the CDCC and the Gross Motor Development Assessment Scale contained relatively complete development steps. The test content, validity and reliability, implementation instruction, and generalizability of self-constructed tools are still lacking. Conclusions: Both international and self-developed motor skills assessment tools have been rarely applied in China, available tools lack enough validation and appropriate adjustments. Cultural differences in motor development between Chinese and Western populations should be considered when constructing a Chinese localized MSAT.

scholarly journals The impact of rater training on rater reliability in an English oral test

2020 ◽  
Vol 10 (2) ◽  
pp. 94-105
Author(s):  
Shiknesvary Karuppaiah ◽  
Abdul Halim Abdul Raof
Keyword(s):  
Secondary School Teachers ◽  
Structured Interview ◽  
Skill Assessment ◽  
Rater Training ◽  
Rater Reliability ◽  
Test Task ◽  
Qualitative Research Design ◽  
Speaking Skill ◽  
The Impact

Speaking skill assessment is gaining great interest in the field of assessment nowadays. Literature has highlighted reliability of raters in rating a speaking performance as one of the challenges due to human’s subjective nature. This study has attempted to explore the influence of rater training on rater reliability in the assessment of a spoken task. A qualitative research design was used and, semi-structured interview was employed to obtain data for this study. A total of 21 secondary school teachers participated in the study. They were raters trained to assess an oral English interaction test. Data were analyzed using thematic content analysis which resulted in three main categories i.e. importance of rater training, effects of rater training on rater reliability, and improvement of rater training. The results show that rater training is essential before any rating is to be done, and its effects include, among others, maintaining rating consistency, exposure to test task, and criteria for grading. While suggestions to improve rater training sessions are related to the length, frequency, and quality of training.

Mediating Effect of Organizational Commitment Between Islamic Human Resource Practices and Organizational Performance Among Islamic Banks of Bangladesh

2018 ◽  
Vol 15 (2) ◽  
pp. 54-65
Author(s):  
Bablu Kumar Dhar ◽  
Rosnia Masruki ◽  
Mahazan Mutalib ◽  
Hatem Mohammed Rahouma ◽  
Farid A. Sobhani ◽  
...  
Keyword(s):  
Organizational Commitment ◽  
Organizational Performance ◽  
Human Resource ◽  
Structural Equation ◽  
Stock Exchange ◽  
Reliability And Validity ◽  
Mediating Effect ◽  
Islamic Banks ◽  
Hr Practices ◽  
The Impact

This paper aims at exploring the impact of Islamic human resource (HR) practices on organizational performance though organizational commitment. Data were collected from randomly selected 170 branch managers of six Islamic Banks listed on Dhaka Stock Exchange of Bangladesh. After collecting data, descriptive analysis and structural equation model were done to examine reliability and validity of the model. By analysis, the study finds that Islamic HR practices have more significant impact on organizational performance though organizational commitment rather than the direct effect of Islamic HR practices to organizational performance. The findings of the study advocate that Islamic banks should emphasize more on Islamic HR practices and organizational commitment to uphold their organizational performance.

Empowering Women Entrepreneurs Through Microfinance - A Case of Rural Bangladesh

2017 ◽  
Vol 3 (2) ◽  
pp. 7
Author(s):  
Saida Parvin
Keyword(s):  
Women's Empowerment ◽  
Assessment Tools ◽  
Women’S Empowerment ◽  
Case Study Method ◽  
Policy Makers ◽  
Theoretical Contribution ◽  
Donor Agencies ◽  
Definition Of ◽  
The Impact

Women’s empowerment has been at the centre of research focus for many decades. Extant literature examined the process, outcome and various challenges. Some claimed substantial success, while others contradicted with evidence of failure. But the success remains a matter of debate due to lack of empirical evidence of actual empowerment of women around the world. The current study aimed to address this gap by taking a case study method. The study critically evaluates 20 cases carefully sampled to include representatives from the entire country of Bangladesh. The study demonstrates popular beliefs about microfinance often misguide even the borrowers and they start living in a fabricated feeling of empowerment, facing real challenges to achieve true empowerment in their lives. The impact of this finding is twofold; firstly there is a theoretical contribution, where the definition of women’s empowerment is proposed to be revisited considering findings from these cases. And lastly, the policy makers at governmental and non-governmental organisations, and multinational donor agencies need to revise their assessment tools for funding.

Sign in / Sign up

Export Citation Format

Share Document