scholarly journals Assessing the Validity Evidence of an Objective Structured Assessment Tool of Technical Skills for Neonatal Lumbar Punctures

2013 ◽  
Vol 20 (3) ◽  
pp. 321-324 ◽  
Author(s):  
Maya S. Iyer ◽  
Sally A. Santen ◽  
Michele Nypaver ◽  
Kavita Warrier ◽  
Stuart Bradin ◽  
...  
Keyword(s):  
Assessment Tool ◽  
Technical Skills ◽  
Validity Evidence

Development and Validation of an Assessment Tool for Technical Skills in Handheld Otoscopy

2020 ◽  
Vol 129 (7) ◽  
pp. 715-721
Author(s):  
Mads J. Guldager ◽  
Jacob Melchiors ◽  
Steven Arild Wuyts Andersen
Keyword(s):  
Medical Students ◽  
Assessment Tool ◽  
Medical Training ◽  
Technical Skills ◽  
University Teaching ◽  
Validity Evidence ◽  
Level Of Evidence ◽  
Training Interventions ◽  
Diagnostic Skills ◽  
Novices And Experts

Objective: Handheld otoscopy requires both technical and diagnostic skills, and is often reported to be insufficient after medical training. We aimed to develop and gather validity evidence for an assessment tool for handheld otoscopy using contemporary medical educational standards. Study Design: Educational study. Setting: University/teaching hospital. Subjects and Methods: A structured Delphi methodology was used to develop the assessment tool: nine key opinion leaders (otologists) in undergraduate training of otoscopy iteratively achieved consensus on the content. Next, validity evidence was gathered by the videotaped assessment of two handheld otoscopy performances of 15 medical students (novices) and 11 specialists in otorhinolaryngology using two raters. Standard setting (pass/fail criteria) was explored using the contrasting groups and Angoff methods. Results: The developed Copenhagen Assessment Tool of Handheld Otoscopy Skills (CATHOS) consists 10 items rated using a 5-point Likert scale with descriptive anchors. Validity evidence was collected and structured according to Messick’s framework: for example the CATHOS had excellent discriminative validity (mean difference in performance between novices and experts 20.4 out of 50 points, P < .001); and high internal consistency (Cronbach’s alpha = 0.94). Finally, a pass/fail score was established at 30 points for medical students and 42 points for specialists in ORL. Conclusion: We have developed and gathered validity evidence for an assessment tool of technical skills of handheld otoscopy and set standards of performance. Standardized assessment allows for individualized learning to the level of proficiency and could be implemented in under- and postgraduate handheld otoscopy training curricula, and is also useful in evaluating training interventions. Level of evidence: NA

scholarly journals Diagnosing delivery capabilities on a large international nature-based solutions project

2021 ◽  
Vol 1 (1) ◽  
Author(s):  
Thami Croeser ◽  
Georgia E. Garrard ◽  
Freya M. Thomas ◽  
Trinh Duc Tran ◽  
Ian Mell ◽  
...  
Keyword(s):  
Assessment Tool ◽  
Technical Skills ◽  
Project Teams ◽  
Strategy Development ◽  
Risk Averse ◽  
Institutional Barriers ◽  
Horizon 2020 ◽  
Significant Control ◽  
Implementation Gap ◽  
Organisational Cultures

AbstractNature-based solutions (NBS) are increasingly at the centre of urban strategies to mitigate heatwaves and flooding, improve public health and restore biodiversity. However, on-ground implementation has been slow, inconsistent and often limited to demonstration sites. A broad literature consistently highlights institutional barriers as a major reason for the observed implementation gap. In this study, we developed and deployed an assessment tool to identify barriers to NBS delivery on a European Commission Horizon 2020 project spanning seven cities. We found that practitioners were effectively navigating challenges in the areas where they had significant control, including community engagement, strategy development and technical skills. The greatest barriers were outside the influence of project teams: understaffing, a lack of intra-organisational processes, and risk-averse organisational cultures. These findings emphasise that after cities embrace NBS at the strategic and political level, it is vital that executives follow through with the necessary pragmatic reforms to enable delivery.

Objective structured assessment of technical skill in temporal bone dissection: validation of a novel tool

2021 ◽  
pp. 1-11
Author(s):  
M Stavrakas ◽  
G Menexes ◽  
S Triaridis ◽  
P Bamidis ◽  
J Constantinidis ◽  
...  
Keyword(s):  
Temporal Bone ◽  
Rating Scale ◽  
Assessment Tool ◽  
Three Dimensional ◽  
Technical Skills ◽  
Technical Skill ◽  
Global Rating Scale ◽  
Global Rating ◽  
Validity And Reliability ◽  
Cortical Mastoidectomy

Abstract Objective This study developed an assessment tool that was based on the objective structured assessment for technical skills principles, to be used for evaluation of surgical skills in cortical mastoidectomy. The objective structured assessment of technical skill is a well-established tool for evaluation of surgical ability. This study also aimed to identify the best material and printing method to make a three-dimensional printed temporal bone model. Methods Twenty-four otolaryngologists in training were asked to perform a cortical mastoidectomy on a three-dimensional printed temporal bone (selective laser sintering resin). They were scored according to the objective structured assessment of technical skill in temporal bone dissection tool developed in this study and an already validated global rating scale. Results Two external assessors scored the candidates, and it was concluded that the objective structured assessment of technical skill in temporal bone dissection tool demonstrated some main aspects of validity and reliability that can be used in training and performance evaluation of technical skills in mastoid surgery. Conclusion Apart from validating the new tool for temporal bone dissection training, the study showed that evolving three-dimensional printing technologies is of high value in simulation training with several advantages over traditional teaching methods.

scholarly journals Assessing Ethical Thinking about AI

2020 ◽  
Vol 34 (09) ◽  
pp. 13525-13528
Author(s):  
Judy Goldsmith ◽  
Emanuelle Burton ◽  
David M. Dueber ◽  
Beth Goldstein ◽  
Shannon Sampson ◽  
...  
Keyword(s):  
Ethics Education ◽  
Assessment Tool ◽  
Ethical Reasoning ◽  
Validity Evidence ◽  
Learner Outcomes ◽  
Ai Ethics ◽  
Ethical Thinking ◽  
Will Force

As is evidenced by the associated AI, Ethics and Society conference, we now take as given the need for ethics education in the AI and general CS curricula. The anticipated surge in AI ethics education will force the field to reckon with delineating and then evaluating learner outcomes to determine what is working and improve what is not. We argue for a more descriptive than normative focus of this ethics education, and propose the development of assessments that can measure descriptive ethical thinking about AI. Such an assessment tool for measuring ethical reasoning capacity in CS contexts must be designed to produce reliable scores for which there is established validity evidence concerning their interpretation and use.

Examining validity evidence for a simulation-based assessment tool for basic robotic surgical skills

2018 ◽  
Vol 13 (1) ◽  
pp. 99-106 ◽  
Author(s):  
Maria Cecilie Havemann ◽  
Torur Dalsgaard ◽  
Jette Led Sørensen ◽  
Kristin Røssaak ◽  
Steffen Brisling ◽  
...  
Keyword(s):  
Assessment Tool ◽  
Surgical Skills ◽  
Validity Evidence ◽  
Simulation Based

Toward an Evidence-Based Approach to Building Evaluation Capacity

2021 ◽  
Vol 36 (1) ◽  
Author(s):  
Btissam El Hassar ◽  
Cheryl Poth ◽  
Rebecca Gokiert ◽  
Okan Bulut
Keyword(s):  
Structural Equation ◽  
Assessment Tool ◽  
Evidence Based ◽  
Validity Evidence ◽  
Factor Analyses ◽  
Future Directions ◽  
Evaluation Capacity ◽  
Organizational Evaluation ◽  
Equation Modelling ◽  
Valid Evidence

Organizations are required to evaluate their programs for both learning and accountability purposes, which has increased the need to build their internal evaluation capacity. A remaining challenge is access to tools that lead to valid evidence supporting internal capacity development. The authors share practical insights from the development and use of the Evaluation Capacity Needs Assessment tool and framework and implications for using its data to make concrete decisions within Canadian contexts. The article refers to validity evidence generated from factor analyses and structural equation modelling and describes how applying the framework can be used to identify individual and organizational evaluation capac­ity strengths and gaps, concluding with practice considerations and future directions for this work.  

Non-technical skills and otolaryngology: systematic review

2020 ◽  
Vol 134 (5) ◽  
pp. 415-418 ◽  
Author(s):  
R Bannon ◽  
K E Stewart ◽  
M Bannister
Keyword(s):  
Systematic Review ◽  
Assessment Tool ◽  
Controlled Trial ◽  
Technical Skills ◽  
Neck Surgery ◽  
Future Research ◽  
Skill Levels ◽  
Simulated Environments ◽  
Simulation Based ◽  
Randomised Controlled

AbstractObjectivesThis study aimed to assess the published literature on non-technical skills in otolaryngology surgery and examine the applicability of any research to others’ practice, and to explore how the published literature can identify areas for further development and guide future research.MethodsA systematic review was conducted using the following key words: ‘otolaryngology’, ‘otorhinolaryngology’, ‘ENT’, ‘ENT surgery’, ‘ear, nose and throat surgery’, ‘head and neck surgery’, ‘thyroid surgery’, ‘parathyroid surgery’, ‘otology’, ‘rhinology’, ‘laryngology’ ‘skull base surgery’, ‘airway surgery’, ‘non-technical skills’, ‘non technical skills for surgeons’, ‘NOTSS’, ‘behavioural markers’ and ‘behavioural assessment tool’.ResultsThree publications were included in the review – 1 randomised, controlled trial and 2 cohort studies – involving 78 participants. All were simulation-based studies involving training otolaryngology surgeons.ConclusionLittle research has been undertaken on non-technical skills in otolaryngology. Training surgeons’ non-technical skill levels are similar across every tested aspect. The research already performed can guide further studies, particularly amongst non-training otolaryngology surgeons and in both emergency and elective non-simulated environments.

Validation evidence of the paediatric Objective Structured Assessment of Debriefing (OSAD) tool

2016 ◽  
Vol 2 (3) ◽  
pp. 61-67 ◽  
Author(s):  
Jane Runnacles ◽  
Libby Thomas ◽  
James Korndorffer ◽  
Sonal Arora ◽  
Nick Sevdalis
Keyword(s):  
Assessment Tool ◽  
Learning Experience ◽  
Pearson Correlation ◽  
Phase 1 ◽  
Evidence Based ◽  
Validity Evidence ◽  
Multiple Sources ◽  
Simulation Based ◽  
Expert Input

IntroductionDebriefing is essential to maximise the simulation-based learning experience, but until recently, there was little guidance on an effective paediatric debriefing. A debriefing assessment tool, Objective Structured Assessment of Debriefing (OSAD), has been developed to measure the quality of feedback in paediatric simulation debriefings. This study gathers and evaluates the validity evidence of OSAD with reference to the contemporary hypothesis-driven approach to validity.MethodsExpert input on the paediatric OSAD tool from 10 paediatric simulation facilitators provided validity evidence based on content and feasibility (phase 1). Evidence for internal structure validity was sought by examining reliability of scores from video ratings of 35 postsimulation debriefings; and evidence for validity based on relationship to other variables was sought by comparing results with trainee ratings of the same debriefings (phase 2).ResultsSimulation experts’ scores were significantly positive regarding the content of OSAD and its instructions. OSAD's feasibility was demonstrated with positive comments regarding clarity and application. Inter-rater reliability was demonstrated with intraclass correlations above 0.45 for 6 of the 7 dimensions of OSAD. The internal consistency of OSAD (Cronbach α) was 0.78. Pearson correlation of trainee total score with OSAD total score was 0.82 (p<0.001) demonstrating validity evidence based on relationships to other variables.ConclusionThe paediatric OSAD tool provides a structured approach to debriefing, which is evidence-based, has multiple sources of validity evidence and is relevant to end-users. OSAD may be used to improve the quality of debriefing after paediatric simulations.

Sign in / Sign up

Export Citation Format

Share Document