Ensuring the Validity of Technical Skill Assessment Outcomes in Technology Education in Nigeria

BACKGROUND: Sensor technologies and data collection practices are changing and improving quality metrics across various domains. Surgical skill assessment in Robot-Assisted Minimally Invasive Surgery (RAMIS) is essential for training and quality assurance. The mental workload on the surgeon (such as time criticality, task complexity, distractions) and non-technical surgical skills (including situational awareness, decision making, stress resilience, communication, leadership) may directly influence the clinical outcome of the surgery. METHODS: A literature search in PubMed, Scopus and PsycNet databases was conducted for relevant scientific publications. The standard PRISMA method was followed to filter the search results, including non-technical skill assessment and mental/cognitive load and workload estimation in RAMIS. Publications related to traditional manual Minimally Invasive Surgery were excluded, and also the usability studies on the surgical tools were not assessed. RESULTS: 50 relevant publications were identified for non-technical skill assessment and mental load and workload estimation in the domain of RAMIS. The identified assessment techniques ranged from self-rating questionnaires and expert ratings to autonomous techniques, citing their most important benefits and disadvantages. CONCLUSIONS: Despite the systematic research, only a limited number of articles was found, indicating that non-technical skill and mental load assessment in RAMIS is not a well-studied area. Workload assessment and soft skill measurement do not constitute part of the regular clinical training and practice yet. Meanwhile, the importance of the research domain is clear based on the publicly available surgical error statistics. Questionnaires and expert-rating techniques are widely employed in traditional surgical skill assessment; nevertheless, recent technological development in sensors and Internet of Things-type devices show that skill assessment approaches in RAMIS can be much more profound employing automated solutions. Measurements and especially big data type analysis may introduce more objectivity and transparency to this critical domain as well. SIGNIFICANCE: Non-technical skill assessment and mental load evaluation in Robot-Assisted Minimally Invasive Surgery is not a well-studied area yet; while the importance of this domain from the clinical outcome’s point of view is clearly indicated by the available surgical error statistics.

Download Full-text

Pengaruh Pembelajaran Fisika Berbasis STEM Terhadap Kemampuan Motorik Siswa

Prosiding SNFA (Seminar Nasional Fisika dan Aplikasinya) ◽

10.20961/prosidingsnfa.v4i0.35906 ◽

2019 ◽

Vol 4 ◽

pp. 1

Author(s):

Afipah Muyassarah ◽

Tursina Ratu ◽

Muhammad Erfan

Keyword(s):

Technology Education ◽

Motor Skills ◽

T Test ◽

Assessment Instrument ◽

Skill Assessment ◽

Motor Ability ◽

Physics Learning ◽

Paired Sample ◽

Science Technology ◽

And Mathematics

Abstract: STEM (Science, Technology, Education, and Mathematics) is learning that not only involves scientific and mathematical concepts (cognitive aspects) but also technically about how a concept is obtained (motoric). The aspect of motor ability is one of the aspects obtained by students in the learning process apart from the cognitive and affective aspects, but generally most studies put forward cognitive aspects (learning outcomes) so that motor skills are rarely exposed. Therefore this study aims to find out how much influence the STEM-based Physics Learning has on students' motor skills. This research is a quasi-experimental study with the design of One Groups Pretest-Posttest. Data was collected by student worksheet and Student Motor Skill Assessment Instrument Sheets and analysed by paired sample t-test. The results showed that the results of paired sample t test (t value = -64,761 far below t table) and the obtained N-Gain value was 0.55 (moderate gain) so it can be concluded that STEM-based physics learning is significantly capable of increasing students' motor ability.Abstrak: Pembelajaran STEM (Science, Technology, Education, and Mathematics) merupakan pembelajaran yang tidak hanya melibatkan kosep-konsep sains dan matematika saja (aspek kognitif) tetapi juga secara teknis mengenai bagaimana suatu konsep itu diperoleh (motorik). Aspek kemampuan motorik merupakan salah satu aspek yang diperoleh siswa dalam proses pembelajaran selain daripada aspek kognitif dan afektif, namun umumnya kebanyakan penelitian lebih mengedepankan aspek kognitif (hasil belajar) sehingga kemampuan motorik jarang terekspose. Oleh karena itu penelitian ini bertuuan untuk mengetahui seberapa besar pengaruh Pembelajaran Fisika berbasis STEM terhadap kemampuan motorik siswa. Penelitian ini termasuk penelitian kuasi eksperimen dengan desain One Groups Pretest-Posttest. Data dikumpulkan dengan LKPD dan Lembar Instrumen Penilaian Kemampuan Motorik Siswa serta dianalisis dengan uji-t sampel berpasangan. Hasil penelitian menunjukkan bahwa hasil uji paired sample t test (nilai t = -64,761 jauh dibawah t tabel (-1.69092), dan nilai N-Gain yang diperoleh sebesar 0,55 (gain sedang) sehingga dapat ditarik kesimpulan bahwa pembelajaran fisika berbasis STEM secara signifikan mampu meningkatnya capaian kemampuan motorik siswa.

Download Full-text

EP.TH.589Comparison of Surgeon Gaze Behaviour Against Objective Skill Assessment in Open Inguinal Hernia Repair – A Pilot Study

British Journal of Surgery ◽

10.1093/bjs/znab309.061 ◽

2021 ◽

Vol 108 (Supplement_7) ◽

Author(s):

Oliver Salazar ◽

Simon Erridge ◽

Jasmine Winter Beatty ◽

Ara Darzi ◽

Sanjay Purkayastha ◽

...

Keyword(s):

Pilot Study ◽

Eye Tracking ◽

Technical Skill ◽

Skill Assessment ◽

Skills Assessment ◽

Power Calculation ◽

Surgical Skill ◽

High Stakes ◽

Objective Skill Assessment ◽

Gaze Behaviour

Abstract Aims Technical skill is associated with improved postoperative outcomes. Adoption of a formalised high-stakes assessment of surgical skill is technically challenging and limited by the financial and human resources available. We aimed to assess the ability to adopt gaze behaviour analysis as an assessment of surgical skill within live open inguinal herniorrhaphy. Methods Surgeons’ gaze was measured with Tobii Pro eye-tracking Glasses 2 (Tobii AB). All grades of surgeons were included. Primary outcomes were dwell time (%) and fixation frequency (count/s), as markers of cognition, on areas of interest correlated to mean Objective Skill Assessment of Technical Skill score. Secondary outcomes assessed effort and concentration levels through maximum pupil diameter (mm) and rate of pupil change (mm/s) correlated to perceived workload (SURG-TLX). Three operative segments underwent analysis: mesh preparation, fixation and muscle closure. Spearman’s and Pearson’s correlation were performed with significance set at p < 0.05. Results 5 cases were analysed, totalling 270 minutes of video footage. All participants were senior surgical trainees and right-hand-dominant. The median number of hernia operations performed was 160 (range:100-500). The median ASA score of each patient participant was 2 (range:1-2). The median operation length was 45 mins (range:40-90 mins). There were no statistically significant primary outcomes from this pilot data (p > 0.05). Conclusions This pilot study demonstrated the feasibility of recording gaze behaviours for comparison against formal skills assessment to determine the role of eye tracking in live high stakes technical skills assessment. A full study will now commence based on formal power calculation.

Download Full-text

PD58-12 SURGEON TECHNICAL SKILL ASSESSMENT USING COMPUTER VISION-BASED ANALYSIS

The Journal of Urology ◽

10.1016/j.juro.2018.02.2800 ◽

2018 ◽

Vol 199 (4S) ◽

Author(s):

Yetong Zhang ◽

Hei Law ◽

Tae-Kyung Kim ◽

David Miller ◽

James Montie ◽

...

Keyword(s):

Computer Vision ◽

Technical Skill ◽

Skill Assessment

Download Full-text

Establishing a Simulation Training Platform for Cardiac Surgery Residents: Improving Surgical Training and Technical Skill Assessment

The Thoracic and Cardiovascular Surgeon ◽

10.1055/s-0038-1628015 ◽

2018 ◽

Vol 66 (S 01) ◽

pp. S1-S110

Author(s):

A. Martens ◽

W. Korte ◽

M. Shrestha ◽

I. Ismail ◽

S. Cebotari ◽

...

Keyword(s):

Cardiac Surgery ◽

Surgical Training ◽

Simulation Training ◽

Technical Skill ◽

Skill Assessment

Download Full-text

Validity evidence as a key marker of quality of technical skill assessment in OTL-HNS

The Laryngoscope ◽

10.1002/lary.27085 ◽

2018 ◽

Vol 128 (10) ◽

pp. 2296-2300 ◽

Cited By ~ 1

Author(s):

Mathilde Labbé ◽

Meredith Young ◽

Lily H.P. Nguyen

Keyword(s):

Technical Skill ◽

Skill Assessment ◽

Validity Evidence

Download Full-text

SMART TECHNICAL SKILL ASSESSMENT ANDROID APPLICATION USING MACHINE LEARNING

International Journal of Engineering Applied Sciences and Technology ◽

10.33564/ijeast.2020.v05i03.116 ◽

2020 ◽

Vol 5 (3) ◽

pp. 671-677

Author(s):

Ishwar Kalbandi ◽

Mohammad Suhaib Ahmed ◽

Rohan Atale ◽

Ajinkya Bhujbal ◽

Nikhunj Saraf

Keyword(s):

Machine Learning ◽

Technical Skill ◽

Skill Assessment ◽

Android Application

Download Full-text

The Intra-rater Reliability of Nine Content-Validated Technical Skill Assessment Instruments (TSAI) for Athletic Taping Skills.

Athletic Training Education Journal ◽

10.4085/1947-380x-3.3.91 ◽

2008 ◽

Vol 3 (3) ◽

pp. 91-101 ◽

Cited By ~ 2

Author(s):

Niko G. Lagumen ◽

Dale J. Butterwick ◽

David M. Paskevich ◽

Tak S. Fung ◽

Tyrone L. Donnon

Keyword(s):

Repeated Measures ◽

Technical Skill ◽

Skill Assessment ◽

Standardized Patient ◽

Assessment Instruments ◽

Rater Reliability ◽

University Of Calgary ◽

Testing Site ◽

Mean Differences ◽

One Way Anova

Objective: To establish the intra-rater reliability of nine content-validated Technical Skill Assessment Instruments (TSAI) for the skills of athletic taping. Setting: University of Calgary. Subjects: Canadian Certified Athletic Therapists, CAT(C), with a mean ± SD of 9.6 ± 10.8 years as a CAT(C), 7.8 ± 10.9 years as a Supervisory Athletic Therapist, 8.5 ± 12.0 years teaching athletic taping skills, and 9.2 ± 11.5 years evaluating athletic taping skills. Design: Six Certified Athletic Therapists from Canada completed the repetitive evaluations of nine different athletic taping scenarios. Each rater evaluated the performance of a student therapist taping a standardized patient while using the appropriate TSAI designed for each athletic taping scenario. Evaluations occurred once per month for five successive months. Raters viewed the performances on a portable DVD player at a central testing site. Measurements: The percent scores of 270 completed TSAIs were used for analysis. ICC (3, k) was used to quantify the intra-rater reliability. We used a One-way ANOVA with repeated measures to determine if mean differences across testing months existed within raters. Significance was achieved with α = 0.05. Results: ICC values for the nine TSAIs ranged from 0.65 to 0.95 with Ankle 3 and Thumb 2 achieving the lowest and highest ICC values respectively. One-way ANOVA with repeated measures did not provide significant mean differences between testing months within each rater. Conclusion: The nine TSAIs possess substantial to almost perfect reliability with seven TSAIs appropriate for summative evaluations and two appropriate for formative evaluations.

Download Full-text

A Generalizability Theory Study of Athletic Taping Using the Technical Skill Assessment Instrument

Journal of Athletic Training ◽

10.4085/1062-6050-49.2.22 ◽

2014 ◽

Vol 49 (3) ◽

pp. 368-372 ◽

Cited By ~ 2

Author(s):

Mark R. Lafave ◽

Dale J. Butterwick

Keyword(s):

Student Performance ◽

Generalizability Theory ◽

Optimal Number ◽

Technical Skill ◽

Assessment Instrument ◽

Skill Assessment ◽

Standardized Patient ◽

Therapy Program ◽

Athletic Therapy ◽

Reliability Coefficients

Context: Athletic taping skills are highly valued clinical competencies in the athletic therapy and training profession. The Technical Skill Assessment Instrument (TSAI) has been content validated and tested for intrarater reliability. Objective: To test the reliability of the TSAI using a more robust measure of reliability, generalizability theory, and to hypothetically and mathematically project the optimal number of raters and scenarios to reliably measure athletic taping skills in the future. Setting: Mount Royal University. Design: Observational study. Patients or Other Participants: A total of 29 university students (8 men, 21 women; age = 20.79 ± 1.59 years) from the Athletic Therapy Program at Mount Royal University. Intervention(s): Participants were allowed 10 minutes per scenario to complete prophylactic taping for a standardized patient presenting with (1) a 4-week-old second-degree ankle sprain and (2) a thumb that had been hyperextended. Two raters judged student performance using the TSAI. Main Outcome Measure(s): Generalizability coefficients were calculated using variance scores for raters, participants, and scenarios. A decision study was calculated to project the optimal number of raters and scenarios to achieve acceptable levels of reliability. Generalizability coefficients were interpreted the same as other reliability coefficients, with 0 indicating no reliability and 1.0 indicating perfect reliability. Results: The result of our study design (2 raters, 1 standardized patient, 2 scenarios) was a generalizability coefficient of 0.67. Decision study projects indicated that 4 scenarios were necessary to reliably measure athletic taping skills. Conclusions: We found moderate reliability coefficients. Researchers should include more scenarios to reliably measure athletic taping skills. They should also focus on the development of evidence-based practice guidelines and standards of athletic taping and should test those standards using a psychometrically sound instrument, such as the TSAI.

Download Full-text