Development and content validation of a competency-based assessment tool for penile prosthesis surgery

Context.— Competency-based medical education relies on frequent formative in-service assessments to ascertain trainee progression. Currently at our institution, trainees receive a summative end-of-rotation In-Training Evaluation Report based on feedback collected from staff pathologists. There is no method of simulating report sign-out. Objective.— To develop a formative in-service assessment tool that is able to simulate report sign-out and provide case-by-case feedback to trainees. Further, to compare time- versus competency-based assessment models. Design.— Twenty-one pathology trainees were assessed for 20 months. Hot Seat Diagnosis by trainees and trainee assessment by pathologists were recorded in the Laboratory Information System. In the first iteration, trainees were assessed by using a time-based assessment scale on their ability to diagnose, report, use ancillary testings, comment on clinical implications, provide intraoperative consultation and/or gross cases. The second iteration used a competency-based assessment scale. Trainees and pathologists completed surveys on the effectiveness of the In-Training Evaluation Report versus the Hot Seat Diagnosis tool. Results.— Scores from both iterations correlated significantly with other assessment tools including the Resident In-Service Examination (r = 0.93, P = .04 and r = 0.87, P = .03). The competency-based model was better able to demonstrate improvement over time and stratify junior versus senior trainees than the time-based model. Trainees and pathologists rated Hot Seat Diagnosis as significantly more objective, detailed, and timely than the In-Training Evaluation Report, and effective at simulating report sign-out. Conclusions.— Hot Seat Diagnosis is an effective tool for the formative in-service assessment of pathology trainees and simulation of report sign-out, with the competency-based model outperforming the time-based model.

Download Full-text

The Reliability of Measuring Nursing Clinical Performance Using A Competency Based Assessment Tool: A Pilot Study

Collegian Journal of the Royal College of Nursing Australia ◽

10.1016/s1322-7696(08)60373-x ◽

2000 ◽

Vol 7 (3) ◽

pp. 21-27 ◽

Cited By ~ 13

Author(s):

Murray Fisher ◽

Marti Parolin

Keyword(s):

Pilot Study ◽

Assessment Tool ◽

Clinical Performance ◽

Competency Based ◽

Nursing Clinical ◽

Competency Based Assessment

Download Full-text

MP16: Oral case presentation: evaluation of a novel curriculum and development of a competency-based assessment tool in Emergency Medicine

CJEM ◽

10.1017/cem.2020.164 ◽

2020 ◽

Vol 22 (S1) ◽

pp. S48-S48

Author(s):

T. Wawrykow ◽

T. McColl ◽

A. Velji ◽

M. Chan

Keyword(s):

Emergency Medicine ◽

Medical Students ◽

Blended Learning ◽

Assessment Tool ◽

Intervention Group ◽

Emergency Setting ◽

Case Presentation ◽

Canadian Association ◽

Competency Based ◽

Competency Based Assessment

Introduction: The oral case presentation is recognized as a core educational and patient care activity but has not been well studied in the emergency setting. The objectives of this study are: 1) to develop a competency-based assessment tool to formally evaluate the emergency medicine oral case presentation (EM-OCP) competency of medical students and ‘transition to discipline’ residents, and 2) to develop, implement and evaluate a curriculum to enhance oral case presentation (OCP) communication skills in the emergency medicine (EM) setting. Methods: Using data from a literature review, a Canadian Association of Emergency Physicians national survey, and local focus groups, the authors designed an OCP framework, blended learning curriculum, and EM-OCP assessment tool. Ninety-six clerkship students were randomly assigned to receive either the control, the standard clerkship curriculum, or intervention, the blended learning curriculum. At the beginning of their emergency medicine rotation, learners completed a pre-test using a standardized patient (SP) case to assess their baseline OCP skills. The intervention group then completed the EM-OCP curriculum. All students completed post-tests with a different SP at the end of the six-week EM rotation. Audio-recordings of pre and post-tests were evaluated using the assessment tool by two blinded evaluators. Results: Using the Kruskal-Wallis test, all students demonstrated improvement in EM-OCP skills between their pre-test and post-test, however, those who received the blended learning curriculum showed significantly greater improvement in synthesis of information (p = 0.044), management (p = 0.006) and overall entrustment decision score (p = 0.000). Conclusion: Implementation of a novel EM-OCP curriculum resulted in more effective communication and higher entrustment scores. This curriculum could improve OCP performance not only in emergency medicine settings but also across specialties where medical students and residents must manage critical patients.

Download Full-text

P057: Performance of a national simulation-based resuscitation OSCE for emergency medicine trainees

CJEM ◽

10.1017/cem.2016.233 ◽

2016 ◽

Vol 18 (S1) ◽

pp. S97-S98 ◽

Cited By ~ 1

Author(s):

C. Hagel ◽

A.K. Hall ◽

D. Klinger ◽

G. McNeil ◽

D. Dagnone

Keyword(s):

Emergency Medicine ◽

Large Scale ◽

Assessment Tool ◽

Postgraduate Medical Education ◽

Perceived Realism ◽

Simulation Based ◽

Competency Based ◽

Generalizability Study ◽

Graduate Trainees ◽

Competency Based Assessment

Introduction: The use of high-fidelity simulation is emerging as an effective method for competency-based assessment in postgraduate medical education. We have previously reported the development of the Queen’s Simulation Assessment Tool (QSAT), for use in simulation-based Objective Structured Clinical Examinations (OSCEs) for Emergency Medicine (EM) trainees. We aimed to demonstrate the feasibility and present an argument for the validity of a simulation-based OSCE utilizing the QSAT with EM residents from multiple Canadian training sites. Methods: EM post-graduate trainees (PGY 2-5) from 9 Canadian EM training programs participated in an 8-station simulation-based resuscitation OSCE at Queen’s University in Kingston, ON. Each station was scored by a single trained rater from a group of 9 expert Canadian EM physicians. Raters utilized a station-specific QSAT and provided an Entrustment Score. A post-examination questionnaire was administered to the trainees to quantify perceived realism, comfort and educational impact. Statistical analyses included analysis of variance to measure the discriminatory capabilities and a generalizability study to examine the sources of variability in the scores. Results: EM postgraduate trainees (N=36) participated in the study. Discriminatory validity was strong, with senior trainees (PGY4-5) outperforming junior trainees (PGY2-3) in 6 of 8 scenarios and in aggregated QSAT and Entrustment Scores across all 8 stations (p<0.01). Generalizability studies found the largest sources of random variability was due to the trainee by station interaction and the error term, with a G coefficient of 0.84. Resident trainees reported reasonable comfort being assessed in the simulation environment (3.6/5), indicated significant perceived realism (4.1/5), and found the OSCE valuable to their learning (4.8/5). Conclusion: Overall, this study demonstrates that a large-scale simulation-based EM resuscitation OSCE is feasible, and an argument has been presented for the validity of such an examination. The incorporation of simulation or a simulation-based OSCE in the national certification process in EM may help to satisfy the increased demand for competency-based assessment required by the Royal College of Physicians & Surgeons of Canada’s Competency by Design transition.

Download Full-text

Developing A Validated Instrument To Assess Paediatric Interns In South Africa

10.21203/rs.2.22362/v1 ◽

2020 ◽

Author(s):

Kimesh Naidoo ◽

Jacqueline M Van Wyk

Keyword(s):

Factor Analysis ◽

Assessment Tool ◽

Clinical Skills ◽

Core Competencies ◽

Methodological Approach ◽

Assessment Process ◽

Emotional Skills ◽

Daunting Task ◽

Competency Based ◽

Competency Based Assessment

Abstract Background: Recently graduated South African medical practitioners (interns) are expected to be efficient and resilient in limited resource contexts that face multiple disease burdens. Work-based assessment in SA internship focuses on clinical skills and neglects the evaluation of non-clinical skills. This sub-optimal assessment process creates a daunting task for clinician supervisors who are expected to certify these interns for independent practice. Methods: Using a mixed methodological approach, the study sampled a cross section of 411 interns in seven hospitals in KwaZulu-Natal and their intern supervisors . Data collection methods included focus groups discussions and surveys. A 4-round modified Delphi-process was followed to reach consensus on the set of core competencies required. The assessment tool that resulted in the process included a variety of competencies and the tool was validated through factor analysis and internal consistency was measured with Cronbach’s alpha. Results : A competency-based assessment tool was developed that allows for the assessment of both clinical and non-clinical skills. This tool was found to be very reliable, with an overall Cronbach alpha of 0.927. Factor analysis for the instrument revealed 61 observable clinical activities aggregated into entrustable professional activities and these measured four major factors. The factors correspond with procedural clinical skills; holistic-care skills; and emotional skills related to social competence and self-management. Conclusion: A locally relevant competency-based assessment tool was developed in SA to assess local paediatric interns. The validated tool’s function as a multidimensional instrument to assess both clinical and non-clinical skills included the assessment of emotional skills which are largely neglected. Validated work-based assessment instruments can be developed that are responsive to local needs and support the development of holistic clinicians in high disease-burdened contexts.

Download Full-text

Development of Competency Based Assessment Model on Job Performance in Family Welfare Education Apprenticeship

Innovation of Vocational Technology Education ◽

10.17509/invotec.v13i1.6262 ◽

2017 ◽

Vol 13 (1) ◽

Author(s):

Yoyoh Jubaedah ◽

Neni Rohaeni ◽

Nenden R. Rinekasari

Keyword(s):

Job Performance ◽

Assessment Tool ◽

Expert Judgment ◽

Assessment Model ◽

Family Welfare ◽

Research Subjects ◽

Learner Performance ◽

Competency Based ◽

Development Approach ◽

Competency Based Assessment

The assessment tool to evaluate learner performance on the apprenticeship is variety. Therefore, we were need a standardized assessment tool. This study was attempt to develop a competency based assessment model on job performance in apprentice of the family welfare education. Problem solving was conducted using research and development approach, through stages of: (1) a study introduction, (2) a development model, and (3) a validation model. The research subjects are internal and external tutors, also a learner intern. Data collection was carried out by interviews, observation, survey and expert judgment. The results showed that the planning, the instrument and the implementation of the assessment was the development of competency based assessment model on job performance in family welfare education apprenticeship. Planning assessment was consisting of components of the purpose, job performance, and assessment methods. An instrument assessment was use a test of performance in the form of assessment rubrics. The implementation of the assessment was covering a preparation, collecting, judging, deciding, and moderation stage. From the validation model through the expert judgment, it turns out that assessment model was already developed feasible to implement on job performance assessment in family welfare education apprenticeship

Download Full-text

MODEL COMPETENCY BASED ASSESSMENT PADA PENGEMBANGAN JOB PERFORMANCE PUBLIC AREA DI HOUSEKEEPING DEPARTMENT

JKKP (Jurnal Kesejahteraan Keluarga dan Pendidikan) ◽

10.21009/jkkp.052.05 ◽

2018 ◽

Vol 5 (2) ◽

pp. 139-146

Author(s):

Yoyoh Jubaedah ◽

Neni Rohaeni ◽

Nenden Rani Rinekasari ◽

Iis Rohmah

Keyword(s):

Job Performance ◽

Assessment Tool ◽

Assessment Tools ◽

Assessment Model ◽

Vocational High School ◽

The Public ◽

Competency Based ◽

Public Area ◽

Judgment Data ◽

Competency Based Assessment

The purpose of this research is to apply a competency-based assessment model on the job performance of Public Area at Housekeeping Department. The scope of the study produced 2 targets in housekeeping department: 1) competency based assessmen model on the job performance of public area; 2) assessment tool on job performance of public area. This study was used a descriptive method with three stages of development study. First is a preliminary study, second stage is development model, and finally is validation test of the model. The research subject was teacher of public area subject matter, and employees of housekeeping department. Data was collected through interviews, observations, and expert judgment. Data was analyzed in accordance with the procedures and characteristic of the findings. It was used both the qualitative and quantitative data. The finding showed that competency-based assessment model on the job performance of public area in housekeeping department included the planning, the instrument and the process of assessment stages. The instrument consisted of the preparation, practices, and outcome of assessment stages. The conclusion is assessment tools on the public area practices using competency-based assessment is very appropriate to apply this competency-based assessment at senior vocational high school, especially for job performance on housekeeping department expertise. Abstrak Tujuan dari penelitian ini adalah untuk menerapkan model Competency Based Assessment pada pengembangan Job Performance Public Area di Housekeeping Department. Target khusus yang ingin dicapai dari penelitian ini adalah menghasilkan: (1) Model Competency Based Assessment pada pengembangan Job Performance Public Area di Housekeeping Department, (2) Alat Penilaian Job Performance Public Area di Housekeeping Department. Penelitian ini mengunakan metode deskriptif dengan pendekatan Research and Development. Studi pengembangan ini dilakukan dalam tiga tahap, meliputi: Tahap pertama merupakan tahap studi pendahuluan, Tahap kedua merupakan tahap pengembangan model dan Tahap ketiga merupakan tahap uji validasi model. Subjek penelitian terdiri dari guru pengampu mata pelajaran Public Area dan praktisi dari housekeeping depatment. Teknik pengumpulan data dilakukan melalui wawancara, observasi, dan expert judgement. Analisis data penelitian dilakukan sesuai dengan prosedur dan karakteristik dari temuan data penelitian; untuk data kuantitatif dianalisis secara kuantitatif dan untuk data kualitatif dianalisis secara kualitatif. Temuan penelitian berkaitan dengan Model Competency Based Assessment pada pengembangan Job Performance Public Area di Housekeeping Department meliputi tahap perencanaan penilaian, instrumen penilaian, dan pelaksanaan penilaian. Instrumen penilaian terdiri dari tahap persiapan praktik, proses praktik, dan hasil praktik. Dari temuan penelitian dan hasil validasi dapat disimpulkan bahwa Alat Penilaian praktik Public Area dengan menerapkan model Competency Based Assessment â€œsangat layakâ€ untuk diimplementasikan di SMK Akomodasi Perhotelan dan di Housekeeping Department pada saat melakukan penilaian terhadap Job Performance Public area. Kata Kunci : Model, Competency Based Assessment, Job Performance, Public Area, Housekeeping Department.

Download Full-text