LO85: Development of a competency based assessment tool for emergency department point of care ultrasound

Innovation Concept: Assessment of residents' Point of Care Ultrasound (PoCUS) competency currently relies on heterogenous and unvalidated methods, such as the completion of a number of proctored studies. Although number of performed studies may be associated with ability, it is not necessarily a surrogate for competence. Our goal was to create a single Ultrasound Competency Assessment Tool (UCAT) using domain-anchored entrustment scoring. Methods: The UCAT was developed as an anchored global assessment score, building on a previously validated simulation-based assessment tool. It was designed to measure performance across the domains of Preparation, Image Acquisition, Image Optimization, and Clinical Integration, in addition to providing a final entrustment score (i.e., OSCORE). A modified Delphi method was used to establish national expert consensus on anchors for each domain. Three surveys were distributed to the CAEP Ultrasound Committee between July-November 2018. The first survey asked members to appraise and modify a list of anchor options created by the authors. Next, collated responses from the first survey were redistributed for a re-appraisal. Finally, anchors obtaining >65% approval from the second survey were condensed and redistributed for final consensus. Curriculum, Tool or Material: Twenty-two, 26, and 22 members responded to the surveys, respectively. Each anchor achieved >90% final agreement. The final anchors for the domains were: Preparation – positioning, initial settings, ensures clean transducer, probe selection, appropriate clinical indication; Image Acquisition – appropriate measurements, hand position, identifies landmarks, visualization of target, efficiency of probe motion, troubleshoots technical limitations; Image Optimization – centers area of interest, overall image quality, troubleshoots patient obstacles, optimizes settings; Clinical Integration – appropriate interpretation, understands limitations, utilizes information appropriately, performs multiple scans if needed, communicates findings, considers false positive and negative causes of findings. Conclusion: The UCAT is a novel assessment tool that has the potential to play a central role in the training and evaluation of residents. Our use of a modified Delphi method, involving key stakeholders in PoCUS education, ensures that the UCAT has a high degree of process and content validity. An important next step in determining its construct validity is to evaluate the use of the UCAT in a multi-centered examination setting.

Download Full-text

Development and Validation of an Assessment Tool for Competency in Critical Care Ultrasound

Journal of Graduate Medical Education ◽

10.4300/jgme-d-14-00613.1 ◽

2015 ◽

Vol 7 (4) ◽

pp. 567-573 ◽

Cited By ~ 12

Author(s):

Paru Patrawalla ◽

Lewis Ari Eisen ◽

Ariel Shiloh ◽

Brijen J. Shah ◽

Oleksandr Savenkov ◽

...

Keyword(s):

Critical Care ◽

Delphi Method ◽

Assessment Tool ◽

Point Of Care ◽

Critical Care Medicine ◽

Competency Assessment ◽

Point Of Care Ultrasound ◽

Modified Delphi Method ◽

Modified Delphi ◽

Critical Care Ultrasound

ABSTRACT Background Point-of-care ultrasound is an emerging technology in critical care medicine. Despite requirements for critical care medicine fellowship programs to demonstrate knowledge and competency in point-of-care ultrasound, tools to guide competency-based training are lacking. Objective We describe the development and validity arguments of a competency assessment tool for critical care ultrasound. Methods A modified Delphi method was used to develop behaviorally anchored checklists for 2 ultrasound applications: “Perform deep venous thrombosis study (DVT)” and “Qualify left ventricular function using parasternal long axis and parasternal short axis views (Echo).” One live rater and 1 video rater evaluated performance of 28 fellows. A second video rater evaluated a subset of 10 fellows. Validity evidence for content, response process, and internal consistency was assessed. Results An expert panel finalized checklists after 2 rounds of a modified Delphi method. The DVT checklist consisted of 13 items, including 1.00 global rating step (GRS). The Echo checklist consisted of 14 items, and included 1.00 GRS for each of 2 views. Interrater reliability evaluated with a Cohen kappa between the live and video rater was 1.00 for the DVT GRS, 0.44 for the PSLA GRS, and 0.58 for the PSSA GRS. Cronbach α was 0.85 for DVT and 0.92 for Echo. Conclusions The findings offer preliminary evidence for the validity of competency assessment tools for 2 applications of critical care ultrasound and data on live versus video raters.

Download Full-text

LO042: Sonography in Hypotension and Cardiac Arrest (SHoC) - Hypotension: derivation of an evidence-based consensus algorithm for the integration of point of care ultrasound into resuscitation of hypotensive patients

CJEM ◽

10.1017/cem.2016.79 ◽

2016 ◽

Vol 18 (S1) ◽

pp. S44-S44 ◽

Cited By ~ 1

Author(s):

P. Atkinson ◽

J. Bowra ◽

J. Milne ◽

M. Lambert ◽

B. Jarman ◽

...

Keyword(s):

Point Of Care ◽

Disease Incidence ◽

Left Ventricular ◽

Consensus Algorithm ◽

Professional Organizations ◽

Clinical Indication ◽

Point Of Care Ultrasound ◽

International Consensus ◽

Modified Delphi ◽

Delphi Approach

Introduction: Point of care ultrasound has become an established tool in the initial management of patients with undifferentiated hypotension. Current established protocols (RUSH, ACES, etc) were developed by expert user opinion, rather than objective, prospective data. We wished to use reported disease incidence to develop an informed approach to PoCUS in hypotension using a “4 F’s” approach: Fluid; Form; Function; Filling. Methods: We summarized the incidence of PoCUS findings from an international multicentre RCT, and using a modified Delphi approach incorporating this data we obtained the input of 24 international experts associated with five professional organizations led by the International Federation of Emergency Medicine. The modified Delphi tool was developed to reach an international consensus on how to integrate PoCUS for hypotensive emergency department patients. Results: Rates of abnormal PoCUS findings from 151 patients with undifferentiated hypotension included left ventricular dynamic changes (43%), IVC abnormalities (27%), pericardial effusion (16%), and pleural fluid (8%). Abdominal pathology was rare (fluid 5%, AAA 2%). After two rounds of the survey, using majority consensus, agreement was reached on a SHoC-hypotension protocol comprising: A. Core: 1. Cardiac views (Sub-xiphoid and parasternal windows for pericardial fluid, cardiac form and ventricular function); 2. Lung views for pleural fluid and B-lines for filling status; and 3. IVC views for filling status; B. Supplementary: Additional cardiac views; and C. Additional views (when indicated) including peritoneal fluid, aorta, pelvic for IUP, and proximal leg veins for DVT. Conclusion: An international consensus process based on prospectively collected disease incidence has led to a proposed SHoC-hypotension PoCUS protocol comprising a stepwise clinical-indication based approach of Core, Supplementary and Additional PoCUS views.

Download Full-text

Ultrasound

Oxford Handbook of Humanitarian Medicine ◽

10.1093/med/9780199565276.003.0055 ◽

2019 ◽

pp. 1001-1028

Author(s):

Matthew Lyon ◽

W. Ted Kuhn

Keyword(s):

Clinical Management ◽

Point Of Care ◽

Image Acquisition ◽

Image Interpretation ◽

Ultrasound Image ◽

Point Of Care Ultrasound ◽

Clinical Integration ◽

Humanitarian Emergency ◽

Emergency Situations ◽

Essential Components

This chapter outlines how clinical management decisions in humanitarian emergency situations can be greatly enhanced by the utilization of point-of-care ultrasound. The chapter recognizes the advancing technology of point-of care ultrasound and its use in humanitarian missions, and provides guidance on the essential components of clinical ultrasound (image acquisition, image interpretation, and clinical integration).

Download Full-text

Consensus-Based Expert Development of Critical Items for Direct Observation of Point-of-Care Ultrasound Skills

Journal of Graduate Medical Education ◽

10.4300/jgme-d-19-00531.1 ◽

2020 ◽

Vol 12 (2) ◽

pp. 176-184 ◽

Cited By ~ 1

Author(s):

Irene W. Y. Ma ◽

Janeve Desy ◽

Michael Y. Woo ◽

Andrew W. Kirkpatrick ◽

Vicki E. Noble

Keyword(s):

Assessment Tool ◽

Point Of Care ◽

Point Of Care Ultrasound ◽

Cut Scores ◽

Medical Specialties ◽

Modified Delphi ◽

Cut Score ◽

Modified Delphi Technique ◽

Workplace Based Assessment ◽

Competency Based

ABSTRACT Background Point-of-care ultrasound (POCUS) is increasingly used in a number of medical specialties. To support competency-based POCUS education, workplace-based assessments are essential. Objective We developed a consensus-based assessment tool for POCUS skills and determined which items are critical for competence. We then performed standards setting to set cut scores for the tool. Methods Using a modified Delphi technique, 25 experts voted on 32 items over 3 rounds between August and December 2016. Consensus was defined as agreement by at least 80% of the experts. Twelve experts then performed 3 rounds of a standards setting procedure in March 2017 to establish cut scores. Results Experts reached consensus for 31 items to include in the tool. Experts reached consensus that 16 of those items were critically important. A final cut score for the tool was established at 65.2% (SD 17.0%). Cut scores for critical items are significantly higher than those for noncritical items (76.5% ± SD 12.4% versus 53.1% ± SD 12.2%, P < .0001). Conclusions We reached consensus on a 31-item workplace-based assessment tool for identifying competence in POCUS. Of those items, 16 were considered critically important. Their importance is further supported by higher cut scores compared with noncritical items.

Download Full-text

Utilization of a Modified Delphi Method to Perform a Needs Assessment and Curriculum Revision of a Senior-Level Beef Systems Management Course

10.31274/ans_air-180814-1139 ◽

2014 ◽

Author(s):

Erika L. Lundy ◽

Daniel D. Loy ◽

Patrick J. Gunn

Keyword(s):

Needs Assessment ◽

Delphi Method ◽

Systems Management ◽

Curriculum Revision ◽

Modified Delphi Method ◽

Modified Delphi

Download Full-text

Multi-organ point-of-care ultrasound for COVID-19 (PoCUS4COVID): international expert consensus

Critical Care ◽

10.1186/s13054-020-03369-5 ◽

2020 ◽

Vol 24 (1) ◽

Author(s):

Arif Hussain ◽

Gabriele Via ◽

Lawrence Melniker ◽

Alberto Goffi ◽

Guido Tavazzi ◽

...

Keyword(s):

Lung Ultrasound ◽

Expert Panel ◽

Point Of Care ◽

Expert Consensus ◽

Future Research ◽

Point Of Care Ultrasound ◽

The Past ◽

Modified Delphi ◽

Grade Methodology

AbstractCOVID-19 has caused great devastation in the past year. Multi-organ point-of-care ultrasound (PoCUS) including lung ultrasound (LUS) and focused cardiac ultrasound (FoCUS) as a clinical adjunct has played a significant role in triaging, diagnosis and medical management of COVID-19 patients. The expert panel from 27 countries and 6 continents with considerable experience of direct application of PoCUS on COVID-19 patients presents evidence-based consensus using GRADE methodology for the quality of evidence and an expedited, modified-Delphi process for the strength of expert consensus. The use of ultrasound is suggested in many clinical situations related to respiratory, cardiovascular and thromboembolic aspects of COVID-19, comparing well with other imaging modalities. The limitations due to insufficient data are highlighted as opportunities for future research.

Download Full-text

Identification of public health priorities, barriers, and solutions for Kuwait using the modified Delphi method for stakeholder consensus

The International Journal of Health Planning and Management ◽

10.1002/hpm.3270 ◽

2021 ◽

Author(s):

Janvier Gasana ◽

Harri Vainio ◽

Joseph Longenecker ◽

Tom Loney ◽

Balázs Ádám ◽

...

Keyword(s):

Public Health ◽

Delphi Method ◽

Health Priorities ◽

Modified Delphi Method ◽

Modified Delphi ◽

Stakeholder Consensus

Download Full-text

Derivation of a procedural performance checklist for ultrasound-guided femoral arterial line placement using the modified Delphi method

The Journal of Vascular Access ◽

10.1177/1129729820904872 ◽

2020 ◽

Vol 21 (5) ◽

pp. 715-722

Author(s):

Jessica Baez ◽

Elizabeth Powell ◽

Megan Leo ◽

Uwe Stolz ◽

Lori Stolz

Keyword(s):

Delphi Method ◽

Competency Assessment ◽

Arterial Line ◽

Ultrasound Guided ◽

Arterial Access ◽

Modified Delphi Method ◽

Modified Delphi ◽

Item Checklist ◽

Standard Quality ◽

Femoral Arterial Access

Background: Many specialties utilize procedural performance checklists as an aid to teach residents and other learners. Procedural checklists ensure that the critical steps of the desired procedure are performed in a specified manner every time. Valid measures of competency are needed to evaluate learners and ensure a standard quality of care. The objective of this study was to employ the modified Delphi method to derive a procedural checklist for use during placement of ultrasound-guided femoral arterial access. Methods: A 27-item procedural checklist was provided to 14 experts from three acute care specialties. Using the modified Delphi method, the checklist was serially modified based on expert feedback. Results: Three rounds of the study were performed resulting in a final 23-item checklist. Each item on the checklist received at least 70% expert agreement on its inclusion in the final checklist. Conclusion: A procedural performance checklist was created for ultrasound-guided femoral arterial access using the modified Delphi method. This is an objective tool to assist procedural training and competency assessment in a variety of clinical and educational settings.

Download Full-text