Assessing physical examination skills using direct observation and volunteer patients

Diagnosis ◽  
2020 ◽  
Vol 0 (0) ◽  
Author(s):  
Bennett W. Clark ◽  
Yi Zhen Joan Lee ◽  
Timothy Niessen ◽  
Sanjay V. Desai ◽  
Brian T. Garibaldi
Keyword(s):  
Physical Examination ◽  
Direct Observation ◽  
Clinical Skill ◽  
Management Plan ◽  
Formative Assessments ◽  
Rater Reliability ◽  
Online Test ◽  
Ongoing Training ◽  
Clinician Educators ◽  
Graduate Trainees

AbstractBackgroundFeedback based on direct observation of the physical examination (PE) is associated with enhanced educational outcomes, yet attending physicians do not frequently observe graduate trainees performing the PE.MethodsWe recruited volunteer patients (VPs), each with an abnormality of the cardiovascular, respiratory, or neurological system. Interns examined each VP, then presented a differential diagnosis and management plan to two clinician educators, who, themselves, had independently examined the VPs. The clinician educators assessed interns along five domains and provided post-examination feedback and teaching. We collected data on intern performance, faculty inter-rater reliability, correlation with a simulation-based measure of clinical skill, and resident and VP perceptions of the assessment.ResultsA total of 72 PGY-1 interns from a large academic training program participated. Performance on the cardiovascular and respiratory system was superior to performance on the neurologic exam. There was no correlation between results of an online test and directly observed cardiovascular skill. Interns preferred feedback from the direct observation sessions. VPs and faculty also rated the experience highly. Inter-rater reliability was good for the respiratory exam, but poor for the cardiovascular and neurologic exams.ConclusionsDirect observation of trainees provides evidence about PE skill that cannot be obtained via simulation. Clinician educators’ ability to provide reliable PE assessment may depend on the portion of the PE being assessed. Our experience highlights the need for ongoing training of clinician educators in direct observation, standard setting, and assessment protocols. This assessment can inform summative or formative assessments of physical exam skill in graduate medical education.

scholarly journals P071 Survey of rheumatology patient satisfaction with telephone consultations during the COVID-19 pandemic

Rheumatology ◽  
2021 ◽  
Vol 60 (Supplement_1) ◽  
Author(s):  
Mia Rodziewicz ◽  
Terence O'Neill ◽  
Audrey Low
Keyword(s):  
Patient Satisfaction ◽  
Physical Examination ◽  
Management Plan ◽  
Service Development ◽  
Telephone Consultation ◽  
Face To Face ◽  
Communication Difficulties ◽  
Strongly Agree ◽  
The Uk

Abstract Background/Aims  Rheumatology departments were required to switch rapidly from face-to-face (F2F) to remote consultations during the COVID-19 pandemic in the UK. We conducted a patient satisfaction survey on the switch to inform future service development. Methods  All patients [new (NP), follow-up (FU)] were identified between 1st to 5th June 2020. Patients who attended or did not attend (DNA) a pre-booked F2F consultation or cancelled were excluded. Of the remainder, half the patients was surveyed by phone using a standardised questionnaire and the other half was posted the same questionnaire. Both groups were offered the opportunity to complete the survey online. Patients were surveyed on the organisation and content of the consultation, whether they were offered a subsequent F2F appointment and future consultation preference. Results  233 consultations were scheduled during the study period. After 53 exclusions (34 pre-booked F2F, 16 DNA, 3 cancellations), 180 eligible consultations were surveyed (85 via mailshot, 95 by telephone). 75/180 patients (42%) responded within 1 month of the telephone consultation (20 NP, 47 FU, 8 missing). The organisation of the switch was positively perceived (Table). Patients were highly satisfied with 4 of the 5 consultation domains but were undecided whether a physical assessment would have changed the outcome of the consultation (Table). After the initial phone consultation, 7 of 20 NP and 19 of 47 FU were not offered subsequent F2F appointments at the clinicians’ discretion. Of those not offered subsequent F2F appointments, proportionally more NP (3/7, 43%) would have liked one, compared to FU (5/19, 26%). Reasons included communication difficulties and a desire for a definitive diagnosis. 48/75 (64%) would be happy for future routine FU to be conducted by phone “most of the time" or "always”; citing patient convenience and disease stability. Caveats were if physical examination was required or if more serious issues (as perceived by the patient) needed F2F discussion. Conclusion  Patients were generally satisfied with telephone consultations and most were happy to be reviewed again this way. NPs should be offered F2F appointments for first visits to maximise patient satisfaction and time efficiency. P071 Table 1:Median age, yearsFemale; n (%)Follow-up; n (%)All eligible for survey; n = 18056122 (68)133 (74)Sent mailshot; n = 855459 (69)65 (76)Surveyed by phone; n = 955663 (66)68 (72)Responder by mail; n = 166911 (69)13 (82)Responder by phone; n = 525437 (71)34 (65)Responder by e-survey; n = 7495 (71)UnknownOrganisation of the telephone consultation, N = 75Yes, n (%)No, n (%)Missing, n (%)Were you informed beforehand about the phone consultation?63 (84)11 (15)1 (1)Were you called within 1-2 hours of the appointed date and time?66 (88)6 (8)3 (4)Domains of the consultation, N = 75Strongly disagree, n (%)Disagree, n (%)Neutral, n (%)Agree, n (%)Strongly agree, n (%)Missing, n (%)During the call, I felt the clinician understood my problem3 (4)1 (1)1 (1)20 (27)49 (65)1 (1)During the call, I had the opportunity to ask questions regarding my clinical care1 (1)02 (3)16 (21)55 (73)1 (1)A physical examination would have changed the outcome of the consultation16 (21)18 (24)20 (27)11 (15)10 (13)0The clinician answered my questions to my satisfaction2 (3)06 (8)18 (24)49 (65)0At the end of the consultation, the clinician agreed a management plan with me3 (4)2 (3)6 (8)24 (32)39 (52)1 (1)Future consultations, N = 75Never, n (%)Sometimes, n (%)Most of the time, n (%)Always, n (%)Missing, n, (%)In the future, would you be happy for routine FU to be conducted by phone?5 (7)20 (27)16 (21)32 (43)2 (3) Disclosure  M. Rodziewicz: None. T. O'Neill: None. A. Low: None.

scholarly journals SP2.1.9Augmented terminal feedback influences cognitive ability of surgical trainees: a randomised controlled trial

2021 ◽  
Vol 108 (Supplement_7) ◽  
Author(s):  
Mohammad Riaz ◽  
Rachel Phing HO ◽  
Benjie Tang ◽  
Afshin Alijani
Keyword(s):  
Error Detection ◽  
Controlled Trial ◽  
Assessment Method ◽  
Formative Assessments ◽  
Current Standard ◽  
Cognitive Approach ◽  
Rater Reliability ◽  
Video Recordings ◽  
Surgical Trainees ◽  
Randomised Controlled

Abstract Introduction Formative assessments in the form of Global assessment (GAS) and procedural based assessment (PBA) are the current methods used for feedback in British laparoscopic surgical training. Video error signature feedback (VESF) has been proposed as an alternative approach to enhance motor skills in laparoscopic training through influencing cognitive approach. Methods Twenty laparoscopic novice students were randomised into Current standard feedback (CSF) and VESF groups. Both groups tied laparoscopic double square knots in four sequential stages. Standard human reliability assessment method (HRA) was utilised to assess unedited video recordings for errors. A validated scoring system by expert trainers assessed proficiency gain. Similar assessment was performed for both groups. Unedited video recordings of the VESF group were annotated for errors at each stage and provided as feedback through video hosting website. CSF group received assessment sheet as their feedback, comparable to current practice. Error numbers, time execution and proficiency gain were the outcomes. Inter-rater reliability among trainers for error detection was established. Results A total of 6490 movements were studied with 1613 errors detected. VESF group committed significantly less errors as compared to the CSF group [1011/1613 (63%) vs 602/1613 (37%), p < 0.01]. VESF group gained proficiency earlier. Time execution was similar. Inter-rater reliability for error detection was high (p = 0.96). Discussion VESF effects cognitive framework of a laparoscopic task in trainee’s mind, ultimately reducing errors. This work demonstrated the practical application of video error signature feedback by demonstrating a simple laparoscopic task and analysing its learning process through novice brains.

scholarly journals Self-Regulated Learning Microanalysis For The Study Of The Performance Of Clinical Examinations By Physiotherapy Students

2020 ◽  
Author(s):  
Raquel Medina ◽  
Daniel David Álamo-Arce ◽  
Felipe Rodríguez de Castro ◽  
Dario Cecilio Fernandez ◽  
John Sandars ◽  
...  
Keyword(s):  
Pilot Study ◽  
Strategic Planning ◽  
Student Performance ◽  
Clinical Skill ◽  
Potential Usefulness ◽  
Rater Reliability ◽  
Self Monitoring ◽  
Self Regulated Learning ◽  
Regulated Learning ◽  
Physiotherapy Students

Abstract Background Students require feedback on their self-regulated learning (SRL) processes to improve the performance of clinical examinations. The key SRL processes used by students can be identified by SRL-micro-analysis but, this method has not been previously applied to physiotherapy students. The aim of this pilot study was to test a research design that might allow the evaluation of the potential usefulness of SRL microanalysis for the identification of key SRL processes used by physiotherapy students during the performance of a clinical examination skill.. The objectives of the pilot study were: 1) to evaluate whether SRL-microanalysis could identify differences in the use of SRL processes between successful and unsuccessful students; 2) to evaluate the reliability of SRL microanalysis ratings produced by different assessors.Methods SRL-microanalysis was used with second year physiotherapy students of a Spanish university (n= 26) as they performed a goniometric task. The task required students to obtain a goniometric measurement of the shoulder joint of a peer. Two assessors evaluated student performance and conducted the SRL- microanalysis with all students. An analysis of inter-rater reliability was performed to evaluate the degree of agreement between assessors. Results The SRL-microanalysis revealed differences in the use of key SRL processes between successful (n= 15: 57.0%) and unsuccessful performers (n= 11: 43.0%): The differences were particularly evident in strategic planning and self-monitoring skills. There was good inter-rater reliability for scoring of strategic planning (k=0.792), self-monitoring (k=0.946) and self-evaluation (k=0.846).Conclusion The use of SRL microanalysis characterized the key SRL processes of physiotherapy students performing a clinical skill with reliability between the assessors. This pilot study supports the potential usefulness of SRL-microanalysis for the identification of key SRL processes in physiotherapy education. Therefore, this study paves the way to the development of a full study, with a larger number of students and more diverse clinical tasks, to evaluate the SRL processes in successful and unsuccessful students.

scholarly journals Experience-Based Guidance for Implementing a Direct Observation Checklist in a Pediatric Emergency Department Setting

2012 ◽  
Vol 4 (4) ◽  
pp. 521-524 ◽  
Author(s):  
Michael FitzGerald ◽  
Mia Mallory ◽  
Matthew Mittiga ◽  
Charles Schubert ◽  
Hamilton Schwartz ◽  
...  
Keyword(s):  
Emergency Department ◽  
Physical Examination ◽  
Direct Observation ◽  
Task Force ◽  
Educational Innovation ◽  
Pediatric Emergency ◽  
Pediatric Emergency Department ◽  
Implementation Challenges ◽  
Observation Methods ◽  
Pilot Implementation

Abstract Background The importance and benefits of direct observation in residency training have been underscored by a number of studies. Yet, implementing direct observation in an effective and sustainable way is hampered by demands on physicians' time and shrinking resources for educational innovation. Objective To describe the development and pilot implementation of a direct observation tool to assess the history and physical examination skills of interns in a pediatric emergency department rotation. Methods A task force developed specific history and physical examination checklists for a range of common conditions. For the pilot implementation, 10 pediatric emergency medicine faculty attendings conducted the initial observations of 34 interns during the course of 1 academic year. At the conclusion of the pilot, the faculty observers and interns were interviewed to assess the feasibility and benefits of the process. Results A total of 33 of the 34 interns were observed during their rotation, with 26 of the observations conducted when the faculty observer was off shift, and it took approximately 20 minutes to complete each observation. In terms of learning benefits, interns and faculty observers reported that it facilitated clear and useful feedback and revealed gaps that would not have otherwise been identified. Faculty observers also mentioned that it helped them focus their teaching effort, built empathy with learners, and gave them a way to demonstrate a true concern for their learning. Conclusion Our results offer evidence for the feasibility and benefits of the direct observation checklists. The description of the implementation, challenges, and response to those challenges may help others avoid some of the common problems faced when implementing direct observation methods.

Feedback and performance scores for direct observation of procedural skills

2014 ◽  
Vol 96 (7) ◽  
pp. e5-e8 ◽  
Author(s):  
H Burnand ◽  
T Fysh ◽  
J Wheeler ◽  
W Allum
Keyword(s):  
Direct Observation ◽  
Medical Training ◽  
Formative Assessments ◽  
Annual Review ◽  
Procedural Skills ◽  
Postgraduate Medical Training ◽  
Surgical Trainees ◽  
And Performance

Direct observation of procedural skills (DOPS) is one of the four main workplace-based assessments (WBAs) in postgraduate medical training. Surgical trainees of all grades and specialties use the Intercollegiate Surgical Curriculum Programme (ISCP) to make an online record of each DOPS. The DOPS are used alongside other WBAs as formative assessments to aid the evaluation of trainee progression at the annual review of competence progression (ARCP).

scholarly journals SELF-REGULATED LEARNING MICROANALYSIS FOR THE STUDY OF THE PERFORMANCE OF CLINICAL EXAMINATIONS BY PHYSIOTHERAPY STUDENTS

2020 ◽  
Author(s):  
Raquel Irina Medina-Ramírez ◽  
Daniel David Álamo-Arce ◽  
Felipe Rodríguez-Castro ◽  
Dario Cecilio-Fernandes ◽  
John Sandars ◽  
...  
Keyword(s):  
Pilot Study ◽  
Strategic Planning ◽  
Student Performance ◽  
Clinical Skill ◽  
Potential Usefulness ◽  
Rater Reliability ◽  
Self Monitoring ◽  
Self Regulated Learning ◽  
Regulated Learning ◽  
Physiotherapy Students

Abstract Background Students require feedback on their self-regulated learning (SRL) processes to improve the performance of clinical examinations. The key SRL processes used by students can be identified by SRL-micro-analysis but, this method has not been previously applied to physiotherapy students. The aim of this pilot study was to test a research design that might allow the evaluation of the potential usefulness of SRL microanalysis for the identification of key SRL processes used by physiotherapy students during the performance of a clinical examination skill. The objectives of the pilot study were: 1) to evaluate whether SRL-microanalysis could identify differences in the use of SRL processes between successful and unsuccessful students; 2) to evaluate the reliability of SRL microanalysis ratings produced by different assessors.Methods SRL-microanalysis was used with second year physiotherapy students of a Spanish university (n= 26) as they performed a goniometric task. The task required students to obtain a goniometric measurement of the shoulder joint of a peer. Two assessors evaluated student performance and conducted the SRL- microanalysis with all students. An analysis of inter-rater reliability was performed to evaluate the degree of agreement between assessors. Results The SRL-microanalysis revealed differences in the use of key SRL processes between successful (n= 15: 57.0%) and unsuccessful performers (n= 11: 43.0%): The differences were particularly evident in strategic planning and self-monitoring skills. There was good inter-rater reliability for scoring of strategic planning (k=0.792), self-monitoring (k=0.946) and self-evaluation (k=0.846).Conclusion The use of SRL microanalysis characterized the key SRL processes of physiotherapy students performing a clinical skill with reliability between the assessors. This pilot study supports the potential usefulness of SRL-microanalysis for the identification of key SRL processes in physiotherapy education. Therefore, this study paves the way to the development of a full study, with a larger number of students and more diverse clinical tasks, to evaluate the SRL processes in successful and unsuccessful students.

scholarly journals Development and Pilot Testing of a Novel Tool for Evaluating Practical Skills in Hematopathology Residents in Pakistan

2019 ◽  
Vol 11 (4s) ◽  
pp. 177-180
Author(s):  
Bushra Moiz ◽  
Syeda Kauser Ali ◽  
Anila Rashid ◽  
Muhammad Shariq ◽  
Farheen Karim
Keyword(s):  
Direct Observation ◽  
Interrater Reliability ◽  
Faculty Members ◽  
Formative Assessments ◽  
Learner Satisfaction ◽  
Practical Skills ◽  
Assessment Scores ◽  
Evaluation Exercise ◽  
Laboratory Procedures ◽  
Mean Time

ABSTRACT Background The mini-Clinical Evaluation Exercise (mini-CEX) and direct observation of procedural skills (DOPS) are reliable tools for work-based assessment of medical trainees. Tools of this type do not yet exist for evaluation of practical laboratory skills of pathology residents. Objective We developed and piloted a 9-item instrument for direct observation of laboratory skills (DOLS). Methods We used the DOLS tool with 10 hematopathology residents (PGY-1 to PGY-5) from Aga Khan University. Each resident was evaluated by 3 faculty members in the laboratory during 4 separate encounters using the DOLS instrument. We assessed construct validity, interrater reliability and G coefficient, feasibility of using DOLS, and learner satisfaction. Results A total of 120 encounters were observed with a mean score (±1 SD) of 56.7% (±12.44). Assessment scores moderately correlated with the number of laboratory procedures previously performed by participants (r = 0.658 and 0.641; P = .0001) and with PGY level. Interrater reliability ranged between 0.47 and 0.96. Cohen's d was 1.64. Residents accounted for a large component of estimated variance (73%), suggesting DOLS can differentiate residents' laboratory skills; variance associated with assessors was small (0.01%). Residents reported being satisfied with the tool. Mean time (±1 SD) taken for observing and feedback was 17.89 ± 5.89 minutes. Conclusions The new DOLS instrument could provide reliable scores for observing laboratory skills. Residents were satisfied with the tool, and rating times make the tool feasible for formative assessments.

Sign in / Sign up

Export Citation Format

Share Document