scholarly journals Application of Objective Structured Clinical Examinations (OSCE) for The Evaluation of Kampo Medicine Training

2021 ◽  
Author(s):  
Marie Amitani ◽  
Haruka Amitani ◽  
Hajime Suzuki ◽  
Suguru Kawazu ◽  
Kimiko Mizuma ◽  
...  
Keyword(s):  
Clinical Competence ◽  
Content Validity ◽  
Assessment Method ◽  
Kampo Medicine ◽  
Simulated Patients ◽  
Rater Reliability ◽  
Video Recordings ◽  
Reliable Assessment ◽  
Test Committee ◽  
Objective Structured Clinical Examinations

Abstract Background The purpose of study was to develop an objective, valid, and reliable assessment method for Kampo medicine using an OSCE for the evaluation of clinical competence in Kampo medicine Methods We developed a blueprint followed by a list of 47 assessment items and three task scenarios related to clinical competence in Kampo medicine. An eight-member test committee checked the relevance and pass/fail criteria of the assessment items on a Likert scale. We calculated a content validity index and content validity ratio, and used the Angoff method to set the passing threshold. We trained a total of nine simulated patients with three assigned to each scenario. We conducted an OSCE for 11 candidates with varying medical abilities, and conducted three stations per person, and one evaluator in one room by direct observation evaluated. We used video recordings to test the inter-rater reliability of the three raters. We used the test results to verify the reliability of the evaluation scale. Results The inter-rater reliability (Cronbach’s α) was 0.92–1.00, intra-examinee reliability was 0.59–0.95, and the reliability of each task was 0.86, 0.89, and 0.86 for Scenarios 1, 2, and 3, respectively. Conclusions We developed a content-valid new OSCE assessment method for Kampo medicine and obtained high inter-rater and test reliabilities. Our findings suggest that this is one of the most reliable and valid evaluation methods for assessing clinical competence in Kampo medicine.

scholarly journals SP2.1.9Augmented terminal feedback influences cognitive ability of surgical trainees: a randomised controlled trial

2021 ◽  
Vol 108 (Supplement_7) ◽  
Author(s):  
Mohammad Riaz ◽  
Rachel Phing HO ◽  
Benjie Tang ◽  
Afshin Alijani
Keyword(s):  
Error Detection ◽  
Controlled Trial ◽  
Assessment Method ◽  
Formative Assessments ◽  
Current Standard ◽  
Cognitive Approach ◽  
Rater Reliability ◽  
Video Recordings ◽  
Surgical Trainees ◽  
Randomised Controlled

Abstract Introduction Formative assessments in the form of Global assessment (GAS) and procedural based assessment (PBA) are the current methods used for feedback in British laparoscopic surgical training. Video error signature feedback (VESF) has been proposed as an alternative approach to enhance motor skills in laparoscopic training through influencing cognitive approach. Methods Twenty laparoscopic novice students were randomised into Current standard feedback (CSF) and VESF groups. Both groups tied laparoscopic double square knots in four sequential stages. Standard human reliability assessment method (HRA) was utilised to assess unedited video recordings for errors. A validated scoring system by expert trainers assessed proficiency gain. Similar assessment was performed for both groups. Unedited video recordings of the VESF group were annotated for errors at each stage and provided as feedback through video hosting website. CSF group received assessment sheet as their feedback, comparable to current practice. Error numbers, time execution and proficiency gain were the outcomes. Inter-rater reliability among trainers for error detection was established. Results A total of 6490 movements were studied with 1613 errors detected. VESF group committed significantly less errors as compared to the CSF group [1011/1613 (63%) vs 602/1613 (37%), p < 0.01]. VESF group gained proficiency earlier. Time execution was similar. Inter-rater reliability for error detection was high (p = 0.96). Discussion VESF effects cognitive framework of a laparoscopic task in trainee’s mind, ultimately reducing errors. This work demonstrated the practical application of video error signature feedback by demonstrating a simple laparoscopic task and analysing its learning process through novice brains.

scholarly journals Judging clinical competence using structured observation tools: A cautionary tale

2019 ◽  
Vol 47 (6) ◽  
pp. 736-744 ◽  
Author(s):  
Anthony D. Roth ◽  
Pamela Myles-Hooton ◽  
Amanda Branson
Keyword(s):  
Clinical Competence ◽  
Rating Scales ◽  
Assessment Tool ◽  
Cautionary Tale ◽  
Specific Context ◽  
Rater Reliability ◽  
Video Recordings ◽  
Standard Set ◽  
Observation Tools ◽  
Realistic Appraisal

AbstractBackground:One method for appraising the competence with which psychological therapy is delivered is to use a structured assessment tool that rates audio or video recordings of therapist performance against a standard set of criteria.Aims:The present study examines the inter-rater reliability of a well-established instrument (the Cognitive Therapy Scale – Revised) and a newly developed scale for assessing competence in CBT.Method:Six experienced raters working independently and blind to each other’s ratings rated 25 video recordings of therapy being undertaken by CBT therapists in training.Results:Inter-rater reliability was found to be low on both instruments.Conclusions:It is argued that the results represent a realistic appraisal of the accuracy of rating scales, and that the figures often cited for inter-rater reliability are unlikely to be generalizable outside the specific context in which they were achieved. The findings raise concerns about the use of these scales for making summative judgements of clinical competence in both educational and research contexts.

Inter-rater Reliability of Preceptors on Clinical Pharmacy Competency Evaluation

2020 ◽  
Vol 15 ◽  
Author(s):  
Dixon Thomas ◽  
Sherief Khalifa ◽  
Jayadevan Sreedharan ◽  
Rucha Bond
Keyword(s):  
United Arab Emirates ◽  
Clinical Pharmacy ◽  
Clinical Competence ◽  
Assessment Tool ◽  
Clinical Skills ◽  
Group Discussion ◽  
Pharmacy Students ◽  
Rater Reliability ◽  
Quasi Experimental ◽  
Competency Evaluation

Background:: Clinical competence of pharmacy students is better evaluated at their practice sites. compared to the classroom. A clinical pharmacy competency evaluation rubric like that of the American College of Clinical Pharmacy (ACCP)is an effective assessment tool for clinical skills and can be used to show item reliability. The preceptors should be trained on how to use the rubrics as many inherent factors could influence inter-rater reliability. Objective:: To evaluate inter-rater reliability among preceptors on evaluating clinical competence of pharmacy students, before and after a group discussion intervention. Methods:: In this quasi experimental study in a United Arab Emirates teaching hospital, Seven clinical pharmacy preceptors rated clinical pharmacy competencies of ten recent PharmD graduates referring to their portfolios and preceptorship. Clinical pharmacy competencies were adopted from ACCP and mildly modified to be relevant for the local settings. Results:: Inter-rater reliability (Cronbach's Alpha) among preceptors was reasonable being practitioners at a single site for 2-4 years. At domain level, inter-rater reliability ranged from 0.79 - 0.93 before intervention and 0.94 - 0.99 after intervention. No inter-rater reliability was observed in relation to certain competency elements ranging from 0.31 – 0.61 before intervention, but improved to 0.79 – 0.97 after intervention. Intra-class correlation coefficient improved among all individual preceptors being reliable with each other after group discussion though some had no reliability with each other before group discussion. Conclusion:: Group discussion among preceptors at the training site was found to be effective in improving inter-rater reliability on all elements of the clinical pharmacy competency evaluation. Removing a preceptor from analysis did not affect inter-rater reliability after group discussion.

Reliability of three linguistically and culturally validated pain assessment tools for sedated ICU patients by ICU nurses in Finland

2018 ◽  
Vol 18 (2) ◽  
pp. 165-173 ◽  
Author(s):  
Sanna-Mari Pudas-Tähkä ◽  
Sanna Salanterä
Keyword(s):  
Intensive Care ◽  
Internal Consistency ◽  
Pain Assessment ◽  
Assessment Tools ◽  
Endotracheal Suctioning ◽  
Icu Patients ◽  
Rater Reliability ◽  
Intensive Care Patients ◽  
Video Recordings ◽  
Test Retest Reliability

Abstract Background and aims: Pain assessment in intensive care is challenging, especially when the patients are sedated. Sedated patients who cannot communicate verbally are at risk of suffering from pain that remains unnoticed without careful pain assessment. Some tools have been developed for use with sedated patients. The Behavioral Pain Scale (BPS), the Critical-Care Pain Observation Tool (CPOT) and the Nonverbal Adult Pain Assessment Scale (NVPS) have shown promising psychometric qualities. We translated and culturally adapted these three tools for the Finnish intensive care environment. The objective of this feasibility study was to test the reliability of the three pain assessment tools translated into Finnish for use with sedated intensive care patients. Methods: Six sedated intensive care patients were videorecorded while they underwent two procedures: an endotracheal suctioning was the nociceptive procedure, and the non-nociceptive treatment was creaming of the feet. Eight experts assessed the patients’ pain by observing video recordings. They assessed the pain using four instruments: the BPS, the CPOT and the NVPS, and the Numeric Rating Scale (NRS) served as a control instrument. Each expert assessed the patients’ pain at five measurement points: (1) right before the procedure, (2) during the endotracheal suctioning, (3) during rest (4) during the creaming of the feet, and (5) after 20 min of rest. Internal consistency and inter-rater reliability of the tools were evaluated. After 6 months, the video recordings were evaluated for testing the test-retest reliability. Results: Using the BPS, the CPOT, the NVPS and the NRS, 960 assessments were obtained. Internal consistency with Cronbach’s alpha coefficient varied greatly with all the instruments. The lowest values were seen at those measurement points where the pain scores were 0. The highest scores were achieved after the endotracheal suctioning at rest: for the BPS, the score was 0.86; for the CPOT, 0.96; and for the NVPS, 0.90. The inter-rater reliability using the Shrout-Fleiss intraclass correlation coefficient (ICC) tests showed the best results after the painful procedure and during the creaming. The scores were slightly lower for the BPS compared to the CPOT and the NVPS. The test-retest results using the Bland-Altman plots show that all instruments gave similar results. Conclusions: To our knowledge, this is the first time all three behavioral pain assessment tools have been evaluated in the same study in a language other than English or French. All three tools had good internal consistency, but it was better for the CPOT and the NVPS compared to the BPS. The inter-rater reliability was best for the NVPS. The test-retest reliability was strongest for the CPOT. The three tools proved to be reliable for further testing in clinical use. Implications: There is a need for feasible, valid and reliable pain assessment tools for pain assessment of sedated ICU patients in Finland. This was the first time the psychometric properties of these tools were tested in Finnish use. Based on the results, all three instruments could be tested further in clinical use for sedated ICU patients in Finland.

scholarly journals Diagnostic Accuracy of Three-dimensional Ultrasonography in Detection of Endometrial Lesions compared with Hysteroscopy in Infertile Women

2016 ◽  
Vol 10 (4) ◽  
pp. 393-397
Author(s):  
Firoozeh Ahmadi ◽  
Farnaz Akhbari ◽  
Fatemeh Niknejad ◽  
Hadieh Haghighi ◽  
Zahra Ghahremani ◽  
...  
Keyword(s):  
Diagnostic Accuracy ◽  
Sensitivity And Specificity ◽  
Predictive Value ◽  
Gold Standard ◽  
Three Dimensional ◽  
Assessment Method ◽  
Diagnostic Efficiency ◽  
Infertile Women ◽  
Reliable Assessment ◽  
Uterine Anomalies

ABSTRACT Introduction Two of the most frequent procedures performed on infertile women are two-dimensional ultrasound (2DUS) and three-dimensional ultrasound (3DUS). Hysteroscopy is considered as the gold standard for evaluation of acquired endometrial lesions in infertile women; however, 3DUS is used as a noninvasive, less expensive, and reliable assessment method for evaluation of the intrauterine lesions in infertile women. We aimed to compare the diagnostic efficiency between 3DUS and hysteroscopy in the detection of lesions (polyps, submucous leiomyoma, and synechiae) in infertile women. Materials and methods In this prospective observational study, infertile women (n = 155) with indication of hysteroscopy were scheduled to undergo 3DUS prior to hysteroscopy from September 2010 to 2011. Women with suspected congenital uterine anomalies were excluded. The sensitivity and specificity values of 3DUS were compared with those of hysteroscopy. Hysteroscopy was used as the gold standard for diagnosis of intrauterine lesions in infertile women. Results Of the 155 women, 50 were found to have an intracavitary abnormality, 36 had polyps, 12 had myomas, and 7 had synechiae on hysteroscopic findings. Examination with 3DUS in the diagnosis of intrauterine lesions reached an accuracy of 94%, and 92.15 and 96.9% of sensitivity and specificity respectively. Positive predictive value (PPV) was 83.9%, and a negative predictive value (NPV) was 91.3% (LR+ = 10.75, LR+ = 0.065). Conclusion According to our results, 3DUS has a reliable diagnostic accuracy for intrauterine lesions, and it may limit unnecessary hysteroscopy in patients with normal results. How to cite this article Ahmadi F, Haghighi H, Ghahremani Z, Niknejad F, Akhbari F, Ramezanali F, Chehrazi M. Diagnostic Accuracy of Three-dimensional Ultrasonography in Detection of Endometrial Lesions compared with Hysteroscopy in Infertile Women. Donald School J Ultrasound Obstet Gynecol 2016;10(4):393-397.

Pilot study on the European Portuguese version of the Confusion Assessment Method

2014 ◽  
Vol 26 (5) ◽  
pp. 321-324
Author(s):  
Sónia Martins ◽  
Patrícia Moldes ◽  
João Pinto-de-Sousa ◽  
Filipe Conceição ◽  
José Artur Paiva ◽  
...  
Keyword(s):  
Pilot Study ◽  
Confusion Assessment Method ◽  
Assessment Method ◽  
Translation Process ◽  
Rater Reliability ◽  
Confusion Assessment ◽  
Research Guidelines ◽  
European Portuguese ◽  
Scale Total Score ◽  
Moderate Sensitivity

ObjectiveTo present the pilot study on the European Portuguese validation of the Confusion Assessment Method (CAM).MethodsThe translation process was carried out according to International Society Pharmacoeconomics and Outcomes Research guidelines with trained researchers and inter-rater reliability assessment. The study included 50 elderly patients, admitted (≥24 h) to two intermediate care units. Exclusion criteria were: Glasgow Coma Scale (total score ≤11), blindness/deafness, inability to communicate and not able to speak Portuguese. The sensitivity and specificity of CAM were assessed, with DSM-IV-TR criteria of delirium used as a reference standard.ResultsFindings revealed excellent inter-rater reliability (k>0.81), moderate sensitivity (73%) and excellent specificity (95%).ConclusionThese preliminary results suggested that this version emerges as a promising diagnostic instrument for delirium.

Video footage from drones for Structure-from-Motion photogrammetry – A practical and rapid assessment method for large wood accumulations in rivers?

2021 ◽  
Author(s):  
Gabriel Spreitzer ◽  
Isabella Schalko ◽  
Robert M. Boes ◽  
Volker Weitbrecht
Keyword(s):  
Structure From Motion ◽  
Estimation Error ◽  
Rapid Assessment ◽  
3D Models ◽  
Assessment Method ◽  
Large Wood ◽  
Video Footage ◽  
Rapid Assessment Method ◽  
Complex Dimensions ◽  
Reliable Assessment

<p>Large wood (LW) and logjams are common and important elements in rivers, yet knowledge about composition, volume and porosity of wooden structures in streams is still limited. Most studies apply a rectangular approach (manually measuring a rough bounding-box of the logjam) to estimate LW accumulation volume and porosity. However, this method cannot capture the complex dimensions of LW accumulations and may introduce an additional human-made estimation error. Furthermore, there is a risk of accidents involved when obtaining manual measurements on logjams in the field. Drones represent a powerful tool in geosciences, yet their potential has not been fully exploited to date. The application of non-intrusive quantification methods is widely available in geosciences and recently also increasing for research related to LW in rivers. Recent studies demonstrated that drone imagery and Structure-from-Motion photogrammetry provide true replicates of prototype logjams in form of 3D-models. In the present study we used video footage of a LW accumulation, obtained via standard drone (DJI Phantom 4 Pro+), to evaluate its potential for a rapid assessment of geometric measures (e.g. length, width, height, volume) of the LW accumulation. The gained results from the 4k drone video footage (4,096 x 2,160 pixels) were scaled solely from the obtained video georeferencing data and verified with a properly scaled 3D-accumulation-model that has been generated from high resolution drone imagery (5,472 x 3,648 pixels). We are interested in the level of detail and accuracy, that can be obtained from georeferenced drone footage, and aim to introduce a practical and more reliable assessment method as a state-of-the-art alternative to the traditionally applied rectangular approach. Our study may be of interest for river managers and engineers to rapidly and safely assess LW accumulation volume and porosity in the field.</p>

Sign in / Sign up

Export Citation Format

Share Document