Abstract
Background
The purpose of study was to develop an objective, valid, and reliable assessment method for Kampo medicine using an OSCE for the evaluation of clinical competence in Kampo medicine
Methods
We developed a blueprint followed by a list of 47 assessment items and three task scenarios related to clinical competence in Kampo medicine. An eight-member test committee checked the relevance and pass/fail criteria of the assessment items on a Likert scale. We calculated a content validity index and content validity ratio, and used the Angoff method to set the passing threshold. We trained a total of nine simulated patients with three assigned to each scenario. We conducted an OSCE for 11 candidates with varying medical abilities, and conducted three stations per person, and one evaluator in one room by direct observation evaluated. We used video recordings to test the inter-rater reliability of the three raters. We used the test results to verify the reliability of the evaluation scale.
Results
The inter-rater reliability (Cronbach’s α) was 0.92–1.00, intra-examinee reliability was 0.59–0.95, and the reliability of each task was 0.86, 0.89, and 0.86 for Scenarios 1, 2, and 3, respectively.
Conclusions
We developed a content-valid new OSCE assessment method for Kampo medicine and obtained high inter-rater and test reliabilities. Our findings suggest that this is one of the most reliable and valid evaluation methods for assessing clinical competence in Kampo medicine.