scholarly journals Automatic analysis of summary statements in virtual patients - a pilot study evaluating a machine learning approach

2020 ◽  
Vol 20 (1) ◽  
Author(s):  
Inga Hege ◽  
Isabel Kiesewetter ◽  
Martin Adler
Keyword(s):  
Language Processing ◽  
Virtual Patient ◽  
Global Rating ◽  
Virtual Patients ◽  
Immediate Feedback ◽  
Assessment Scores ◽  
Reasoning Abilities ◽  
Reliable Assessment ◽  
Machine Learning Approach ◽  
Computer Based

Abstract Background The ability to compose a concise summary statement about a patient is a good indicator for the clinical reasoning abilities of healthcare students. To assess such summary statements manually a rubric based on five categories - use of semantic qualifiers, narrowing, transformation, accuracy, and global rating has been published. Our aim was to explore whether computer-based methods can be applied to automatically assess summary statements composed by learners in virtual patient scenarios based on the available rubric in real-time to serve as a basis for immediate feedback to learners. Methods We randomly selected 125 summary statements in German and English composed by learners in five different virtual patient scenarios. Then we manually rated these statements based on the rubric plus an additional category for the use of the virtual patients’ name. We implemented a natural language processing approach in combination with our own algorithm to automatically assess 125 randomly selected summary statements and compared the results of the manual and automatic rating in each category. Results We found a moderate agreement of the manual and automatic rating in most of the categories. However, some further analysis and development is needed, especially for a more reliable assessment of the factual accuracy and the identification of patient names in the German statements. Conclusions Despite some areas of improvement we believe that our results justify a careful display of the computer-calculated assessment scores as feedback to the learners. It will be important to emphasize that the rating is an approximation and give learners the possibility to complain about supposedly incorrect assessments, which will also help us to further improve the rating algorithms.

Sorting Out the Virtual Patient

2012 ◽  
Vol 4 (3) ◽  
pp. 1-19 ◽  
Author(s):  
Thomas B. Talbot ◽  
Kenji Sagae ◽  
Bruce John ◽  
Albert A. Rizzo
Keyword(s):  
Virtual Patient ◽  
Standardized Patient ◽  
Virtual Patients ◽  
Conversational Agents ◽  
Patient Encounter ◽  
Computers In Medicine ◽  
Interaction Structure ◽  
Medical Encounter ◽  
Optimal Technology ◽  
Computer Based

Since Dr. Howard Barrows (1964) introduced the human standardized patient in 1963, there have been attempts to game a computer-based simulacrum of a patient encounter; the first being a heart attack simulation using the online PLATO system (Bitzer, 1966). With the now ubiquitous use of computers in medicine, interest and effort have expended in the area of Virtual Patients (VPs). One problem in trying to understand VPs is that there are several quite distinct educational approaches that are all called a ‘virtual patient.’ This article is not a general review of virtual patients as current reviews of excellent quality exist (Poulton & Balasubramaniam, 2011; Cook & Triola, 2009). Also, research that demonstrates the efficacy of virtual patients is ample (Triola, et al., 2006). This article assesses the different kinds of things the authors call “virtual patients”, which are often mutually exclusive approaches, then analyzes their interaction structure or ‘game-play’, and considers the best use scenarios for that design strategy. This article also explores dialogue-based conversational agents as virtual patients and the technology approaches to creating them. Finally, the authors offer a theoretical approach that synthesizes several educational approaches over the course of a medical encounter and recommend the optimal technology for the type of encounter desired.

scholarly journals Putting the ghost in the machine: exploring human-machine hybrid virtual patient systems for health professional education

2016 ◽  
Author(s):  
David Topps ◽  
Michelle L Cullen ◽  
Nishan Sharma ◽  
Rachel H Ellaway
Keyword(s):  
Health Professional ◽  
Language Processing ◽  
Professional Education ◽  
Virtual Patient ◽  
Interactive Methods ◽  
Research Approach ◽  
Virtual Patients ◽  
Text Input ◽  
Project Description ◽  
Study Results

Background: Virtual patient authoring tools provide a simple means of creating rich and complex online cases for health professional students to explore. However, the responses available to the learner are usually predefined, which limits the utility of virtual patients, both in terms of replayability and adaptability. Using artificial intelligence or natural language processing is expensive and hard to design. This project description lays out an alternative approach to making virtual patients more adaptable and interactive. Methods: Using OpenLabyrinth, an open-source educational research platform, we modified the interface and functionality to provide a human-computer hybrid interface, where a human facilitator can interact with learners from within the online case scenario. Using a design-based research approach, we have iteratively improved our case designs, workflows and scripts and interface designs. The next step is to robustly test this new functionality in action. This report describes the piloting and background as well as the rationale, objectives, software development implications, learning designs, and educational intervention designs for the planned study. Results: The costs and time required to modify the software were much lower than anticipated. Facilitators have been able to handle text input from multiple concurrent learners. Learners were not discouraged waiting for the facilitator to respond. Discussion: The implementation and use of this new technique seems very promising and there are a great many ways in which it might be used for training and assessment purposes. This report also explores the provisional implications arising from the study so far.

scholarly journals Putting the ghost in the machine: exploring human-machine hybrid virtual patient systems for health professional education

2016 ◽  
Author(s):  
David Topps ◽  
Michelle L Cullen ◽  
Nishan Sharma ◽  
Rachel H Ellaway
Keyword(s):  
Health Professional ◽  
Language Processing ◽  
Professional Education ◽  
Virtual Patient ◽  
Interactive Methods ◽  
Research Approach ◽  
Virtual Patients ◽  
Text Input ◽  
Project Description ◽  
Study Results

Background: Virtual patient authoring tools provide a simple means of creating rich and complex online cases for health professional students to explore. However, the responses available to the learner are usually predefined, which limits the utility of virtual patients, both in terms of replayability and adaptability. Using artificial intelligence or natural language processing is expensive and hard to design. This project description lays out an alternative approach to making virtual patients more adaptable and interactive. Methods: Using OpenLabyrinth, an open-source educational research platform, we modified the interface and functionality to provide a human-computer hybrid interface, where a human facilitator can interact with learners from within the online case scenario. Using a design-based research approach, we have iteratively improved our case designs, workflows and scripts and interface designs. The next step is to robustly test this new functionality in action. This report describes the piloting and background as well as the rationale, objectives, software development implications, learning designs, and educational intervention designs for the planned study. Results: The costs and time required to modify the software were much lower than anticipated. Facilitators have been able to handle text input from multiple concurrent learners. Learners were not discouraged waiting for the facilitator to respond. Discussion: The implementation and use of this new technique seems very promising and there are a great many ways in which it might be used for training and assessment purposes. This report also explores the provisional implications arising from the study so far.

Mol2vec: Unsupervised Machine Learning Approach with Chemical Intuition

2017 ◽  
Author(s):  
Sabrina Jaeger ◽  
Simone Fulle ◽  
Samo Turk
Keyword(s):  
Machine Learning ◽  
Language Processing ◽  
Supervised Machine Learning ◽  
Learning Approach ◽  
Learning Approaches ◽  
Unsupervised Machine Learning ◽  
Feature Representations ◽  
Machine Learning Approach ◽  
The Individual ◽  
Vector Representations

Inspired by natural language processing techniques we here introduce Mol2vec which is an unsupervised machine learning approach to learn vector representations of molecular substructures. Similarly, to the Word2vec models where vectors of closely related words are in close proximity in the vector space, Mol2vec learns vector representations of molecular substructures that are pointing in similar directions for chemically related substructures. Compounds can finally be encoded as vectors by summing up vectors of the individual substructures and, for instance, feed into supervised machine learning approaches to predict compound properties. The underlying substructure vector embeddings are obtained by training an unsupervised machine learning approach on a so-called corpus of compounds that consists of all available chemical matter. The resulting Mol2vec model is pre-trained once, yields dense vector representations and overcomes drawbacks of common compound feature representations such as sparseness and bit collisions. The prediction capabilities are demonstrated on several compound property and bioactivity data sets and compared with results obtained for Morgan fingerprints as reference compound representation. Mol2vec can be easily combined with ProtVec, which employs the same Word2vec concept on protein sequences, resulting in a proteochemometric approach that is alignment independent and can be thus also easily used for proteins with low sequence similarities.

scholarly journals Evaluating the Difference between Virtual and Paper-Based Clinical Cases in Family Medicine Undergraduate Education

2018 ◽  
Vol 2018 ◽  
pp. 1-7 ◽  
Author(s):  
Zalika Klemenc-Ketis ◽  
Branka Cagran ◽  
Dejan Dinevski
Keyword(s):  
Family Medicine ◽  
Virtual Patient ◽  
Educational Outcome ◽  
Factual Knowledge ◽  
Virtual Patients ◽  
Multiple Choice Questions ◽  
Significant Difference ◽  
The Difference ◽  
And Control ◽  
Clinical Cases

Introduction. A “virtual patient” is defined as a computer program which simulates real patients’ cases. The aim of this study was to determine whether the inclusion of virtual patients affects the level of factual knowledge of family medicine students at the undergraduate level. Methods. This was a case-controlled prospective study. The students were randomly divided into experimental (EG: N=51) and control (CG: N=48) groups. The students in the EG were asked to practice diagnosis using virtual patients instead of the paper-based clinical cases which were solved by the students in the CG. The main observed variable in the study was knowledge of family medicine, determined by 50 multiple choice questions (MCQs) about knowledge of family medicine. Results. There were no statistically significant differences in the groups’ initial knowledge. At the final assessment of knowledge, there were no statistically significant differences between the groups, but there was a statistically significant difference between their initial and final knowledge. Conclusions. The study showed that adding virtual patient cases to the curriculum, instead of paper clinical cases, did not affect the level of factual knowledge about family medicine. Virtual patients can be used, but a significant educational outcome is not expected.

scholarly journals The affectability of writing assessment scores: a G-theory analysis of rater, task, and scoring method contribution

2021 ◽  
Vol 11 (1) ◽  
Author(s):  
Ali Khodi
Keyword(s):  
Writing Assessment ◽  
Generalizability Theory ◽  
Error Variance ◽  
Total Variance ◽  
Scoring Method ◽  
Writing Assessments ◽  
Assessment Scores ◽  
Peer Rating ◽  
Reliable Assessment ◽  
Writing Scores

AbstractThe present study attempted to to investigate  factors  which affect EFL writing scores through using generalizability theory (G-theory). To this purpose, one hundred and twenty students participated in one independent and one integrated writing tasks. Proceeding, their performances were scored by six raters: one self-rating,  three peers,-rating and two instructors-rating. The main purpose of the sudy was to determine the relative and absolute contributions of different facets such as student, rater, task, method of scoring, and background of education  to the validity of writing assessment scores. The results indicated three major sources of variance: (a) the student by task by method of scoring (nested in background of education) interaction (STM:B) with 31.8% contribution to the total variance, (b) the student by rater by task by method of scoring (nested in background of education) interaction (SRTM:B) with 26.5% of contribution to the total variance, and (c) the student by rater by method of scoring (nested in background of education) interaction (SRM:B) with 17.6% of the contribution. With regard to the G-coefficients in G-study (relative G-coefficient ≥ 0.86), it was also found that the result of the assessment was highly valid and reliable. The sources of error variance were detected as the student by rater (nested in background of education) (SR:B) and rater by background of education with 99.2% and 0.8% contribution to the error variance, respectively. Additionally, ten separate G-studies were conducted to investigate the contribution of different facets across rater, task, and methods of scoring as differentiation facet. These studies suggested that peer rating, analytical scoring method, and integrated writing tasks were the most reliable and generalizable designs of the writing assessments. Finally, five decision-making studies (D-studies) in optimization level were conducted and it was indicated that at least four raters (with G-coefficient = 0.80) are necessary for a valid and reliable assessment. Based on these results, to achieve the greatest gain in generalizability, teachers should have their students take two writing assessments and their performance should be rated on at least two scoring methods by at least four raters.

Learner Acceptance of Using Virtual Patient Encounters to Train Foreign Healthcare Professionals in Swedish

2017 ◽  
pp. 379-393
Author(s):  
Uno G. H. Fors ◽  
Olivier Courteille
Keyword(s):  
Learning Environments ◽  
Communication Skills ◽  
Healthcare Professionals ◽  
Virtual Patient ◽  
Patient Communication ◽  
Virtual Patients ◽  
Educational Value ◽  
Good Communication ◽  
Professional Needs ◽  
Almost All

Healthcare professionals need good communication skills to be able to communicate with patients. In such provider-patient communication, the professional needs to be well understood by the patient, but also be able to understand subtle parts of a medical history taking dialogue with worried, sick or mentally affected patients. Virtual Patients (VPs) – learning environments that simulate encounters between a patient and a physician – were used to prepare 26 immigrating professionals in Swedish for healthcare practitioners. The professionals were speaking nine different foreign languages and used two different VP systems to train patient communication. Almost all participants welcomed the use of VPs for training communication in healthcare Swedish and 19 of the 26 users indicated that they considered that VPs should be mandatory to use in future courses. Targeted individual training in provider-patient communication with Virtual Patients seems to be of great educational value and well accepted by immigrating healthcare professionals.

Developing the Assessment Questions Automatically to Determine the Cognitive Level of the E-Learner Using NLP Techniques

2020 ◽  
Vol 11 (2) ◽  
pp. 95-110
Author(s):  
Deena G. ◽  
Raja K. ◽  
Nizar Banu P.K. ◽  
Kannan K.
Keyword(s):  
Language Processing ◽  
Vital Role ◽  
Multiple Choice Questions ◽  
Digital World ◽  
Online Mode ◽  
E Learning ◽  
Computer Based ◽  
Expertise Level ◽  
Teaching Learning ◽  
Assessment Questions

The key objective of the teaching-learning process (TLP) is to impart the knowledge to the learner. In the digital world, the computer-based system emphasis teaching through online mode known as e-learning. The expertise level of the learner in learned subjects can be measured through e-assessment in which multiple choice questions (MCQ) is considered to be an effective one. The assessment questions play the vital role which decides the ability level of a learner. In manual preparation, covering all the topics is difficult and time consumable. Hence, this article proposes a system which automatically generates two different types of question helps to identify the skill level of a learner. First, the MCQ questions with the distractor set are created using named entity recognizer (NER). Further, based on blooms taxonomy the Subjective questions are generated using natural language processing (NLP). The objective of the proposed system is to generate the questions dynamically which helps to reduce the occupation of memory concept.

Parameterizing mental model ascription across intelligent agents

2014 ◽  
Vol 15 (3) ◽  
pp. 404-425 ◽  
Author(s):  
Marjorie McShane
Keyword(s):  
Social Interactions ◽  
Intelligent Agents ◽  
Mental Model ◽  
Mental States ◽  
Virtual Patient ◽  
Prototype System ◽  
Virtual Patients ◽  
Artificial Intelligent ◽  
Value Sets ◽  
Real Patients

Mental model ascription – also called mindreading – is the process of inferring the mental states of others, which happens as a matter of course in social interactions. But although ubiquitous, mindreading is presumably a highly variable process: people mindread to different extents and with different results. We hypothesize that human mindreading ability relies on a large number of personal and contextual features: the inherent abilities of specific individuals, their current physical and mental states, their knowledge of the domain of discourse, their familiarity with the interlocutor, the risks associated with an incorrect assessment of intent, and so on. This paper presents a theory of mindreading that models diverse artificial intelligent agents using an inventory of parameters and value sets that represent traits of humans and features of discourse contexts. Examples are drawn from Maryland Virtual Patient, a prototype system that will permit medical trainees to diagnose and treat cognitively modeled virtual patients with the optional assistance of a virtual tutor. Since real patients vary greatly with respect to physiological and cognitive features, so must a society of virtual patients. Modeling such variation is one of the goals of the overall OntoAgent program of research and development.

Sign in / Sign up

Export Citation Format

Share Document