Automatic Classification of Screen Gaze and Dialogue in Doctor-Patient-Computer Interactions: Computational Ethnography Algorithm Development and Validation

Background The study of doctor-patient-computer interactions is a key research area for examining doctor-patient relationships; however, studying these interactions is costly and obtrusive as researchers usually set up complex mechanisms or intrude on consultations to collect, then manually analyze the data. Objective We aimed to facilitate human-computer and human-human interaction research in clinics by providing a computational ethnography tool: an unobtrusive automatic classifier of screen gaze and dialogue combinations in doctor-patient-computer interactions. Methods The classifier’s input is video taken by doctors using their computers' internal camera and microphone. By estimating the key points of the doctor's face and the presence of voice activity, we estimate the type of interaction that is taking place. The classification output of each video segment is 1 of 4 interaction classes: (1) screen gaze and dialogue, wherein the doctor is gazing at the computer screen while conversing with the patient; (2) dialogue, wherein the doctor is gazing away from the computer screen while conversing with the patient; (3) screen gaze, wherein the doctor is gazing at the computer screen without conversing with the patient; and (4) other, wherein no screen gaze or dialogue are detected. We evaluated the classifier using 30 minutes of video provided by 5 doctors simulating consultations in their clinics both in semi- and fully inclusive layouts. Results The classifier achieved an overall accuracy of 0.83, a performance similar to that of a human coder. Similar to the human coder, the classifier was more accurate in fully inclusive layouts than in semi-inclusive layouts. Conclusions The proposed classifier can be used by researchers, care providers, designers, medical educators, and others who are interested in exploring and answering questions related to screen gaze and dialogue in doctor-patient-computer interactions.

Deep Networks for Collaboration Analytics: Promoting Automatic Analysis of Face-to-Face Interaction in the Context of Inquiry-Based Learning

Scholars have applied automatic content analysis to study computer-mediated communication in computer-supported collaborative learning (CSCL). Since CSCL also takes place in face-to-face interactions, we studied the automatic coding accuracy of manually transcribed face-to-face communication. We conducted our study in an authentic higher-education physics context where computer-supported collaborative inquiry-based learning (CSCIL) is a popular pedagogical approach. Since learners’ needs for support in CSCIL vary in the different inquiry phases (orientation, conceptualization, investigation, conclusion, and discussion), we studied, first, how the coding accuracy of five computational models (based on word embeddings and deep neural networks with attention layers) differed in the various inquiry-based learning (IBL) phases when compared to human coding. Second, we investigated how the different features of the best performing computational model improved the coding accuracy. The study indicated that the accuracy of the best performing computational model (differentiated attention with pre-trained static embeddings) was slightly better than that of the human coder (58.9% vs. 54.3%). We also found that considering the previous and following utterances, as well as the relative position of the utterance, improved the model’s accuracy. Our method illustrates how computational models can be trained for specific purposes (e.g., to code IBL phases) with small data sets by using pre-trained models.

Trends in reasons for emergency calls during the COVID-19 crisis in the department of Gironde, France using automatic classification.

Objectives During periods such as the COVID-19 crisis, there is a need for responsive public health surveillance indicators related to the epidemic and to preventative measures such as lockdown. The automatic classification of the content of calls to emergency medical communication centers could provide relevant and responsive indicators. Methods We retrieved all 796,209 free-text call reports from the emergency medical communication center of the Gironde department, France, between 2018 and 2020. We trained a natural language processing neural network model with a mixed unsupervised/supervised method to classify all reasons for calls in 2020. Validation and parameter adjustment were performed using a sample of 20,000 manually-coded free-text reports. Results The number of daily calls for flu-like symptoms began to increase from February 21, 2020 and reached an unprecedented level by February 28, 2020 and peaked on March 14, 2020, 3 days before lockdown. It was strongly correlated with daily emergency room admissions, with a delay of 14 days. Calls for chest pain, stress, but also those mentioning dyspnea, ageusia and anosmia peaked 12 days later. Calls for malaises with loss of consciousness, non-voluntary injuries and alcohol intoxications sharply decreased, starting one month before lockdown. Discussion This example of the COVID-19 crisis shows how the availability of reliable and unbiased surveillance platforms can be useful for a timely and relevant monitoring of all events with public health consequences. The use of an automatic classification system using artificial intelligence makes it possible to free itself from the context that could influence a human coder, especially in a crisis situation. Conclusion The content of calls to emergency medical communication centers is an efficient epidemiological surveillance data source that provides insights into the societal upheavals induced by a health crisis.

Automatic Classification of Screen Gaze and Dialogue in Doctor-Patient-Computer Interactions: Computational Ethnography Algorithm Development and Validation (Preprint)

BACKGROUND The study of doctor-patient-computer interactions is a key research area for examining doctor-patient relationships; however, studying these interactions is costly and obtrusive as researchers usually set up complex mechanisms or intrude on consultations to collect, then manually analyze the data. OBJECTIVE We aimed to facilitate human-computer and human-human interaction research in clinics by providing a computational ethnography tool: an unobtrusive automatic classifier of screen gaze and dialogue combinations in doctor-patient-computer interactions. METHODS The classifier’s input is video taken by doctors using their computers' internal camera and microphone. By estimating the key points of the doctor's face and the presence of voice activity, we estimate the type of interaction that is taking place. The classification output of each video segment is 1 of 4 interaction classes: (1) screen gaze and dialogue, wherein the doctor is gazing at the computer screen while conversing with the patient; (2) dialogue, wherein the doctor is gazing away from the computer screen while conversing with the patient; (3) screen gaze, wherein the doctor is gazing at the computer screen without conversing with the patient; and (4) other, wherein no screen gaze or dialogue are detected. We evaluated the classifier using 30 minutes of video provided by 5 doctors simulating consultations in their clinics both in semi- and fully inclusive layouts. RESULTS The classifier achieved an overall accuracy of 0.83, a performance similar to that of a human coder. Similar to the human coder, the classifier was more accurate in fully inclusive layouts than in semi-inclusive layouts. CONCLUSIONS The proposed classifier can be used by researchers, care providers, designers, medical educators, and others who are interested in exploring and answering questions related to screen gaze and dialogue in doctor-patient-computer interactions.

Lane Formation Beyond Intuition Towards an Automated Characterization of Lanes in Counter-flows

Pedestrian behavioural dynamics have been growingly investigated by means of (semi)automated computing techniques for almost two decades, exploiting advancements on computing power, sensor accuracy and availability, computer vision algorithms. This has led to a unique consensus on the existence of significant difference between unidirectional and bidirectional flows of pedestrians, where the phenomenon of lane formation seems to play a major role. The collective behaviour of lane formation emerges in condition of variable density and due to a self-organisation dynamic, for which pedestrians are induced to walk following preceding persons to avoid and minimize conflictual situations. Although the formation of lanes is a well-known phenomenon in this field of study, there is still a lack of methods offering the possibility to provide an (even semi-) automatic identification and a quantitative characterization. In this context, the paper proposes an unsupervised learning approach for an automatic detection of lanes in multi-directional pedestrian flows, based on the DBSCAN clustering algorithm. The reliability of the approach is evaluated through an inter-rater agreement test between the results achieved by a human coder and by the algorithm.

Social group behaviour of triads. Dependence on purpose and gender

We analysed a set of uninstructed pedestrian trajectories automatically tracked in a public area, and we asked a human coder to assess their group relationships. For those pedestrians who belong to the groups, we asked the coder to identify their apparent purpose of visit to the tracking area and apparent gender. We studied the quantitative dependence of the group dynamics on such properties in the case of triads (three people groups) and compared them to the two pedestrian group case (dyads), studied in a previous work. We found that the group velocity strongly depends on relation and gender for both triads and dyads, while the influence of these properties on spatial structure of groups is less clear in the triadic case. We discussed the relevance of these results to the modelling of pedestrian and crowd dynamics, and examined the possibility of the future works on this subject.

SPeeding Up the Detection of Non-iconic and Iconic Gestures (SPUDNIG): a toolkit for the automatic detection of hand movements and gestures in video data.

In human face-to-face communication, speech is frequently accompanied by visual signals, especially communicative hand gestures. Analyzing these visual signals requires detailed manual annotation of video data, which is often a labor-intensive and time-consuming process. To facilitate this process, we here present SPUDNIG (SPeeding Up the Detection of Non-iconic and Iconic Gestures), a tool to automatize the detection and annotation of hand movements in video data. We provide a detailed description of how SPUDNIG detects hand movement initiation and termination, as well as open-source code and a short tutorial on an easy-to-use graphical user interface (GUI) of our tool. We then provide a proof-of-principle and validation of our method by comparing SPUDNIG’s output to manual annotations of gestures by a human coder. While the tool does not entirely eliminate the need of a human coder (e.g., for false positives detection), our results demonstrate that SPUDNIG can detect both iconic and non-iconic gestures with very high accuracy, and could successfully detect all iconic gestures in our validation dataset. Importantly, SPUDNIG’s output can directly be imported into commonly used annotation tools such as ELAN and ANVIL. We therefore believe that SPUDNIG will be highly relevant for researchers studying multimodal communication due to its annotations significantly accelerating the analysis of large video corpora.

A comparative study of the effectiveness of sentiment tools and human coding in sarcasm detection

Purpose Sarcasm is often used in everyday speech and writing and is prevalent in online contexts. The purpose of this paper is to investigate the analogy between sarcasm comments from sentiment tools and the human coder. Design/methodology/approach Using the Verbal Irony Procedure, eight human coders were engaged to analyse comments collected from an online commercial page, and a dissimilarity analysis was conducted with sentiment tools. Three constants were tested, namely, polarity from sentiment tools, polarity rating by human coders; and sarcasm-level ratings by human coders. Findings Results found an inconsistent ratio between these three constants. Sentiment tools used did not have the capability or reliability to detect the subtle, contextualized meanings of sarcasm statements that human coders could detect. Further research is required to refine the sentiment tools to enhance their sensitivity and capability. Practical implications With these findings, it is recommended that further research and commercialization efforts be directed at improving current sentiment tools – for example, to incorporate sophisticated human sarcasm texts in their analytical systems. Sarcasm exists frequently in media, politics and human forms of communications in society. Therefore, more highly sophisticated sentiment tools with the abilities to detect human sarcasm would be vital in research and industry. Social implications The findings suggest that presently, of the sentiment tools investigated, most are still unable to pick up subtle contexts within the text which can reverse or change the message that the writer intends to send to his/her receiver. Hence, the use of the relevant hashtags (e.g. #sarcasm; #irony) are of fundamental importance in detection tools. This would aid the evaluation of product reviews online for commercial usage. Originality/value The value of this study lies in its original, empirical findings on the inconsistencies between sentiment tools and human coders in sarcasm detection. The current study proves these inconsistencies are detected between human and sentiment tools in social media texts and points to the inadequacies of current sentiment tools. With these findings, it is recommended that further research and commercialization efforts be directed at improving current sentiment tools – to incorporate sophisticated human sarcasm texts in their analytical systems. The system can then be used as a reference for psychologists, media analysts, researchers and speech writers to detect cues in the inconsistencies in behaviour and language.

Answering Mobile Surveys With Images: An Exploration Using a Computer Vision API

Most mobile devices nowadays have a camera. Besides, posting and sharing images have been found as one of the most frequent and engaging Internet activities. However, to our knowledge, no research has explored the feasibility of asking respondents of online surveys to upload images to answer survey questions. The main goal of this article is to investigate the viability of asking respondents of an online opt-in panel to upload during a mobile web survey: First, a photo taken in the moment, and second, an image already saved on their smartphone. In addition, we want to test to what extent the Google Vision application programming interface (API), which can label images into categories, produces similar tags than a human coder. Overall, results from a survey conducted among millennials in Spain and Mexico ( N = 1,614) show that more than half of the respondents uploaded an image. Of those, 77.3% and 83.4%, respectively, complied with what the question asked. Moreover, respectively, 52.4% and 65.0% of the images were similarly codified by the Google Vision API and the human coder. In addition, the API codified 1,818 images in less than 5 min, whereas the human coder spent nearly 35 hours to complete the same task.

A method for modeling co-occurrence propensity of clinical codes with application to ICD-10-PCS auto-coding

Objective Natural language processing methods for medical auto-coding, or automatic generation of medical billing codes from electronic health records, generally assign each code independently of the others. They may thus assign codes for closely related procedures or diagnoses to the same document, even when they do not tend to occur together in practice, simply because the right choice can be difficult to infer from the clinical narrative. Methods We propose a method that injects awareness of the propensities for code co-occurrence into this process. First, a model is trained to estimate the conditional probability that one code is assigned by a human coder, given than another code is known to have been assigned to the same document. Then, at runtime, an iterative algorithm is used to apply this model to the output of an existing statistical auto-coder to modify the confidence scores of the codes. Results We tested this method in combination with a primary auto-coder for International Statistical Classification of Diseases-10 procedure codes, achieving a 12% relative improvement in F -score over the primary auto-coder baseline. The proposed method can be used, with appropriate features, in combination with any auto-coder that generates codes with different levels of confidence. Conclusions The promising results obtained for International Statistical Classification of Diseases-10 procedure codes suggest that the proposed method may have wider applications in auto-coding.