A Multimodal Approach to Investigate the Neural Mechanisms of Real World Social Vision

Eye tracking and other behavioral measurements collected from patient-participants in their hospital rooms afford a unique opportunity to study immersive natural behavior for basic and clinical-translational research, and also requires addressing important logistical, technical, and ethical challenges. Hospital rooms provide the opportunity to richly capture both clinically relevant and ordinary natural behavior. As clinical settings, they add the potential to study the relationship between behavior and physiology by collecting physiological data synchronized to behavioral measures from participants. Combining eye-tracking, other behavioral measures, and physiological measurements enables clinical-translational research into understanding the participants' disorders and clinician-patient interactions, as well as basic research into natural, real world behavior as participants eat, read, converse with friends and family, etc. Here we describe a paradigm in individuals undergoing surgical treatment for epilepsy who spend 1-2 weeks in the hospital with electrodes implanted in their brain to determine the source of their seizures. This provides the unique opportunity to record behavior using eye tracking glasses customized to address clinically-related ergonomic concerns, synchronized direct neural recordings, use computer vision to assist with video annotation, and apply multivariate machine learning analyses to multimodal data encompassing hours of natural behavior. We discuss the acquisition, quality control, annotation, and analysis pipelines to study the neural basis of real world social and affective perception during natural conversations with friends and family in participants with epilepsy. We also discuss clinical, logistical, and ethical and privacy considerations that must be addressed to acquire high quality multimodal data in this setting.

Current Trends and Future Directions of Large Scale Image and Video Annotation: Observations From Four Years of BIIGLE 2.0

Marine imaging has evolved from small, narrowly focussed applications to large-scale applications covering areas of several hundred square kilometers or time series covering observation periods of several months. The analysis and interpretation of the accumulating large volume of digital images or videos will continue to challenge the marine science community to keep this process efficient and effective. It is safe to say that any strategy will rely on some software platform supporting manual image and video annotation, either for a direct manual annotation-based analysis or for collecting training data to deploy a machine learning–based approach for (semi-)automatic annotation. This paper describes how computer-assisted manual full-frame image and video annotation is currently performed in marine science and how it can evolve to keep up with the increasing demand for image and video annotation and the growing volume of imaging data. As an example, observations are presented how the image and video annotation tool BIIGLE 2.0 has been used by an international community of more than one thousand users in the last 4 years. In addition, new features and tools are presented to show how BIIGLE 2.0 has evolved over the same time period: video annotation, support for large images in the gigapixel range, machine learning assisted image annotation, improved mobility and affordability, application instance federation and enhanced label tree collaboration. The observations indicate that, despite novel concepts and tools introduced by BIIGLE 2.0, full-frame image and video annotation is still mostly done in the same way as two decades ago, where single users annotated subsets of image collections or single video frames with limited computational support. We encourage researchers to review their protocols for education and annotation, making use of newer technologies and tools to improve the efficiency and effectivity of image and video annotation in marine science.

The Use of Video Annotation in Education: A Review

Video annotation (VA), a tool which allows commentaries to be synchronized with video content has recently received significant research attention in education. However, the application contexts of these studies are varied and fragmented. A review was therefore undertaken with the objectives to find out the extent to which the use of VA has been explored for different instructional purposes and summarize the potential affordances of VA in supporting student learning. Articles related to the use of VA in education context were searched from 2011 to 2020 (Nov). Of the final 32 eligible studies, it was found that VA tools were used predominantly to develop teaching practices, enhance learners’ conceptual understanding of video content and develop workplace skills as well as clinical practices. Five most dominant educational affordances of VA tools were summarized as follows: (1) facilitating learners’ reflection (2) facilitating feedback process (3) enhancing comprehension of video content (4) promoting students’ learning satisfaction and positive attitude and (5) convenience and ease. With the outstanding weight of research evidence gained on educational affordances offered by VA, it is convincing that advancing the use of VA in education can further expand the learning opportunities in 21st century classrooms. Keywords: Affordances, Education, Feedback, Learners’ reflection, Video annotation

Research on Aerobics Training and Evaluation Method Based on Artificial Intelligence-Aided Modeling

Traditional aerobics training methods have the problems of lack of auxiliary teaching conditions and low-training efficiency. With the in-depth application of artificial intelligence and computer-aided training methods in the field of aerobics teaching and practice, this paper proposes a local space-time preserving Fisher vector (FV) coding method and monocular motion video automatic scoring technology. Firstly, the gradient direction histogram and optical flow histogram are extracted to describe the motion posture and motion characteristics of the human body in motion video. After normalization and data dimensionality reduction based on the principal component analysis, the human motion feature vector with discrimination ability is obtained. Then, the spatiotemporal pyramid method is used to embed spatiotemporal features in FV coding to improve the ability to identify the correctness and coordination of human behavior. Finally, the linear model of different action classifications is established to determine the action score. In the key frame extraction experiment of the aerobics action video, the ST-FMP model improves the recognition accuracy of uncertain human parts in the flexible hybrid joint human model by about 15 percentage points, and the key frame extraction accuracy reaches 81%, which is better than the traditional algorithm. This algorithm is not only sensitive to human motion characteristics and human posture but also suitable for sports video annotation evaluation, which has a certain reference significance for improving the level of aerobics training.

The Stumblemeter: Design and Validation of a System That Detects and Classifies Stumbles during Gait

Stumbling during gait is commonly encountered in patients who suffer from mild to serious walking problems, e.g., after stroke, in osteoarthritis, or amputees using a lower leg prosthesis. Instead of self-reporting, an objective assessment of the number of stumbles in daily life would inform clinicians more accurately and enable the evaluation of treatments that aim to achieve a safer walking pattern. An easy-to-use wearable might fulfill this need. The goal of the present study was to investigate whether a single inertial measurement unit (IMU) placed at the shank and machine learning algorithms could be used to detect and classify stumbling events in a dataset comprising of a wide variety of daily movements. Ten healthy test subjects were deliberately tripped by an unexpected and unseen obstacle while walking on a treadmill. The subjects stumbled a total of 276 times, both using an elevating recovery strategy and a lowering recovery strategy. Subjects also performed multiple Activities of Daily Living. During data processing, an event-defined window segmentation technique was used to trace high peaks in acceleration that could potentially be stumbles. In the reduced dataset, time windows were labelled with the aid of video annotation. Subsequently, discriminative features were extracted and fed to train seven different types of machine learning algorithms. Trained machine learning algorithms were validated using leave-one-subject-out cross-validation. Support Vector Machine (SVM) algorithms were most successful, and could detect and classify stumbles with 100% sensitivity, 100% specificity, and 96.7% accuracy in the independent testing dataset. The SVM algorithms were implemented in a user-friendly, freely available, stumble detection app named Stumblemeter. This work shows that stumble detection and classification based on SVM is accurate and ready to apply in clinical practice.

Estudio de los videopóster mediante anotaciones de vídeo como metodología para eventos científicos

Los encuentros científicos en línea motivados por la Covid19 han sido una práctica muy común recientemente. No es nueva esta modalidad, pero hoy por su crecimiento solicitan soluciones tecnológicas estables y experimentadas. En un año se ha avanzado mucho en diferentes funciones en los sistemas de videoconferencias, como también en nuevas fórmulas para presentar los trabajos científicos como son los videopóster, que utilizando tecnologías emergentes de anotaciones de vídeo permiten una mayor interacción y debate en los trabajos científicos en un evento en línea. El estudio analiza la cantidad y la calidad de las anotaciones dentro de las cuales se recogen los debates e interacciones entre los autores y los oyentes de una mesa de comunicaciones con videopóster frente a otra presencial. La evaluación de la experimentación se realiza bajo estudio descriptivo de datos cuantitativos y cualitativos, y técnicas de análisis de contenidos “Q-análisis” de las 437 anotaciones y las 238 intervenciones en los 16 videopóster de un evento en línea. Los resultados muestran el doble de participación en la modalidad de videopóster frente a las Comunicaciones presenciales, al tiempo que se enumera un número relevante de beneficios que ofrecen los videopóster para el desarrollo de eventos en línea y redes profesionales de aprendizaje. Online scientific meetings motivated by Covid19 have been a very common practice recently. This modality is not new, but today, due to its growth, it requires stable and experienced technological solutions. In one year, much progress has been made in different functions in videoconferencing systems, as well as in new formulas for presenting scientific papers, such as videoposters, which, using emerging video annotation technologies, allow greater interaction and discussion of scientific papers in an online event. The study analyzes the quantity and quality generated by discussions between authors and listeners at a videoposter presentation table versus a face-to-face one. The evaluation of the experiment is performed under a descriptive study of quantitative and qualitative data, and content analysis techniques "Q-analysis" of the 450 annotations and 355 interventions in the 16 videoposters of an online event. We found twice the production of participation in the video-poster modality compared to face-to-face communications, while listing a relevant number of benefits offered by videoposters for the development of online events and professional learning networks.

Life beneath the ice: jellyfish and ctenophores from the Ross Sea, Antarctica, with an image-based training set for machine learning

Southern Ocean ecosystems are currently experiencing increased environmental changes and anthropogenic pressures, urging scientists to report on their biodiversity and biogeography. Two major taxonomically diverse and trophically important gelatinous zooplankton groups that have, however, stayed largely understudied until now are the cnidarian jellyfish and ctenophores. This data scarcity is predominantly due to many of these fragile, soft-bodied organisms being easily fragmented and/or destroyed with traditional net sampling methods. Progress in alternative survey methods including, for instance, optics-based methods is slowly starting to overcome these obstacles. As video annotation by human observers is both time-consuming and financially costly, machine-learning techniques should be developed for the analysis of in situ /in aqua image-based datasets. This requires taxonomically accurate training sets for correct species identification and the present paper is the first to provide such data. In this study, we twice conducted three week-long in situ optics-based surveys of jellyfish and ctenophores found under the ice in the McMurdo Sound, Antarctica. Our study constitutes the first optics-based survey of gelatinous zooplankton in the Ross Sea and the first study to use in situ / in aqua observations to describe taxonomic and some trophic and behavioural characteristics of gelatinous zooplankton from the Southern Ocean. Despite the small geographic and temporal scales of our study, we provided new undescribed morphological traits for all observed gelatinous zooplankton species (eight cnidarian and four ctenophore species). Three ctenophores and one leptomedusa likely represent undescribed species. Furthermore, along with the photography and videography, we prepared a Common Objects in Context (COCO) dataset, so that this study is the first to provide a taxonomist-ratified image training set for future machine-learning algorithm development concerning Southern Ocean gelatinous zooplankton species.

Fast Machine Learning Annotation in the Medical Domain: A Semi-Automated Video Annotation Tool for Gastroenterologists

Background: Machine learning, especially deep learning, is becoming more and more relevant in research and development in the medical domain. For all of the supervised deep learning applications, data is the most critical factor in securing successful implementation and sustaining the progress of the machine learning model. Especially gastroenterological data, which often involves endoscopic videos, are cumbersome to annotate. Domain experts are needed to interpret and annotate the videos. To support those domain experts, we generated a framework. With this framework, instead of annotating every frame in the video sequence, experts are just performing key annotations at the beginning and the end of sequences with pathologies, e.g. visible polyps. Subsequently, non-expert annotators supported by machine learning add the missing annotations for the frames in-between. Results: Using this framework we were able to reduce work load of domain experts on average by a factor of 20. This is primarily due to the structure of the framework, which is designed to minimize the workload of the domain expert. Pairing this framework with a state-of-the-art semi-automated pre-annotation model enhances the annotation speed further. Through a study with 10 participants we show that semi-automated annotation using our tool doubles the annotation speed of non-expert annotators compared to a well-known state-of-the-art annotation tool. Conclusion: In summary, we introduce a framework for fast expert annotation for gastroenterologists, which reduces the workload of the domain expert considerably while maintaining a very high annotation quality. The framework incorporates a semi-automated annotation system utilizing trained object detection models. The software and framework are open-source.