Augmenting Social Science Research with Multimodal Data Collection: The EZ-MMLA Toolkit

While the majority of social scientists still rely on traditional research instruments (e.g., surveys, self-reports, qualitative observations), multimodal sensing is becoming an emerging methodology for capturing human behaviors. Sensing technology has the potential to complement and enrich traditional measures by providing high frequency data on people’s behavior, cognition and affects. However, there is currently no easy-to-use toolkit for recording multimodal data streams. Existing methodologies rely on the use of physical sensors and custom-written code for accessing sensor data. In this paper, we present the EZ-MMLA toolkit. This toolkit was implemented as a website and provides easy access to multimodal data collection algorithms. One can collect a variety of data modalities: data on users’ attention (eye-tracking), physiological states (heart rate), body posture (skeletal data), gestures (from hand motion), emotions (from facial expressions and speech) and lower-level computer vision algorithms (e.g., fiducial/color tracking). This toolkit can run from any browser and does not require dedicated hardware or programming experience. We compare this toolkit with traditional methods and describe a case study where the EZ-MMLA toolkit was used by aspiring educational researchers in a classroom context. We conclude by discussing future work and other applications of this toolkit, potential limitations and implications.

SensorHub: Multimodal Sensing in Real-Life Enables Home-Based Studies

Observational studies are an important tool for determining whether the findings from controlled experiments can be transferred into scenarios that are closer to subjects’ real-life circumstances. A rigorous approach to observational studies involves collecting data from different sensors to comprehensively capture the situation of the subject. However, this leads to technical difficulties especially if the sensors are from different manufacturers, as multiple data collection tools have to run simultaneously. We present SensorHub, a system that can collect data from various wearable devices from different manufacturers, such as inertial measurement units, portable electrocardiographs, portable electroencephalographs, portable photoplethysmographs, and sensors for electrodermal activity. Additionally, our tool offers the possibility to include ecological momentary assessments (EMAs) in studies. Hence, SensorHub enables multimodal sensor data collection under real-world conditions and allows direct user feedback to be collected through questionnaires, enabling studies at home. In a first study with 11 participants, we successfully used SensorHub to record multiple signals with different devices and collected additional information with the help of EMAs. In addition, we evaluated SensorHub’s technical capabilities in several trials with up to 21 participants recording simultaneously using multiple sensors with sampling frequencies as high as 1000 Hz. We could show that although there is a theoretical limitation to the transmissible data rate, in practice this limitation is not an issue and data loss is rare. We conclude that with modern communication protocols and with the increasingly powerful smartphones and wearables, a system like our SensorHub establishes an interoperability framework to adequately combine consumer-grade sensing hardware which enables observational studies in real life.

Pain fingerprinting using multimodal sensing: pilot study

Pain is a complex phenomenon, the experience of which varies widely across individuals. At worst, chronic pain can lead to anxiety and depression. Cost-effective strategies are urgently needed to improve the treatment of pain, and thus we propose a novel home-based pain measurement system for the longitudinal monitoring of pain experience and variation in different patients with chronic low back pain. The autonomous nervous system and audio-visual features are analyzed from heart rate signals, voice characteristics and facial expressions using a unique measurement protocol. Self-reporting is utilized for the follow-up of changes in pain intensity, induced by well-designed physical maneuvers, and for studying the consecutive trends in pain. We describe the study protocol, including hospital measurements and questionnaires and the implementation of the home measurement devices. We also present different methods for analyzing the multimodal data: electroencephalography, audio, video and heart rate. Our intention is to provide new insights using technical methodologies that will be beneficial in the future not only for patients with low back pain but also patients suffering from any chronic pain.

Polymer skulls with integrated transparent electrode arrays for cortex-wide opto-electrophysiological recordings

Electrophysiology and optical imaging provide complementary neural sensing capabilities; electrophysiological recordings have the highest temporal resolution, while optical imaging allows recording the activities of genetically defined populations at high spatial resolution. Combining these complementary, yet orthogonal modalities to perform simultaneous large-scale, multimodal sensing of neural activity across multiple brain regions would be very powerful. Here we show that transparent, inkjet-printed electrocorticography (ECoG) electrode arrays can be seamlessly integrated with morphologically conformant transparent polymer skulls for multimodal recordings across the cortex. These eSee-Shells, were implanted on transgenic mice expressing the Ca2+ indicator GCaMP6f in cortical excitatory cells and provided a robust opto-electrophysiological interface for over 100 days. eSee-Shells enable simultaneous mesoscale Ca2+ imaging and ECoG acquisition under anesthesia as well as in awake animals presented with sensory stimuli. eSee-Shells further show sufficient clarity and transparency to observe single-cell Ca2+ signals directly below the electrodes and interconnects. Simultaneous multimodal measurement of cortical dynamics reveals changes in both ECoG and Ca2+ signals that depend on the behavioral state.

In-situ-grown silver–polymer framework with coordination complexes for functional artificial tissues

Sensorized actuators are critical to imitate proprio-/exteroception properties of biological neuromuscular systems. Existing add-on approaches, which physically blend heterogeneous sensor/actuator components, fall short of yielding satisfactory solutions, considering their suboptimal interfaces, poor adhesion, and electronic/mechanical property mismatch. Here, we report a single homogeneous material comprising seamless sensing-actuation unification properties at nano-/molecule levels, in which built-in sensing functions originate from the actuator architecture itself. In-situ-grown silver nanoparticles and metal-ligand complexes cooperatively create a silver–polymer framework (SPF) that is stretchable (1200%), conductive (0.076 S/m), and strong (0.76 MPa in-strength). SPF displays concomitant multimodal sensing (mechanical and thermal cues) and sensorized actuation capabilities, which include proprio-deformation and external stimuli perceptions (simultaneous with load-lifting ability up to 3700× of own weight). In view of its human somatosensitive muscular systems imitative functionality, the reported SPFs bode well for use with next generation functional tissues including artificial skins, human-machine interfaces, self-sensing robots, and otherwise dynamic materials.