SonicFace

Accurate recognition of facial expressions and emotional gestures is promising to understand the audience's feedback and engagement on the entertainment content. Existing methods are primarily based on various cameras or wearable sensors, which either raise privacy concerns or demand extra devices. To this aim, we propose a novel ubiquitous sensing system based on the commodity microphone array --- SonicFace, which provides an accessible, unobtrusive, contact-free, and privacy-preserving solution to monitor the user's emotional expressions continuously without playing hearable sound. SonicFace utilizes a pair of speaker and microphone array to recognize various fine-grained facial expressions and emotional hand gestures by emitted ultrasound and received echoes. Based on a set of experimental evaluations, the accuracy of recognizing 6 common facial expressions and 4 emotional gestures can reach around 80%. Besides, the extensive system evaluations with distinct configurations and an extended real-life case study have demonstrated the robustness and generalizability of the proposed SonicFace system.

CSMC

The rise concern about mobile communication performance has driven the growing demand for the construction of mobile network signal maps which are widely utilized in network monitoring, spectrum management, and indoor/outdoor localization. Existing studies such as time-consuming and labor-intensive site surveys are difficult to maintain an update-to-date finegrained signal map within a large area. The mobile crowdsensing (MCS) paradigm is a promising approach for building signal maps because collecting large-scale MCS data is low-cost and with little extra-efforts. However, the dynamic environment and the mobility of the crowd cause spatio-temporal uncertainty and sparsity of MCS. In this work, we leverage MCS as an opportunity to conduct the city-wide mobile network signal map construction. We propose a fine-grained city-wide Cellular Signal Map Construction (CSMC) framework to address two challenges including (i) the problem of missing and unreliable MCS data; (ii) spatio-temporal uncertainty of signal propagation. In particular, CSMC captures spatio-temporal characteristics of signals from both inter- and intra- cellular base stations and conducts missing signal recovery with Bayesian tensor decomposition to build large-area fine-grained signal maps. Furthermore, CSMC develops a context-aware multi-view fusion network to make full use of external information and enhance signal map construction accuracy. To evaluate the performance of CSMC, we conduct extensive experiments and ablation studies on a large-scale dataset with over 200GB MCS signal records collected from Shanghai. Experimental results demonstrate that our model outperforms state-of-the-art baselines in the accuracy of signal estimation and user localization.

DAFI

WiFi-based Device-free Passive (DfP) indoor localization systems liberate their users from carrying dedicated sensors or smartphones, and thus provide a non-intrusive and pleasant experience. Although existing fingerprint-based systems achieve sub-meter-level localization accuracy by training location classifiers/regressors on WiFi signal fingerprints, they are usually vulnerable to small variations in an environment. A daily change, e.g., displacement of a chair, may cause a big inconsistency between the recorded fingerprints and the real-time signals, leading to significant localization errors. In this paper, we introduce a Domain Adaptation WiFi (DAFI) localization approach to address the problem. DAFI formulates this fingerprint inconsistency issue as a domain adaptation problem, where the original environment is the source domain and the changed environment is the target domain. Directly applying existing domain adaptation methods to our specific problem is challenging, since it is generally hard to distinguish the variations in the different WiFi domains (i.e., signal changes caused by different environmental variations). DAFI embraces the following techniques to tackle this challenge. 1) DAFI aligns both marginal and conditional distributions of features in different domains. 2) Inside the target domain, DAFI squeezes the marginal distribution of every class to be more concentrated at its center. 3) Between two domains, DAFI conducts fine-grained alignment by forcing every target-domain class to better align with its source-domain counterpart. By doing these, DAFI outperforms the state of the art by up to 14.2% in real-world experiments.

Constructing Floor Plan through Smoke Using Ultra Wideband Radar

Floor plan construction has been one of the key techniques in many important applications such as indoor navigation, location-based services, and emergency rescue. Existing floor plan construction methods require expensive dedicated hardware (e.g., Lidar or depth camera), and may not work in low-visibility environments (e.g., smoke, fog or dust). In this paper, we develop a low-cost Ultra Wideband (UWB)-based system (named UWBMap) that is mounted on a mobile robot platform to construct floor plan through smoke. UWBMap leverages on low-cost and off-the-shelf UWB radar, and it is able to construct an indoor map with an accuracy comparable to Lidar (i.e., the state-of-the-art). The underpinning technique is to take advantage of the mobility of radar to form virtual antennas and gather spatial information of a target. UWBMap also eliminates both robot motion noise and environmental noise to enhance weak reflection from small objects for the robust construction process. In addition, we overcome the limited view of single radar by combining multi-view from multiple radars. Extensive experiments in different indoor environments show that UWBMap achieves a map construction with a median error of 11 cm and a 90-percentile error of 26 cm, and it operates effectively in indoor scenarios with glass wall and dense smoke.

FaceBit

The COVID-19 pandemic has dramatically increased the use of face masks across the world. Aside from physical distancing, they are among the most effective protection for healthcare workers and the general population. Face masks are passive devices, however, and cannot alert the user in case of improper fit or mask degradation. Additionally, face masks are optimally positioned to give unique insight into some personal health metrics. Recognizing this limitation and opportunity, we present FaceBit: an open-source research platform for smart face mask applications. FaceBit's design was informed by needfinding studies with a cohort of health professionals. Small and easily secured into any face mask, FaceBit is accompanied by a mobile application that provides a user interface and facilitates research. It monitors heart rate without skin contact via ballistocardiography, respiration rate via temperature changes, and mask-fit and wear time from pressure signals, all on-device with an energy-efficient runtime system. FaceBit can harvest energy from breathing, motion, or sunlight to supplement its tiny primary cell battery that alone delivers a battery lifetime of 11 days or more. FaceBit empowers the mobile computing community to jumpstart research in smart face mask sensing and inference, and provides a sustainable, convenient form factor for health management, applicable to COVID-19 frontline workers and beyond.

VREED

The paper introduces a multimodal affective dataset named VREED (VR Eyes: Emotions Dataset) in which emotions were triggered using immersive 360° Video-Based Virtual Environments (360-VEs) delivered via Virtual Reality (VR) headset. Behavioural (eye tracking) and physiological signals (Electrocardiogram (ECG) and Galvanic Skin Response (GSR)) were captured, together with self-reported responses, from healthy participants (n=34) experiencing 360-VEs (n=12, 1--3 min each) selected through focus groups and a pilot trial. Statistical analysis confirmed the validity of the selected 360-VEs in eliciting the desired emotions. Preliminary machine learning analysis was carried out, demonstrating state-of-the-art performance reported in affective computing literature using non-immersive modalities. VREED is among the first multimodal VR datasets in emotion recognition using behavioural and physiological signals. VREED is made publicly available on Kaggle1. We hope that this contribution encourages other researchers to utilise VREED further to understand emotional responses in VR and ultimately enhance VR experiences design in applications where emotional elicitation plays a key role, i.e. healthcare, gaming, education, etc.

SmartKC

Keratoconus is a severe eye disease affecting the cornea (the clear, dome-shaped outer surface of the eye), causing it to become thin and develop a conical bulge. The diagnosis of keratoconus requires sophisticated ophthalmic devices which are non-portable and very expensive. This makes early detection of keratoconus inaccessible to large populations in low-and middle-income countries, making it a leading cause for partial/complete blindness among such populations. We propose SmartKC, a low-cost, smartphone-based keratoconus diagnosis system comprising of a 3D-printed placido's disc attachment, an LED light strip, and an intelligent smartphone app to capture the reflection of the placido rings on the cornea. An image processing pipeline analyzes the corneal image and uses the smartphone's camera parameters, the placido rings' 3D location, the pixel location of the reflected placido rings and the setup's working distance to construct the corneal surface, via the Arc-Step method and Zernike polynomials based surface fitting. In a clinical study with 101 distinct eyes, we found that SmartKC achieves a sensitivity of 87.8% and a specificity of 80.4%. Moreover, the quantitative curvature estimates (sim-K) strongly correlate with a gold-standard medical device (Pearson correlation coefficient = 0.77). Our results indicate that SmartKC has the potential to be used as a keratoconus screening tool under real-world medical settings.

Towards Device Independent Eavesdropping on Telephone Conversations with Built-in Accelerometer

Motion sensors in modern smartphones have been exploited for audio eavesdropping in loudspeaker mode due to their sensitivity to vibrations. In this paper, we further move one step forward to explore the feasibility of using built-in accelerometer to eavesdrop on the telephone conversation of caller/callee who takes the phone against cheek-ear and design our attack Vibphone. The inspiration behind Vibphone is that the speech-induced vibrations (SIV) can be transmitted through the physical contact of phone-cheek to accelerometer with the traces of voice content. To this end, Vibphone faces three main challenges: i) Accurately detecting SIV signals from miscellaneous disturbance; ii) Combating the impact of device diversity to work with a variety of attack scenarios; and iii) Enhancing feature-agnostic recognition model to generalize to newly issued devices and reduce training overhead. To address these challenges, we first conduct an in-depth investigation on SIV features to figure out the root cause of device diversity impacts and identify a set of critical features that are highly relevant to the voice content retained in SIV signals and independent of specific devices. On top of these pivotal observations, we propose a combo method that is the integration of extracted critical features and deep neural network to recognize speech information from the spectrogram representation of acceleration signals. We implement the attack using commodity smartphones and the results show it is highly effective. Our work brings to light a fundamental design vulnerability in the vast majority of currently deployed smartphones, which may put people's speech privacy at risk during phone calls. We also propose a practical and effective defense solution. We validate that it is feasible to prevent audio eavesdropping by using random variation of sampling rate.

ChestLive

Voice-based authentication is prevalent on smart devices to verify the legitimacy of users, but is vulnerable to replay attacks. In this paper, we propose to leverage the distinctive chest motions during speaking to establish a secure multi-factor authentication system, named ChestLive. Compared with other biometric-based authentication systems, ChestLive does not require users to remember any complicated information (e.g., hand gestures, doodles) and the working distance is much longer (30cm). We use acoustic sensing to monitor chest motions with a built-in speaker and microphone on smartphones. To obtain fine-grained chest motion signals during speaking for reliable user authentication, we derive Channel Energy (CE) of acoustic signals to capture the chest movement, and then remove the static and non-static interference from the aggregated CE signals. Representative features are extracted from the correlation between voice signal and corresponding chest motion signal. Unlike learning-based image or speech recognition models with millions of available training samples, our system needs to deal with a limited number of samples from legitimate users during enrollment. To address this problem, we resort to meta-learning, which initializes a general model with good generalization property that can be quickly fine-tuned to identify a new user. We implement ChestLive as an application and evaluate its performance in the wild with 61 volunteers using their smartphones. Experiment results show that ChestLive achieves an authentication accuracy of 98.31% and less than 2% of false accept rate against replay attacks and impersonation attacks. We also validate that ChestLive is robust to various factors, including training set size, distance, angle, posture, phone models, and environment noises.

Janus

Proximity detection is at the core of several mobile and ubiquitous computing applications. These include reactive use cases, e.g., alerting individuals of hazards or interaction opportunities, and others concerned only with logging proximity data, e.g., for offline analysis and modeling. Common approaches rely on Bluetooth Low Energy (BLE) or ultra-wideband (UWB) radios. Nevertheless, these strike opposite tradeoffs between the accuracy of distance estimates quantifying proximity and the energy efficiency affecting system lifetime, effectively forcing a choice between the two and ultimately constraining applicability. Janus reconciles these dimensions in a dual-radio protocol enabling accurate and energy-efficient proximity detection, where the energy-savvy BLE is exploited to discover devices and coordinate their distance measurements, acquired via the energy-hungry UWB. A model supports domain experts in configuring Janus for their use cases with predictable performance. The latency, reliability, and accuracy of Janus are evaluated experimentally, including realistic scenarios endowed with the mm-level ground truth provided by a motion capture system. Energy measurements show that Janus achieves weeks to months of autonomous operation, depending on the use case configuration. Finally, several large-scale campaigns exemplify its practical usefulness in real-world contexts.