Deep Convolutional Neural Network with RNNs for Complex Activity Recognition Using Wrist-Worn Wearable Sensor Data

Sensor-based human activity recognition (S-HAR) has become an important and high-impact topic of research within human-centered computing. In the last decade, successful applications of S-HAR have been presented through fruitful academic research and industrial applications, including for healthcare monitoring, smart home controlling, and daily sport tracking. However, the growing requirements of many current applications for recognizing complex human activities (CHA) have begun to attract the attention of the HAR research field when compared with simple human activities (SHA). S-HAR has shown that deep learning (DL), a type of machine learning based on complicated artificial neural networks, has a significant degree of recognition efficiency. Convolutional neural networks (CNNs) and recurrent neural networks (RNNs) are two different types of DL methods that have been successfully applied to the S-HAR challenge in recent years. In this paper, we focused on four RNN-based DL models (LSTMs, BiLSTMs, GRUs, and BiGRUs) that performed complex activity recognition tasks. The efficiency of four hybrid DL models that combine convolutional layers with the efficient RNN-based models was also studied. Experimental studies on the UTwente dataset demonstrated that the suggested hybrid RNN-based models achieved a high level of recognition performance along with a variety of performance indicators, including accuracy, F1-score, and confusion matrix. The experimental results show that the hybrid DL model called CNN-BiGRU outperformed the other DL models with a high accuracy of 98.89% when using only complex activity data. Moreover, the CNN-BiGRU model also achieved the highest recognition performance in other scenarios (99.44% by using only simple activity data and 98.78% with a combination of simple and complex activities).

Download Full-text

Similarity Embedding Networks for Robust Human Activity Recognition

ACM Transactions on Knowledge Discovery from Data ◽

10.1145/3448021 ◽

2021 ◽

Vol 15 (6) ◽

pp. 1-17

Author(s):

Chenglin Li ◽

Carrie Lu Tong ◽

Di Niu ◽

Bei Jiang ◽

Xiao Zuo ◽

...

Keyword(s):

Activity Recognition ◽

Human Activity ◽

Short Term Memory ◽

Real Space ◽

Human Activity Recognition ◽

Sensor Data ◽

Activity Data ◽

Extensive Evaluation ◽

Sensor Signals ◽

Public Datasets

Deep learning models for human activity recognition (HAR) based on sensor data have been heavily studied recently. However, the generalization ability of deep models on complex real-world HAR data is limited by the availability of high-quality labeled activity data, which are hard to obtain. In this article, we design a similarity embedding neural network that maps input sensor signals onto real vectors through carefully designed convolutional and Long Short-Term Memory (LSTM) layers. The embedding network is trained with a pairwise similarity loss, encouraging the clustering of samples from the same class in the embedded real space, and can be effectively trained on a small dataset and even on a noisy dataset with mislabeled samples. Based on the learned embeddings, we further propose both nonparametric and parametric approaches for activity recognition. Extensive evaluation based on two public datasets has shown that the proposed similarity embedding network significantly outperforms state-of-the-art deep models on HAR classification tasks, is robust to mislabeled samples in the training set, and can also be used to effectively denoise a noisy dataset.

Download Full-text

Self-Adaptive Multi-Sensor Activity Recognition Systems Based on Gaussian Mixture Models

Informatics ◽

10.3390/informatics5030038 ◽

2018 ◽

Vol 5 (3) ◽

pp. 38 ◽

Cited By ~ 3

Author(s):

Martin Jänicke ◽

Bernhard Sick ◽

Sven Tomforde

Keyword(s):

Activity Recognition ◽

Gaussian Mixture Models ◽

Gaussian Mixture ◽

Recognition System ◽

Training Data ◽

Sensor Data ◽

Activity Data ◽

High Loss ◽

New Sensors ◽

Self Adaptive

Personal wearables such as smartphones or smartwatches are increasingly utilized in everyday life. Frequently, activity recognition is performed on these devices to estimate the current user status and trigger automated actions according to the user’s needs. In this article, we focus on the creation of a self-adaptive activity recognition system based on IMU that includes new sensors during runtime. Starting with a classifier based on GMM, the density model is adapted to new sensor data fully autonomously by issuing the marginalization property of normal distributions. To create a classifier from that, label inference is done, either based on the initial classifier or based on the training data. For evaluation, we used more than 10 h of annotated activity data from the publicly available PAMAP2 benchmark dataset. Using the data, we showed the feasibility of our approach and performed 9720 experiments, to get resilient numbers. One approach performed reasonably well, leading to a system improvement on average, with an increase in the F-score of 0.0053, while the other one shows clear drawbacks due to a high loss of information during label inference. Furthermore, a comparison with state of the art techniques shows the necessity for further experiments in this area.

Download Full-text

Multimodal Sensor Data Fusion for Activity Recognition Using Filtered Classifier

Proceedings ◽

10.3390/proceedings2191262 ◽

2018 ◽

Vol 2 (19) ◽

pp. 1262 ◽

Cited By ~ 4

Author(s):

Muhammad Razzaq ◽

Ian Cleland ◽

Chris Nugent ◽

Sungyoung Lee

Keyword(s):

Activity Recognition ◽

Confusion Matrix ◽

Machine Learning Algorithms ◽

Training Data ◽

Sensor Data ◽

Test Dataset ◽

Sensory Data ◽

Multimodal Sensors ◽

Imbalanced Class Distribution ◽

Single Modality

Activity recognition (AR) is a subtask in pervasive computing and context-aware systems, which presents the physical state of human in real-time. These systems offer a new dimension to the widely spread applications by fusing recognized activities obtained from the raw sensory data generated by the obtrusive as well as unobtrusive revolutionary digital technologies. In recent years, an exponential growth has been observed for AR technologies and much literature exists focusing on applying machine learning algorithms on obtrusive single modality sensor devices. However, University of Jaén Ambient Intelligence (UJAmI), a Smart Lab in Spain has initiated a 1st UCAmI Cup challenge by sharing aforementioned varieties of the sensory data in order to recognize the human activities in the smart environment. This paper presents the fusion, both at the feature level and decision level for multimodal sensors by preprocessing and predicting the activities within the context of training and test datasets. Though it achieves 94% accuracy for training data and 47% accuracy for test data. However, this study further evaluates post-confusion matrix also and draws a conclusion for various discrepancies such as imbalanced class distribution within the training and test dataset. Additionally, this study also highlights challenges associated with the datasets for which, could improve further analysis.

Download Full-text

Human Activity Recognition for Indoor Localization Using Smartphone Inertial Sensors

Sensors ◽

10.3390/s21186316 ◽

2021 ◽

Vol 21 (18) ◽

pp. 6316

Author(s):

Dinis Moreira ◽

Marília Barandas ◽

Tiago Rocha ◽

Pedro Alves ◽

Ricardo Santos ◽

...

Keyword(s):

Activity Recognition ◽

Human Activity ◽

Human Activities ◽

Short Term Memory ◽

Indoor Localization ◽

Inertial Sensor ◽

Human Activity Recognition ◽

Location Based Services ◽

Sensor Data ◽

Indoor Environments

With the fast increase in the demand for location-based services and the proliferation of smartphones, the topic of indoor localization is attracting great interest. In indoor environments, users’ performed activities carry useful semantic information. These activities can then be used by indoor localization systems to confirm users’ current relative locations in a building. In this paper, we propose a deep-learning model based on a Convolutional Long Short-Term Memory (ConvLSTM) network to classify human activities within the indoor localization scenario using smartphone inertial sensor data. Results show that the proposed human activity recognition (HAR) model accurately identifies nine types of activities: not moving, walking, running, going up in an elevator, going down in an elevator, walking upstairs, walking downstairs, or going up and down a ramp. Moreover, predicted human activities were integrated within an existing indoor positioning system and evaluated in a multi-story building across several testing routes, with an average positioning error of 2.4 m. The results show that the inclusion of human activity information can reduce the overall localization error of the system and actively contribute to the better identification of floor transitions within a building. The conducted experiments demonstrated promising results and verified the effectiveness of using human activity-related information for indoor localization.

Download Full-text

Privacy Issues Regarding the Application of DNNs to Activity-Recognition using Wearables and Its Countermeasures by Use of Adversarial Training

Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence ◽

10.24963/ijcai.2017/268 ◽

2017 ◽

Cited By ~ 5

Author(s):

Yusuke Iwasawa ◽

Kotaro Nakayama ◽

Ikuko Yairi ◽

Yutaka Matsuo

Keyword(s):

Neural Networks ◽

Activity Recognition ◽

Information Disclosure ◽

Deep Neural Networks ◽

Recognition Performance ◽

Empirical Validation ◽

Privacy Concerns ◽

Proposed Model ◽

Significant Performance ◽

Adversarial Training

Deep neural networks have been successfully applied to activity recognition with wearables in terms of recognition performance. However, the black-box nature of neural networks could lead to privacy concerns. Namely, generally it is hard to expect what neural networks learn from data, and so they possibly learn features that highly discriminate user-information unintentionally, which increases the risk of information-disclosure. In this study, we analyzed the features learned by conventional deep neural networks when applied to data of wearables to confirm this phenomenon.Based on the results of our analysis, we propose the use of an adversarial training framework to suppress the risk of sensitive/unintended information disclosure. Our proposed model considers both an adversarial user classifier and a regular activity-classifier during training, which allows the model to learn representations that help the classifier to distinguish the activities but which, at the same time, prevents it from accessing user-discriminative information. This paper provides an empirical validation of the privacy issue and efficacy of the proposed method using three activity recognition tasks based on data of wearables. The empirical validation shows that our proposed method suppresses the concerns without any significant performance degradation, compared to conventional deep nets on all three tasks.

Download Full-text

A multilabel classification approach for complex human activities using a combination of emerging patterns and fuzzy sets

International Journal of Electrical and Computer Engineering (IJECE) ◽

10.11591/ijece.v9i4.pp2993-3001 ◽

2019 ◽

Vol 9 (4) ◽

pp. 2993

Author(s):

Nehal A. Sakr ◽

Mervat Abu-ElKheir ◽

A. Atwan ◽

H. H. Soliman

Keyword(s):

Fuzzy Sets ◽

Activity Recognition ◽

Pattern Mining ◽

Scoring Function ◽

Sensor Data ◽

Training Dataset ◽

Complex Activity ◽

Multilabel Classification ◽

Classification Approach ◽

Emerging Patterns

In our daily lives, humans perform different Activities of Daily Living (ADL), such as cooking, and studying. According to the nature of humans, they perform these activities in a sequential/simple or an overlapping/complex scenario. Many research attempts addressed simple activity recognition, but complex activity recognition is still a challenging issue. Recognition of complex activities is a multilabel classification problem, such that a test instance is assigned to a multiple overlapping activities. Existing data-driven techniques for complex activity recognition can recognize a maximum number of two overlapping activities and require a training dataset of complex (i.e. multilabel) activities. In this paper, we propose a multilabel classification approach for complex activity recognition using a combination of Emerging Patterns and Fuzzy Sets. In our approach, we require a training dataset of only simple (i.e. single-label) activities. First, we use a pattern mining technique to extract discriminative features called Strong Jumping Emerging Patterns (SJEPs) that exclusively represent each activity. Then, our scoring function takes SJEPs and fuzzy membership values of incoming sensor data and outputs the activity label(s). We validate our approach using two different dataset. Experimental results demonstrate the efficiency and superiority of our approach against other approaches.

Download Full-text

Fusing Object Information and Inertial Data for Activity Recognition

Sensors ◽

10.3390/s19194119 ◽

2019 ◽

Vol 19 (19) ◽

pp. 4119 ◽

Cited By ~ 3

Author(s):

Alexander Diete ◽

Heiner Stuckenschmidt

Keyword(s):

Activity Recognition ◽

Pervasive Computing ◽

Human Activities ◽

Inertial Sensors ◽

Inertial Sensor ◽

Arm Movement ◽

Sensor Data ◽

Object A ◽

Camera View ◽

Different Types

In the field of pervasive computing, wearable devices have been widely used for recognizing human activities. One important area in this research is the recognition of activities of daily living where especially inertial sensors and interaction sensors (like RFID tags with scanners) are popular choices as data sources. Using interaction sensors, however, has one drawback: they may not differentiate between proper interaction and simple touching of an object. A positive signal from an interaction sensor is not necessarily caused by a performed activity e.g., when an object is only touched but no interaction occurred afterwards. There are, however, many scenarios like medicine intake that rely heavily on correctly recognized activities. In our work, we aim to address this limitation and present a multimodal egocentric-based activity recognition approach. Our solution relies on object detection that recognizes activity-critical objects in a frame. As it is infeasible to always expect a high quality camera view, we enrich the vision features with inertial sensor data that monitors the users’ arm movement. This way we try to overcome the drawbacks of each respective sensor. We present our results of combining inertial and video features to recognize human activities on different types of scenarios where we achieve an F 1 -measure of up to 79.6%.

Download Full-text

Sensor Data Acquisition and Multimodal Sensor Fusion for Human Activity Recognition Using Deep Learning

Sensors ◽

10.3390/s19071716 ◽

2019 ◽

Vol 19 (7) ◽

pp. 1716 ◽

Cited By ~ 20

Author(s):

Seungeun Chung ◽

Jiyoun Lim ◽

Kyoung Ju Noh ◽

Gague Kim ◽

Hyuntae Jeong

Keyword(s):

Deep Learning ◽

Data Acquisition ◽

Sensor Fusion ◽

Activity Recognition ◽

Human Activity ◽

Short Term Memory ◽

Sampling Rate ◽

Human Activity Recognition ◽

Sensor Data ◽

Activity Data

In this paper, we perform a systematic study about the on-body sensor positioning and data acquisition details for Human Activity Recognition (HAR) systems. We build a testbed that consists of eight body-worn Inertial Measurement Units (IMU) sensors and an Android mobile device for activity data collection. We develop a Long Short-Term Memory (LSTM) network framework to support training of a deep learning model on human activity data, which is acquired in both real-world and controlled environments. From the experiment results, we identify that activity data with sampling rate as low as 10 Hz from four sensors at both sides of wrists, right ankle, and waist is sufficient in recognizing Activities of Daily Living (ADLs) including eating and driving activity. We adopt a two-level ensemble model to combine class-probabilities of multiple sensor modalities, and demonstrate that a classifier-level sensor fusion technique can improve the classification performance. By analyzing the accuracy of each sensor on different types of activity, we elaborate custom weights for multimodal sensor fusion that reflect the characteristic of individual activities.

Download Full-text

CARTMAN: Complex Activity Recognition Using Topic Models for Feature Generation from Wearable Sensor Data

10.1109/smartcomp52413.2021.00026 ◽

2021 ◽

Author(s):

Kavin Chandrasekaran ◽

Walter Gerych ◽

Luke Buquicchio ◽

Abdulaziz Alajaji ◽

Emmanuel Agu ◽

...

Keyword(s):

Activity Recognition ◽

Topic Models ◽

Sensor Data ◽

Wearable Sensor ◽

Feature Generation ◽

Complex Activity

Download Full-text

A Method for Sensor-Based Activity Recognition in Missing Data Scenario

Sensors ◽

10.3390/s20143811 ◽

2020 ◽

Vol 20 (14) ◽

pp. 3811

Author(s):

Tahera Hossain ◽

Md. Atiqur Rahman Ahad ◽

Sozo Inoue

Keyword(s):

Machine Learning ◽

Missing Data ◽

Activity Recognition ◽

Human Activity ◽

Human Activities ◽

Real Life ◽

Sensor Data ◽

Quality Data ◽

Support Vector ◽

Random Pattern

Sensor-based human activity recognition has various applications in the arena of healthcare, elderly smart-home, sports, etc. There are numerous works in this field—to recognize various human activities from sensor data. However, those works are based on data patterns that are clean data and have almost no missing data, which is a genuine concern for real-life healthcare centers. Therefore, to address this problem, we explored the sensor-based activity recognition when some partial data were lost in a random pattern. In this paper, we propose a novel method to improve activity recognition while having missing data without any data recovery. For the missing data pattern, we considered data to be missing in a random pattern, which is a realistic missing pattern for sensor data collection. Initially, we created different percentages of random missing data only in the test data, while the training was performed on good quality data. In our proposed approach, we explicitly induce different percentages of missing data randomly in the raw sensor data to train the model with missing data. Learning with missing data reinforces the model to regulate missing data during the classification of various activities that have missing data in the test module. This approach demonstrates the plausibility of the machine learning model, as it can learn and predict from an identical domain. We exploited several time-series statistical features to extricate better features in order to comprehend various human activities. We explored both support vector machine and random forest as machine learning models for activity classification. We developed a synthetic dataset to empirically evaluate the performance and show that the method can effectively improve the recognition accuracy from 80.8% to 97.5%. Afterward, we tested our approach with activities from two challenging benchmark datasets: the human activity sensing consortium (HASC) dataset and single chest-mounted accelerometer dataset. We examined the method for different missing percentages, varied window sizes, and diverse window sliding widths. Our explorations demonstrated improved recognition performances even in the presence of missing data. The achieved results provide persuasive findings on sensor-based activity recognition in the presence of missing data.

Download Full-text