A Fast and Robust Deep Convolutional Neural Networks for Complex Human Activity Recognition Using Smartphone

As a significant role in healthcare and sports applications, human activity recognition (HAR) techniques are capable of monitoring humans’ daily behavior. It has spurred the demand for intelligent sensors and has been giving rise to the explosive growth of wearable and mobile devices. They provide the most availability of human activity data (big data). Powerful algorithms are required to analyze these heterogeneous and high-dimension streaming data efficiently. This paper proposes a novel fast and robust deep convolutional neural network structure (FR-DCNN) for human activity recognition (HAR) using a smartphone. It enhances the effectiveness and extends the information of the collected raw data from the inertial measurement unit (IMU) sensors by integrating a series of signal processing algorithms and a signal selection module. It enables a fast computational method for building the DCNN classifier by adding a data compression module. Experimental results on the sampled 12 complex activities dataset show that the proposed FR-DCNN model is the best method for fast computation and high accuracy recognition. The FR-DCNN model only needs 0.0029 s to predict activity in an online way with 95.27% accuracy. Meanwhile, it only takes 88 s (average) to establish the DCNN classifier on the compressed dataset with less precision loss 94.18%.

Download Full-text

Similarity Embedding Networks for Robust Human Activity Recognition

ACM Transactions on Knowledge Discovery from Data ◽

10.1145/3448021 ◽

2021 ◽

Vol 15 (6) ◽

pp. 1-17

Author(s):

Chenglin Li ◽

Carrie Lu Tong ◽

Di Niu ◽

Bei Jiang ◽

Xiao Zuo ◽

...

Keyword(s):

Activity Recognition ◽

Human Activity ◽

Short Term Memory ◽

Real Space ◽

Human Activity Recognition ◽

Sensor Data ◽

Activity Data ◽

Extensive Evaluation ◽

Sensor Signals ◽

Public Datasets

Deep learning models for human activity recognition (HAR) based on sensor data have been heavily studied recently. However, the generalization ability of deep models on complex real-world HAR data is limited by the availability of high-quality labeled activity data, which are hard to obtain. In this article, we design a similarity embedding neural network that maps input sensor signals onto real vectors through carefully designed convolutional and Long Short-Term Memory (LSTM) layers. The embedding network is trained with a pairwise similarity loss, encouraging the clustering of samples from the same class in the embedded real space, and can be effectively trained on a small dataset and even on a noisy dataset with mislabeled samples. Based on the learned embeddings, we further propose both nonparametric and parametric approaches for activity recognition. Extensive evaluation based on two public datasets has shown that the proposed similarity embedding network significantly outperforms state-of-the-art deep models on HAR classification tasks, is robust to mislabeled samples in the training set, and can also be used to effectively denoise a noisy dataset.

Download Full-text

Personalized Human Activity Recognition Based on Integrated Wearable Sensor and Transfer Learning

Sensors ◽

10.3390/s21030885 ◽

2021 ◽

Vol 21 (3) ◽

pp. 885 ◽

Cited By ~ 1

Author(s):

Zhongzheng Fu ◽

Xinrun He ◽

Enkai Wang ◽

Jun Huo ◽

Jian Huang ◽

...

Keyword(s):

Transfer Learning ◽

Activity Recognition ◽

Human Activity ◽

Sensor Node ◽

Learning Algorithm ◽

Joint Probability ◽

Human Activity Recognition ◽

Sensor Technology ◽

Measurement Unit ◽

Wearable Sensor

Human activity recognition (HAR) based on the wearable device has attracted more attention from researchers with sensor technology development in recent years. However, personalized HAR requires high accuracy of recognition, while maintaining the model’s generalization capability is a major challenge in this field. This paper designed a compact wireless wearable sensor node, which combines an air pressure sensor and inertial measurement unit (IMU) to provide multi-modal information for HAR model training. To solve personalized recognition of user activities, we propose a new transfer learning algorithm, which is a joint probability domain adaptive method with improved pseudo-labels (IPL-JPDA). This method adds the improved pseudo-label strategy to the JPDA algorithm to avoid cumulative errors due to inaccurate initial pseudo-labels. In order to verify our equipment and method, we use the newly designed sensor node to collect seven daily activities of 7 subjects. Nine different HAR models are trained by traditional machine learning and transfer learning methods. The experimental results show that the multi-modal data improve the accuracy of the HAR system. The IPL-JPDA algorithm proposed in this paper has the best performance among five HAR models, and the average recognition accuracy of different subjects is 93.2%.

Download Full-text

On-Device Deep Learning Inference for Efficient Activity Data Collection

Sensors ◽

10.3390/s19153434 ◽

2019 ◽

Vol 19 (15) ◽

pp. 3434 ◽

Cited By ~ 2

Author(s):

Nattaya Mairittha ◽

Tittaya Mairittha ◽

Sozo Inoue

Keyword(s):

Deep Learning ◽

Data Collection ◽

Activity Recognition ◽

Human Activity ◽

Short Term Memory ◽

Human Activity Recognition ◽

Classification Model ◽

Activity Data ◽

Ground Truth Data ◽

Recognition Systems

Labeling activity data is a central part of the design and evaluation of human activity recognition systems. The performance of the systems greatly depends on the quantity and “quality” of annotations; therefore, it is inevitable to rely on users and to keep them motivated to provide activity labels. While mobile and embedded devices are increasingly using deep learning models to infer user context, we propose to exploit on-device deep learning inference using a long short-term memory (LSTM)-based method to alleviate the labeling effort and ground truth data collection in activity recognition systems using smartphone sensors. The novel idea behind this is that estimated activities are used as feedback for motivating users to collect accurate activity labels. To enable us to perform evaluations, we conduct the experiments with two conditional methods. We compare the proposed method showing estimated activities using on-device deep learning inference with the traditional method showing sentences without estimated activities through smartphone notifications. By evaluating with the dataset gathered, the results show our proposed method has improvements in both data quality (i.e., the performance of a classification model) and data quantity (i.e., the number of data collected) that reflect our method could improve activity data collection, which can enhance human activity recognition systems. We discuss the results, limitations, challenges, and implications for on-device deep learning inference that support activity data collection. Also, we publish the preliminary dataset collected to the research community for activity recognition.

Download Full-text

Sensor Data Acquisition and Multimodal Sensor Fusion for Human Activity Recognition Using Deep Learning

Sensors ◽

10.3390/s19071716 ◽

2019 ◽

Vol 19 (7) ◽

pp. 1716 ◽

Cited By ~ 20

Author(s):

Seungeun Chung ◽

Jiyoun Lim ◽

Kyoung Ju Noh ◽

Gague Kim ◽

Hyuntae Jeong

Keyword(s):

Deep Learning ◽

Data Acquisition ◽

Sensor Fusion ◽

Activity Recognition ◽

Human Activity ◽

Short Term Memory ◽

Sampling Rate ◽

Human Activity Recognition ◽

Sensor Data ◽

Activity Data

In this paper, we perform a systematic study about the on-body sensor positioning and data acquisition details for Human Activity Recognition (HAR) systems. We build a testbed that consists of eight body-worn Inertial Measurement Units (IMU) sensors and an Android mobile device for activity data collection. We develop a Long Short-Term Memory (LSTM) network framework to support training of a deep learning model on human activity data, which is acquired in both real-world and controlled environments. From the experiment results, we identify that activity data with sampling rate as low as 10 Hz from four sensors at both sides of wrists, right ankle, and waist is sufficient in recognizing Activities of Daily Living (ADLs) including eating and driving activity. We adopt a two-level ensemble model to combine class-probabilities of multiple sensor modalities, and demonstrate that a classifier-level sensor fusion technique can improve the classification performance. By analyzing the accuracy of each sensor on different types of activity, we elaborate custom weights for multimodal sensor fusion that reflect the characteristic of individual activities.

Download Full-text

Sensitivity of Sensor Locations in a Wearable IMU-based Human Activity Recognition System

10.21203/rs.3.rs-779170/v1 ◽

2021 ◽

Author(s):

Mehdi Ejtehadi ◽

Amin M. Nasrabadi ◽

Saeed Behzadipour

Keyword(s):

Activity Recognition ◽

Human Activity ◽

Human Subjects ◽

Kinematic Model ◽

Recognition System ◽

Human Activity Recognition ◽

The Body ◽

Measurement Unit ◽

Body Segments ◽

Body Kinematic

Abstract Background: The advent of Inertial measurement unit (IMU) sensors has significantly extended the application domain of Human Activity Recognition (HAR) systems to healthcare, tele-rehabilitation & daily life monitoring. IMU’s are categorized as body-worn sensors and therefore their output signals and the HAR performance naturally depends on their exact location on the body segments. Objectives: This research aims to introduce a methodology to investigate the effects of misplacing the sensors on the performance of the HAR systems. Methods: The properly placed sensors and their misplaced variations were modeled on a human body kinematic model. The model was then actuated using measured motions from human subjects. The model was then used to run a sensitivity analysis. Results: The results indicated that the transverse misplacement of the sensors on the left arm and right thigh and the rotation of the left thigh sensor significantly decrease the rate of activity recognition. It was also shown that the longitudinal displacements of the sensors (along the body segments) have minor impacts on the HAR performance. A Monte Carlo simulation indicated that if the sensitive sensors are mounted with extra care, the performance can be maintained at a higher than 95% level.Conclusions: Accurate mounting of the IMU’s on the body impacts the performance of the HAR. Particularly, the transverse position and rotation of the IMU’s are more sensitive. The users of such systems need to be informed about the more sensitive sensors and directions to maintain an acceptable performance for the HAR.

Download Full-text

Feature extraction for human activity recognition on streaming data

2015 International Symposium on Innovations in Intelligent SysTems and Applications (INISTA) ◽

10.1109/inista.2015.7276759 ◽

2015 ◽

Cited By ~ 14

Author(s):

Nawel Yala ◽

Belkacem Fergani ◽

Anthony Fleury

Keyword(s):

Feature Extraction ◽

Activity Recognition ◽

Human Activity ◽

Human Activity Recognition ◽

Streaming Data

Download Full-text

A Comparative Study of Feature Selection Approaches for Human Activity Recognition Using Multimodal Sensory Data

Sensors ◽

10.3390/s21072368 ◽

2021 ◽

Vol 21 (7) ◽

pp. 2368

Author(s):

Fatima Amjad ◽

Muhammad Hassan Khan ◽

Muhammad Adeel Nisar ◽

Muhammad Shahid Farid ◽

Marcin Grzegorzek

Keyword(s):

Activity Recognition ◽

Human Activity ◽

Human Body ◽

Human Activities ◽

Wearable Sensors ◽

Human Activity Recognition ◽

Motion Sensors ◽

Activity Data ◽

Sensory Data ◽

Atomic Actions

Human activity recognition (HAR) aims to recognize the actions of the human body through a series of observations and environmental conditions. The analysis of human activities has drawn the attention of the research community in the last two decades due to its widespread applications, diverse nature of activities, and recording infrastructure. Lately, one of the most challenging applications in this framework is to recognize the human body actions using unobtrusive wearable motion sensors. Since the human activities of daily life (e.g., cooking, eating) comprises several repetitive and circumstantial short sequences of actions (e.g., moving arm), it is quite difficult to directly use the sensory data for recognition because the multiple sequences of the same activity data may have large diversity. However, a similarity can be observed in the temporal occurrence of the atomic actions. Therefore, this paper presents a two-level hierarchical method to recognize human activities using a set of wearable sensors. In the first step, the atomic activities are detected from the original sensory data, and their recognition scores are obtained. Secondly, the composite activities are recognized using the scores of atomic actions. We propose two different methods of feature extraction from atomic scores to recognize the composite activities, and they include handcrafted features and the features obtained using the subspace pooling technique. The proposed method is evaluated on the large publicly available CogAge dataset, which contains the instances of both atomic and composite activities. The data is recorded using three unobtrusive wearable devices: smartphone, smartwatch, and smart glasses. We also investigated the performance evaluation of different classification algorithms to recognize the composite activities. The proposed method achieved 79% and 62.8% average recognition accuracies using the handcrafted features and the features obtained using subspace pooling technique, respectively. The recognition results of the proposed technique and their comparison with the existing state-of-the-art techniques confirm its effectiveness.

Download Full-text

Human Activity Recognition Based on Residual Network and BiLSTM

Sensors ◽

10.3390/s22020635 ◽

2022 ◽

Vol 22 (2) ◽

pp. 635

Author(s):

Yong Li ◽

Luping Wang

Keyword(s):

Deep Learning ◽

Activity Recognition ◽

Human Activity ◽

Inertial Sensors ◽

Human Activity Recognition ◽

Activity Data ◽

Multidimensional Signals ◽

Proposed Model ◽

Public Datasets ◽

Residual Block

Due to the wide application of human activity recognition (HAR) in sports and health, a large number of HAR models based on deep learning have been proposed. However, many existing models ignore the effective extraction of spatial and temporal features of human activity data. This paper proposes a deep learning model based on residual block and bi-directional LSTM (BiLSTM). The model first extracts spatial features of multidimensional signals of MEMS inertial sensors automatically using the residual block, and then obtains the forward and backward dependencies of feature sequence using BiLSTM. Finally, the obtained features are fed into the Softmax layer to complete the human activity recognition. The optimal parameters of the model are obtained by experiments. A homemade dataset containing six common human activities of sitting, standing, walking, running, going upstairs and going downstairs is developed. The proposed model is evaluated on our dataset and two public datasets, WISDM and PAMAP2. The experimental results show that the proposed model achieves the accuracy of 96.95%, 97.32% and 97.15% on our dataset, WISDM and PAMAP2, respectively. Compared with some existing models, the proposed model has better performance and fewer parameters.

Download Full-text

Virtual Sensors for Optimal Integration of Human Activity Data

Sensors ◽

10.3390/s19092017 ◽

2019 ◽

Vol 19 (9) ◽

pp. 2017 ◽

Cited By ~ 2

Author(s):

Antonio A. Aguileta ◽

Ramon F. Brena ◽

Oscar Mayora ◽

Erik Molino-Minero-Re ◽

Luis A. Trejo

Keyword(s):

Machine Learning ◽

Activity Recognition ◽

Human Activity ◽

Human Activity Recognition ◽

Combination Method ◽

Learning Methods ◽

Activity Data ◽

Virtual Sensors ◽

Machine Learning Methods ◽

Source Of Information

Sensors are becoming more and more ubiquitous as their price and availability continue to improve, and as they are the source of information for many important tasks. However, the use of sensors has to deal with noise and failures. The lack of reliability in the sensors has led to many forms of redundancy, but simple solutions are not always the best, and the precise way in which several sensors are combined has a big impact on the overall result. In this paper, we discuss how to deal with the combination of information coming from different sensors, acting thus as “virtual sensors”, in the context of human activity recognition, in a systematic way, aiming for optimality. To achieve this goal, we construct meta-datasets containing the “signatures” of individual datasets, and apply machine-learning methods in order to distinguish when each possible combination method could be actually the best. We present specific results based on experimentation, supporting our claims of optimality.

Download Full-text

Human Activity Recognition for People with Knee Osteoarthritis—A Proof-of-Concept

Sensors ◽

10.3390/s21103381 ◽

2021 ◽

Vol 21 (10) ◽

pp. 3381

Author(s):

Jay-Shian Tan ◽

Behrouz Khabbaz Beheshti ◽

Tara Binnie ◽

Paul Davey ◽

J.P. Caneiro ◽

...

Keyword(s):

Knee Osteoarthritis ◽

Activity Recognition ◽

Human Activity ◽

Network Models ◽

Human Activity Recognition ◽

Sensor Data ◽

Measurement Unit ◽

Proof Of Concept ◽

Neural Network Models ◽

Direction Of Movement

Clinicians lack objective means for monitoring if their knee osteoarthritis patients are improving outside of the clinic (e.g., at home). Previous human activity recognition (HAR) models using wearable sensor data have only used data from healthy people and such models are typically imprecise for people who have medical conditions affecting movement. HAR models designed for people with knee osteoarthritis have classified rehabilitation exercises but not the clinically relevant activities of transitioning from a chair, negotiating stairs and walking, which are commonly monitored for improvement during therapy for this condition. Therefore, it is unknown if a HAR model trained on data from people who have knee osteoarthritis can be accurate in classifying these three clinically relevant activities. Therefore, we collected inertial measurement unit (IMU) data from 18 participants with knee osteoarthritis and trained convolutional neural network models to identify chair, stairs and walking activities, and phases. The model accuracy was 85% at the first level of classification (activity), 89–97% at the second (direction of movement) and 60–67% at the third level (phase). This study is the first proof-of-concept that an accurate HAR system can be developed using IMU data from people with knee osteoarthritis to classify activities and phases of activities.

Download Full-text