Temporal action detection based on two-stream You Only Look Once network for elderly care service robot

Human action segmentation and recognition from the continuous untrimmed sensor data stream is a challenging issue known as temporal action detection. This article provides a two-stream You Only Look Once-based network method, which fuses video and skeleton streams captured by a Kinect sensor, and our data encoding method is used to turn the spatiotemporal temporal action detection into a one-dimensional object detection problem in constantly augmented feature space. The proposed approach extracts spatial–temporal three-dimensional convolutional neural network features from video stream and view-invariant features from skeleton stream, respectively. Furthermore, these two streams are encoded into three-dimensional feature spaces, which are represented as red, green, and blue images for subsequent network input. We proposed the two-stream You Only Look Once-based networks which are capable of fusing video and skeleton information by using the processing pipeline to provide two fusion strategies, boxes-fusion or layers-fusion. We test the temporal action detection performance of two-stream You Only Look Once network based on our data set High-Speed Interplanetary Tug/Cocoon Vehicles-v1, which contains seven activities in the home environment and achieve a particularly high mean average precision. We also test our model on the public data set PKU-MMD that contains 51 activities, and our method also has a good performance on this data set. To prove that our method can work efficiently on robots, we transplanted it to the robotic platform and an online fall down detection experiment.

Download Full-text

CARE: Context-awareness for elderly care

Health and Technology ◽

10.1007/s12553-020-00512-8 ◽

2020 ◽

Author(s):

Simon Klakegg ◽

Kennedy Opoku Asare ◽

Niels van Berkel ◽

Aku Visuri ◽

Eija Ferreira ◽

...

Keyword(s):

Nursing Home ◽

Context Awareness ◽

Care Service ◽

Elderly Care ◽

Sensor Data ◽

Context Aware ◽

Integrated Technology ◽

Potential Impact ◽

Post Hoc

AbstractWe present CARE, a context-aware tool for nurses in nursing homes. The system utilises a sensors infrastructure to quantify the behaviour and wellbeing (e.g., activity, mood, social and nurse interactions) of elderly residents. The sensor data is offloaded, processed and analysed in the cloud, to generate daily and long-term summaries of residents’ health. These insights are then presented to nurses via an Android tablet application. We aim to create a tool that can assist nurses and increase their awareness to residents’ needs. We deployed CARE in a local nursing home for two months and evaluated the system through a post-hoc exploratory analysis and interviews with the nurses. The results indicate that CARE can reveal essential insights on the wellbeing of elderly residents and improve the care service. In the discussion, we reflect on our understanding and potential impact of future integrated technology in elderly care environments.

Download Full-text

A dynamic zone estimation method using cumulative voxels for autonomous driving

International Journal of Advanced Robotic Systems ◽

10.1177/1729881416687130 ◽

2017 ◽

Vol 14 (1) ◽

pp. 172988141668713 ◽

Cited By ~ 2

Author(s):

Seongjo Lee ◽

Seoungjae Cho ◽

Sungdae Sim ◽

Kiho Kwak ◽

Yong Woon Park ◽

...

Keyword(s):

High Speed ◽

Moving Objects ◽

Three Dimensional ◽

Estimation Method ◽

Autonomous Driving ◽

Significant Feature ◽

Data Set ◽

Ground Truth Data ◽

Unmanned Vehicle ◽

Space Data

Obstacle avoidance and available road identification technologies have been investigated for autonomous driving of an unmanned vehicle. In order to apply research results to autonomous driving in real environments, it is necessary to consider moving objects. This article proposes a preprocessing method to identify the dynamic zones where moving objects exist around an unmanned vehicle. This method accumulates three-dimensional points from a light detection and ranging sensor mounted on an unmanned vehicle in voxel space. Next, features are identified from the cumulative data at high speed, and zones with significant feature changes are estimated as zones where dynamic objects exist. The approach proposed in this article can identify dynamic zones even for a moving vehicle and processes data quickly using several features based on the geometry, height map and distribution of three-dimensional space data. The experiment for evaluating the performance of proposed approach was conducted using ground truth data on simulation and real environment data set.

Download Full-text

A data set with kinematic and ground reaction forces of human balance

10.7287/peerj.preprints.2865 ◽

2017 ◽

Author(s):

Damiana A dos Santos ◽

Claudiane A Fukuchi ◽

Reginaldo K Fukuchi ◽

Marcos Duarte

Keyword(s):

Center Of Pressure ◽

Reaction Force ◽

Kinematic Model ◽

Three Dimensional ◽

Force Platform ◽

Whole Body ◽

Ground Reaction Forces ◽

Data Set ◽

Public Data ◽

Reaction Forces

This article describes a public data set with the three-dimensional kinematics of the whole body and the ground reaction forces (with a dual force platform setup) of subjects standing still for 60 s in different conditions, in which the vision and the standing surface were manipulated. Twenty-seven young subjects and 22 old subjects were evaluated. The data set comprises a file with metadata plus 1,813 files with the ground reaction force (GRF) and kinematics data for the 49 subjects (three files for each of the 12 trials plus one file for each subject). The file with metadata has information about each subject’s sociocultural, demographic, and health characteristics. The files with the GRF have the data from each force platform and from the resultant GRF (including the center of pressure data). The files with the kinematics have the three-dimensional position of the 42 markers used for the kinematic model of the whole body and the 73 calculated angles. In this text, we illustrate how to access, analyze, and visualize the data set. All the data is available at Figshare (DOI: 10.6084/m9.figshare.4525082 ), and a companion Jupyter Notebook (available at https://github.com/demotu/datasets ) presents the programming code to generate analyses and other examples.

Download Full-text

A data set with kinematic and ground reaction forces of human balance

10.7287/peerj.preprints.2865v1 ◽

2017 ◽

Author(s):

Damiana A dos Santos ◽

Claudiane A Fukuchi ◽

Reginaldo K Fukuchi ◽

Marcos Duarte

Keyword(s):

Center Of Pressure ◽

Reaction Force ◽

Kinematic Model ◽

Three Dimensional ◽

Force Platform ◽

Whole Body ◽

Ground Reaction Forces ◽

Data Set ◽

Public Data ◽

Reaction Forces

Download Full-text

An improved traffic lights recognition algorithm for autonomous driving in complex scenarios

International Journal of Distributed Sensor Networks ◽

10.1177/15501477211018374 ◽

2021 ◽

Vol 17 (5) ◽

pp. 155014772110183

Author(s):

Ziyue Li ◽

Qinghua Zeng ◽

Yuchao Liu ◽

Jianye Liu ◽

Lin Li

Keyword(s):

Data Fusion ◽

Autonomous Driving ◽

Recognition Algorithm ◽

Sensor Data ◽

Optimal Size ◽

Data Set ◽

Traffic Lights ◽

Sensor Data Fusion ◽

Multi Sensor Data Fusion ◽

Public Data

Image recognition is susceptible to interference from the external environment. It is challenging to accurately and reliably recognize traffic lights in all-time and all-weather conditions. This article proposed an improved vision-based traffic lights recognition algorithm for autonomous driving, integrating deep learning and multi-sensor data fusion assist (MSDA). We introduce a method to obtain the best size of the region of interest (ROI) dynamically, including four aspects. First, based on multi-sensor data (RTK BDS/GPS, IMU, camera, and LiDAR) acquired in a normal environment, we generated a prior map that contained sufficient traffic lights information. And then, by analyzing the relationship between the error of the sensors and the optimal size of ROI, the adaptively dynamic adjustment (ADA) model was built. Furthermore, according to the multi-sensor data fusion positioning and ADA model, the optimal ROI can be obtained to predict the location of traffic lights. Finally, YOLOv4 is employed to extract and identify the image features. We evaluated our algorithm using a public data set and actual city road test at night. The experimental results demonstrate that the proposed algorithm has a relatively high accuracy rate in complex scenarios and can promote the engineering application of autonomous driving technology.

Download Full-text

Experimental Investigation to Study Convective Mixing, Spatial Uniformity, and Cycle-to-Cycle Variation During the Intake Stroke in an IC Engine

Journal of Engineering for Gas Turbines and Power ◽

10.1115/1.1286626 ◽

1999 ◽

Vol 122 (3) ◽

pp. 493-501 ◽

Cited By ~ 5

Author(s):

Woong-Chul Choi ◽

Yann G. Guezennec

Keyword(s):

Flow Field ◽

High Speed ◽

Large Scale ◽

Three Dimensional ◽

Data Set ◽

Ic Engine ◽

Engine Simulation ◽

Spatial Uniformity ◽

Intake Stroke ◽

Advective Mixing

The work described in this paper focuses on experiments to quantify the initial fuel mixing and gross fuel distribution in the cylinder during the intake stroke and its relationship to the large-scale convective flow field. The experiments were carried out in a water analog engine simulation rig, and, hence, limited to the intake stroke. The same engine head configuration was used for the three-dimensional PTV flow field and the PLIF fuel concentration measurements. High-speed CCD cameras were used to record the time evolution of the dye convection and mixing with a 1/4 deg of crank angle resolution (and were also used for the three-dimensional PTV measurements). The captured sequences of images were digitally processed to correct for background light non-uniformity and other spurious effects. The results are finely resolved evolution of the dye concentration maps in the center tumble plane. The three-dimensional PTV measurements show that the flow is characterized by a strong tumble, as well as pairs of cross-tumble, counter-rotating eddies. The results clearly show the advection of a fuel-rich zone along the wall opposite to the intake valves and later along the piston crown. It also shows that strong out-of-plane motions further contribute to the cross-stream mixing to result in a relatively uniform concentration at BDC, albeit slightly stratified by the lean fluid entering the cylinder later in the intake stroke. In addition to obtaining phase-averaged concentration maps at various crank angles throughout the intake stroke, the same data set is processed for a large number of cycle to extract spatial statistics of the cycle-to-cycle variability and spatial non-uniformity of the concentration maps. The combination of the three-dimensional PTV and PLIF measurements provides a very detailed understanding of the advective mixing properties of the intake-generated flow field. [S0742-4795(00)00103-4]

Download Full-text

TIP4.0: Industrial Internet of Things Platform for Predictive Maintenance

Sensors ◽

10.3390/s21144676 ◽

2021 ◽

Vol 21 (14) ◽

pp. 4676

Author(s):

Carlos Resende ◽

Duarte Folgado ◽

João Oliveira ◽

Bernardo Franco ◽

Waldir Moreira ◽

...

Keyword(s):

Edge Computing ◽

Machine Learning Algorithms ◽

Predictive Maintenance ◽

Sensor Data ◽

Shop Floor ◽

Processing Unit ◽

Data Set ◽

Central Processing ◽

Modular Software ◽

Public Data

Industry 4.0, allied with the growth and democratization of Artificial Intelligence (AI) and the advent of IoT, is paving the way for the complete digitization and automation of industrial processes. Maintenance is one of these processes, where the introduction of a predictive approach, as opposed to the traditional techniques, is expected to considerably improve the industry maintenance strategies with gains such as reduced downtime, improved equipment effectiveness, lower maintenance costs, increased return on assets, risk mitigation, and, ultimately, profitable growth. With predictive maintenance, dedicated sensors monitor the critical points of assets. The sensor data then feed into machine learning algorithms that can infer the asset health status and inform operators and decision-makers. With this in mind, in this paper, we present TIP4.0, a platform for predictive maintenance based on a modular software solution for edge computing gateways. TIP4.0 is built around Yocto, which makes it readily available and compliant with Commercial Off-the-Shelf (COTS) or proprietary hardware. TIP4.0 was conceived with an industry mindset with communication interfaces that allow it to serve sensor networks in the shop floor and modular software architecture that allows it to be easily adjusted to new deployment scenarios. To showcase its potential, the TIP4.0 platform was validated over COTS hardware, and we considered a public data-set for the simulation of predictive maintenance scenarios. We used a Convolution Neural Network (CNN) architecture, which provided competitive performance over the state-of-the-art approaches, while being approximately four-times and two-times faster than the uncompressed model inference on the Central Processing Unit (CPU) and Graphical Processing Unit, respectively. These results highlight the capabilities of distributed large-scale edge computing over industrial scenarios.

Download Full-text

A data set with kinematic and ground reaction forces of human balance

PeerJ ◽

10.7717/peerj.3626 ◽

2017 ◽

Vol 5 ◽

pp. e3626 ◽

Cited By ~ 5

Author(s):

Damiana A. dos Santos ◽

Claudiane A. Fukuchi ◽

Reginaldo K. Fukuchi ◽

Marcos Duarte

Keyword(s):

Center Of Pressure ◽

Reaction Force ◽

Three Dimensional ◽

Force Platform ◽

Ground Reaction Forces ◽

Data Set ◽

Public Data ◽

Human Balance ◽

Reaction Forces ◽

Young Subjects

This article describes a public data set containing the three-dimensional kinematics of the whole human body and the ground reaction forces (with a dual force platform setup) of subjects who were standing still for 60 s in different conditions, in which the subjects’ vision and the standing surface were manipulated. Twenty-seven young subjects and 22 old subjects were evaluated. The data set comprises a file with metadata plus 1,813 files with the ground reaction force (GRF) and kinematics data for the 49 subjects (three files for each of the 12 trials plus one file for each subject). The file with metadata has information about each subject’s sociocultural, demographic, and health characteristics. The files with the GRF have the data from each force platform and from the resultant GRF (including the center of pressure data). The files with the kinematics contain the three-dimensional positions of 42 markers that were placed on each subject’s body and 73 calculated joint angles. In this text, we illustrate how to access, analyze, and visualize the data set. All the data is available at Figshare (DOI:10.6084/m9.figshare.4525082), and a companion Jupyter Notebook presents programming code to access the data set, generate analyses and other examples. The availability of a public data set on the Internet that contains these measurements and information about how to access and process this data can potentially boost the research on human postural control, increase the reproducibility of studies, and be used for training and education, among other applications.

Download Full-text