Robot Vision System for Human Detection and Action Recognition

2020 ◽  
Vol 24 (3) ◽  
pp. 346-356
Author(s):  
Satoshi Hoshino ◽  
◽  
Kyohei Niimura
Keyword(s):  
Optical Flow ◽  
Real Time ◽  
Action Recognition ◽  
Autonomous Navigation ◽  
Vision System ◽  
Robot Vision ◽  
Human Detection ◽  
Time Performance ◽  
Camera Sensors ◽  
Camera Sensor

Mobile robots equipped with camera sensors are required to perceive humans and their actions for safe autonomous navigation. For simultaneous human detection and action recognition, the real-time performance of the robot vision is an important issue. In this paper, we propose a robot vision system in which original images captured by a camera sensor are described by the optical flow. These images are then used as inputs for the human and action classifications. For the image inputs, two classifiers based on convolutional neural networks are developed. Moreover, we describe a novel detector (a local search window) for clipping partial images around the target human from the original image. Since the camera sensor moves together with the robot, the camera movement has an influence on the calculation of optical flow in the image, which we address by further modifying the optical flow for changes caused by the camera movement. Through the experiments, we show that the robot vision system can detect humans and recognize the action in real time. Furthermore, we show that a moving robot can achieve human detection and action recognition by modifying the optical flow.

scholarly journals Optical Flow for Real-Time Human Detection and Action Recognition Based on CNN Classifiers

2019 ◽  
Vol 23 (4) ◽  
pp. 735-742
Author(s):  
Satoshi Hoshino ◽  
◽  
Kyohei Niimura
Keyword(s):  
Optical Flow ◽  
Real Time ◽  
Autonomous Navigation ◽  
Vision System ◽  
Robot Vision ◽  
Human Detection ◽  
Original Image ◽  
Search Window ◽  
Time Performance ◽  
Camera Sensors

Mobile robots equipped with camera sensors are required to perceive surrounding humans and their actions for safe and autonomous navigation. In this work, moving humans are the target objects. For robot vision, real-time performance is an important requirement. Therefore, we propose a robot vision system in which the original images captured by a camera sensor are described by optical flow. These images are then used as inputs to a classifier. For classifying images into human and not-human classifications, and the actions, we use a convolutional neural network (CNN), rather than coding invariant features. Moreover, we present a local search window as a novel detector for clipping partial images around target objects in an original image. Through the experiments, we ultimately show that the robot vision system is able to detect moving humans and recognize action in real time.

A Novel Robot Vision Employing a Silicon Retina

2001 ◽  
Vol 13 (6) ◽  
pp. 614-620 ◽  
Author(s):  
Kazuhiro Shimonomura ◽  
◽  
Seiji Kameda ◽  
Kazuo Ishii ◽  
Tetsuya Yagi ◽  
...  
Keyword(s):  
Real Time ◽  
Integrated Circuit ◽  
Large Scale ◽  
Analog Circuit ◽  
Vision System ◽  
Robot Vision ◽  
Illumination Condition ◽  
Output Image ◽  
Analog Cmos ◽  
Silicon Retina

A Robot vision system was designed using a silicon retina, which has been developed to mimick the parallel circuit structure of the vertebrate retina. The silicon retina used here is an analog CMOS very large-scale integrated circuit, which executes Laplacian-Gaussian like filtering on the image in real time. The processing is robust to change of illumination condition. Analog circuit modules were designed to detect the contour from the output image of the silicon retina and to binarize the output image. The images processed by the silicon retina as well as those by the analog circuit modules are received by the DOS/V-compatible mother-board with NTSC signal, which enables higher level processings using digital image processing techniques. This novel robot vision system can achieve real time and robust processings in natural illumination condition with a compact hardware and a low power consumption.

scholarly journals A Robot Vision Navigation Method using deep Learning in Edge Computing Environment

2021 ◽  
Author(s):  
Jing Li ◽  
Jialin Yin ◽  
Lin Deng
Keyword(s):  
Deep Learning ◽  
Autonomous Navigation ◽  
Vision System ◽  
Robot Vision ◽  
Image Understanding ◽  
Visual Navigation ◽  
Mechanical Equipment ◽  
Agricultural Machinery ◽  
Convolutional Network ◽  
Agricultural Robots

Abstract In the development of modern agriculture, the intelligent use of mechanical equipment is one of the main signs for agricultural modernization. Navigation technology is the key technology for agricultural machinery to control autonomously in operating environment, and it is a hotspot in the field of intelligent research on agricultural machinery. Facing the accuracy requirements of autonomous navigation for intelligent agricultural robots, this paper proposes a visual navigation algorithm for agricultural robots based on deep learning image understanding. The method first uses cascaded deep convolutional network and hybrid dilated convolution fusion method to process images collected by vision system. Then it extracts the route of processed images based on improved Hough transform algorithm. At the same time, the posture of agricultural robots is adjusted to realize autonomous navigation. Finally, our proposed method is verified by using non-interference experimental scenes and noisy experimental scenes. Experimental results show that the method can perform autonomous navigation in complex and noisy environments, and has good practicability and applicability.

scholarly journals Method for detection of unsafe actions in power field based on edge computing architecture

2021 ◽  
Vol 10 (1) ◽  
Author(s):  
Yanfang Yin ◽  
Jinjiao Lin ◽  
Nongliang Sun ◽  
Qigang Zhu ◽  
Shuaishuai Zhang ◽  
...  
Keyword(s):  
Electric Power ◽  
Real Time ◽  
Action Recognition ◽  
Electric Power Industry ◽  
Edge Computing ◽  
Classification Rule ◽  
Computing Architecture ◽  
The Real ◽  
Time Performance ◽  
Temporal Features

AbstractDue to the high risk factors in the electric power industry, the safety of power system can be improved by using the surveillance system to predict and warn the operators’ nonstandard and unsafe actions in real time. In this paper, aiming at the real-time and accuracy requirements in video intelligent surveillance, a method based on edge computing architecture is proposed to judge unsafe actions of electric power operations in time. In this method, the service of unsafe actions judgment is deployed to the edge cloud, which improves the real-time performance. In order to identify the action being executed, the end-to-end action recognition model proposed in this paper uses the Temporal Convolutional Neural Network (TCN) to extract local temporal features and a Gate Recurrent Unit (GRU) layer to extract global temporal features, which increases the accuracy of action fragment recognition. The result of action recognition is combined with the result of equipment target recognition based on the yolov3 model, and the classification rule is used to determine whether the current action is safe. Experiments show that the proposed method has better real-time performance, and the proposed action cognition is verified on the MSRAction Dataset, which improves the recognition accuracy of action segments. At the same time, the judgment results of unsafe actions also prove the effectiveness of the proposed method.

scholarly journals Research on Vision System Calibration Method of Forestry Mobile Robots

2021 ◽  
Vol 14 ◽  
pp. 1107-1114
Author(s):  
Ruting Yao ◽  
Yili Zheng ◽  
Fengjun Chen ◽  
Jian Wu ◽  
Hui Wang
Keyword(s):  
Mobile Robots ◽  
Autonomous Navigation ◽  
Time Synchronization ◽  
Vision System ◽  
Robot Vision ◽  
Nonlinear Least Squares ◽  
Calibration Method ◽  
Time Operation ◽  
Spatial Calibration ◽  
Low Efficiency

Forestry mobile robots can effectively solve the problems of low efficiency and poor safety in the forestry operation process. To realize the autonomous navigation of forestry mobile robots, a vision system consisting of a monocular camera and two-dimensional LiDAR and its calibration method are investigated. First, the adaptive algorithm is used to synchronize the data captured by the two in time. Second, a calibration board with a convex checkerboard is designed for the spatial calibration of the devices. The nonlinear least squares algorithm is employed to solve and optimize the external parameters. The experimental results show that the time synchronization precision of this calibration method is 0.0082s, the communication rate is 23Hz, and the gradient tolerance of spatial calibration is 8.55e−07. The calibration results satisfy the requirements of real-time operation and accuracy of the forestry mobile robot vision system. Furthermore, the engineering applications of the vision system are discussed herein. This study lays the foundation for further forestry mobile robots research, which is relevant to intelligent forest machines.

Sign in / Sign up

Export Citation Format

Share Document