Real-time monocular 3D perception with ORB-Features

2018 ◽  
Vol 45 (6) ◽  
pp. 776-783
Author(s):  
Babing Ji ◽  
Qixin Cao
Keyword(s):  
Real Time ◽  
3D Structure ◽  
Depth Map ◽  
Depth Estimation ◽  
Industrial Robots ◽  
3D Perception ◽  
Content Type ◽  
Structure Information ◽  
Monocular Camera ◽  
Indoor Scenarios

Purpose This paper aims to propose a new solution for real-time 3D perception with monocular camera. Most of the industrial robots’ solutions use active sensors to acquire 3D structure information, which limit their applications to indoor scenarios. By only using monocular camera, some state of art method provides up-to-scale 3D structure information, but scale information of corresponding objects is uncertain. Design/methodology/approach First, high-accuracy and scale-informed camera pose and sparse 3D map are provided by leveraging ORB-SLAM and marker. Second, for each frame captured by a camera, a specially designed depth estimation pipeline is used to compute corresponding 3D structure called depth map in real-time. Finally, depth map is integrated into volumetric scene model. A feedback module has been designed for users to visualize intermediate scene surface in real-time. Findings The system provides more robust tracking performance and compelling results. The implementation runs near 25 Hz on mainstream laptop based on parallel computation technique. Originality/value A new solution for 3D perception is using monocular camera by leveraging ORB-SLAM systems. Results in our system are visually comparable to active sensor systems such as elastic fusion in small scenes. The system is also both efficient and easy to implement, and algorithms and specific configurations involved are introduced in detail.

Real-time velocity scaling and obstacle avoidance for industrial robots using fuzzy dynamic movement primitives and virtual impedances

2018 ◽  
Vol 45 (1) ◽  
pp. 110-126 ◽  
Author(s):  
Iman Kardan ◽  
Alireza Akbarzadeh ◽  
Ali Mousavi Mohammadi
Keyword(s):  
Real Time ◽  
Obstacle Avoidance ◽  
Time Constant ◽  
Fuzzy System ◽  
Industrial Robots ◽  
Simple Method ◽  
Content Type ◽  
Dynamic Movement ◽  
Steering Mechanism ◽  
Novel Method

Purpose This paper aims to increase the safety of the robots’ operation by developing a novel method for real-time implementation of velocity scaling and obstacle avoidance as the two widely accepted safety increasing concepts. Design/methodology/approach A fuzzy version of dynamic movement primitive (DMP) framework is proposed as a real-time trajectory generator with imbedded velocity scaling capability. Time constant of the DMP system is determined by a fuzzy system which makes decisions based on the distance from obstacle to the robot’s workspace and its velocity projection toward the workspace. Moreover, a combination of the DMP framework with a human-like steering mechanism and a novel configuration of virtual impedances is proposed for real-time obstacle avoidance. Findings The results confirm the effectiveness of the proposed method in real-time implementation of the velocity scaling and obstacle avoidance concepts in different cases of single and multiple stationary obstacles as well as moving obstacles. Practical implications As the provided experiments indicate, the proposed method can effectively increase the real-time safety of the robots’ operations. This is achieved by developing a simple method with low computational loads. Originality/value This paper proposes a novel method for real-time implementation of velocity scaling and obstacle avoidance concepts. This method eliminates the need for modification of original DMP formulation. The velocity scaling concept is implemented by using a fuzzy system to adjust the DMP’s time constant. Furthermore, the novel impedance configuration makes it possible to obtain a non-oscillatory convergence to the desired path, in all degrees of freedom.

scholarly journals Fast Depth Estimation in a Single Image Using Lightweight Efficient Neural Network

Sensors ◽  
2019 ◽  
Vol 19 (20) ◽  
pp. 4434 ◽  
Author(s):  
Sangwon Kim ◽  
Jaeyeal Nam ◽  
Byoungchul Ko
Keyword(s):  
Neural Network ◽  
Real Time ◽  
Fundamental Problem ◽  
Depth Map ◽  
Ground Truth ◽  
Depth Estimation ◽  
Depth Range ◽  
Single Image ◽  
Special Equipment ◽  
Multiple Images

Depth estimation is a crucial and fundamental problem in the computer vision field. Conventional methods re-construct scenes using feature points extracted from multiple images; however, these approaches require multiple images and thus are not easily implemented in various real-time applications. Moreover, the special equipment required by hardware-based approaches using 3D sensors is expensive. Therefore, software-based methods for estimating depth from a single image using machine learning or deep learning are emerging as new alternatives. In this paper, we propose an algorithm that generates a depth map in real time using a single image and an optimized lightweight efficient neural network (L-ENet) algorithm instead of physical equipment, such as an infrared sensor or multi-view camera. Because depth values have a continuous nature and can produce locally ambiguous results, pixel-wise prediction with ordinal depth range classification was applied in this study. In addition, in our method various convolution techniques are applied to extract a dense feature map, and the number of parameters is greatly reduced by reducing the network layer. By using the proposed L-ENet algorithm, an accurate depth map can be generated from a single image quickly and, in a comparison with the ground truth, we can produce depth values closer to those of the ground truth with small errors. Experiments confirmed that the proposed L-ENet can achieve a significantly improved estimation performance over the state-of-the-art algorithms in depth estimation based on a single image.

Acquisition of welding skills in industrial robots

2015 ◽  
Vol 42 (2) ◽  
pp. 156-166 ◽  
Author(s):  
J.F. Aviles-Viñas ◽  
I. Lopez-Juarez ◽  
R. Rios-Cabrera
Keyword(s):  
Real Time ◽  
Learning Process ◽  
Vision System ◽  
Industrial Robot ◽  
Industrial Robots ◽  
Welding Parameters ◽  
Bead Geometry ◽  
Content Type ◽  
Welding Machine ◽  
Vision Algorithm

Purpose – The purpose of this paper was to propose a method based on an Artificial Neural Network and a real-time vision algorithm, to learn welding skills in industrial robotics. Design/methodology/approach – By using an optic camera to measure the bead geometry (width and height), the authors propose a real-time computer vision algorithm to extract training patterns and to enable an industrial robot to acquire and learn autonomously the welding skill. To test the approach, an industrial KUKA robot and a welding gas metal arc welding machine were used in a manufacturing cell. Findings – Several data analyses are described, showing empirically that industrial robots can acquire the skill even if the specific welding parameters are unknown. Research limitations/implications – The approach considers only stringer beads. Weave bead and bead penetration are not considered. Practical implications – With the proposed approach, it is possible to learn specific welding parameters despite of the material, type of robot or welding machine. This is due to the fact that the feedback system produces automatic measurements that are labelled prior to the learning process. Originality/value – The main contribution is that the complex learning process is reduced into an input-process-output system, where the process part is learnt automatically without human supervision, by registering the patterns with an automatically calibrated vision system.

Interfacing an industrial robot and MATLAB for predictive visual servoing

2020 ◽  
Vol ahead-of-print (ahead-of-print) ◽  
Author(s):  
Megha G. Krishnan ◽  
Abhilash T. Vijayan ◽  
Ashok S.
Keyword(s):  
Real Time ◽  
Control Algorithm ◽  
Visual Servoing ◽  
Industrial Robot ◽  
Effective Communication ◽  
Industrial Robots ◽  
Content Type ◽  
Robot Controller ◽  
Predictive Controller ◽  
And Control

Purpose Real-time implementation of sophisticated algorithms on robotic systems demands a rewarding interface between hardware and software components. Individual robot manufacturers have dedicated controllers and languages. However, robot operation would require either the knowledge of additional software or expensive add-on installations for effective communication between the robot controller and the computation software. This paper aims to present a novel method of interfacing the commercial robot controllers with most widely used simulation platform, e.g. MATLAB in real-time with a demonstration of visual predictive controller. Design/methodology/approach A remote personal computer (PC), running MATLAB, is connected with the IRC5 controller of an ABB robotic arm through the File Transfer Protocol (FTP). FTP server on the IRC5 responds to a request from an FTP client (MATLAB) on a remote computer. MATLAB provides the basic platform for programming and control algorithm development. The controlled output is transferred to the robot controller through Ethernet port as files and, thereby, the proposed scheme ensures connection and control of the robot using the control algorithms developed by the researchers without the additional cost of buying add-on packages or mastering vendor-specific programming languages. Findings New control strategies and contrivances can be developed with numerous conditions and constraints in simulation platforms. When the results are to be implemented in real-time systems, the proposed method helps to establish a simple, fast and cost-effective communication with commercial robot controllers for validating the real-time performance of the developed control algorithm. Practical implications The proposed method is used for real-time implementation of visual servo control with predictive controller, for accurate pick-and-place application with different initial conditions. The same strategy has been proven effective in supervisory control using two cameras and artificial neural network-based visual control of robotic manipulators. Originality/value This paper elaborates a real-time example using visual servoing for researchers working with industrial robots, enabling them to understand and explore the possibilities of robot communication.

Real-time control of bilateral teleoperation system with adaptive computed torque method

2017 ◽  
Vol 44 (3) ◽  
pp. 299-311 ◽  
Author(s):  
Tayfun Abut ◽  
Servet Soyguder
Keyword(s):  
Real Time ◽  
Industrial Robots ◽  
Torque Control ◽  
Position Tracking ◽  
Environment Interaction ◽  
Content Type ◽  
Teleoperation System ◽  
Six Dof ◽  
Visual Interface ◽  
Computed Torque

Purpose This paper aims to use the adaptive computed torque control (ACTC) method to eliminate the kinematic and dynamic uncertainties of master and slave robots and for the control of the system in the presence of forces originating from human and environment interaction. Design/methodology/approach In case of uncertainties in the robot parameters that are utilized in teleoperation studies and when the environment where interactions take place is not known and when there is a time delay, very serious problems take place in system performance. An adaptation rule was created to update uncertain parameters. In addition to this, disturbance observer was designed for slave robot. Lyapunov function was used to analyze the system’s position tracking and stability. A visual interface was designed to ensure that the movements of the master robot provided a visual feedback to the user. Findings In this study, a visual interface was created, and position and velocity control was achieved utilizing teleoperation; the system’s position tracking and stability were analyzed using the Lyapunov method; a simulation was applied in a real-time environment, and the performance results were analyzed. Originality/value This study consisted of both simulation and real-time studies. The teleoperation system, which was created in a laboratory environment, consisted of six-degree-of-freedom (DOF) master robots, six-DOF industrial robots and six-DOF virtual robots.

scholarly journals Real-time Estimation of Road Surfaces using Fast Monocular Depth Estimation and Normal Vector Clustering

2021 ◽  
Vol 5 (3) ◽  
pp. 206
Author(s):  
Chuho Yi ◽  
Jungwon Cho
Keyword(s):  
Real Time ◽  
Autonomous Vehicle ◽  
Depth Map ◽  
Time Estimation ◽  
Weather Conditions ◽  
Depth Estimation ◽  
Normal Vector ◽  
Single Camera ◽  
Distance Information ◽  
Lower Accuracy

Estimating a road surface or planes for applying AR(Augmented Reality) or an autonomous vehicle using a camera requires significant computation. Vision sensors have lower accuracy in distance measurement than other types of sensor, and have the difficulty that additional algorithms for estimating data must be included. However, using a camera has the advantage of being able to extract various information such as weather conditions, sign information, and road markings that are difficult to measure with other sensors. Various methods differing in sensor type and configuration have been applied. Many of the existing studies had generally researched by performing the depth estimation after the feature extraction. However, recent studies have suggested using deep learning to skip multiple processes and use a single DNN(Deep Neural Network). Also, a method using a limited single camera instead of a method using a plurality of sensors has been proposed. This paper presents a single-camera method that performs quickly and efficiently by employing a DNN to extract distance information using a single camera, and proposes a modified method for using a depth map to obtain real-time surface characteristics. First, a DNN is used to estimate the depth map, and then for quick operation, normal vector that can connect similar planes to depth is calculated, and a clustering method that can be connected is provided. An experiment is used to show the validity of our method, and to evaluate the calculation time.

scholarly journals View Synthesis: LiDAR Camera versus Depth Estimation

2021 ◽  
Author(s):  
Yupeng Xie ◽  
Sarah Fachada ◽  
Daniele Bonatto ◽  
Mehrdad Teratani ◽  
Gauthier Lafruit
Keyword(s):  
Real Time ◽  
Low Cost ◽  
Depth Map ◽  
Depth Estimation ◽  
View Synthesis ◽  
Depth Image ◽  
Estimation Methods ◽  
Quality Performance ◽  
Virtual View ◽  
Multiview Images

Depth-Image-Based Rendering (DIBR) can synthesize a virtual view image from a set of multiview images and corresponding depth maps. However, this requires an accurate depth map estimation that incurs a high compu- tational cost over several minutes per frame in DERS (MPEG-I’s Depth Estimation Reference Software) even by using a high-class computer. LiDAR cameras can thus be an alternative solution to DERS in real-time DIBR ap- plications. We compare the quality of a low-cost LiDAR camera, the Intel Realsense LiDAR L515 calibrated and configured adequately, with DERS using MPEG-I’s Reference View Synthesizer (RVS). In IV-PSNR, the LiDAR camera reaches 32.2dB view synthesis quality with a 15cm camera baseline and 40.3dB with a 2cm baseline. Though DERS outperforms the LiDAR camera with 4.2dB, the latter provides a better quality-performance trade- off. However, visual inspection demonstrates that LiDAR’s virtual views have even slightly higher quality than with DERS in most tested low-texture scene areas, except for object borders. Overall, we highly recommend using LiDAR cameras over advanced depth estimation methods (like DERS) in real-time DIBR applications. Neverthe- less, this requires delicate calibration with multiple tools further exposed in the paper.

Robust industrial robot real-time positioning system using VW-camera-space manipulation method

2014 ◽  
Vol 41 (1) ◽  
pp. 70-81 ◽  
Author(s):  
Yong Liu ◽  
Dingbing Shi ◽  
Steven Baard Skaar
Keyword(s):  
Real Time ◽  
Camera Calibration ◽  
Industrial Robot ◽  
Research Problem ◽  
Industrial Applications ◽  
Industrial Robots ◽  
Positioning System ◽  
Content Type ◽  
Time Performance ◽  
The Impact

Purpose – Vision-based positioning without camera calibration, using uncalibrated industrial robots, is a challenging research problem. To address the issue, an uncalibrated industrial robot real-time positioning system has been developed in this paper. The paper aims to discuss these issues. Design/methodology/approach – The software and hardware of this system as well as the methodology are described. Direct and inverse kinematics equations that map joint space into “camera space” are developed. The camera-space manipulation (CSM) algorithm has been employed and improved with varying weights on camera samples of the robot end effector, and the improved CSM is named VW-CSM. The experiments of robot positioning have been conducted using the traditional CSM algorithm and the varying-weight CSM (VW-CSM) algorithm, respectively, both without separate camera calibration. The impact on the accuracy and real-time performance of the system caused by different weights has been examined and discussed. Findings – The experimental results show that the accuracy and real-time performance of the system with the VW-CSM algorithm is better than the one with using the original CSM algorithm, and the impact on the accuracy and real-time performance of the system caused by different weights has been revealed. Originality/value – The accuracy and real-time performance of the system with the VW-CSM algorithm is verified. These results prove that the developed system using the VW-CSM algorithm can satisfy the requirements of most industrial applications and can be widely used in the field of industrial robots.

Real-time 3D work-piece tracking with monocular camera based on static and dynamic model libraries

2017 ◽  
Vol 37 (2) ◽  
pp. 219-229 ◽  
Author(s):  
Wenjun Zhu ◽  
Peng Wang ◽  
Rui Li ◽  
Xiangli Nie
Keyword(s):  
Real Time ◽  
High Precision ◽  
Computer Aided Design ◽  
High Efficiency ◽  
Selection Process ◽  
Work Piece ◽  
Small Range ◽  
Generation Process ◽  
Content Type ◽  
Monocular Camera

Purpose This paper aims to propose a novel real-time three-dimensional (3D) model-based work-piece tracking method with monocular camera for high-precision assembly. Tracking of 3D work-pieces with real-time speed is becoming more and more important for some industrial tasks, such as work-pieces grasping and assembly, especially in complex environment. Design/methodology/approach A three-step process method was provided, i.e. the offline static global library generation process, the online dynamic local library updating and selection process and the 3D work-piece localization process. In the offline static global library generation process, the computer-aided design models of the work-piece are used to generate a set of discrete two-dimensional (2D) hierarchical views matching libraries. In the online dynamic library updating and selection process, the previous 3D location information of the work-piece is used to predict the following location range, and a discrete matching library with a small number of 2D hierarchical views is selected from dynamic local library for localization. Then, the work-piece is localized with high-precision and real-time speed in the 3D work-piece localization process. Findings The method is suitable for the texture-less work-pieces in industrial applications. Originality/value The small range of the library enables a real-time matching. Experimental results demonstrate the high accuracy and high efficiency of the proposed method.

scholarly journals The Design and Implementation of Postprocessing for Depth Map on Real-Time Extraction System

2014 ◽  
Vol 2014 ◽  
pp. 1-10 ◽  
Author(s):  
Zhiwei Tang ◽  
Bin Li ◽  
Huosheng Li ◽  
Zheng Xu
Keyword(s):  
Real Time ◽  
Stereo Matching ◽  
Depth Map ◽  
Depth Estimation ◽  
Consistency Check ◽  
Key Technology ◽  
Design And Implementation ◽  
Depth Maps ◽  
Matching Cost

Depth estimation becomes the key technology to resolve the communications of the stereo vision. We can get the real-time depth map based on hardware, which cannot implement complicated algorithm as software, because there are some restrictions in the hardware structure. Eventually, some wrong stereo matching will inevitably exist in the process of depth estimation by hardware, such as FPGA. In order to solve the problem a postprocessing function is designed in this paper. After matching cost unique test, the both left-right and right-left consistency check solutions are implemented, respectively; then, the cavities in depth maps can be filled by right depth values on the basis of right-left consistency check solution. The results in the experiments have shown that the depth map extraction and postprocessing function can be implemented in real time in the same system; what is more, the quality of the depth maps is satisfactory.

Sign in / Sign up

Export Citation Format

Share Document