DRNet: A Depth-Based Regression Network for 6D Object Pose Estimation

This paper focuses on 6Dof object pose estimation from a single RGB image. We tackle this challenging problem with a two-stage optimization framework. More specifically, we first introduce a translation estimation module to provide an initial translation based on an estimated depth map. Then, a pose regression module combines the ROI (Region of Interest) and the original image to predict the rotation and refine the translation. Compared with previous end-to-end methods that directly predict rotations and translations, our method can utilize depth information as weak guidance and significantly reduce the searching space for the subsequent module. Furthermore, we design a new loss function function for symmetric objects, an approach that has handled such exceptionally difficult cases in prior works. Experiments show that our model achieves state-of-the-art object pose estimation for the YCB- video dataset (Yale-CMU-Berkeley).

Download Full-text

CrossFuNet: RGB and Depth Cross-Fusion Network for Hand Pose Estimation

Sensors ◽

10.3390/s21186095 ◽

2021 ◽

Vol 21 (18) ◽

pp. 6095

Author(s):

Xiaojing Sun ◽

Bin Wang ◽

Longxiang Huang ◽

Qian Zhang ◽

Sulei Zhu ◽

...

Keyword(s):

Pose Estimation ◽

Depth Map ◽

Depth Information ◽

Feature Maps ◽

Hand Pose Estimation ◽

Depth Sensors ◽

Key Points ◽

Rgb Images ◽

Public Datasets ◽

Hand Pose

Despite recent successes in hand pose estimation from RGB images or depth maps, inherent challenges remain. RGB-based methods suffer from heavy self-occlusions and depth ambiguity. Depth sensors rely heavily on distance and can only be used indoors, thus there are many limitations to the practical application of depth-based methods. The aforementioned challenges have inspired us to combine the two modalities to offset the shortcomings of the other. In this paper, we propose a novel RGB and depth information fusion network to improve the accuracy of 3D hand pose estimation, which is called CrossFuNet. Specifically, the RGB image and the paired depth map are input into two different subnetworks, respectively. The feature maps are fused in the fusion module in which we propose a completely new approach to combine the information from the two modalities. Then, the common method is used to regress the 3D key-points by heatmaps. We validate our model on two public datasets and the results reveal that our model outperforms the state-of-the-art methods.

Download Full-text

Planar Prior Assisted PatchMatch Multi-View Stereo

Proceedings of the AAAI Conference on Artificial Intelligence ◽

10.1609/aaai.v34i07.6940 ◽

2020 ◽

Vol 34 (07) ◽

pp. 12516-12523

Author(s):

Qingshan Xu ◽

Wenbing Tao

Keyword(s):

Graphical Model ◽

State Of The Art ◽

Depth Estimation ◽

3D Models ◽

The Other ◽

Depth Information ◽

Probabilistic Graphical Model ◽

Challenging Problem ◽

Matching Cost ◽

Planar Regions

The completeness of 3D models is still a challenging problem in multi-view stereo (MVS) due to the unreliable photometric consistency in low-textured areas. Since low-textured areas usually exhibit strong planarity, planar models are advantageous to the depth estimation of low-textured areas. On the other hand, PatchMatch multi-view stereo is very efficient for its sampling and propagation scheme. By taking advantage of planar models and PatchMatch multi-view stereo, we propose a planar prior assisted PatchMatch multi-view stereo framework in this paper. In detail, we utilize a probabilistic graphical model to embed planar models into PatchMatch multi-view stereo and contribute a novel multi-view aggregated matching cost. This novel cost takes both photometric consistency and planar compatibility into consideration, making it suited for the depth estimation of both non-planar and planar regions. Experimental results demonstrate that our method can efficiently recover the depth information of extremely low-textured areas, thus obtaining high complete 3D models and achieving state-of-the-art performance.

Download Full-text

6DoF Pose Estimation of Transparent Object from a Single RGB-D Image

Sensors ◽

10.3390/s20236790 ◽

2020 ◽

Vol 20 (23) ◽

pp. 6790

Author(s):

Chi Xu ◽

Jiale Chen ◽

Mengyang Yao ◽

Jun Zhou ◽

Lijun Zhang ◽

...

Keyword(s):

Pose Estimation ◽

Point Cloud ◽

State Of The Art ◽

Transparent Material ◽

Automatic Driving ◽

Second Stage ◽

Transparent Objects ◽

Object Pose Estimation ◽

Transparent Object ◽

Extended Point

6DoF object pose estimation is a foundation for many important applications, such as robotic grasping, automatic driving, and so on. However, it is very challenging to estimate 6DoF pose of transparent object which is commonly seen in our daily life, because the optical characteristics of transparent material lead to significant depth error which results in false estimation. To solve this problem, a two-stage approach is proposed to estimate 6DoF pose of transparent object from a single RGB-D image. In the first stage, the influence of the depth error is eliminated by transparent segmentation, surface normal recovering, and RANSAC plane estimation. In the second stage, an extended point-cloud representation is presented to accurately and efficiently estimate object pose. As far as we know, it is the first deep learning based approach which focuses on 6DoF pose estimation of transparent objects from a single RGB-D image. Experimental results show that the proposed approach can effectively estimate 6DoF pose of transparent object, and it out-performs the state-of-the-art baselines by a large margin.

Download Full-text

Color Attenuation Prior (CAP) for Single Image Dehazing

International Journal Of Engineering And Computer Science ◽

10.18535/ijecs/v7i2.10 ◽

2018 ◽

Vol 7 (02) ◽

pp. 23578-23584

Author(s):

Miss. Anjana Navale ◽

Prof. Namdev Sawant ◽

Prof. Umaji Bagal

Keyword(s):

State Of The Art ◽

Depth Map ◽

Depth Information ◽

Single Image ◽

Image Dehazing ◽

Atmospheric Scattering ◽

Haze Removal ◽

Ill Posed ◽

Single Input ◽

Scene Depth

Single image haze removal has been a challenging problem due to its ill-posed nature. In this paper, we have used a simple but powerful color attenuation prior for haze removal from a single input hazy image. By creating a linear model for modeling the scene depth of the hazy image under this novel prior and learning the parameters of the model with a supervised learning method, the depth information can be well recovered. With the depth map of the hazy image, we can easily estimate the transmission and restore the scene radiance via the atmospheric scattering model, and thus effectively remove the haze from a single image. Experimental results show that the proposed approach outperforms state-of-the-art haze removal algorithms in terms of both efficiency and the dehazing effect.

Download Full-text

Coarse-to-Fine Hand–Object Pose Estimation with Interaction-Aware Graph Convolutional Network

Sensors ◽

10.3390/s21238092 ◽

2021 ◽

Vol 21 (23) ◽

pp. 8092

Author(s):

Maomao Zhang ◽

Ao Li ◽

Honglei Liu ◽

Minghui Wang

Keyword(s):

Object Relations ◽

Pose Estimation ◽

State Of The Art ◽

Convolutional Network ◽

Specific Relation ◽

Key Factor ◽

Rgb Images ◽

Object Pose Estimation ◽

Coarse To Fine ◽

Rgb Image

The analysis of hand–object poses from RGB images is important for understanding and imitating human behavior and acts as a key factor in various applications. In this paper, we propose a novel coarse-to-fine two-stage framework for hand–object pose estimation, which explicitly models hand–object relations in 3D pose refinement rather than in the process of converting 2D poses to 3D poses. Specifically, in the coarse stage, 2D heatmaps of hand and object keypoints are obtained from RGB image and subsequently fed into pose regressor to derive coarse 3D poses. As for the fine stage, an interaction-aware graph convolutional network called InterGCN is introduced to perform pose refinement by fully leveraging the hand–object relations in 3D context. One major challenge in 3D pose refinement lies in the fact that relations between hand and object change dynamically according to different HOI scenarios. In response to this issue, we leverage both general and interaction-specific relation graphs to significantly enhance the capacity of the network to cover variations of HOI scenarios for successful 3D pose refinement. Extensive experiments demonstrate state-of-the-art performance of our approach on benchmark hand–object datasets.

Download Full-text

Marker-Less 3d Object Recognition and 6d Pose Estimation for Homogeneous Textureless Objects: An RGB-D Approach

Sensors ◽

10.3390/s20185098 ◽

2020 ◽

Vol 20 (18) ◽

pp. 5098

Author(s):

Nasim Hajari ◽

Gabriel Lugo Bustillo ◽

Harsh Sharma ◽

Irene Cheng

Keyword(s):

Object Recognition ◽

Pose Estimation ◽

Template Matching ◽

Low Cost ◽

Region Of Interest ◽

Depth Information ◽

Natural Scenes ◽

Proposed Model ◽

Industrial Assembly ◽

Rgb Images

The task of recognising an object and estimating its 6d pose in a scene has received considerable attention in recent years. The accessibility and low-cost of consumer RGB-D cameras, make object recognition and pose estimation feasible even for small industrial businesses. An example is the industrial assembly line, where a robotic arm should pick a small, textureless and mostly homogeneous object and place it in a designated location. Despite all the recent advancements of object recognition and pose estimation techniques in natural scenes, the problem remains challenging for industrial parts. In this paper, we present a framework to simultaneously recognise the object’s class and estimate its 6d pose from RGB-D data. The proposed model adapts a global approach, where an object and the Region of Interest (ROI) are first recognised from RGB images. The object’s pose is then estimated from the corresponding depth information. We train various classifiers based on extracted Histogram of Oriented Gradient (HOG) features to detect and recognize the objects. We then perform template matching on the point cloud based on surface normal and Fast Point Feature Histograms (FPFH) to estimate the pose of the object. Experimental results show that our system is quite efficient, accurate and robust to illumination and background changes, even for the challenging objects of Tless dataset.

Download Full-text

Iterative Pose Refinement for Object Pose Estimation Based on RGBD Data

Sensors ◽

10.3390/s20154114 ◽

2020 ◽

Vol 20 (15) ◽

pp. 4114

Author(s):

Shao-Kang Huang ◽

Chen-Chien Hsu ◽

Wei-Yen Wang ◽

Cheng-Hung Lin

Keyword(s):

Augmented Reality ◽

Pose Estimation ◽

State Of The Art ◽

Iterative Refinement ◽

Accurate Estimation ◽

Geometric Features ◽

Wide Range ◽

Final Output ◽

Refinement Process ◽

Object Pose Estimation

Accurate estimation of 3D object pose is highly desirable in a wide range of applications, such as robotics and augmented reality. Although significant advancement has been made for pose estimation, there is room for further improvement. Recent pose estimation systems utilize an iterative refinement process to revise the predicted pose to obtain a better final output. However, such refinement process only takes account of geometric features for pose revision during the iteration. Motivated by this approach, this paper designs a novel iterative refinement process that deals with both color and geometric features for object pose refinement. Experiments show that the proposed method is able to reach 94.74% and 93.2% in ADD(-S) metric with only 2 iterations, outperforming the state-of-the-art methods on the LINEMOD and YCB-Video datasets, respectively.

Download Full-text

Automatic Carotid Artery Detection Using Attention Layer Region-Based Convolution Neural Network

International Journal of Humanoid Robotics ◽

10.1142/s0219843619500154 ◽

2019 ◽

Vol 16 (04) ◽

pp. 1950015

Author(s):

Xiaoyan Wang ◽

Xingyu Zhong ◽

Ming Xia ◽

Weiwei Jiang ◽

Xiaojie Huang ◽

...

Keyword(s):

Carotid Artery ◽

Object Detection ◽

State Of The Art ◽

Region Of Interest ◽

Feature Maps ◽

Interactive Approach ◽

Detection Systems ◽

Art Object ◽

Layer Region ◽

Artery Detection

Localization of vessel Region of Interest (ROI) from medical images provides an interactive approach that can assist doctors in evaluating carotid artery diseases. Accurate vessel detection is a prerequisite for the following procedures, like wall segmentation, plaque identification and 3D reconstruction. Deep learning models such as CNN have been widely used in medical image processing, and achieve state-of-the-art performance. Faster R-CNN is one of the most representative and successful methods for object detection. Using outputs of feature maps in different layers has been proved to be a useful way to improve the detection performance, however, the common method is to ensemble outputs of different layers directly, and the special characteristic and different importance of each layer haven’t been considered. In this work, we introduce a new network named Attention Layer R-CNN(AL R-CNN) and use it for automatic carotid artery detection, in which we integrate a new module named Attention Layer Part (ALP) into a basic Faster R-CNN system for better assembling feature maps of different layers. Experimental results on carotid dataset show that our method surpasses other state-of-the-art object detection systems.

Download Full-text

A New Multi-Person Pose Estimation Method Using the Partitioned CenterPose Network

Applied Sciences ◽

10.3390/app11094241 ◽

2021 ◽

Vol 11 (9) ◽

pp. 4241

Author(s):

Jiahua Wu ◽

Hyo Jong Lee

Keyword(s):

Pose Estimation ◽

Human Body ◽

State Of The Art ◽

Estimation Method ◽

Bottom Up ◽

Center Point ◽

Novel Approach ◽

Body Joints

In bottom-up multi-person pose estimation, grouping joint candidates into the appropriately structured corresponding instance of a person is challenging. In this paper, a new bottom-up method, the Partitioned CenterPose (PCP) Network, is proposed to better cluster the detected joints. To achieve this goal, we propose a novel approach called Partition Pose Representation (PPR) which integrates the instance of a person and its body joints based on joint offset. PPR leverages information about the center of the human body and the offsets between that center point and the positions of the body’s joints to encode human poses accurately. To enhance the relationships between body joints, we divide the human body into five parts, and then, we generate a sub-PPR for each part. Based on this PPR, the PCP Network can detect people and their body joints simultaneously, then group all body joints according to joint offset. Moreover, an improved l1 loss is designed to more accurately measure joint offset. Using the COCO keypoints and CrowdPose datasets for testing, it was found that the performance of the proposed method is on par with that of existing state-of-the-art bottom-up methods in terms of accuracy and speed.

Download Full-text

Bottom-up and Layerwise Domain Adaptation for Pedestrian Detection in Thermal Images

ACM Transactions on Multimedia Computing Communications and Applications ◽

10.1145/3418213 ◽

2021 ◽

Vol 17 (1) ◽

pp. 1-19

Author(s):

My Kieu ◽

Andrew D. Bagdanov ◽

Marco Bertini

Keyword(s):

Domain Adaptation ◽

State Of The Art ◽

Pedestrian Detection ◽

Challenging Problem ◽

Top Down ◽

Bottom Up ◽

Security Applications ◽

Lighting Conditions ◽

Initial Layers ◽

Single Modality

Pedestrian detection is a canonical problem for safety and security applications, and it remains a challenging problem due to the highly variable lighting conditions in which pedestrians must be detected. This article investigates several domain adaptation approaches to adapt RGB-trained detectors to the thermal domain. Building on our earlier work on domain adaptation for privacy-preserving pedestrian detection, we conducted an extensive experimental evaluation comparing top-down and bottom-up domain adaptation and also propose two new bottom-up domain adaptation strategies. For top-down domain adaptation, we leverage a detector pre-trained on RGB imagery and efficiently adapt it to perform pedestrian detection in the thermal domain. Our bottom-up domain adaptation approaches include two steps: first, training an adapter segment corresponding to initial layers of the RGB-trained detector adapts to the new input distribution; then, we reconnect the adapter segment to the original RGB-trained detector for final adaptation with a top-down loss. To the best of our knowledge, our bottom-up domain adaptation approaches outperform the best-performing single-modality pedestrian detection results on KAIST and outperform the state of the art on FLIR.

Download Full-text