2D-to-3D Projection for Monocular and Multi-View 3D Object Detection in Outdoor Scenes

2021 ◽  
Vol 12 (7) ◽  
pp. 373-384
Author(s):  
D. D. Rukhovich ◽  
Keyword(s):  
Object Detection ◽  
Detection Methods ◽  
Navigation Systems ◽  
Detection Problem ◽  
3D Object ◽  
Car Detection ◽  
Extensive Evaluation ◽  
Outdoor Scenes ◽  
Semantic Scene ◽  
3D Object Detection

In this article, we introduce the task of multi-view RGB-based 3D object detection as an end-to-end optimization problem. In a multi-view formulation of the 3D object detection problem, several images of a static scene are used to detect objects in the scene. To address the 3D object detection problem in a multi-view formulation, we propose a novel 3D object detection method named ImVoxelNet. ImVoxelNet is based on a fully convolutional neural network. Unlike existing 3D object detection methods, ImVoxelNet works directly with 3D representations and does not mediate 3D object detection through 2D object detection. The proposed method accepts multi-view inputs. The number of monocular images in each multi-view input can vary during training and inference; actually, this number might be unique for each multi-view input. Moreover, we propose to treat a single RGB image as a special case of a multi-view input. Accordingly, the proposed method can also accept monocular inputs with no modifications. Through extensive evaluation, we demonstrate that the proposed method successfully handles a variety of outdoor scenes. Specifically, it achieves state-of-the-art results in car detection on KITTI (monocular) and nuScenes (multi-view) benchmarks among all methods that accept RGB images. The proposed method operates in real-time, which makes it possible to integrate it into the navigation systems of autonomous devices. The results of this study can be used to address tasks of navigation, path planning, and semantic scene mapping.

scholarly journals A Survey on Deep Learning Based Methods and Datasets for Monocular 3D Object Detection

Electronics ◽  
2021 ◽  
Vol 10 (4) ◽  
pp. 517
Author(s):  
Seong-heum Kim ◽  
Youngbae Hwang
Keyword(s):  
Deep Learning ◽  
Object Detection ◽  
Low Cost ◽  
Detection Methods ◽  
Future Research ◽  
3D Object ◽  
Practical Applications ◽  
Depth Sensors ◽  
Significant Research ◽  
3D Object Detection

Owing to recent advancements in deep learning methods and relevant databases, it is becoming increasingly easier to recognize 3D objects using only RGB images from single viewpoints. This study investigates the major breakthroughs and current progress in deep learning-based monocular 3D object detection. For relatively low-cost data acquisition systems without depth sensors or cameras at multiple viewpoints, we first consider existing databases with 2D RGB photos and their relevant attributes. Based on this simple sensor modality for practical applications, deep learning-based monocular 3D object detection methods that overcome significant research challenges are categorized and summarized. We present the key concepts and detailed descriptions of representative single-stage and multiple-stage detection solutions. In addition, we discuss the effectiveness of the detection models on their baseline benchmarks. Finally, we explore several directions for future research on monocular 3D object detection.

scholarly journals 3D Object Detection Using Scale Invariant and Feature Reweighting Networks

2019 ◽  
Vol 33 ◽  
pp. 9267-9274 ◽  
Author(s):  
Xin Zhao ◽  
Zhe Liu ◽  
Ruolan Hu ◽  
Kaiqi Huang
Keyword(s):  
Object Detection ◽  
Network Architecture ◽  
Point Clouds ◽  
Scale Invariant ◽  
3D Object ◽  
Outdoor Scenes ◽  
Indoor Scenes ◽  
Bounding Boxes ◽  
The One ◽  
3D Object Detection

3D object detection plays an important role in a large number of real-world applications. It requires us to estimate the localizations and the orientations of 3D objects in real scenes. In this paper, we present a new network architecture which focuses on utilizing the front view images and frustum point clouds to generate 3D detection results. On the one hand, a PointSIFT module is utilized to improve the performance of 3D segmentation. It can capture the information from different orientations in space and the robustness to different scale shapes. On the other hand, our network obtains the useful features and suppresses the features with less information by a SENet module. This module reweights channel features and estimates the 3D bounding boxes more effectively. Our method is evaluated on both KITTI dataset for outdoor scenes and SUN-RGBD dataset for indoor scenes. The experimental results illustrate that our method achieves better performance than the state-of-the-art methods especially when point clouds are highly sparse.

scholarly journals A Novel Regional Fusion Network for 3D Object Detection based on RGB Images and Point Clouds

2021 ◽  
Author(s):  
Hung-Hao Chen ◽  
Chia-Hung Wang ◽  
Hsueh-Wei Chen ◽  
Pei-Yung Hsiao ◽  
Li-Chen Fu ◽  
...  
Keyword(s):  
Object Detection ◽  
Receptive Fields ◽  
Point Clouds ◽  
Detection Methods ◽  
Lidar Data ◽  
3D Object ◽  
Multi Scale ◽  
Interest Level ◽  
Rgb Images ◽  
3D Object Detection

The current fusion-based methods transform LiDAR data into bird’s eye view (BEV) representations or 3D voxel, leading to information loss and heavy computation cost of 3D convolution. In contrast, we directly consume raw point clouds and perform fusion between two modalities. We employ the concept of region proposal network to generate proposals from two streams, respectively. In order to make two sensors compensate the weakness of each other, we utilize the calibration parameters to project proposals from one stream onto the other. With the proposed multi-scale feature aggregation module, we are able to combine the extracted regionof-interest-level (RoI-level) features of RGB stream from different receptive fields, resulting in fertilizing feature richness. Experiments on KITTI dataset show that our proposed network outperforms other fusion-based methods with meaningful improvements as compared to 3D object detection methods under challenging setting.

scholarly journals A Survey on 3D Object Detection Methods for Autonomous Driving Applications

2019 ◽  
Vol 20 (10) ◽  
pp. 3782-3795 ◽  
Author(s):  
Eduardo Arnold ◽  
Omar Y. Al-Jarrah ◽  
Mehrdad Dianati ◽  
Saber Fallah ◽  
David Oxtoby ◽  
...  
Keyword(s):  
Object Detection ◽  
Autonomous Driving ◽  
Detection Methods ◽  
3D Object ◽  
3D Object Detection

scholarly journals Scale-Aware Attention-Based PillarsNet (SAPN) Based 3D Object Detection for Point Cloud

2020 ◽  
Vol 2020 ◽  
pp. 1-12
Author(s):  
Xiang Song ◽  
Weiqin Zhan ◽  
Xiaoyu Che ◽  
Huilin Jiang ◽  
Biao Yang
Keyword(s):  
Object Detection ◽  
Autonomous Navigation ◽  
Point Clouds ◽  
Detection Performance ◽  
Detection Methods ◽  
Feature Maps ◽  
3D Object ◽  
3D Point Clouds ◽  
Detection Approach ◽  
3D Object Detection

Three-dimensional object detection can provide precise positions of objects, which can be beneficial to many robotics applications, such as self-driving cars, housekeeping robots, and autonomous navigation. In this work, we focus on accurate object detection in 3D point clouds and propose a new detection pipeline called scale-aware attention-based PillarsNet (SAPN). SAPN is a one-stage 3D object detection approach similar to PointPillar. However, SAPN achieves better performance than PointPillar by introducing the following strategies. First, we extract multiresolution pillar-level features from the point clouds to make the detection approach more scale-aware. Second, a spatial-attention mechanism is used to highlight the object activations in the feature maps, which can improve detection performance. Finally, SE-attention is employed to reweight the features fed into the detection head, which performs 3D object detection in a multitask learning manner. Experiments on the KITTI benchmark show that SAPN achieved similar or better performance compared with several state-of-the-art LiDAR-based 3D detection methods. The ablation study reveals the effectiveness of each proposed strategy. Furthermore, strategies used in this work can be embedded easily into other LiDAR-based 3D detection approaches, which improve their detection performance with slight modifications.

2D-to-3D Projection for Monocular and Multi-View 3D Multi-class Object Detection in Indoor Scenes

2021 ◽  
Vol 12 (9) ◽  
pp. 459-469
Author(s):  
D. D. Rukhovich ◽  
Keyword(s):  
Path Planning ◽  
Object Detection ◽  
Mobile Robots ◽  
3D Object ◽  
Indoor Scenes ◽  
Novel Method ◽  
Semantic Scene ◽  
Almost All ◽  
3D Object Detection ◽  
2D To 3D

In this paper, we propose a novel method of joint 3D object detection and room layout estimation. The proposed method surpasses all existing methods of 3D object detection from monocular images on the indoor SUN RGB-D dataset. Moreover, the proposed method shows competitive results on the ScanNet dataset in multi-view mode. Both these datasets are collected in various residential, administrative, educational and industrial spaces, and altogether they cover almost all possible use cases. Moreover, we are the first to formulate and solve a problem of multi-class 3D object detection from multi-view inputs in indoor scenes. The proposed method can be integrated into the controlling systems of mobile robots. The results of this study can be used to address a navigation task, as well as path planning, capturing and manipulating scene objects, and semantic scene mapping.

scholarly journals Enhancing Point Features with Spatial Information for Point-Based 3D Object Detection

2021 ◽  
Vol 2021 ◽  
pp. 1-11
Author(s):  
Huaijin Liu ◽  
Jixiang Du ◽  
Yong Zhang ◽  
Hongbo Zhang
Keyword(s):  
Object Detection ◽  
Point Cloud ◽  
Spatial Information ◽  
Structural Information ◽  
Feature Representation ◽  
Detection Methods ◽  
Detection Accuracy ◽  
Long Distance ◽  
3D Object ◽  
3D Object Detection

Currently, there are many kinds of voxel-based multisensor 3D object detectors, while point-based multisensor 3D object detectors have not been fully studied. In this paper, we propose a new 3D two-stage object detection method based on point cloud and image fusion to improve the detection accuracy. To address the problem of insufficient semantic information of point cloud, we perform multiscale deep fusion of LiDAR point and camera image in a point-wise manner to enhance point features. Due to the imbalance of LiDAR points, the object point cloud in the long-distance area is sparse. We design a point cloud completion module to predict the spatial shape of objects in the candidate boxes and extract the structural information to improve the feature representation ability to further refine the boxes. The framework is evaluated on widely used KITTI and SUN-RGBD dataset. Experimental results show that our method outperforms all state-of-the-art point-based 3D object detection methods and has comparable performance to voxel-based methods as well.

Sign in / Sign up

Export Citation Format

Share Document