Generating Adversarial Point Clouds on Multi-modal Fusion Based 3D Object Detection Model

2021 ◽  
pp. 187-203
Author(s):  
Huiying Wang ◽  
Huixin Shen ◽  
Boyang Zhang ◽  
Yu Wen ◽  
Dan Meng
Keyword(s):  
Object Detection ◽  
Point Clouds ◽  
3D Object ◽  
Detection Model ◽  
3D Object Detection

scholarly journals ZoomNet: Part-Aware Adaptive Zooming Neural Network for 3D Object Detection

2020 ◽  
Vol 34 (07) ◽  
pp. 12557-12564 ◽  
Author(s):  
Zhenbo Xu ◽  
Wei Zhang ◽  
Xiaoqing Ye ◽  
Xiao Tan ◽  
Wei Yang ◽  
...  
Keyword(s):  
Object Detection ◽  
Point Clouds ◽  
Autonomous Driving ◽  
Disparity Estimation ◽  
3D Object ◽  
Detection Model ◽  
Occluded Objects ◽  
Bounding Boxes ◽  
Detection Quality ◽  
3D Object Detection

3D object detection is an essential task in autonomous driving and robotics. Though great progress has been made, challenges remain in estimating 3D pose for distant and occluded objects. In this paper, we present a novel framework named ZoomNet for stereo imagery-based 3D detection. The pipeline of ZoomNet begins with an ordinary 2D object detection model which is used to obtain pairs of left-right bounding boxes. To further exploit the abundant texture cues in rgb images for more accurate disparity estimation, we introduce a conceptually straight-forward module – adaptive zooming, which simultaneously resizes 2D instance bounding boxes to a unified resolution and adjusts the camera intrinsic parameters accordingly. In this way, we are able to estimate higher-quality disparity maps from the resized box images then construct dense point clouds for both nearby and distant objects. Moreover, we introduce to learn part locations as complementary features to improve the resistance against occlusion and put forward the 3D fitting score to better estimate the 3D detection quality. Extensive experiments on the popular KITTI 3D detection dataset indicate ZoomNet surpasses all previous state-of-the-art methods by large margins (improved by 9.4% on APbv (IoU=0.7) over pseudo-LiDAR). Ablation study also demonstrates that our adaptive zooming strategy brings an improvement of over 10% on AP3d (IoU=0.7). In addition, since the official KITTI benchmark lacks fine-grained annotations like pixel-wise part locations, we also present our KFG dataset by augmenting KITTI with detailed instance-wise annotations including pixel-wise part location, pixel-wise disparity, etc.. Both the KFG dataset and our codes will be publicly available at https://github.com/detectRecog/ZoomNet.

scholarly journals Strong-Weak Feature Alignment for 3D Object Detection

Electronics ◽  
2021 ◽  
Vol 10 (10) ◽  
pp. 1205
Author(s):  
Zhiyu Wang ◽  
Li Wang ◽  
Bin Dai
Keyword(s):  
Object Detection ◽  
Point Clouds ◽  
Autonomous Driving ◽  
Feature Representation ◽  
Alignment Algorithm ◽  
3D Object ◽  
3D Point Clouds ◽  
Object Feature ◽  
3D Object Detection ◽  
Feature Alignment

Object detection in 3D point clouds is still a challenging task in autonomous driving. Due to the inherent occlusion and density changes of the point cloud, the data distribution of the same object will change dramatically. Especially, the incomplete data with sparsity or occlusion can not represent the complete characteristics of the object. In this paper, we proposed a novel strong–weak feature alignment algorithm between complete and incomplete objects for 3D object detection, which explores the correlations within the data. It is an end-to-end adaptive network that does not require additional data and can be easily applied to other object detection networks. Through a complete object feature extractor, we achieve a robust feature representation of the object. It serves as a guarding feature to help the incomplete object feature generator to generate effective features. The strong–weak feature alignment algorithm reduces the gap between different states of the same object and enhances the ability to represent the incomplete object. The proposed adaptation framework is validated on the KITTI object benchmark and gets about 6% improvement in detection average precision on 3D moderate difficulty compared to the basic model. The results show that our adaptation method improves the detection performance of incomplete 3D objects.

Real-Time 3D object detection using improved convolutional neural network based on image-driven point cloud

2021 ◽  
Vol 14 ◽  
Author(s):  
Zhiyong Gao ◽  
Jianhong Xiang
Keyword(s):  
Neural Network ◽  
Object Detection ◽  
Convolutional Neural Network ◽  
Real Time ◽  
Point Cloud ◽  
Point Clouds ◽  
3D Point Cloud ◽  
3D Object ◽  
3D Object Detection ◽  
Instance Segmentation

Background: While detecting the object directly from the 3D point cloud, the natural 3D patterns and invariance of 3D data are often obscure. Objective: In this work, we aimed at studying the 3D object detection from discrete, disordered and sparse 3D point clouds. Methods: The CNN is composed of the frustum sequence module, 3D instance segmentation module S-NET, 3D point cloud transformation module T-NET, and 3D boundary box estimation module E-NET. The search space of the object is determined by the frustum sequence module. The instance segmentation of the point cloud is performed by the 3D instance segmentation module. The 3D coordinates of the object are confirmed by the transformation module and the 3D bounding box estimation module. Results: Evaluated on KITTI benchmark dataset, our method outperforms the state of the art by remarkable margins while having real-time capability. Conclusion: We achieve real-time 3D object detection by proposing an improved convolutional neural network (CNN) based on image-driven point clouds.

Adversarial Training on Point Clouds for Sim-to-Real 3D Object Detection

2021 ◽  
pp. 1-1
Author(s):  
Robert Debortoli ◽  
Fuxin Li ◽  
Ashish Kapoor ◽  
Geoffrey Hollinger
Keyword(s):  
Object Detection ◽  
Point Clouds ◽  
3D Object ◽  
Adversarial Training ◽  
3D Object Detection

scholarly journals 3D Object Detection Using Scale Invariant and Feature Reweighting Networks

2019 ◽  
Vol 33 ◽  
pp. 9267-9274 ◽  
Author(s):  
Xin Zhao ◽  
Zhe Liu ◽  
Ruolan Hu ◽  
Kaiqi Huang
Keyword(s):  
Object Detection ◽  
Network Architecture ◽  
Point Clouds ◽  
Scale Invariant ◽  
3D Object ◽  
Outdoor Scenes ◽  
Indoor Scenes ◽  
Bounding Boxes ◽  
The One ◽  
3D Object Detection

3D object detection plays an important role in a large number of real-world applications. It requires us to estimate the localizations and the orientations of 3D objects in real scenes. In this paper, we present a new network architecture which focuses on utilizing the front view images and frustum point clouds to generate 3D detection results. On the one hand, a PointSIFT module is utilized to improve the performance of 3D segmentation. It can capture the information from different orientations in space and the robustness to different scale shapes. On the other hand, our network obtains the useful features and suppresses the features with less information by a SENet module. This module reweights channel features and estimates the 3D bounding boxes more effectively. Our method is evaluated on both KITTI dataset for outdoor scenes and SUN-RGBD dataset for indoor scenes. The experimental results illustrate that our method achieves better performance than the state-of-the-art methods especially when point clouds are highly sparse.

scholarly journals An LSTM Approach to Temporal 3D Object Detection in LiDAR Point Clouds

2020 ◽  
pp. 266-282
Author(s):  
Rui Huang ◽  
Wanyue Zhang ◽  
Abhijit Kundu ◽  
Caroline Pantofaru ◽  
David A. Ross ◽  
...  
Keyword(s):  
Object Detection ◽  
Point Clouds ◽  
3D Object ◽  
3D Object Detection

scholarly journals 3D object detection combining semantic and geometric features from point clouds

Cobot ◽  
2022 ◽  
Vol 1 ◽  
pp. 2
Author(s):  
Hao Peng ◽  
Guofeng Tong ◽  
Zheng Li ◽  
Yaqi Wang ◽  
Yuyuan Shao
Keyword(s):  
Object Detection ◽  
Feature Learning ◽  
Point Clouds ◽  
Difficulty Level ◽  
Geometric Feature ◽  
Geometric Features ◽  
3D Object ◽  
Point Module ◽  
Institute Of Technology ◽  
3D Object Detection

Background: 3D object detection based on point clouds in road scenes has attracted much attention recently. The voxel-based methods voxelize the scene to regular grids, which can be processed with the advanced feature learning frameworks based on convolutional layers for semantic feature learning. The point-based methods can extract the geometric feature of the point due to the coordinate reservations. The combination of the two is effective for 3D object detection. However, the current methods use a voxel-based detection head with anchors for classification and localization. Although the preset anchors cover the entire scene, it is not suitable for detection tasks with larger scenes and multiple categories of objects, due to the limitation of the voxel size. Additionally, the misalignment between the predicted confidence and proposals in the Regions of the Interest (ROI) selection bring obstacles to 3D object detection. Methods: We investigate the combination of voxel-based methods and point-based methods for 3D object detection. Additionally, a voxel-to-point module that captures semantic and geometric features is proposed in the paper. The voxel-to-point module is conducive to the detection of small-size objects and avoids the presets of anchors in the inference stage. Moreover, a confidence adjustment module with the center-boundary-aware confidence attention is proposed to solve the misalignment between the predicted confidence and proposals in the regions of the interest selection. Results: The proposed method has achieved state-of-the-art results for 3D object detection in the  Karlsruhe Institute of Technology and Toyota Technological Institute (KITTI) object detection dataset. Actually, as of September 19, 2021, our method ranked 1st in the 3D and Bird Eyes View (BEV) detection of cyclists tagged with difficulty level ‘easy’, and ranked 2nd in the 3D detection of cyclists tagged with ‘moderate’. Conclusions: We propose an end-to-end two-stage 3D object detector with voxel-to-point module and confidence adjustment module.

scholarly journals Optimization of the PointPillars network for 3D object detection in point clouds

2020 ◽  
Author(s):  
Joanna Stanisz ◽  
Konrad Lis ◽  
Tomasz Kryjak ◽  
Marek Gorgon
Keyword(s):  
Object Detection ◽  
Point Cloud ◽  
Main Part ◽  
Point Clouds ◽  
Lidar Data ◽  
Detection Accuracy ◽  
3D Object ◽  
Fold Reduction ◽  
Low Energy Consumption ◽  
3D Object Detection

In this paper we present our research on the optimisation of a deep neural network for 3D object detection in a point cloud. Techniques like quantisation and pruning available in the Brevitas and PyTorch tools were used. We performed the experiments for the PointPillars network, which offers a reasonable compromise between detection accuracy and calculation complexity. The aim of this work was to propose a variant of the network which we will ultimately implement in an FPGA device. This will allow for real-time LiDAR data processing with low energy consumption. The obtained results indicate that even a significant quantisation from 32-bit floating point to 2-bit integer in the main part of the algorithm, results in 5%-9% decrease of the detection accuracy, while allowing for almost a 16-fold reduction in size of the model.

scholarly journals A Novel Regional Fusion Network for 3D Object Detection based on RGB Images and Point Clouds

2021 ◽  
Author(s):  
Hung-Hao Chen ◽  
Chia-Hung Wang ◽  
Hsueh-Wei Chen ◽  
Pei-Yung Hsiao ◽  
Li-Chen Fu ◽  
...  
Keyword(s):  
Object Detection ◽  
Receptive Fields ◽  
Point Clouds ◽  
Detection Methods ◽  
Lidar Data ◽  
3D Object ◽  
Multi Scale ◽  
Interest Level ◽  
Rgb Images ◽  
3D Object Detection

The current fusion-based methods transform LiDAR data into bird’s eye view (BEV) representations or 3D voxel, leading to information loss and heavy computation cost of 3D convolution. In contrast, we directly consume raw point clouds and perform fusion between two modalities. We employ the concept of region proposal network to generate proposals from two streams, respectively. In order to make two sensors compensate the weakness of each other, we utilize the calibration parameters to project proposals from one stream onto the other. With the proposed multi-scale feature aggregation module, we are able to combine the extracted regionof-interest-level (RoI-level) features of RGB stream from different receptive fields, resulting in fertilizing feature richness. Experiments on KITTI dataset show that our proposed network outperforms other fusion-based methods with meaningful improvements as compared to 3D object detection methods under challenging setting.

Sign in / Sign up

Export Citation Format

Share Document