SFGAN: Unsupervised Generative Adversarial Learning of 3D Scene Flow from the 3D Scene Self

3D scene flow presents the 3D motion of each point in the 3D space, which forms the fundamental 3D motion perception for autonomous driving and server robots. Although the RGBD camera or LiDAR capture discrete 3D points in space, the objects and motions usually are continuous in the macro world. That is, the objects keep themselves consistent as they flow from the current frame to the next frame. Based on this insight, the Generative Adversarial Networks (GAN) is utilized to self-learn 3D scene flow with no need for ground truth. The fake point cloud of the second frame is synthesized from the predicted scene flow and the point cloud of the first frame. The adversarial training of the generator and discriminator is realized through synthesizing indistinguishable fake point cloud and discriminating the real point cloud and the synthesized fake point cloud. The experiments on KITTI scene flow dataset show that our method realizes promising results without ground truth. Just like a human observing a real-world scene, the proposed approach is capable of determining the consistency of the scene at different moments in spite of the exact flow value of each point is unknown in advance. Corresponding author(s) Email: [email protected]

Download Full-text

GENERATION OF GROUND TRUTH DATASETS FOR THE ANALYSIS OF 3D POINT CLOUDS IN URBAN SCENES ACQUIRED VIA DIFFERENT SENSORS

ISPRS - International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences ◽

10.5194/isprs-archives-xlii-3-2009-2018 ◽

2018 ◽

Vol XLII-3 ◽

pp. 2009-2015 ◽

Cited By ~ 1

Author(s):

Y. Xu ◽

Z. Sun ◽

R. Boerner ◽

T. Koch ◽

L. Hoegner ◽

...

Keyword(s):

Point Cloud ◽

Spatial Information ◽

Ground Truth ◽

Semantic Segmentation ◽

Point Clouds ◽

Urban Scenes ◽

3D Space ◽

3D Point Clouds ◽

Testing Site ◽

Voxel Grid

In this work, we report a novel way of generating ground truth dataset for analyzing point cloud from different sensors and the validation of algorithms. Instead of directly labeling large amount of 3D points requiring time consuming manual work, a multi-resolution 3D voxel grid for the testing site is generated. Then, with the help of a set of basic labeled points from the reference dataset, we can generate a 3D labeled space of the entire testing site with different resolutions. Specifically, an octree-based voxel structure is applied to voxelize the annotated reference point cloud, by which all the points are organized by 3D grids of multi-resolutions. When automatically annotating the new testing point clouds, a voting based approach is adopted to the labeled points within multiple resolution voxels, in order to assign a semantic label to the 3D space represented by the voxel. Lastly, robust line- and plane-based fast registration methods are developed for aligning point clouds obtained via various sensors. Benefiting from the labeled 3D spatial information, we can easily create new annotated 3D point clouds of different sensors of the same scene directly by considering the corresponding labels of 3D space the points located, which would be convenient for the validation and evaluation of algorithms related to point cloud interpretation and semantic segmentation.

Download Full-text

Evaluation of four point cloud similarity measures for the use in autonomous driving

at - Automatisierungstechnik ◽

10.1515/auto-2020-0140 ◽

2021 ◽

Vol 69 (6) ◽

pp. 499-510

Author(s):

Felix Berens ◽

Stefan Elser ◽

Markus Reischl

Keyword(s):

Point Cloud ◽

Similarity Measures ◽

Ground Truth ◽

Point Clouds ◽

Autonomous Driving ◽

Wasserstein Metric ◽

3D Perception ◽

Chamfer Distance ◽

Environmental Representation ◽

Ratio Measure

Abstract Measuring the similarity between point clouds is required in many areas. In autonomous driving, point clouds for 3D perception are estimated from camera images but these estimations are error-prone. Furthermore, there is a lack of measures for quality quantification using ground truth. In this paper, we derive conditions point cloud comparisons need to fulfill and accordingly evaluate the Chamfer distance, a lower bound of the Gromov Wasserstein metric, and the ratio measure. We show that the ratio measure is not affected by erroneous points and therefore introduce the new measure “average ratio”. All measures are evaluated and compared using exemplary point clouds. We discuss characteristics, advantages and drawbacks with respect to interpretability, noise resistance, environmental representation, and computation.

Download Full-text

Bounding Boxes, Segmentations and Object Coordinates: How Important is Recognition for 3D Scene Flow Estimation in Autonomous Driving Scenarios?

2017 IEEE International Conference on Computer Vision (ICCV) ◽

10.1109/iccv.2017.281 ◽

2017 ◽

Cited By ~ 38

Author(s):

Aseem Behl ◽

Omid Hosseini Jafari ◽

Siva Karthik Mustikovela ◽

Hassan Abu Alhaija ◽

Carsten Rother ◽

...

Keyword(s):

Autonomous Driving ◽

Flow Estimation ◽

Scene Flow ◽

Bounding Boxes ◽

3D Scene

Download Full-text

Synchronizing 3D point cloud from 3D scene flow estimation with 3D Lidar and RGB camera

Electronic Imaging ◽

10.2352/issn.2470-1173.2018.18.3dipm-426 ◽

2018 ◽

Vol 2018 (18) ◽

pp. 426-1-426-6

Author(s):

Hiroki Usami ◽

Hideo Saito ◽

Jun Kawai ◽

Noriko Itani

Keyword(s):

Point Cloud ◽

3D Point Cloud ◽

Flow Estimation ◽

Scene Flow ◽

3D Scene ◽

3D Lidar

Download Full-text

THERMAL TEXTURE GENERATION AND 3D MODEL RECONSTRUCTION USING SFM AND GAN

ISPRS - International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences ◽

10.5194/isprs-archives-xlii-2-519-2018 ◽

2018 ◽

Vol XLII-2 ◽

pp. 519-524 ◽

Cited By ~ 2

Author(s):

V. V. Kniaz ◽

V. A. Mizginov

Keyword(s):

Thermal Emission ◽

Ground Truth ◽

Autonomous Driving ◽

3D Models ◽

Automatic Generation ◽

Generative Adversarial Networks ◽

Visible Range ◽

Geometrical Shape ◽

Infrared Imagery ◽

Thermal Range

Realistic 3D models with textures representing thermal emission of the object are widely used in such fields as dynamic scene analysis, autonomous driving, and video surveillance. Structure from Motion (SfM) methods provide a robust approach for the generation of textured 3D models in the visible range. Still, automatic generation of 3D models from the infrared imagery is challenging due to an absence of the feature points and low sensor resolution. Recent advances in Generative Adversarial Networks (GAN) have proved that they can perform complex image-to-image transformations such as a transformation of day to night and generation of imagery in a different spectral range. In this paper, we propose a novel method for generation of realistic 3D models with thermal textures using the SfM pipeline and GAN. The proposed method uses visible range images as an input. The images are processed in two ways. Firstly, they are used for point matching and dense point cloud generation. Secondly, the images are fed into a GAN that performs the transformation from the visible range to the thermal range. We evaluate the proposed method using real infrared imagery captured with a FLIR ONE PRO camera. We generated a dataset with 2000 pairs of real images captured in thermal and visible range. The dataset is used to train the GAN network and to generate 3D models using SfM. The evaluation of the generated 3D models and infrared textures proved that they are similar to the ground truth model in both thermal emissivity and geometrical shape.

Download Full-text

Parallel Point Clouds: Hybrid Point Cloud Generation and 3D Model Enhancement via Virtual–Real Integration

Remote Sensing ◽

10.3390/rs13152868 ◽

2021 ◽

Vol 13 (15) ◽

pp. 2868

Author(s):

Yonglin Tian ◽

Xiao Wang ◽

Yu Shen ◽

Zhongzheng Guo ◽

Zilei Wang ◽

...

Keyword(s):

Point Cloud ◽

Closed Loop ◽

Real Data ◽

Point Clouds ◽

Autonomous Driving ◽

Training Model ◽

Labor Cost ◽

Open Loop ◽

Training Dataset ◽

Data Annotation

Three-dimensional information perception from point clouds is of vital importance for improving the ability of machines to understand the world, especially for autonomous driving and unmanned aerial vehicles. Data annotation for point clouds is one of the most challenging and costly tasks. In this paper, we propose a closed-loop and virtual–real interactive point cloud generation and model-upgrading framework called Parallel Point Clouds (PPCs). To our best knowledge, this is the first time that the training model has been changed from an open-loop to a closed-loop mechanism. The feedback from the evaluation results is used to update the training dataset, benefiting from the flexibility of artificial scenes. Under the framework, a point-based LiDAR simulation model is proposed, which greatly simplifies the scanning operation. Besides, a group-based placing method is put forward to integrate hybrid point clouds, via locating candidate positions for virtual objects in real scenes. Taking advantage of the CAD models and mobile LiDAR devices, two hybrid point cloud datasets, i.e., ShapeKITTI and MobilePointClouds, are built for 3D detection tasks. With almost zero labor cost on data annotation for newly added objects, the models (PointPillars) trained with ShapeKITTI and MobilePointClouds achieved 78.6% and 60.0% of the average precision of the model trained with real data on 3D detection, respectively.

Download Full-text