UNDERSTANDING 3D POINT CLOUD DEEP NEURAL NETWORKS BY VISUALIZATION TECHNIQUES

Abstract. In the wake of the success of Deep Learning Networks (DLN) for image recognition, object detection, shape classification and semantic segmentation, this approach has proven to be both a major breakthrough and an excellent tool in point cloud classification. However, understanding how different types of DLN achieve still lacks. In several studies the output of segmentation/classification process is compared against benchmarks, but the network is treated as a “black-box” and intermediate steps are not deeply analysed. Specifically, here the following questions are discussed: (1) what exactly did DLN learn from a point cloud? (2) On the basis of what information do DLN make decisions? To conduct such a quantitative investigation of these DLN applied to point clouds, this paper investigates the visual interpretability for the decision-making process. Firstly, we introduce a reconstruction network able to reconstruct and visualise the learned features, in order to face with question (1). Then, we propose 3DCAM to indicate the discriminative point cloud regions used by these networks to identify that category, thus dealing with question (2). Through answering the above two questions, the paper would like to offer some initial solutions to better understand the application of DLN to point clouds.

Download Full-text

Multispectral LiDAR Point Cloud Classification Using SE-PointNet++

Remote Sensing ◽

10.3390/rs13132516 ◽

2021 ◽

Vol 13 (13) ◽

pp. 2516

Author(s):

Zhuangwei Jing ◽

Haiyan Guan ◽

Peiran Zhao ◽

Dilong Li ◽

Yongtao Yu ◽

...

Keyword(s):

Point Cloud ◽

Three Dimensional ◽

Semantic Segmentation ◽

Point Clouds ◽

Cloud Classification ◽

Feature Saliency ◽

Intensity Information ◽

Multi Wavelength ◽

Data Volume ◽

Point Cloud Classification

A multispectral light detection and ranging (LiDAR) system, which simultaneously collects spatial geometric data and multi-wavelength intensity information, opens the door to three-dimensional (3-D) point cloud classification and object recognition. Because of the irregular distribution property of point clouds and the massive data volume, point cloud classification directly from multispectral LiDAR data is still challengeable and questionable. In this paper, a point-wise multispectral LiDAR point cloud classification architecture termed as SE-PointNet++ is proposed via integrating a Squeeze-and-Excitation (SE) block with an improved PointNet++ semantic segmentation network. PointNet++ extracts local features from unevenly sampled points and represents local geometrical relationships among the points through multi-scale grouping. The SE block is embedded into PointNet++ to strengthen important channels to increase feature saliency for better point cloud classification. Our SE-PointNet++ architecture has been evaluated on the Titan multispectral LiDAR test datasets and achieved an overall accuracy, a mean Intersection over Union (mIoU), an F1-score, and a Kappa coefficient of 91.16%, 60.15%, 73.14%, and 0.86, respectively. Comparative studies with five established deep learning models confirmed that our proposed SE-PointNet++ achieves promising performance in multispectral LiDAR point cloud classification tasks.

Download Full-text

Identifying roadside objects in mobile laser scanning data using image-based point cloud segmentation

Journal of Information Technology in Construction ◽

10.36680/j.itcon.2020.031 ◽

2020 ◽

Vol 25 ◽

pp. 545-560

Author(s):

Gustaf Uggla ◽

Milan Horemuz

Keyword(s):

Point Cloud ◽

Laser Scanning ◽

Large Data ◽

Semantic Segmentation ◽

Point Clouds ◽

Object Identification ◽

Data Sets ◽

Visual Appearance ◽

Segmentation Of Images ◽

Point Cloud Classification

Capturing geographic information from a mobile platform, a method known as mobile mapping, is today one of the best methods for rapid and safe data acquisition along roads and railroads. The digitalization of society and the use of information technology in the construction industry is increasing the need for structured geometric and semantic information about the built environment. This puts an emphasis on automatic object identification in data such as point clouds. Most point clouds are accompanied by RGB images, and a recent literature review showed that these are possibly underutilized for object identification. This article presents a method (image-based point cloud segmentations – IBPCS) where semantic segmentation of images is used to filter point clouds, which drastically reduces the number of points that have to be considered in object identification and allows simpler algorithms to be used. An example implementation where IBPCS is used to identify roadside game fences along a country road is provided, and the accuracy and efficiency of the method is compared to the performance of PointNet, which is a neural network designed for end-to-end point cloud classification and segmentation. The results show that our implementation of IBPCS outperforms PointNet for the given task. The strengths of IBPCS are the ability to filter point clouds based on visual appearance and that it efficiently can process large data sets. This makes the method a suitable candidate for object identification along rural roads and railroads, where the objects of interest are scattered over long distances.

Download Full-text

GEOMETRY-BASED POINT CLOUD CLASSIFICATION USING HEIGHT DISTRIBUTIONS

ISPRS Annals of Photogrammetry Remote Sensing and Spatial Information Sciences ◽

10.5194/isprs-annals-v-2-2020-259-2020 ◽

2020 ◽

Vol V-2-2020 ◽

pp. 259-266

Author(s):

F. Politz ◽

M. Sester ◽

C. Brenner

Keyword(s):

Point Cloud ◽

Laser Scanning ◽

Ground Level ◽

Semantic Segmentation ◽

Point Clouds ◽

Training Data ◽

Mountainous Regions ◽

Dense Image ◽

The Mean ◽

Point Cloud Classification

Abstract. Semantic segmentation is one of the main steps in the processing chain for Airborne Laser Scanning (ALS) point clouds, but it is also one of the most labour intensive steps, as it requires many labelled examples to train a classifier. National mapping agencies (NMAs) have to acquire nationwide ALS data every couple of years for their duties. Having point clouds cover different terrains such as flat or mountainous regions, a classifier often requires a refinement using additional data from those specific terrains. In this study, we present an algorithm, which is able to classify point clouds of similar terrain types without requiring any additional training data and which is still able to achieve overall F1-Scores of over 90% in most setups. Our algorithm uses up to two height distributions within a single cell in a rasterized point cloud. For each distribution, the empirical mean and standard deviation are calculated, which are the input for a Convolutional Neural Network (CNN) classifier. Consequently, our approach only requires the geometry of point clouds, which enables also the usage of the same network structure for point clouds from other sensor systems such as Dense Image Matching. Since the mean ground level varies with the observed area, we also examined five different normalisation methods for our input in order to reduce the ground influence on the point clouds and thus increase its transferability towards other datasets. We test our trained networks on four different tests sets with the classes’ ground, building, water, non-ground and bridge.

Download Full-text

EXPLORING ALS AND DIM DATA FOR SEMANTIC SEGMENTATION USING CNNS

ISPRS - International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences ◽

10.5194/isprs-archives-xlii-1-347-2018 ◽

2018 ◽

Vol XLII-1 ◽

pp. 347-354 ◽

Cited By ~ 5

Author(s):

F. Politz ◽

M. Sester

Keyword(s):

Point Cloud ◽

Laser Scanning ◽

Semantic Segmentation ◽

Point Clouds ◽

Good Alternative ◽

Aerial Images ◽

Learning Approaches ◽

Advantages And Disadvantages ◽

Sensing Applications ◽

High Level

Abstract. Over the past years, the algorithms for dense image matching (DIM) to obtain point clouds from aerial images improved significantly. Consequently, DIM point clouds are now a good alternative to the established Airborne Laser Scanning (ALS) point clouds for remote sensing applications. In order to derive high-level applications such as digital terrain models or city models, each point within a point cloud must be assigned a class label. Usually, ALS and DIM are labelled with different classifiers due to their varying characteristics. In this work, we explore both point cloud types in a fully convolutional encoder-decoder network, which learns to classify ALS as well as DIM point clouds. As input, we project the point clouds onto a 2D image raster plane and calculate the minimal, average and maximal height values for each raster cell. The network then differentiates between the classes ground, non-ground, building and no data. We test our network in six training setups using only one point cloud type, both point clouds as well as several transfer-learning approaches. We quantitatively and qualitatively compare all results and discuss the advantages and disadvantages of all setups. The best network achieves an overall accuracy of 96% in an ALS and 83% in a DIM test set.

Download Full-text

Classification of Point Clouds for Indoor Components Using Few Labeled Samples

Remote Sensing ◽

10.3390/rs12142181 ◽

2020 ◽

Vol 12 (14) ◽

pp. 2181

Author(s):

Hangbin Wu ◽

Huimin Yang ◽

Shengyu Huang ◽

Doudou Zeng ◽

Chun Liu ◽

...

Keyword(s):

Deep Learning ◽

Point Cloud ◽

Point Clouds ◽

Neighborhood Search ◽

Learning Methods ◽

Semantic Classification ◽

Cloud Classification ◽

Mixed Features ◽

Indoor Scenarios ◽

Point Cloud Classification

The existing deep learning methods for point cloud classification are trained using abundant labeled samples and used to test only a few samples. However, classification tasks are diverse, and not all tasks have enough labeled samples for training. In this paper, a novel point cloud classification method for indoor components using few labeled samples is proposed to solve the problem of the requirement for abundant labeled samples for training with deep learning classification methods. This method is composed of four parts: mixing samples, feature extraction, dimensionality reduction, and semantic classification. First, the few labeled point clouds are mixed with unlabeled point clouds. Next, the mixed high-dimensional features are extracted using a deep learning framework. Subsequently, a nonlinear manifold learning method is used to embed the mixed features into a low-dimensional space. Finally, the few labeled point clouds in each cluster are identified, and semantic labels are provided for unlabeled point clouds in the same cluster by a neighborhood search strategy. The validity and versatility of the proposed method were validated by different experiments and compared with three state-of-the-art deep learning methods. Our method uses fewer than 30 labeled point clouds to achieve an accuracy that is 1.89–19.67% greater than existing methods. More importantly, the experimental results suggest that this method is not only suitable for single-attribute indoor scenarios but also for comprehensive complex indoor scenarios.

Download Full-text

Virtual Disassembling of Historical Edifices: Experiments and Assessments of an Automatic Approach for Classifying Multi-Scalar Point Clouds into Architectural Elements

Sensors ◽

10.3390/s20082161 ◽

2020 ◽

Vol 20 (8) ◽

pp. 2161 ◽

Cited By ~ 4

Author(s):

Arnadi Murtiyoso ◽

Pierre Grussenmeyer

Keyword(s):

Point Cloud ◽

Laser Scanning ◽

Semantic Annotation ◽

Semantic Segmentation ◽

Point Clouds ◽

Training Data ◽

Algorithmic Approach ◽

Architectural Elements ◽

Semantic Labeling ◽

Geometric Point

3D heritage documentation has seen a surge in the past decade due to developments in reality-based 3D recording techniques. Several methods such as photogrammetry and laser scanning are becoming ubiquitous amongst architects, archaeologists, surveyors, and conservators. The main result of these methods is a 3D representation of the object in the form of point clouds. However, a solely geometric point cloud is often insufficient for further analysis, monitoring, and model predicting of the heritage object. The semantic annotation of point clouds remains an interesting research topic since traditionally it requires manual labeling and therefore a lot of time and resources. This paper proposes an automated pipeline to segment and classify multi-scalar point clouds in the case of heritage object. This is done in order to perform multi-level segmentation from the scale of a historical neighborhood up until that of architectural elements, specifically pillars and beams. The proposed workflow involves an algorithmic approach in the form of a toolbox which includes various functions covering the semantic segmentation of large point clouds into smaller, more manageable and semantically labeled clusters. The first part of the workflow will explain the segmentation and semantic labeling of heritage complexes into individual buildings, while a second part will discuss the use of the same toolbox to segment the resulting buildings further into architectural elements. The toolbox was tested on several historical buildings and showed promising results. The ultimate intention of the project is to help the manual point cloud labeling, especially when confronted with the large training data requirements of machine learning-based algorithms.

Download Full-text

A Single Point-Based Multilevel Features Fusion and Pyramid Neighborhood Optimization Method for ALS Point Cloud Classification

Applied Sciences ◽

10.3390/app9050951 ◽

2019 ◽

Vol 9 (5) ◽

pp. 951 ◽

Cited By ~ 6

Author(s):

Yong Li ◽

Guofeng Tong ◽

Xiance Du ◽

Xiang Yang ◽

Jianjun Zhang ◽

...

Keyword(s):

Point Cloud ◽

Single Point ◽

Point Clouds ◽

Optimization Method ◽

Gaussian Kernel ◽

Support Vector ◽

Angle Distribution ◽

Features Fusion ◽

Cloud Classification ◽

Point Cloud Classification

3D point cloud classification has wide applications in the field of scene understanding. Point cloud classification based on points can more accurately segment the boundary region between adjacent objects. In this paper, a point cloud classification algorithm based on a single point multilevel features fusion and pyramid neighborhood optimization are proposed for a Airborne Laser Scanning (ALS) point cloud. First, the proposed algorithm determines the neighborhood region of each point, after which the features of each single point are extracted. For the characteristics of the ALS point cloud, two new feature descriptors are proposed, i.e., a normal angle distribution histogram and latitude sampling histogram. Following this, multilevel features of a single point are constructed by multi-resolution of the point cloud and multi-neighborhood spaces. Next, the features are trained by the Support Vector Machine based on a Gaussian kernel function, and the points are classified by the trained model. Finally, a classification results optimization method based on a multi-scale pyramid neighborhood constructed by a multi-resolution point cloud is used. In the experiment, the algorithm is tested by a public dataset. The experimental results show that the proposed algorithm can effectively classify large-scale ALS point clouds. Compared with the existing algorithms, the proposed algorithm has a better classification performance.

Download Full-text

Voxel-based 3D Point Cloud Semantic Segmentation: Unsupervised Geometric and Relationship Featuring vs Deep Learning Methods

ISPRS International Journal of Geo-Information ◽

10.3390/ijgi8050213 ◽

2019 ◽

Vol 8 (5) ◽

pp. 213 ◽

Cited By ~ 19

Author(s):

Florent Poux ◽

Roland Billen

Keyword(s):

Deep Learning ◽

Point Cloud ◽

Semantic Representation ◽

Structural Connectivity ◽

Three Dimensional ◽

Strong Support ◽

Semantic Segmentation ◽

Point Clouds ◽

Learning Methods ◽

Cloud Data

Automation in point cloud data processing is central in knowledge discovery within decision-making systems. The definition of relevant features is often key for segmentation and classification, with automated workflows presenting the main challenges. In this paper, we propose a voxel-based feature engineering that better characterize point clusters and provide strong support to supervised or unsupervised classification. We provide different feature generalization levels to permit interoperable frameworks. First, we recommend a shape-based feature set (SF1) that only leverages the raw X, Y, Z attributes of any point cloud. Afterwards, we derive relationship and topology between voxel entities to obtain a three-dimensional (3D) structural connectivity feature set (SF2). Finally, we provide a knowledge-based decision tree to permit infrastructure-related classification. We study SF1/SF2 synergy on a new semantic segmentation framework for the constitution of a higher semantic representation of point clouds in relevant clusters. Finally, we benchmark the approach against novel and best-performing deep-learning methods while using the full S3DIS dataset. We highlight good performances, easy-integration, and high F1-score (> 85%) for planar-dominant classes that are comparable to state-of-the-art deep learning.

Download Full-text

Automating Parameter Learning for Classifying Terrestrial LiDAR Point Cloud Using 2D Land Cover Maps

Remote Sensing ◽

10.3390/rs10081192 ◽

2018 ◽

Vol 10 (8) ◽

pp. 1192 ◽

Cited By ~ 2

Author(s):

Chen-Chieh Feng ◽

Zhou Guo

Keyword(s):

Land Cover ◽

Point Cloud ◽

Laser Scanning ◽

Point Clouds ◽

Parameter Learning ◽

Cloud Classification ◽

Land Cover Map ◽

Point Cloud Classification ◽

Optimal Feature

The automating classification of point clouds capturing urban scenes is critical for supporting applications that demand three-dimensional (3D) models. Achieving this goal, however, is met with challenges because of the varying densities of the point clouds and the complexity of the 3D data. In order to increase the level of automation in the point cloud classification, this study proposes a segment-based parameter learning method that incorporates a two-dimensional (2D) land cover map, in which a strategy of fusing the 2D land cover map and the 3D points is first adopted to create labelled samples, and a formalized procedure is then implemented to automatically learn the following parameters of point cloud classification: the optimal scale of the neighborhood for segmentation, optimal feature set, and the training classifier. It comprises four main steps, namely: (1) point cloud segmentation; (2) sample selection; (3) optimal feature set selection; and (4) point cloud classification. Three datasets containing the point cloud data were used in this study to validate the efficiency of the proposed method. The first two datasets cover two areas of the National University of Singapore (NUS) campus while the third dataset is a widely used benchmark point cloud dataset of Oakland, Pennsylvania. The classification parameters were learned from the first dataset consisting of a terrestrial laser-scanning data and a 2D land cover map, and were subsequently used to classify both of the NUS datasets. The evaluation of the classification results showed overall accuracies of 94.07% and 91.13%, respectively, indicating that the transition of the knowledge learned from one dataset to another was satisfactory. The classification of the Oakland dataset achieved an overall accuracy of 97.08%, which further verified the transferability of the proposed approach. An experiment of the point-based classification was also conducted on the first dataset and the result was compared to that of the segment-based classification. The evaluation revealed that the overall accuracy of the segment-based classification is indeed higher than that of the point-based classification, demonstrating the advantage of the segment-based approaches.

Download Full-text

A SEMANTIC 3D POINT CLOUD SEGMENTATION APPROACH BASED ON OPTIMAL VIEW SELECTION FOR 2D IMAGE FEATURE EXTRACTION

ISPRS - International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences ◽

10.5194/isprs-archives-xlii-2-w17-9-2019 ◽

2019 ◽

Vol XLII-2/W17 ◽

pp. 9-14

Author(s):

A. Adam ◽

L. Grammatikopoulos ◽

G. Karras ◽

E. Protopapadakis ◽

K. Karantzalos

Keyword(s):

Feature Extraction ◽

Point Cloud ◽

Structural Information ◽

Semantic Segmentation ◽

Point Clouds ◽

Image Feature ◽

Hybrid Features ◽

Semantic Class ◽

Segmentation Approach ◽

Traditional Approaches

Abstract. 3D semantic segmentation is the joint task of partitioning a point cloud into semantically consistent 3D regions and assigning them to a semantic class/label. While the traditional approaches for 3D semantic segmentation typically rely only on structural information of the objects (i.e. object geometry and shape), the last years many techniques combining both visual and geometric features have emerged, taking advantage of the progress in SfM/MVS algorithms that reconstruct point clouds from multiple overlapping images. Our work describes a hybrid methodology for 3D semantic segmentation, relying both on 2D and 3D space and aiming at exploring whether image selection is critical as regards the accuracy of 3D semantic segmentation of point clouds. Experimental results are demonstrated on a free online dataset depicting city blocks around Paris. The experimental procedure not only validates that hybrid features (geometric and visual) can achieve a more accurate semantic segmentation, but also demonstrates the importance of the most appropriate view for the 2D feature extraction.

Download Full-text