Torch-Points3D: A Modular Multi-Task Framework for Reproducible Deep Learning on 3D Point Clouds

Recently, performance improvement in facade parsing from 3D point clouds has been brought about by designing more complex network structures, which cost huge computing resources and do not take full advantage of prior knowledge of facade structure. Instead, from the perspective of data distribution, we construct a new hierarchical mesh multi-view data domain based on the characteristics of facade objects to achieve fusion of deep-learning models and prior knowledge, thereby significantly improving segmentation accuracy. We comprehensively evaluate the current mainstream method on the RueMonge 2014 data set and demonstrate the superiority of our method. The mean intersection-over-union index on the facade-parsing task reached 76.41%, which is 2.75% higher than the current best result. In addition, through comparative experiments, the reasons for the performance improvement of the proposed method are further analyzed.

Download Full-text

FWNet: Semantic Segmentation for Full-Waveform LiDAR Data Using Deep Learning

Sensors ◽

10.3390/s20123568 ◽

2020 ◽

Vol 20 (12) ◽

pp. 3568 ◽

Cited By ~ 2

Author(s):

Takayuki Shinohara ◽

Haoyi Xiu ◽

Masashi Matsuoka

Keyword(s):

Deep Learning ◽

Semantic Segmentation ◽

Point Clouds ◽

Lidar Data ◽

Global Features ◽

Waveform Data ◽

Full Waveform ◽

3D Point Clouds ◽

Waveform Lidar ◽

Full Waveform Lidar

In the computer vision field, many 3D deep learning models that directly manage 3D point clouds (proposed after PointNet) have been published. Moreover, deep learning-based-techniques have demonstrated state-of-the-art performance for supervised learning tasks on 3D point cloud data, such as classification and segmentation tasks for open datasets in competitions. Furthermore, many researchers have attempted to apply these deep learning-based techniques to 3D point clouds observed by aerial laser scanners (ALSs). However, most of these studies were developed for 3D point clouds without radiometric information. In this paper, we investigate the possibility of using a deep learning method to solve the semantic segmentation task of airborne full-waveform light detection and ranging (lidar) data that consists of geometric information and radiometric waveform data. Thus, we propose a data-driven semantic segmentation model called the full-waveform network (FWNet), which handles the waveform of full-waveform lidar data without any conversion process, such as projection onto a 2D grid or calculating handcrafted features. Our FWNet is based on a PointNet-based architecture, which can extract the local and global features of each input waveform data, along with its corresponding geographical coordinates. Subsequently, the classifier consists of 1D convolutional operational layers, which predict the class vector corresponding to the input waveform from the extracted local and global features. Our trained FWNet achieved higher scores in its recall, precision, and F1 score for unseen test data—higher scores than those of previously proposed methods in full-waveform lidar data analysis domain. Specifically, our FWNet achieved a mean recall of 0.73, a mean precision of 0.81, and a mean F1 score of 0.76. We further performed an ablation study, that is assessing the effectiveness of our proposed method, of the above-mentioned metric. Moreover, we investigated the effectiveness of our PointNet based local and global feature extraction method using the visualization of the feature vector. In this way, we have shown that our network for local and global feature extraction allows training with semantic segmentation without requiring expert knowledge on full-waveform lidar data or translation into 2D images or voxels.

Download Full-text

Comparing Machine and Deep Learning Methods for Large 3D Heritage Semantic Segmentation

ISPRS International Journal of Geo-Information ◽

10.3390/ijgi9090535 ◽

2020 ◽

Vol 9 (9) ◽

pp. 535

Author(s):

Francesca Matrone ◽

Eleonora Grilli ◽

Massimo Martini ◽

Marina Paolanti ◽

Roberto Pierdicca ◽

...

Keyword(s):

Deep Learning ◽

Cultural Heritage ◽

Laser Scanning ◽

Semantic Segmentation ◽

Point Clouds ◽

Classification Algorithms ◽

Learning Methods ◽

3D Point Clouds ◽

The Subject

In recent years semantic segmentation of 3D point clouds has been an argument that involves different fields of application. Cultural heritage scenarios have become the subject of this study mainly thanks to the development of photogrammetry and laser scanning techniques. Classification algorithms based on machine and deep learning methods allow to process huge amounts of data as 3D point clouds. In this context, the aim of this paper is to make a comparison between machine and deep learning methods for large 3D cultural heritage classification. Then, considering the best performances of both techniques, it proposes an architecture named DGCNN-Mod+3Dfeat that combines the positive aspects and advantages of these two methodologies for semantic segmentation of cultural heritage point clouds. To demonstrate the validity of our idea, several experiments from the ArCH benchmark are reported and commented.

Download Full-text

Deep Learning Applied to Vegetation Identification and Removal Using Multidimensional Aerial Data

Sensors ◽

10.3390/s20216187 ◽

2020 ◽

Vol 20 (21) ◽

pp. 6187

Author(s):

Milena F. Pinto ◽

Aurelio G. Melo ◽

Leonardo M. Honório ◽

André L. M. Marcato ◽

André G. S. Conceição ◽

...

Keyword(s):

Deep Learning ◽

Three Dimensional ◽

Point Clouds ◽

Color Filter ◽

Structural Problems ◽

3D Point Clouds ◽

Common Resource ◽

Complete Inspection ◽

Colored Point ◽

Covering Vegetation

When performing structural inspection, the generation of three-dimensional (3D) point clouds is a common resource. Those are usually generated from photogrammetry or through laser scan techniques. However, a significant drawback for complete inspection is the presence of covering vegetation, hiding possible structural problems, and making difficult the acquisition of proper object surfaces in order to provide a reliable diagnostic. Therefore, this research’s main contribution is developing an effective vegetation removal methodology through the use of a deep learning structure that is capable of identifying and extracting covering vegetation in 3D point clouds. The proposed approach uses pre and post-processing filtering stages that take advantage of colored point clouds, if they are available, or operate independently. The results showed high classification accuracy and good effectiveness when compared with similar methods in the literature. After this step, if color is available, then a color filter is applied, enhancing the results obtained. Besides, the results are analyzed in light of real Structure From Motion (SFM) reconstruction data, which further validates the proposed method. This research also presented a colored point cloud library of bushes built for the work used by other studies in the field.

Download Full-text

Point Cloud Semantic Segmentation Using a Deep Learning Framework for Cultural Heritage

Remote Sensing ◽

10.3390/rs12061005 ◽

2020 ◽

Vol 12 (6) ◽

pp. 1005 ◽

Cited By ~ 7

Author(s):

Roberto Pierdicca ◽

Marina Paolanti ◽

Francesca Matrone ◽

Massimo Martini ◽

Christian Morbidoni ◽

...

Keyword(s):

Deep Learning ◽

Cultural Heritage ◽

Point Cloud ◽

Semantic Segmentation ◽

Point Clouds ◽

Information Modeling ◽

Dynamic Graph ◽

Historical Building ◽

Architectural Elements ◽

3D Point Clouds

In the Digital Cultural Heritage (DCH) domain, the semantic segmentation of 3D Point Clouds with Deep Learning (DL) techniques can help to recognize historical architectural elements, at an adequate level of detail, and thus speed up the process of modeling of historical buildings for developing BIM models from survey data, referred to as HBIM (Historical Building Information Modeling). In this paper, we propose a DL framework for Point Cloud segmentation, which employs an improved DGCNN (Dynamic Graph Convolutional Neural Network) by adding meaningful features such as normal and colour. The approach has been applied to a newly collected DCH Dataset which is publicy available: ArCH (Architectural Cultural Heritage) Dataset. This dataset comprises 11 labeled points clouds, derived from the union of several single scans or from the integration of the latter with photogrammetric surveys. The involved scenes are both indoor and outdoor, with churches, chapels, cloisters, porticoes and loggias covered by a variety of vaults and beared by many different types of columns. They belong to different historical periods and different styles, in order to make the dataset the least possible uniform and homogeneous (in the repetition of the architectural elements) and the results as general as possible. The experiments yield high accuracy, demonstrating the effectiveness and suitability of the proposed approach.

Download Full-text

Classification of Soft-Story Buildings Using Deep Learning with Density Features Extracted from 3D Point Clouds

Journal of Computing in Civil Engineering ◽

10.1061/(asce)cp.1943-5487.0000968 ◽

2021 ◽

Vol 35 (3) ◽

pp. 04021005

Author(s):

Peng-Yu Chen ◽

Zheng Yi Wu ◽

Ertugrul Taciroglu

Keyword(s):

Deep Learning ◽

Point Clouds ◽

Soft Story ◽

3D Point Clouds

Download Full-text

Detection of Surface Defects in Logs Using Point Cloud Data and Deep Learning

International Journal of Circuits, Systems and Signal Processing ◽

10.46300/9106.2021.15.67 ◽

2021 ◽

Vol 15 ◽

pp. 607-616

Author(s):

Shengbo Liu ◽

Pengyuan Fu ◽

Lei Yan ◽

Jian Wu ◽

Yandong Zhao

Keyword(s):

Deep Learning ◽

Surface Defects ◽

Recognition Rate ◽

Point Clouds ◽

Data Set ◽

Cloud Data ◽

Wood Processing ◽

3D Point Clouds ◽

3D Data ◽

Deep Learning Network

Deep learning classification based on 3D point clouds has gained considerable research interest in recent years.The classification and quantitative analysis of wood defects are of great significance to the wood processing industry. In order to solve the problems of slow processing and low robustness of 3D data. This paper proposes an improvement based on littlepoint CNN lightweight deep learning network, adding BN layer. And based on the data set made by ourselves, the test is carried out. The new network bnlittlepoint CNN has been improved in speed and recognition rate. The correct rate of recognition for non defect log, non defect log and defect log as well as defect knot and dead knot can reach 95.6%.Finally, the "dead knot" and "loose knot" are quantitatively analyzed based on the "integral" idea, and the volume and surface area of the defect are obtained to a certain extent,the error is not more than 1.5% and the defect surface reconstruction is completed based on the triangulation idea.

Download Full-text

Deep Learning Anthropomorphic 3D Point Clouds from a Single Depth Map Camera Viewpoint

2017 IEEE International Conference on Computer Vision Workshops (ICCVW) ◽

10.1109/iccvw.2017.87 ◽

2017 ◽

Cited By ~ 1

Author(s):

Nolan Lunscher ◽

John Zelek

Keyword(s):

Deep Learning ◽

Depth Map ◽

Point Clouds ◽

3D Point Clouds

Download Full-text

Structure-Aware Convolution for 3D Point Cloud Classification and Segmentation

Remote Sensing ◽

10.3390/rs12040634 ◽

2020 ◽

Vol 12 (4) ◽

pp. 634 ◽

Cited By ~ 2

Author(s):

Lei Wang ◽

Yuxuan Liu ◽

Shenman Zhang ◽

Jixing Yan ◽

Pengjie Tao

Keyword(s):

Deep Learning ◽

Template Matching ◽

Point Cloud ◽

Structure Learning ◽

Feature Learning ◽

Point Clouds ◽

Learning Networks ◽

Geometric Structures ◽

Learning Capability ◽

3D Point Clouds

Semantic feature learning on 3D point clouds is quite challenging because of their irregular and unordered data structure. In this paper, we propose a novel structure-aware convolution (SAC) to generalize deep learning on regular grids to irregular 3D point clouds. Similar to the template-matching process of convolution on 2D images, the key of our SAC is to match the point clouds’ neighborhoods with a series of 3D kernels, where each kernel can be regarded as a “geometric template” formed by a set of learnable 3D points. Thus, the interested geometric structures of the input point clouds can be activated by the corresponding kernels. To verify the effectiveness of the proposed SAC, we embedded it into three recently developed point cloud deep learning networks (PointNet, PointNet++, and KCNet) as a lightweight module, and evaluated its performance on both classification and segmentation tasks. Experimental results show that, benefiting from the geometric structure learning capability of our SAC, all these back-end networks achieved better classification and segmentation performance (e.g., +2.77% mean accuracy for classification and +4.99% mean intersection over union (IoU) for segmentation) with few additional parameters. Furthermore, results also demonstrate that the proposed SAC is helpful in improving the robustness of networks with the constraints of geometric structures.

Download Full-text

Review: Deep Learning on 3D Point Clouds

Remote Sensing ◽

10.3390/rs12111729 ◽

2020 ◽

Vol 12 (11) ◽

pp. 1729 ◽

Cited By ~ 4

Author(s):

Saifullahi Aminu Bello ◽

Shangshu Yu ◽

Cheng Wang ◽

Jibril Muhmmad Adam ◽

Jonathan Li

Keyword(s):

Deep Learning ◽

Point Cloud ◽

General Structure ◽

Point Clouds ◽

Autonomous Driving ◽

Point Cloud Data ◽

3D Vision ◽

Cloud Data ◽

3D Point Clouds ◽

Learning Techniques

A point cloud is a set of points defined in a 3D metric space. Point clouds have become one of the most significant data formats for 3D representation and are gaining increased popularity as a result of the increased availability of acquisition devices, as well as seeing increased application in areas such as robotics, autonomous driving, and augmented and virtual reality. Deep learning is now the most powerful tool for data processing in computer vision and is becoming the most preferred technique for tasks such as classification, segmentation, and detection. While deep learning techniques are mainly applied to data with a structured grid, the point cloud, on the other hand, is unstructured. The unstructuredness of point clouds makes the use of deep learning for its direct processing very challenging. This paper contains a review of the recent state-of-the-art deep learning techniques, mainly focusing on raw point cloud data. The initial work on deep learning directly with raw point cloud data did not model local regions; therefore, subsequent approaches model local regions through sampling and grouping. More recently, several approaches have been proposed that not only model the local regions but also explore the correlation between points in the local regions. From the survey, we conclude that approaches that model local regions and take into account the correlation between points in the local regions perform better. Contrary to existing reviews, this paper provides a general structure for learning with raw point clouds, and various methods were compared based on the general structure. This work also introduces the popular 3D point cloud benchmark datasets and discusses the application of deep learning in popular 3D vision tasks, including classification, segmentation, and detection.

Download Full-text