Collapsed Building Detection Using 3D Point Clouds and Deep Learning

Collapsed buildings should be detected with the highest priority during earthquake emergency response, due to the associated fatality rates. Although deep learning-based damage detection using vertical aerial images can achieve high performance, as depth information cannot be obtained, it is difficult to detect collapsed buildings when their roofs are not heavily damaged. Airborne LiDAR can efficiently obtain the 3D geometries of buildings (in the form of point clouds) and thus has greater potential to detect various collapsed buildings. However, there have been few previous studies on deep learning-based damage detection using point cloud data, due to a lack of large-scale datasets. Therefore, in this paper, we aim to develop a dataset tailored to point cloud-based building damage detection, in order to investigate the potential of point cloud data in collapsed building detection. Two types of building data are created: building roof and building patch, which contains the building and its surroundings. Comprehensive experiments are conducted under various data availability scenarios (pre–post-building patch, post-building roof, and post-building patch) with varying reference data. The pre–post scenario tries to detect damage using pre-event and post-event data, whereas post-building patch and roof only use post-event data. Damage detection is implemented using both basic and modern 3D point cloud-based deep learning algorithms. To adapt a single-input network, which can only accept one building’s data for a prediction, to the pre–post (double-input) scenario, a general extension framework is proposed. Moreover, a simple visual explanation method is proposed, in order to conduct sensitivity analyses for validating the reliability of model decisions under the post-only scenario. Finally, the generalization ability of the proposed approach is tested using buildings with different architectural styles acquired by a distinct sensor. The results show that point cloud-based methods can achieve high accuracy and are robust under training data reduction. The sensitivity analysis reveals that the trained models are able to locate roof deformations precisely, but have difficulty recognizing global damage, such as that relating to the roof inclination. Additionally, it is revealed that the model decisions are overly dependent on debris-like objects when surroundings information is available, which leads to misclassifications. By training on the developed dataset, the model can achieve moderate accuracy on another dataset with different architectural styles without additional training.

Download Full-text

Multi-Dimensional Underwater Point Cloud Detection Based on Deep Learning

Sensors ◽

10.3390/s21030884 ◽

2021 ◽

Vol 21 (3) ◽

pp. 884

Author(s):

Chia-Ming Tsai ◽

Yi-Horng Lai ◽

Yung-Da Sun ◽

Yu-Jen Chung ◽

Jau-Woei Perng

Keyword(s):

Deep Learning ◽

Point Cloud ◽

Three Dimensional ◽

Point Clouds ◽

Training Data ◽

Network Architectures ◽

Point Cloud Data ◽

Data Types ◽

Raw Data ◽

Cloud Data

Numerous sensors can obtain images or point cloud data on land, however, the rapid attenuation of electromagnetic signals and the lack of light in water have been observed to restrict sensing functions. This study expands the utilization of two- and three-dimensional detection technologies in underwater applications to detect abandoned tires. A three-dimensional acoustic sensor, the BV5000, is used in this study to collect underwater point cloud data. Some pre-processing steps are proposed to remove noise and the seabed from raw data. Point clouds are then processed to obtain two data types: a 2D image and a 3D point cloud. Deep learning methods with different dimensions are used to train the models. In the two-dimensional method, the point cloud is transferred into a bird’s eye view image. The Faster R-CNN and YOLOv3 network architectures are used to detect tires. Meanwhile, in the three-dimensional method, the point cloud associated with a tire is cut out from the raw data and is used as training data. The PointNet and PointConv network architectures are then used for tire classification. The results show that both approaches provide good accuracy.

Download Full-text

Voxel-based 3D Point Cloud Semantic Segmentation: Unsupervised Geometric and Relationship Featuring vs Deep Learning Methods

ISPRS International Journal of Geo-Information ◽

10.3390/ijgi8050213 ◽

2019 ◽

Vol 8 (5) ◽

pp. 213 ◽

Cited By ~ 19

Author(s):

Florent Poux ◽

Roland Billen

Keyword(s):

Deep Learning ◽

Point Cloud ◽

Semantic Representation ◽

Structural Connectivity ◽

Three Dimensional ◽

Strong Support ◽

Semantic Segmentation ◽

Point Clouds ◽

Learning Methods ◽

Cloud Data

Automation in point cloud data processing is central in knowledge discovery within decision-making systems. The definition of relevant features is often key for segmentation and classification, with automated workflows presenting the main challenges. In this paper, we propose a voxel-based feature engineering that better characterize point clusters and provide strong support to supervised or unsupervised classification. We provide different feature generalization levels to permit interoperable frameworks. First, we recommend a shape-based feature set (SF1) that only leverages the raw X, Y, Z attributes of any point cloud. Afterwards, we derive relationship and topology between voxel entities to obtain a three-dimensional (3D) structural connectivity feature set (SF2). Finally, we provide a knowledge-based decision tree to permit infrastructure-related classification. We study SF1/SF2 synergy on a new semantic segmentation framework for the constitution of a higher semantic representation of point clouds in relevant clusters. Finally, we benchmark the approach against novel and best-performing deep-learning methods while using the full S3DIS dataset. We highlight good performances, easy-integration, and high F1-score (> 85%) for planar-dominant classes that are comparable to state-of-the-art deep learning.

Download Full-text

Review: Deep Learning on 3D Point Clouds

Remote Sensing ◽

10.3390/rs12111729 ◽

2020 ◽

Vol 12 (11) ◽

pp. 1729 ◽

Cited By ~ 4

Author(s):

Saifullahi Aminu Bello ◽

Shangshu Yu ◽

Cheng Wang ◽

Jibril Muhmmad Adam ◽

Jonathan Li

Keyword(s):

Deep Learning ◽

Point Cloud ◽

General Structure ◽

Point Clouds ◽

Autonomous Driving ◽

Point Cloud Data ◽

3D Vision ◽

Cloud Data ◽

3D Point Clouds ◽

Learning Techniques

A point cloud is a set of points defined in a 3D metric space. Point clouds have become one of the most significant data formats for 3D representation and are gaining increased popularity as a result of the increased availability of acquisition devices, as well as seeing increased application in areas such as robotics, autonomous driving, and augmented and virtual reality. Deep learning is now the most powerful tool for data processing in computer vision and is becoming the most preferred technique for tasks such as classification, segmentation, and detection. While deep learning techniques are mainly applied to data with a structured grid, the point cloud, on the other hand, is unstructured. The unstructuredness of point clouds makes the use of deep learning for its direct processing very challenging. This paper contains a review of the recent state-of-the-art deep learning techniques, mainly focusing on raw point cloud data. The initial work on deep learning directly with raw point cloud data did not model local regions; therefore, subsequent approaches model local regions through sampling and grouping. More recently, several approaches have been proposed that not only model the local regions but also explore the correlation between points in the local regions. From the survey, we conclude that approaches that model local regions and take into account the correlation between points in the local regions perform better. Contrary to existing reviews, this paper provides a general structure for learning with raw point clouds, and various methods were compared based on the general structure. This work also introduces the popular 3D point cloud benchmark datasets and discusses the application of deep learning in popular 3D vision tasks, including classification, segmentation, and detection.

Download Full-text

Automatic Building Outline Extraction from ALS Point Clouds by Ordered Points Aided Hough Transform

Remote Sensing ◽

10.3390/rs11141727 ◽

2019 ◽

Vol 11 (14) ◽

pp. 1727 ◽

Cited By ~ 4

Author(s):

Elyta Widyaningrum ◽

Ben Gorte ◽

Roderik Lindenbergh

Keyword(s):

Hough Transform ◽

Point Cloud ◽

Point Clouds ◽

Airborne Lidar ◽

Positional Accuracy ◽

High Quality ◽

Building Roof ◽

Cloud Data ◽

Edge Points ◽

Benchmark Datasets

Many urban applications require building polygons as input. However, manual extraction from point cloud data is time- and labor-intensive. Hough transform is a well-known procedure to extract line features. Unfortunately, current Hough-based approaches lack flexibility to effectively extract outlines from arbitrary buildings. We found that available point order information is actually never used. Using ordered building edge points allows us to present a novel ordered points–aided Hough Transform (OHT) for extracting high quality building outlines from an airborne LiDAR point cloud. First, a Hough accumulator matrix is constructed based on a voting scheme in parametric line space (θ, r). The variance of angles in each column is used to determine dominant building directions. We propose a hierarchical filtering and clustering approach to obtain accurate line based on detected hotspots and ordered points. An Ordered Point List matrix consisting of ordered building edge points enables the detection of line segments of arbitrary direction, resulting in high-quality building roof polygons. We tested our method on three different datasets of different characteristics: one new dataset in Makassar, Indonesia, and two benchmark datasets in Vaihingen, Germany. To the best of our knowledge, our algorithm is the first Hough method that is highly adaptable since it works for buildings with edges of different lengths and arbitrary relative orientations. The results prove that our method delivers high completeness (between 90.1% and 96.4%) and correctness percentages (all over 96%). The positional accuracy of the building corners is between 0.2–0.57 m RMSE. The quality rate (89.6%) for the Vaihingen-B benchmark outperforms all existing state of the art methods. Other solutions for the challenging Vaihingen-A dataset are not yet available, while we achieve a quality score of 93.2%. Results with arbitrary directions are demonstrated on the complex buildings around the EYE museum in Amsterdam.

Download Full-text

Benchmarking Deep Learning Models on Point Cloud Segmentation

10.3233/faia210152 ◽

2021 ◽

Author(s):

Thanasis Zoumpekas ◽

Guillem Molina ◽

Maria Salamó ◽

Anna Puig

Keyword(s):

Deep Learning ◽

Point Cloud ◽

Performance Metrics ◽

Point Clouds ◽

Learning Models ◽

Cloud Data ◽

Point Cloud Segmentation ◽

Segmentation Analysis ◽

Intelligent Analysis ◽

Segmentation Task

Point clouds are currently used for a variety of applications, such as detection tasks in medical and geological domains. Intelligent analysis of point clouds is considered a highly computationally demanding and challenging task, especially the segmentation task among the points. Although numerous deep learning models have recently been proposed to segment point cloud data, there is no clear instruction of which exactly neural network to utilize and then incorporate into a system dealing with point cloud segmentation analysis. Besides, the majority of the developed models emphasize more on the accuracy rather than the efficiency, in order to achieve great results. Consequently, the training, validation and testing phases of the models require a great number of processing hours and a huge amount of memory. These high computational requirements are commonly difficult to deal with for many users. In this article, we analyse five state-of-the-art deep learning models for part segmentation task and give meaningful insights into the utilization of each one. We advance guidelines based on different properties, considering both learning-related metrics, such as accuracy, and system-related metrics, such as run time and memory footprint. We further propose and analyse generalized performance metrics, which facilitate the model evaluation phase in segmentation tasks allowing users to select the most appropriate approach for their context in terms of accuracy and efficiency.

Download Full-text

AN EXTRACTION METHOD FOR ROOF POINT CLOUD OF ANCIENT BUILDING USING DEEP LEARNING FRAMEWORK

ISPRS - International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences ◽

10.5194/isprs-archives-xlvi-m-1-2021-321-2021 ◽

2021 ◽

Vol XLVI-M-1-2021 ◽

pp. 321-327

Author(s):

Y. Ji ◽

Y. Dong ◽

M. Hou ◽

Y. Qi ◽

A. Li

Keyword(s):

Deep Learning ◽

Point Cloud ◽

Learning Algorithm ◽

Point Clouds ◽

Structural Features ◽

Real Point ◽

Labor Costs ◽

Dynamic Graph ◽

Cloud Data ◽

Roof Structure

Abstract. Chinese ancient architecture is a valuable heritage wealth, especially for roof that reflects the construction age, structural features and cultural connotation. Point cloud data, as a flexible representation with characteristics of fast, precise, non-contact, plays a crucial role in a variety of applications for ancient architectural heritage, such as 3D fine reconstruction, HBIM, disaster monitoring etc. However, there are still many limitations in data editing tasks that need to be worked out manually, which is time-consuming, labor-intensive and error-prone. In recent years, the theoretical advance on deep learning has stimulated the development of various domains, and digital heritage is not in exception. Whenever, deep learning algorithm need to consume a huge amount of labeled date to achieve the purpose for segmentation, resulting a actuality that high labor costs also be acquired. In this paper, inspired by the architectural style similarity between mimetic model and real building, we proposed a method supported by deep learning, which aims to give a solution for the point cloud automatic extraction of roof structure. Firstly, to generate real point cloud, Baoguang Temple, unmanned Aerial Vehicle (UAV) is presented to obtain image collections that are subsequently processed by reconstruction technology. Secondly, a modified Dynamic Graph Convolutional Neural Network (DGCNN) which can learn local features with taking advantage of an edge attention convolution is trained using simulated data and additional attributes of geometric attributes. The mimetic data is sampled from 3DMAX model surface. Finally, we try to extract roof structure of ancient building from real point clouds scenes utilizing the trained model. The experimental results show that the proposed method can extract the rooftop structure from real scene of Baoguang, which illustrates not only effectiveness of approach but also a fact that the simulated source perform potential value when real point cloud datasets are scarce.

Download Full-text

Registration and Fusion of UAV LiDAR System Sequence Images and Laser Point Clouds

Journal of Imaging Science and Technology ◽

10.2352/j.imagingsci.technol.2021.65.1.01050110.2352/j.imagingsci.technol.2021.65.1.010501 ◽

2020 ◽

Author(s):

Jiayong Yu ◽

Longchen Ma ◽

Maoyi Tian, ◽

Xiushan Lu

Keyword(s):

Point Cloud ◽

Image Data ◽

Point Clouds ◽

Optical Image ◽

Point Cloud Data ◽

Feature Points ◽

Lidar System ◽

Registration Accuracy ◽

Cloud Data ◽

Intensity Image

The unmanned aerial vehicle (UAV)-mounted mobile LiDAR system (ULS) is widely used for geomatics owing to its efficient data acquisition and convenient operation. However, due to limited carrying capacity of a UAV, sensors integrated in the ULS should be small and lightweight, which results in decrease in the density of the collected scanning points. This affects registration between image data and point cloud data. To address this issue, the authors propose a method for registering and fusing ULS sequence images and laser point clouds, wherein they convert the problem of registering point cloud data and image data into a problem of matching feature points between the two images. First, a point cloud is selected to produce an intensity image. Subsequently, the corresponding feature points of the intensity image and the optical image are matched, and exterior orientation parameters are solved using a collinear equation based on image position and orientation. Finally, the sequence images are fused with the laser point cloud, based on the Global Navigation Satellite System (GNSS) time index of the optical image, to generate a true color point cloud. The experimental results show the higher registration accuracy and fusion speed of the proposed method, thereby demonstrating its accuracy and effectiveness.

Download Full-text

Linear-Based Incremental Co-Registration of MLS and Photogrammetric Point Clouds

Remote Sensing ◽

10.3390/rs13112195 ◽

2021 ◽

Vol 13 (11) ◽

pp. 2195

Author(s):

Shiming Li ◽

Xuming Ge ◽

Shengfu Li ◽

Bo Xu ◽

Zhendong Wang

Keyword(s):

Point Cloud ◽

Laser Scanning ◽

Point Clouds ◽

Registration Method ◽

Cloud Data ◽

Linear Feature ◽

Data Registration ◽

Source Point ◽

Urban Remote Sensing ◽

Single Data

Today, mobile laser scanning and oblique photogrammetry are two standard urban remote sensing acquisition methods, and the cross-source point-cloud data obtained using these methods have significant differences and complementarity. Accurate co-registration can make up for the limitations of a single data source, but many existing registration methods face critical challenges. Therefore, in this paper, we propose a systematic incremental registration method that can successfully register MLS and photogrammetric point clouds in the presence of a large number of missing data, large variations in point density, and scale differences. The robustness of this method is due to its elimination of noise in the extracted linear features and its 2D incremental registration strategy. There are three main contributions of our work: (1) the development of an end-to-end automatic cross-source point-cloud registration method; (2) a way to effectively extract the linear feature and restore the scale; and (3) an incremental registration strategy that simplifies the complex registration process. The experimental results show that this method can successfully achieve cross-source data registration, while other methods have difficulty obtaining satisfactory registration results efficiently. Moreover, this method can be extended to more point-cloud sources.

Download Full-text

Label3DMaize: toolkit for 3D point cloud data annotation of maize shoots

GigaScience ◽

10.1093/gigascience/giab031 ◽

2021 ◽

Vol 10 (5) ◽

Author(s):

Teng Miao ◽

Weiliang Wen ◽

Yinglun Li ◽

Sheng Wu ◽

Chao Zhu ◽

...

Keyword(s):

Deep Learning ◽

Point Cloud ◽

Training Dataset ◽

Growth Stages ◽

3D Point Cloud ◽

Plant Structure ◽

Data Annotation ◽

Cloud Data ◽

Point Cloud Segmentation ◽

Different Growth Stages

Abstract Background The 3D point cloud is the most direct and effective data form for studying plant structure and morphology. In point cloud studies, the point cloud segmentation of individual plants to organs directly determines the accuracy of organ-level phenotype estimation and the reliability of the 3D plant reconstruction. However, highly accurate, automatic, and robust point cloud segmentation approaches for plants are unavailable. Thus, the high-throughput segmentation of many shoots is challenging. Although deep learning can feasibly solve this issue, software tools for 3D point cloud annotation to construct the training dataset are lacking. Results We propose a top-to-down point cloud segmentation algorithm using optimal transportation distance for maize shoots. We apply our point cloud annotation toolkit for maize shoots, Label3DMaize, to achieve semi-automatic point cloud segmentation and annotation of maize shoots at different growth stages, through a series of operations, including stem segmentation, coarse segmentation, fine segmentation, and sample-based segmentation. The toolkit takes ∼4–10 minutes to segment a maize shoot and consumes 10–20% of the total time if only coarse segmentation is required. Fine segmentation is more detailed than coarse segmentation, especially at the organ connection regions. The accuracy of coarse segmentation can reach 97.2% that of fine segmentation. Conclusion Label3DMaize integrates point cloud segmentation algorithms and manual interactive operations, realizing semi-automatic point cloud segmentation of maize shoots at different growth stages. The toolkit provides a practical data annotation tool for further online segmentation research based on deep learning and is expected to promote automatic point cloud processing of various plants.

Download Full-text

LiDAR Point Cloud Recognition and Visualization with Deep Learning for Overhead Contact Inspection

Sensors ◽

10.3390/s20216387 ◽

2020 ◽

Vol 20 (21) ◽

pp. 6387 ◽

Cited By ~ 1

Author(s):

Xiaohan Tu ◽

Cheng Xu ◽

Siping Liu ◽

Shuai Lin ◽

Lipei Chen ◽

...

Keyword(s):

Deep Learning ◽

Computational Complexity ◽

Point Cloud ◽

High Speed ◽

Point Cloud Data ◽

Embedded Devices ◽

Cloud Data ◽

Manual Inspection ◽

Order Of Magnitude ◽

Supply Systems

As overhead contact (OC) is an essential part of power supply systems in high-speed railways, it is necessary to regularly inspect and repair abnormal OC components. Relative to manual inspection, applying LiDAR (light detection and ranging) to OC inspection can improve efficiency, accuracy, and safety, but it faces challenges to efficiently and effectively segment LiDAR point cloud data and identify catenary components. Recent deep learning-based recognition methods are rarely employed to recognize OC components, because they have high computational complexity, while their accuracy needs to be improved. To track these problems, we first propose a lightweight model, RobotNet, with depthwise and pointwise convolutions and an attention module to recognize the point cloud. Second, we optimize RobotNet to accelerate its recognition speed on embedded devices using an existing compilation tool. Third, we design software to facilitate the visualization of point cloud data. Our software can not only display a large amount of point cloud data, but also visualize the details of OC components. Extensive experiments demonstrate that RobotNet recognizes OC components more accurately and efficiently than others. The inference speed of the optimized RobotNet increases by an order of magnitude. RobotNet has lower computational complexity than other studies. The visualization results also show that our recognition method is effective.

Download Full-text