Label noise tolerance of deep semantic segmentation networks for extracting buildings in ultra-high-resolution aerial images of semi-built environments

<p><strong>Abstract.</strong> Recent advances in machine learning techniques for image classification have led to the development of robust approaches to both object detection and extraction. Traditional CNN architectures, such as LeNet, AlexNet and CaffeNet, usually use as input images of fixed sizes taken from objects and attempt to assign labels to those images. Another possible approach is the Fast Region-based CNN (or Fast R-CNN), which works by using two models: (i) a Region Proposal Network (RPN) which generates a set of potential Regions of Interest (RoI) in the image; and (ii) a traditional CNN which assigns labels to the proposed RoI. As an alternative, this study proposes an approach to automatic object extraction from aerial images similar to the Fast R-CNN architecture, the main difference being the use of the Simple Linear Iterative Clustering (SLIC) algorithm instead of an RPN to generate the RoI. The dataset used is composed of high-resolution aerial images and the following classes were considered: house, sport court, hangar, building, swimming pool, tree, and street/road. The proposed method can generate RoI with different sizes by running a multi-scale SLIC approach. The overall accuracy obtained for object detection was 89% and the major advantage is that the proposed method is capable of semantic segmentation by assigning a label to each selected RoI. Some of the problems encountered are related to object proximity, in which different instances appeared merged in the results.</p>

Download Full-text

Semantic Segmentation of Urban Buildings Using a High-Resolution Network (HRNet) with Channel and Spatial Attention Gates

Remote Sensing ◽

10.3390/rs13163087 ◽

2021 ◽

Vol 13 (16) ◽

pp. 3087

Author(s):

Seonkyeong Seong ◽

Jaewan Choi

Keyword(s):

Deep Learning ◽

High Resolution ◽

Spatial Attention ◽

Semantic Segmentation ◽

Aerial Images ◽

Building Extraction ◽

Learning Models ◽

Urban Buildings

In this study, building extraction in aerial images was performed using csAG-HRNet by applying HRNet-v2 in combination with channel and spatial attention gates. HRNet-v2 consists of transition and fusion processes based on subnetworks according to various resolutions. The channel and spatial attention gates were applied in the network to efficiently learn important features. A channel attention gate assigns weights in accordance with the importance of each channel, and a spatial attention gate assigns weights in accordance with the importance of each pixel position for the entire channel. In csAG-HRNet, csAG modules consisting of a channel attention gate and a spatial attention gate were applied to each subnetwork of stage and fusion modules in the HRNet-v2 network. In experiments using two datasets, it was confirmed that csAG-HRNet could minimize false detections based on the shapes of large buildings and small nonbuilding objects compared to existing deep learning models.

Download Full-text

Multi-Object Segmentation in Complex Urban Scenes from High-Resolution Remote Sensing Data

Remote Sensing ◽

10.3390/rs13183710 ◽

2021 ◽

Vol 13 (18) ◽

pp. 3710

Author(s):

Abolfazl Abdollahi ◽

Biswajeet Pradhan ◽

Nagesh Shukla ◽

Subrata Chakraborty ◽

Abdullah Alamri

Keyword(s):

Remote Sensing ◽

High Resolution ◽

Urban Areas ◽

Object Segmentation ◽

Remote Sensing Data ◽

Semantic Segmentation ◽

Aerial Images ◽

Urban Scenes ◽

Sensing Data ◽

Boundary Information

Terrestrial features extraction, such as roads and buildings from aerial images using an automatic system, has many usages in an extensive range of fields, including disaster management, change detection, land cover assessment, and urban planning. This task is commonly tough because of complex scenes, such as urban scenes, where buildings and road objects are surrounded by shadows, vehicles, trees, etc., which appear in heterogeneous forms with lower inter-class and higher intra-class contrasts. Moreover, such extraction is time-consuming and expensive to perform by human specialists manually. Deep convolutional models have displayed considerable performance for feature segmentation from remote sensing data in the recent years. However, for the large and continuous area of obstructions, most of these techniques still cannot detect road and building well. Hence, this work’s principal goal is to introduce two novel deep convolutional models based on UNet family for multi-object segmentation, such as roads and buildings from aerial imagery. We focused on buildings and road networks because these objects constitute a huge part of the urban areas. The presented models are called multi-level context gating UNet (MCG-UNet) and bi-directional ConvLSTM UNet model (BCL-UNet). The proposed methods have the same advantages as the UNet model, the mechanism of densely connected convolutions, bi-directional ConvLSTM, and squeeze and excitation module to produce the segmentation maps with a high resolution and maintain the boundary information even under complicated backgrounds. Additionally, we implemented a basic efficient loss function called boundary-aware loss (BAL) that allowed a network to concentrate on hard semantic segmentation regions, such as overlapping areas, small objects, sophisticated objects, and boundaries of objects, and produce high-quality segmentation maps. The presented networks were tested on the Massachusetts building and road datasets. The MCG-UNet improved the average F1 accuracy by 1.85%, and 1.19% and 6.67% and 5.11% compared with UNet and BCL-UNet for road and building extraction, respectively. Additionally, the presented MCG-UNet and BCL-UNet networks were compared with other state-of-the-art deep learning-based networks, and the results proved the superiority of the networks in multi-object segmentation tasks.

Download Full-text

A Dual-Path and Lightweight Convolutional Neural Network for High-Resolution Aerial Image Segmentation

ISPRS International Journal of Geo-Information ◽

10.3390/ijgi8120582 ◽

2019 ◽

Vol 8 (12) ◽

pp. 582 ◽

Cited By ~ 6

Author(s):

Gang Zhang ◽

Tao Lei ◽

Yi Cui ◽

Ping Jiang

Keyword(s):

Neural Network ◽

Image Segmentation ◽

High Resolution ◽

Convolutional Neural Network ◽

Feature Learning ◽

Semantic Segmentation ◽

Aerial Images ◽

Aerial Image ◽

Sensing Applications ◽

Edge Path

Semantic segmentation on high-resolution aerial images plays a significant role in many remote sensing applications. Although the Deep Convolutional Neural Network (DCNN) has shown great performance in this task, it still faces the following two challenges: intra-class heterogeneity and inter-class homogeneity. To overcome these two problems, a novel dual-path DCNN, which contains a spatial path and an edge path, is proposed for high-resolution aerial image segmentation. The spatial path, which combines the multi-level and global context features to encode the local and global information, is used to address the intra-class heterogeneity challenge. For inter-class homogeneity problem, a Holistically-nested Edge Detection (HED)-like edge path is employed to detect the semantic boundaries for the guidance of feature learning. Furthermore, we improve the computational efficiency of the network by employing the backbone of MobileNetV2. We enhance the performance of MobileNetV2 with two modifications: (1) replacing the standard convolution in the last four Bottleneck Residual Blocks (BRBs) with atrous convolution; and (2) removing the convolution stride of 2 in the first layer of BRBs 4 and 6. Experimental results on the ISPRS Vaihingen and Potsdam 2D labeling dataset show that the proposed DCNN achieved real-time inference speed on a single GPU card with better performance, compared with the state-of-the-art baselines.

Download Full-text

High-Resolution Aerial Imagery Semantic Labeling with Dense Pyramid Network

Sensors ◽

10.3390/s18113774 ◽

2018 ◽

Vol 18 (11) ◽

pp. 3774 ◽

Cited By ~ 9

Author(s):

Xuran Pan ◽

Lianru Gao ◽

Bing Zhang ◽

Fan Yang ◽

Wenzhi Liao

Keyword(s):

High Resolution ◽

Class Imbalance ◽

Semantic Segmentation ◽

Aerial Imagery ◽

Aerial Images ◽

Sensor Data ◽

Median Frequency ◽

Feature Maps ◽

Class Imbalance Problem ◽

Semantic Labeling

Semantic segmentation of high-resolution aerial images is of great importance in certain fields, but the increasing spatial resolution brings large intra-class variance and small inter-class differences that can lead to classification ambiguities. Based on high-level contextual features, the deep convolutional neural network (DCNN) is an effective method to deal with semantic segmentation of high-resolution aerial imagery. In this work, a novel dense pyramid network (DPN) is proposed for semantic segmentation. The network starts with group convolutions to deal with multi-sensor data in channel wise to extract feature maps of each channel separately; by doing so, more information from each channel can be preserved. This process is followed by the channel shuffle operation to enhance the representation ability of the network. Then, four densely connected convolutional blocks are utilized to both extract and take full advantage of features. The pyramid pooling module combined with two convolutional layers are set to fuse multi-resolution and multi-sensor features through an effective global scenery prior manner, producing the probability graph for each class. Moreover, the median frequency balanced focal loss is proposed to replace the standard cross entropy loss in the training phase to deal with the class imbalance problem. We evaluate the dense pyramid network on the International Society for Photogrammetry and Remote Sensing (ISPRS) Vaihingen and Potsdam 2D semantic labeling dataset, and the results demonstrate that the proposed framework exhibits better performances, compared to the state of the art baseline.

Download Full-text

Improved Anchor-Free Instance Segmentation for Building Extraction from High-Resolution Remote Sensing Images

Remote Sensing ◽

10.3390/rs12182910 ◽

2020 ◽

Vol 12 (18) ◽

pp. 2910

Author(s):

Tong Wu ◽

Yuan Hu ◽

Ling Peng ◽

Ruonan Chen

Keyword(s):

Remote Sensing ◽

High Resolution ◽

Semantic Segmentation ◽

Aerial Images ◽

Building Extraction ◽

Remote Sensing Images ◽

Highly Sensitive ◽

Segmentation Methods ◽

Speed And Accuracy ◽

Instance Segmentation

Building extraction from high-resolution remote sensing images plays a vital part in urban planning, safety supervision, geographic databases updates, and some other applications. Several researches are devoted to using convolutional neural network (CNN) to extract buildings from high-resolution satellite/aerial images. There are two major methods, one is the CNN-based semantic segmentation methods, which can not distinguish different objects of the same category and may lead to edge connection. The other one is CNN-based instance segmentation methods, which rely heavily on pre-defined anchors, and result in the highly sensitive, high computation/storage cost and imbalance between positive and negative samples. Therefore, in this paper, we propose an improved anchor-free instance segmentation method based on CenterMask with spatial and channel attention-guided mechanisms and improved effective backbone network for accurate extraction of buildings in high-resolution remote sensing images. Then we analyze the influence of different parameters and network structure on the performance of the model, and compare the performance for building extraction of Mask R-CNN, Mask Scoring R-CNN, CenterMask, and the improved CenterMask in this paper. Experimental results show that our improved CenterMask method can successfully well-balanced performance in terms of speed and accuracy, which achieves state-of-the-art performance at real-time speed.

Download Full-text

Attention-Based Context Aware Network for Semantic Comprehension of Aerial Scenery

Sensors ◽

10.3390/s21061983 ◽

2021 ◽

Vol 21 (6) ◽

pp. 1983

Author(s):

Weipeng Shi ◽

Wenhu Qin ◽

Zhonghua Yun ◽

Peng Ping ◽

Kaiyang Wu ◽

...

Keyword(s):

High Resolution ◽

Semantic Segmentation ◽

Aerial Images ◽

Aerial Image ◽

Convolutional Network ◽

Convolutional Networks ◽

Fully Convolutional Networks ◽

Semantic Labeling ◽

Autonomous Cars ◽

High Resolution Images

It is essential for researchers to have a proper interpretation of remote sensing images (RSIs) and precise semantic labeling of their component parts. Although FCN (Fully Convolutional Networks)-like deep convolutional network architectures have been widely applied in the perception of autonomous cars, there are still two challenges in the semantic segmentation of RSIs. The first is to identify details in high-resolution images with complex scenes and to solve the class-mismatch issues; the second is to capture the edge of objects finely without being confused by the surroundings. HRNET has the characteristics of maintaining high-resolution representation by fusing feature information with parallel multi-resolution convolution branches. We adopt HRNET as a backbone and propose to incorporate the Class-Oriented Region Attention Module (CRAM) and Class-Oriented Context Fusion Module (CCFM) to analyze the relationships between classes and patch regions and between classes and local or global pixels, respectively. Thus, the perception capability of the model for the detailed part in the aerial image can be enhanced. We leverage these modules to develop an end-to-end semantic segmentation model for aerial images and validate it on the ISPRS Potsdam and Vaihingen datasets. The experimental results show that our model improves the baseline accuracy and outperforms some commonly used CNN architectures.

Download Full-text

Relation Matters: Relational Context-Aware Fully Convolutional Network for Semantic Segmentation of High-Resolution Aerial Images

IEEE Transactions on Geoscience and Remote Sensing ◽

10.1109/tgrs.2020.2979552 ◽

2020 ◽

Vol 58 (11) ◽

pp. 7557-7569 ◽

Cited By ~ 1

Author(s):

Lichao Mou ◽

Yuansheng Hua ◽

Xiao Xiang Zhu

Keyword(s):

High Resolution ◽

Semantic Segmentation ◽

Aerial Images ◽

Context Aware ◽

Convolutional Network ◽

Relational Context ◽

Fully Convolutional Network

Download Full-text

CPISNet: Delving into Consistent Proposals of Instance Segmentation Network for High-Resolution Aerial Images

Remote Sensing ◽

10.3390/rs13142788 ◽

2021 ◽

Vol 13 (14) ◽

pp. 2788

Author(s):

Xiangfeng Zeng ◽

Shunjun Wei ◽

Jinshan Wei ◽

Zichen Zhou ◽

Jun Shi ◽

...

Keyword(s):

High Resolution ◽

Semantic Segmentation ◽

Aerial Images ◽

False Alarms ◽

Feature Maps ◽

Sensing Applications ◽

Multi Level ◽

Bounding Boxes ◽

Adaptive Feature Extraction ◽

Instance Segmentation

Instance segmentation of high-resolution aerial images is challenging when compared to object detection and semantic segmentation in remote sensing applications. It adopts boundary-aware mask predictions, instead of traditional bounding boxes, to locate the objects-of-interest in pixel-wise. Meanwhile, instance segmentation can distinguish the densely distributed objects within a certain category by a different color, which is unavailable in semantic segmentation. Despite the distinct advantages, there are rare methods which are dedicated to the high-quality instance segmentation for high-resolution aerial images. In this paper, a novel instance segmentation method, termed consistent proposals of instance segmentation network (CPISNet), for high-resolution aerial images is proposed. Following top-down instance segmentation formula, it adopts the adaptive feature extraction network (AFEN) to extract the multi-level bottom-up augmented feature maps in design space level. Then, elaborated RoI extractor (ERoIE) is designed to extract the mask RoIs via the refined bounding boxes from proposal consistent cascaded (PCC) architecture and multi-level features from AFEN. Finally, the convolution block with shortcut connection is responsible for generating the binary mask for instance segmentation. Experimental conclusions can be drawn on the iSAID and NWPU VHR-10 instance segmentation dataset: (1) Each individual module in CPISNet acts on the whole instance segmentation utility; (2) CPISNet* exceeds vanilla Mask R-CNN 3.4%/3.8% AP on iSAID validation/test set and 9.2% AP on NWPU VHR-10 instance segmentation dataset; (3) The aliasing masks, missing segmentations, false alarms, and poorly segmented masks can be avoided to some extent for CPISNet; (4) CPISNet receives high precision of instance segmentation for aerial images and interprets the objects with fitting boundary.

Download Full-text