An Approach to Semantically Segmenting Building Components and Outdoor Scenes Based on Multichannel Aerial Imagery Datasets

As-is building modeling plays an important role in energy audits and retrofits. However, in order to understand the source(s) of energy loss, researchers must know the semantic information of the buildings and outdoor scenes. Thermal information can potentially be used to distinguish objects that have similar surface colors but are composed of different materials. To utilize both the red–green–blue (RGB) color model and thermal information for the semantic segmentation of buildings and outdoor scenes, we deployed and adapted various pioneering deep convolutional neural network (DCNN) tools that combine RGB information with thermal information to improve the semantic and instance segmentation processes. When both types of information are available, the resulting DCNN models allow us to achieve better segmentation performance. By deploying three case studies, we experimented with our proposed DCNN framework, deploying datasets of building components and outdoor scenes, and testing the models to determine whether the segmentation performance had improved or not. In our observation, the fusion of RGB and thermal information can help the segmentation task in specific cases, but it might also make the neural networks hard to train or deteriorate their prediction performance in some cases. Additionally, different algorithms perform differently in semantic and instance segmentation.

Download Full-text

RESIDUAL SHUFFLING CONVOLUTIONAL NEURAL NETWORKS FOR DEEP SEMANTIC IMAGE SEGMENTATION USING MULTI-MODAL DATA

ISPRS Annals of Photogrammetry Remote Sensing and Spatial Information Sciences ◽

10.5194/isprs-annals-iv-2-65-2018 ◽

2018 ◽

Vol IV-2 ◽

pp. 65-72 ◽

Cited By ~ 3

Author(s):

K. Chen ◽

M. Weinmann ◽

X. Gao ◽

M. Yan ◽

S. Hinz ◽

...

Keyword(s):

Semantic Segmentation ◽

Aerial Imagery ◽

Geometric Features ◽

Modal Data ◽

Learning Framework ◽

Semantic Labeling ◽

Surface Models ◽

Data Representations ◽

Segmentation Task ◽

Feature Selection Techniques

In this paper, we address the deep semantic segmentation of aerial imagery based on multi-modal data. Given multi-modal data composed of true orthophotos and the corresponding Digital Surface Models (DSMs), we extract a variety of hand-crafted radiometric and geometric features which are provided separately and in different combinations as input to a modern deep learning framework. The latter is represented by a Residual Shuffling Convolutional Neural Network (RSCNN) combining the characteristics of a Residual Network with the advantages of atrous convolution and a shuffling operator to achieve a dense semantic labeling. Via performance evaluation on a benchmark dataset, we analyze the value of different feature sets for the semantic segmentation task. The derived results reveal that the use of radiometric features yields better classification results than the use of geometric features for the considered dataset. Furthermore, the consideration of data on both modalities leads to an improvement of the classification results. However, the derived results also indicate that the use of all defined features is less favorable than the use of selected features. Consequently, data representations derived via feature extraction and feature selection techniques still provide a gain if used as the basis for deep semantic segmentation.

Download Full-text

SEMANTIC SEGMENTATION OF AERIAL IMAGERY VIA MULTI-SCALE SHUFFLING CONVOLUTIONAL NEURAL NETWORKS WITH DEEP SUPERVISION

ISPRS Annals of Photogrammetry Remote Sensing and Spatial Information Sciences ◽

10.5194/isprs-annals-iv-1-29-2018 ◽

2018 ◽

Vol IV-1 ◽

pp. 29-36 ◽

Cited By ~ 4

Author(s):

K. Chen ◽

M. Weinmann ◽

X. Sun ◽

M. Yan ◽

S. Hinz ◽

...

Keyword(s):

Neural Network ◽

Convolutional Neural Network ◽

Semantic Segmentation ◽

Aerial Imagery ◽

Geometric Features ◽

Feature Maps ◽

Multi Scale ◽

Intermediate Layers ◽

Segmentation Task ◽

The Impact

<p><strong>Abstract.</strong> In this paper, we address the semantic segmentation of aerial imagery based on the use of multi-modal data given in the form of true orthophotos and the corresponding Digital Surface Models (DSMs). We present the Deeply-supervised Shuffling Convolutional Neural Network (DSCNN) representing a multi-scale extension of the Shuffling Convolutional Neural Network (SCNN) with deep supervision. Thereby, we take the advantage of the SCNN involving the shuffling operator to effectively upsample feature maps and then fuse multiscale features derived from the intermediate layers of the SCNN, which results in the Multi-scale Shuffling Convolutional Neural Network (MSCNN). Based on the MSCNN, we derive the DSCNN by introducing additional losses into the intermediate layers of the MSCNN. In addition, we investigate the impact of using different sets of hand-crafted radiometric and geometric features derived from the true orthophotos and the DSMs on the semantic segmentation task. For performance evaluation, we use a commonly used benchmark dataset. The achieved results reveal that both multi-scale fusion and deep supervision contribute to an improvement in performance. Furthermore, the use of a diversity of hand-crafted radiometric and geometric features as input for the DSCNN does not provide the best numerical results, but smoother and improved detections for several objects.</p>

Download Full-text

Dense cellular segmentation for EM using 2D–3D neural network ensembles

Scientific Reports ◽

10.1038/s41598-021-81590-0 ◽

2021 ◽

Vol 11 (1) ◽

Author(s):

Matthew D. Guay ◽

Zeyad A. S. Emam ◽

Adam B. Anderson ◽

Maria A. Aronova ◽

Irina D. Pokrovskaya ◽

...

Keyword(s):

Blood Platelets ◽

Semantic Segmentation ◽

3D Models ◽

Image Features ◽

Cell Models ◽

Whole Cells ◽

Large Numbers ◽

The Creation ◽

Segmentation Task ◽

Neural Network Ensembles

AbstractBiologists who use electron microscopy (EM) images to build nanoscale 3D models of whole cells and their organelles have historically been limited to small numbers of cells and cellular features due to constraints in imaging and analysis. This has been a major factor limiting insight into the complex variability of cellular environments. Modern EM can produce gigavoxel image volumes containing large numbers of cells, but accurate manual segmentation of image features is slow and limits the creation of cell models. Segmentation algorithms based on convolutional neural networks can process large volumes quickly, but achieving EM task accuracy goals often challenges current techniques. Here, we define dense cellular segmentation as a multiclass semantic segmentation task for modeling cells and large numbers of their organelles, and give an example in human blood platelets. We present an algorithm using novel hybrid 2D–3D segmentation networks to produce dense cellular segmentations with accuracy levels that outperform baseline methods and approach those of human annotators. To our knowledge, this work represents the first published approach to automating the creation of cell models with this level of structural detail.

Download Full-text

Adversarial Data Augmentation on Breast MRI Segmentation

Applied Sciences ◽

10.3390/app11104554 ◽

2021 ◽

Vol 11 (10) ◽

pp. 4554

Author(s):

João F. Teixeira ◽

Mariana Dias ◽

Eva Batista ◽

Joana Costa ◽

Luís F. Teixeira ◽

...

Keyword(s):

Data Augmentation ◽

Medical Image Analysis ◽

Synthetic Data ◽

Breast Mri ◽

Semantic Segmentation ◽

Adversarial Networks ◽

Augmentation Strategy ◽

Magnetic Resonance Imaging Mri ◽

Segmentation Task ◽

The Impact

The scarcity of balanced and annotated datasets has been a recurring problem in medical image analysis. Several researchers have tried to fill this gap employing dataset synthesis with adversarial networks (GANs). Breast magnetic resonance imaging (MRI) provides complex, texture-rich medical images, with the same annotation shortage issues, for which, to the best of our knowledge, no previous work tried synthesizing data. Within this context, our work addresses the problem of synthesizing breast MRI images from corresponding annotations and evaluate the impact of this data augmentation strategy on a semantic segmentation task. We explored variations of image-to-image translation using conditional GANs, namely fitting the generator’s architecture with residual blocks and experimenting with cycle consistency approaches. We studied the impact of these changes on visual verisimilarity and how an U-Net segmentation model is affected by the usage of synthetic data. We achieved sufficiently realistic-looking breast MRI images and maintained a stable segmentation score even when completely replacing the dataset with the synthetic set. Our results were promising, especially when concerning to Pix2PixHD and Residual CycleGAN architectures.

Download Full-text

InstantDL: an easy-to-use deep learning pipeline for image segmentation and classification

BMC Bioinformatics ◽

10.1186/s12859-021-04037-3 ◽

2021 ◽

Vol 22 (1) ◽

Author(s):

Dominik Jens Elias Waibel ◽

Sayedali Shetab Boushehri ◽

Carsten Marr

Keyword(s):

Image Processing ◽

Deep Learning ◽

Specific Problem ◽

State Of The Art ◽

Image Data ◽

Semantic Segmentation ◽

Parameter Tuning ◽

Cellular Processes ◽

Minimal Effort ◽

Instance Segmentation

Abstract Background Deep learning contributes to uncovering molecular and cellular processes with highly performant algorithms. Convolutional neural networks have become the state-of-the-art tool to provide accurate and fast image data processing. However, published algorithms mostly solve only one specific problem and they typically require a considerable coding effort and machine learning background for their application. Results We have thus developed InstantDL, a deep learning pipeline for four common image processing tasks: semantic segmentation, instance segmentation, pixel-wise regression and classification. InstantDL enables researchers with a basic computational background to apply debugged and benchmarked state-of-the-art deep learning algorithms to their own data with minimal effort. To make the pipeline robust, we have automated and standardized workflows and extensively tested it in different scenarios. Moreover, it allows assessing the uncertainty of predictions. We have benchmarked InstantDL on seven publicly available datasets achieving competitive performance without any parameter tuning. For customization of the pipeline to specific tasks, all code is easily accessible and well documented. Conclusions With InstantDL, we hope to empower biomedical researchers to conduct reproducible image processing with a convenient and easy-to-use pipeline.

Download Full-text

Fault Detection by Using Instance Segmentation

10.2523/iptc-21249-ms ◽

2021 ◽

Author(s):

Donglin Zhu ◽

Lei Li ◽

Rui Guo ◽

Shifan Zhan

Keyword(s):

Fault Detection ◽

Seismic Data ◽

Semantic Segmentation ◽

Data Interpretation ◽

Machine Learning Methods ◽

Segmentation Algorithms ◽

Direct Recognition ◽

First Time ◽

Seismic Images ◽

Instance Segmentation

Abstract Fault detection is an important, but time-consuming task in seismic data interpretation. Traditionally, seismic attributes, such as coherency (Marfurt et al., 1998) and curvature (Al-Dossary et al., 2006) are used to detect faults. Recently, machine learning methods, such as convolution neural networks (CNNs) are used to detect faults, by applying various semantic segmentation algorithms to the seismic data (Wu et al., 2019). The most used algorithm is U-Net (Ronneberger et al., 2015), which can accurately and efficiently provide probability maps of faults. However, probabilities of faults generated by semantic segmentation algorithms are not sufficient for direct recognition of fault types and reconstruction of fault surfaces. To address this problem, we propose, for the first time, a workflow to use instance segmentation algorithm to detect different fault lines. Specifically, a modified CNN (LaneNet; Neven et al., 2018) is trained using automatically generated synthetic seismic images and corresponding labels. We then test the trained CNN using both synthetic and field collected seismic data. Results indicate that the proposed workflow is accurate and effective at detecting faults.

Download Full-text

Constructing a Convolutional Neural Network with a Suitable Capacity for a Semantic Segmentation Task

Deep Learning: Concepts and Architectures - Studies in Computational Intelligence ◽

10.1007/978-3-030-31756-0_8 ◽

2019 ◽

pp. 237-268

Author(s):

Yalong Jiang ◽

Zheru Chi

Keyword(s):

Neural Network ◽

Convolutional Neural Network ◽

Semantic Segmentation ◽

Segmentation Task

Download Full-text

Mapping the Unseen: Exploiting Super-Resolution for Semantic Segmentation in Low-Resolution Images

10.5753/sibgrapi.est.2020.12987 ◽

2020 ◽

Author(s):

Matheus B. Pereira ◽

Jefersson Alex Dos Santos

Keyword(s):

Remote Sensing ◽

Pattern Recognition ◽

Super Resolution ◽

Remote Sensing Data ◽

Semantic Segmentation ◽

The Other ◽

Aerial Imagery ◽

Aerial Images ◽

Remote Sensing Images ◽

Low Resolution

High-resolution aerial images are usually not accessible or affordable. On the other hand, low-resolution remote sensing data is easily found in public open repositories. The problem is that the low-resolution representation can compromise pattern recognition algorithms, especially semantic segmentation. In this M.Sc. dissertation1 , we design two frameworks in order to evaluate the effectiveness of super-resolution in the semantic segmentation of low-resolution remote sensing images. We carried out an extensive set of experiments on different remote sensing datasets. The results show that super-resolution is effective to improve semantic segmentation performance on low-resolution aerial imagery, outperforming unsupervised interpolation and achieving semantic segmentation results comparable to highresolution data.

Download Full-text