USING SEMANTICALLY PAIRED IMAGES TO IMPROVE DOMAIN ADAPTATION FOR THE SEMANTIC SEGMENTATION OF AERIAL IMAGES

Abstract. Modern machine learning, especially deep learning, which is used in a variety of applications, requires a lot of labelled data for model training. Having an insufficient amount of training examples leads to models which do not generalize well to new input instances. This is a particular significant problem for tasks involving aerial images: often training data is only available for a limited geographical area and a narrow time window, thus leading to models which perform poorly in different regions, at different times of day, or during different seasons. Domain adaptation can mitigate this issue by using labelled source domain training examples and unlabeled target domain images to train a model which performs well on both domains. Modern adversarial domain adaptation approaches use unpaired data. We propose using pairs of semantically similar images, i.e., whose segmentations are accurate predictions of each other, for improved model performance. In this paper we show that, as an upper limit based on ground truth, using semantically paired aerial images during training almost always increases model performance with an average improvement of 4.2% accuracy and .036 mean intersection-over-union (mIoU). Using a practical estimate of semantic similarity, we still achieve improvements in more than half of all cases, with average improvements of 2.5% accuracy and .017 mIoU in those cases.

Download Full-text

ITERATIVE RE-WEIGHTED INSTANCE TRANSFER FOR DOMAIN ADAPTATION

ISPRS Annals of Photogrammetry Remote Sensing and Spatial Information Sciences ◽

10.5194/isprsannals-iii-3-339-2016 ◽

2016 ◽

Vol III-3 ◽

pp. 339-346 ◽

Cited By ~ 1

Author(s):

A. Paul ◽

F. Rottensteiner ◽

C. Heipke

Keyword(s):

Domain Adaptation ◽

Training Data ◽

Aerial Images ◽

Target Domain ◽

Data Set ◽

Source Domain ◽

The Difference ◽

A New Technique ◽

Class Labels ◽

Decision Boundaries

Domain adaptation techniques in transfer learning try to reduce the amount of training data required for classification by adapting a classifier trained on samples from a source domain to a new data set (target domain) where the features may have different distributions. In this paper, we propose a new technique for domain adaptation based on logistic regression. Starting with a classifier trained on training data from the source domain, we iteratively include target domain samples for which class labels have been obtained from the current state of the classifier, while at the same time removing source domain samples. In each iteration the classifier is re-trained, so that the decision boundaries are slowly transferred to the distribution of the target features. To make the transfer procedure more robust we introduce weights as a function of distance from the decision boundary and a new way of regularisation. Our methodology is evaluated using a benchmark data set consisting of aerial images and digital surface models. The experimental results show that in the majority of cases our domain adaptation approach can lead to an improvement of the classification accuracy without additional training data, but also indicate remaining problems if the difference in the feature distributions becomes too large.

Download Full-text

ADVERSARIAL DOMAIN ADAPTATION FOR THE CLASSIFICATION OF AERIAL IMAGES AND HEIGHT DATA USING CONVOLUTIONAL NEURAL NETWORKS

ISPRS Annals of Photogrammetry Remote Sensing and Spatial Information Sciences ◽

10.5194/isprs-annals-iv-2-w7-197-2019 ◽

2019 ◽

Vol IV-2/W7 ◽

pp. 197-204 ◽

Cited By ~ 1

Author(s):

D. Wittich ◽

F. Rottensteiner

Keyword(s):

Neural Networks ◽

Convolutional Neural Networks ◽

Domain Adaptation ◽

Feature Representation ◽

Training Data ◽

Aerial Images ◽

Remotely Sensed Data ◽

Target Domain ◽

Deep Convolutional Neural Networks

<p><strong>Abstract.</strong> Domain adaptation (DA) can drastically decrease the amount of training data needed to obtain good classification models by leveraging available data from a source domain for the classification of a new (target) domains. In this paper, we address deep DA, i.e. DA with deep convolutional neural networks (CNN), a problem that has not been addressed frequently in remote sensing. We present a new method for semi-supervised DA for the task of pixel-based classification by a CNN. After proposing an encoder-decoder-based fully convolutional neural network (FCN), we adapt a method for adversarial discriminative DA to be applicable to the pixel-based classification of remotely sensed data based on this network. It tries to learn a feature representation that is domain invariant; domain-invariance is measured by a classifier’s incapability of predicting from which domain a sample was generated. We evaluate our FCN on the ISPRS labelling challenge, showing that it is close to the best-performing models. DA is evaluated on the basis of three domains. We compare different network configurations and perform the representation transfer at different layers of the network. We show that when using a proper layer for adaptation, our method achieves a positive transfer and thus an improved classification accuracy in the target domain for all evaluated combinations of source and target domains.</p>

Download Full-text

Unsupervised Domain Adaptation Using Generative Adversarial Networks for Semantic Segmentation of Aerial Images

Remote Sensing ◽

10.3390/rs11111369 ◽

2019 ◽

Vol 11 (11) ◽

pp. 1369 ◽

Cited By ~ 21

Author(s):

Bilel Benjdira ◽

Yakoub Bazi ◽

Anis Koubaa ◽

Kais Ouni

Keyword(s):

Urban Areas ◽

Traffic Management ◽

Domain Adaptation ◽

Semantic Segmentation ◽

Aerial Images ◽

Generative Adversarial Networks ◽

Target Domain ◽

Adversarial Networks ◽

Segmentation Accuracy ◽

New City

Segmenting aerial images is of great potential in surveillance and scene understanding of urban areas. It provides a mean for automatic reporting of the different events that happen in inhabited areas. This remarkably promotes public safety and traffic management applications. After the wide adoption of convolutional neural networks methods, the accuracy of semantic segmentation algorithms could easily surpass 80% if a robust dataset is provided. Despite this success, the deployment of a pretrained segmentation model to survey a new city that is not included in the training set significantly decreases accuracy. This is due to the domain shift between the source dataset on which the model is trained and the new target domain of the new city images. In this paper, we address this issue and consider the challenge of domain adaptation in semantic segmentation of aerial images. We designed an algorithm that reduces the domain shift impact using generative adversarial networks (GANs). In the experiments, we tested the proposed methodology on the International Society for Photogrammetry and Remote Sensing (ISPRS) semantic segmentation dataset and found that our method improves overall accuracy from 35% to 52% when passing from the Potsdam domain (considered as source domain) to the Vaihingen domain (considered as target domain). In addition, the method allows efficiently recovering the inverted classes due to sensor variation. In particular, it improves the average segmentation accuracy of the inverted classes due to sensor variation from 14% to 61%.

Download Full-text

ITERATIVE RE-WEIGHTED INSTANCE TRANSFER FOR DOMAIN ADAPTATION

ISPRS Annals of Photogrammetry Remote Sensing and Spatial Information Sciences ◽

10.5194/isprs-annals-iii-3-339-2016 ◽

2016 ◽

Vol III-3 ◽

pp. 339-346 ◽

Cited By ~ 1

Author(s):

A. Paul ◽

F. Rottensteiner ◽

C. Heipke

Keyword(s):

Domain Adaptation ◽

Training Data ◽

Aerial Images ◽

Target Domain ◽

Data Set ◽

Source Domain ◽

The Difference ◽

A New Technique ◽

Class Labels ◽

Decision Boundaries

Download Full-text

From shallow to deep: exploiting feature-based classifiers for domain adaptation in semantic segmentation

10.1101/2021.11.09.467925 ◽

2021 ◽

Author(s):

Alex Matskevych ◽

Adrian Wolny ◽

Constantin Pape ◽

Anna Kreshuk

Keyword(s):

Random Forest ◽

Domain Adaptation ◽

Semantic Segmentation ◽

Training Data ◽

Target Domain ◽

Segmentation Accuracy ◽

Challenging Tasks ◽

Feature Based ◽

The Cost ◽

Nuclear Segmentation

The remarkable performance of Convolutional Neural Networks on image segmentation tasks comes at the cost of a large amount of pixelwise annotated images that have to be segmented for training. In contrast, feature-based learning methods, such as the Random Forest, require little training data, but never reach the segmentation accuracy of CNNs. This work bridges the two approaches in a transfer learning setting. We show that a CNN can be trained to correct the errors of the Random Forest in the source domain and then be applied to correct such errors in the target domain without retraining, as the domain shift between the Random Forest predictions is much smaller than between the raw data. By leveraging a few brushstrokes as annotations in the target domain, the method can deliver segmentations that are sufficiently accurate to act as pseudo-labels for target-domain CNN training. We demonstrate the performance of the method on several datasets with the challenging tasks of mitochondria, membrane and nuclear segmentation. It yields excellent performance compared to microscopy domain adaptation baselines, especially when a significant domain shift is involved.

Download Full-text

Harmonised segmentation of neonatal brain MRI

10.1101/2021.02.17.431611 ◽

2021 ◽

Author(s):

Irina Grigorescu ◽

Lucy Vanes ◽

Alena Uus ◽

Dafnis Batalle ◽

Lucilio Cordero-Grande ◽

...

Keyword(s):

Domain Adaptation ◽

Brain Mri ◽

Ground Truth ◽

Clinical Analysis ◽

Medical Image Segmentation ◽

Training Data ◽

Target Domain ◽

Tissue Segmentation ◽

Language Outcome ◽

Preterm Born

ABSTRACTDeep learning based medical image segmentation has shown great potential in becoming a key part of the clinical analysis pipeline. However, many of these models rely on the assumption that the train and test data come from the same distribution. This means that such methods cannot guarantee high quality predictions when the source and target domains are dissimilar due to different acquisition protocols, or biases in patient cohorts. Recently, unsupervised domain adaptation (DA) techniques have shown great potential in alleviating this problem by minimizing the shift between the source and target distributions, without requiring the use of labelled data in the target domain. In this work, we aim to predict tissue segmentation maps on T2-weighted (T2w) magnetic resonance imaging (MRI) data of an unseen preterm-born neonatal population, which has both different acquisition parameters and population bias when compared to our training data. We achieve this by investigating two unsupervised DA techniques with the objective of finding the best solution for our problem. We compare the two methods with a baseline fully-supervised segmentation network and report our results in terms of Dice scores obtained on our ground truth test dataset. Moreover, we analyse tissue volumes and cortical thickness (CT) measures of the harmonised data on a subset of the population matched for gestational age (GA) at birth and postmenstrual age (PMA) at scan. Finally, we demonstrate the applicability of the harmonised cortical gray matter maps with an analysis comparing term and preterm-born neonates and a proof-of-principle investigation of the association between CT and a language outcome measure.

Download Full-text

BiFDANet: Unsupervised Bidirectional Domain Adaptation for Semantic Segmentation of Remote Sensing Images

Remote Sensing ◽

10.3390/rs14010190 ◽

2022 ◽

Vol 14 (1) ◽

pp. 190

Author(s):

Yuxiang Cai ◽

Yingchun Yang ◽

Qiyi Zheng ◽

Zhengwei Shen ◽

Yongheng Shang ◽

...

Keyword(s):

Remote Sensing ◽

Domain Adaptation ◽

Semantic Segmentation ◽

Aerial Images ◽

Combination Method ◽

Remote Sensing Images ◽

Target Domain ◽

Semantic Consistency ◽

Image Translation ◽

Segmentation Models

When segmenting massive amounts of remote sensing images collected from different satellites or geographic locations (cities), the pre-trained deep learning models cannot always output satisfactory predictions. To deal with this issue, domain adaptation has been widely utilized to enhance the generalization abilities of the segmentation models. Most of the existing domain adaptation methods, which based on image-to-image translation, firstly transfer the source images to the pseudo-target images, adapt the classifier from the source domain to the target domain. However, these unidirectional methods suffer from the following two limitations: (1) they do not consider the inverse procedure and they cannot fully take advantage of the information from the other domain, which is also beneficial, as confirmed by our experiments; (2) these methods may fail in the cases where transferring the source images to the pseudo-target images is difficult. In this paper, in order to solve these problems, we propose a novel framework BiFDANet for unsupervised bidirectional domain adaptation in the semantic segmentation of remote sensing images. It optimizes the segmentation models in two opposite directions. In the source-to-target direction, BiFDANet learns to transfer the source images to the pseudo-target images and adapts the classifier to the target domain. In the opposite direction, BiFDANet transfers the target images to the pseudo-source images and optimizes the source classifier. At test stage, we make the best of the source classifier and the target classifier, which complement each other with a simple linear combination method, further improving the performance of our BiFDANet. Furthermore, we propose a new bidirectional semantic consistency loss for our BiFDANet to maintain the semantic consistency during the bidirectional image-to-image translation process. The experiments on two datasets including satellite images and aerial images demonstrate the superiority of our method against existing unidirectional methods.

Download Full-text

Domain Adaptation for Semantic Segmentation of Historical Panchromatic Orthomosaics in Central Africa

ISPRS International Journal of Geo-Information ◽

10.3390/ijgi10080523 ◽

2021 ◽

Vol 10 (8) ◽

pp. 523

Author(s):

Nicholus Mboga ◽

Stefano D’Aronco ◽

Tais Grippa ◽

Charlotte Pelletier ◽

Stefanos Georganos ◽

...

Keyword(s):

Land Cover ◽

Domain Adaptation ◽

Central Africa ◽

Semantic Segmentation ◽

Target Domain ◽

Convolutional Networks ◽

Fully Convolutional Networks ◽

The Cost ◽

Performance Gains

Multitemporal environmental and urban studies are essential to guide policy making to ultimately improve human wellbeing in the Global South. Land-cover products derived from historical aerial orthomosaics acquired decades ago can provide important evidence to inform long-term studies. To reduce the manual labelling effort by human experts and to scale to large, meaningful regions, we investigate in this study how domain adaptation techniques and deep learning can help to efficiently map land cover in Central Africa. We propose and evaluate a methodology that is based on unsupervised adaptation to reduce the cost of generating reference data for several cities and across different dates. We present the first application of domain adaptation based on fully convolutional networks for semantic segmentation of a dataset of historical panchromatic orthomosaics for land-cover generation for two focus cities Goma-Gisenyi and Bukavu. Our experimental evaluation shows that the domain adaptation methods can reach an overall accuracy between 60% and 70% for different regions. If we add a small amount of labelled data from the target domain, too, further performance gains can be achieved.

Download Full-text

An intelligent fault diagnosis method based on domain adaptation for rolling bearings under variable load conditions

Proceedings of the Institution of Mechanical Engineers Part C Journal of Mechanical Engineering Science ◽

10.1177/09544062211032995 ◽

2021 ◽

pp. 095440622110329

Author(s):

Jianqun Zhang ◽

Qing Zhang ◽

Xianrong Qin ◽

Yuantao Sun

Keyword(s):

Feature Extraction ◽

Fault Diagnosis ◽

Domain Adaptation ◽

Rolling Bearing ◽

Training Data ◽

Variable Load ◽

K Nearest Neighbor ◽

Target Domain ◽

Bearing Faults ◽

Load Conditions

To identify rolling bearing faults under variable load conditions, a method named DISA-KNN is proposed in this paper, which is based on the strategy of feature extraction-domain adaptation-classification. To be specific, the time-domain and frequency-domain indicators are used for feature extraction. Discriminative and domain invariant subspace alignment (DISA) is used to minimize the data distributions’ discrepancies between the training data (source domain) and testing data (target domain). K-nearest neighbor (KNN) is applied to identify rolling bearing faults. DISA-KNN’s validation is proved by the experimental signal collected under different load conditions. The identification accuracies obtained by the DISA-KNN method are more than 90% on four datasets, including one dataset with 99.5% accuracy. The strength of the proposed method is further highlighted by comparisons with the other 8 methods. These results reveal that the proposed method is promising for the rolling bearing fault diagnosis in real rotating machinery.

Download Full-text

Multi-Source Distilling Domain Adaptation

Proceedings of the AAAI Conference on Artificial Intelligence ◽

10.1609/aaai.v34i07.6997 ◽

2020 ◽

Vol 34 (07) ◽

pp. 12975-12983

Author(s):

Sicheng Zhao ◽

Guangzhi Wang ◽

Shanghang Zhang ◽

Yang Gu ◽

Yaxian Li ◽

...

Keyword(s):

Domain Adaptation ◽

Feature Space ◽

Wasserstein Distance ◽

Training Data ◽

Source Distribution ◽

Single Source ◽

Multiple Sources ◽

Target Domain ◽

Target Feature ◽

Training Samples

Deep neural networks suffer from performance decay when there is domain shift between the labeled source domain and unlabeled target domain, which motivates the research on domain adaptation (DA). Conventional DA methods usually assume that the labeled data is sampled from a single source distribution. However, in practice, labeled data may be collected from multiple sources, while naive application of the single-source DA algorithms may lead to suboptimal solutions. In this paper, we propose a novel multi-source distilling domain adaptation (MDDA) network, which not only considers the different distances among multiple sources and the target, but also investigates the different similarities of the source samples to the target ones. Specifically, the proposed MDDA includes four stages: (1) pre-train the source classifiers separately using the training data from each source; (2) adversarially map the target into the feature space of each source respectively by minimizing the empirical Wasserstein distance between source and target; (3) select the source training samples that are closer to the target to fine-tune the source classifiers; and (4) classify each encoded target feature by corresponding source classifier, and aggregate different predictions using respective domain weight, which corresponds to the discrepancy between each source and target. Extensive experiments are conducted on public DA benchmarks, and the results demonstrate that the proposed MDDA significantly outperforms the state-of-the-art approaches. Our source code is released at: https://github.com/daoyuan98/MDDA.

Download Full-text