DC-FUDA: Improving Deep Clustering via Fully Unsupervised Domain Adaptation

<div>By transferring knowledge from a source domain, the performance of deep clustering on an unlabeled target domain can be improved. When achieving this, traditional approaches make the assumption that adequate amount of labeled data is available in a source domain. However, this assumption is usually unrealistic in practice. The source domain should be carefully selected to share some characteristics with the target domain, and it can not be guaranteed that rich labeled samples are always available in the selected source domain.</div><div>We propose a novel framework to improve deep clustering by transferring knowledge from a source domain without any labeled data. To select reliable instances in the source domain for transferring, we propose a novel adaptive threshold algorithm to select low entropy instances. To transfer important features of the selected instances, we propose a feature-level domain adaptation network (FeatureDA) which cancels unstable generation process. With extensive experiments, we validate that our method effectively improves deep clustering, without using any labeled data in the source domain. Besides, without using any labeled data in the source domain, our method achieves competitive results, compared to the state-of-the-art methods using labeled data in the source domain.</div>

Download Full-text

DC-FUDA: Improving Deep Clustering via Fully Unsupervised Domain Adaptation

10.36227/techrxiv.14995086 ◽

2021 ◽

Author(s):

Zhimeng Yang ◽

Zirui Wu ◽

Ming Zeng ◽

Yazhou Ren ◽

Xiaorong Pu ◽

...

Keyword(s):

Domain Adaptation ◽

State Of The Art ◽

Adaptive Threshold ◽

Generation Process ◽

Target Domain ◽

Source Domain ◽

Unsupervised Domain Adaptation ◽

Adequate Amount ◽

Traditional Approaches ◽

Low Entropy

Download Full-text

Benchmarking Domain Adaptation Methods on Aerial Datasets

Sensors ◽

10.3390/s21238070 ◽

2021 ◽

Vol 21 (23) ◽

pp. 8070

Author(s):

Navya Nagananda ◽

Abu Md Niamul Taufique ◽

Raaga Madappa ◽

Chowdhury Sadman Jahan ◽

Breton Minnehan ◽

...

Keyword(s):

Deep Learning ◽

Supervised Classification ◽

Domain Adaptation ◽

State Of The Art ◽

Classification Performance ◽

Target Domain ◽

Source Domain ◽

Unsupervised Domain Adaptation ◽

Testing Data ◽

Classification Tasks

Deep learning grew in importance in recent years due to its versatility and excellent performance on supervised classification tasks. A core assumption for such supervised approaches is that the training and testing data are drawn from the same underlying data distribution. This may not always be the case, and in such cases, the performance of the model is degraded. Domain adaptation aims to overcome the domain shift between the source domain used for training and the target domain data used for testing. Unsupervised domain adaptation deals with situations where the network is trained on labeled data from the source domain and unlabeled data from the target domain with the goal of performing well on the target domain data at the time of deployment. In this study, we overview seven state-of-the-art unsupervised domain adaptation models based on deep learning and benchmark their performance on three new domain adaptation datasets created from publicly available aerial datasets. We believe this is the first study on benchmarking domain adaptation methods for aerial data. In addition to reporting classification performance for the different domain adaptation models, we present t-SNE visualizations that illustrate the benefits of the adaptation process.

Download Full-text

Correlation alignment with attention mechanism for unsupervised domain adaptation

Web Intelligence ◽

10.3233/web-210447 ◽

2021 ◽

pp. 1-7

Author(s):

Rong Chen ◽

Chongguang Ren

Keyword(s):

Natural Language Processing ◽

Language Processing ◽

Transfer Process ◽

Negative Transfer ◽

Domain Adaptation ◽

Attention Mechanism ◽

Target Domain ◽

Source Domain ◽

Second Order Statistics ◽

Unsupervised Domain Adaptation

Domain adaptation aims to solve the problems of lacking labels. Most existing works of domain adaptation mainly focus on aligning the feature distributions between the source and target domain. However, in the field of Natural Language Processing, some of the words in different domains convey different sentiment. Thus not all features of the source domain should be transferred, and it would cause negative transfer when aligning the untransferable features. To address this issue, we propose a Correlation Alignment with Attention mechanism for unsupervised Domain Adaptation (CAADA) model. In the model, an attention mechanism is introduced into the transfer process for domain adaptation, which can capture the positively transferable features in source and target domain. Moreover, the CORrelation ALignment (CORAL) loss is utilized to minimize the domain discrepancy by aligning the second-order statistics of the positively transferable features extracted by the attention mechanism. Extensive experiments on the Amazon review dataset demonstrate the effectiveness of CAADA method.

Download Full-text

Unsupervised Domain Adaptation by Statistics Alignment for Deep Sleep Staging Networks

10.36227/techrxiv.17212184.v1 ◽

2021 ◽

Author(s):

Jiahao Fan ◽

Hangyu Zhu ◽

Xinyu Jiang ◽

Long Meng ◽

Cong Fu ◽

...

Keyword(s):

Large Scale ◽

Domain Adaptation ◽

Source Model ◽

Deep Sleep ◽

Sleep Staging ◽

Target Domain ◽

Source Domain ◽

Unsupervised Domain Adaptation ◽

Generalization Problem ◽

Source Models

Deep sleep staging networks have reached top performance on large-scale datasets. However, these models perform poorer when training and testing on small sleep cohorts due to data inefficiency. Transferring well-trained models from large-scale datasets (source domain) to small sleep cohorts (target domain) is a promising solution but still remains challenging due to the domain-shift issue. In this work, an unsupervised domain adaptation approach, domain statistics alignment (DSA), is developed to bridge the gap between the data distribution of source and target domains. DSA adapts the source models on the target domain by modulating the domain-specific statistics of deep features stored in the Batch Normalization (BN) layers. Furthermore, we have extended DSA by introducing cross-domain statistics in each BN layer to perform DSA adaptively (AdaDSA). The proposed methods merely need the well-trained source model without access to the source data, which may be proprietary and inaccessible. DSA and AdaDSA are universally applicable to various deep sleep staging networks that have BN layers. We have validated the proposed methods by extensive experiments on two state-of-the-art deep sleep staging networks, DeepSleepNet+ and U-time. The performance was evaluated by conducting various transfer tasks on six sleep databases, including two large-scale databases, MASS and SHHS, as the source domain, four small sleep databases as the target domain. Thereinto, clinical sleep records acquired in Huashan Hospital, Shanghai, were used. The results show that both DSA and AdaDSA could significantly improve the performance of source models on target domains, providing novel insights into the domain generalization problem in sleep staging tasks.<br>

Download Full-text

Distributional Correspondence Indexing for Cross-Lingual and Cross-Domain Sentiment Classification (Extended Abstract)

Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence ◽

10.24963/ijcai.2018/802 ◽

2018 ◽

Author(s):

Alejandro Moreo Fernández ◽

Andrea Esuli ◽

Fabrizio Sebastiani

Keyword(s):

Domain Adaptation ◽

State Of The Art ◽

Sentiment Classification ◽

Training Data ◽

Target Domain ◽

Source Domain ◽

Machine Learning Methods ◽

Cross Domain ◽

Current State ◽

Cross Lingual

Domain Adaptation (DA) techniques aim at enabling machine learning methods learn effective classifiers for a “target” domain when the only available training data belongs to a different “source” domain. In this extended abstract, we briefly describe our new DA method called Distributional Correspondence Indexing (DCI) for sentiment classification. DCI derives term representations in a vector space common to both domains where each dimension reflects its distributional correspondence to a pivot, i.e., to a highly predictive term that behaves similarly across domains. The experiments we have conducted show that DCI obtains better performance than current state-of-the-art techniques for cross-lingual and cross-domain sentiment classification.

Download Full-text

Self-Ensembling Attention Networks: Addressing Domain Shift for Semantic Segmentation

Proceedings of the AAAI Conference on Artificial Intelligence ◽

10.1609/aaai.v33i01.33015581 ◽

2019 ◽

Vol 33 ◽

pp. 5581-5588 ◽

Cited By ~ 3

Author(s):

Yonghao Xu ◽

Bo Du ◽

Lefei Zhang ◽

Qian Zhang ◽

Guoli Wang ◽

...

Keyword(s):

Domain Adaptation ◽

State Of The Art ◽

Semantic Segmentation ◽

Great Success ◽

Learning Models ◽

Target Domain ◽

Attention Networks ◽

Source Domain ◽

Benchmark Datasets ◽

Different Levels

Recent years have witnessed the great success of deep learning models in semantic segmentation. Nevertheless, these models may not generalize well to unseen image domains due to the phenomenon of domain shift. Since pixel-level annotations are laborious to collect, developing algorithms which can adapt labeled data from source domain to target domain is of great significance. To this end, we propose self-ensembling attention networks to reduce the domain gap between different datasets. To the best of our knowledge, the proposed method is the first attempt to introduce selfensembling model to domain adaptation for semantic segmentation, which provides a different view on how to learn domain-invariant features. Besides, since different regions in the image usually correspond to different levels of domain gap, we introduce the attention mechanism into the proposed framework to generate attention-aware features, which are further utilized to guide the calculation of consistency loss in the target domain. Experiments on two benchmark datasets demonstrate that the proposed framework can yield competitive performance compared with the state of the art methods.

Download Full-text

Adversarial Training Based Multi-Source Unsupervised Domain Adaptation for Sentiment Analysis

Proceedings of the AAAI Conference on Artificial Intelligence ◽

10.1609/aaai.v34i05.6262 ◽

2020 ◽

Vol 34 (05) ◽

pp. 7618-7625

Author(s):

Yong Dai ◽

Jian Liu ◽

Xiancong Ren ◽

Zenglin Xu

Keyword(s):

Sentiment Analysis ◽

Domain Adaptation ◽

State Of The Art ◽

Weak Assumption ◽

Target Domain ◽

Smoothness Assumption ◽

Unsupervised Domain Adaptation ◽

Good Target ◽

Adversarial Training ◽

Learning Frameworks

Multi-source unsupervised domain adaptation (MS-UDA) for sentiment analysis (SA) aims to leverage useful information in multiple source domains to help do SA in an unlabeled target domain that has no supervised information. Existing algorithms of MS-UDA either only exploit the shared features, i.e., the domain-invariant information, or based on some weak assumption in NLP, e.g., smoothness assumption. To avoid these problems, we propose two transfer learning frameworks based on the multi-source domain adaptation methodology for SA by combining the source hypotheses to derive a good target hypothesis. The key feature of the first framework is a novel Weighting Scheme based Unsupervised Domain Adaptation framework ((WS-UDA), which combine the source classifiers to acquire pseudo labels for target instances directly. While the second framework is a Two-Stage Training based Unsupervised Domain Adaptation framework (2ST-UDA), which further exploits these pseudo labels to train a target private extractor. Importantly, the weights assigned to each source classifier are based on the relations between target instances and source domains, which measured by a discriminator through the adversarial training. Furthermore, through the same discriminator, we also fulfill the separation of shared features and private features.Experimental results on two SA datasets demonstrate the promising performance of our frameworks, which outperforms unsupervised state-of-the-art competitors.

Download Full-text

Multi-Source Domain Adaptation for Text Classification via DistanceNet-Bandits

Proceedings of the AAAI Conference on Artificial Intelligence ◽

10.1609/aaai.v34i05.6288 ◽

2020 ◽

Vol 34 (05) ◽

pp. 7830-7838 ◽

Cited By ~ 1

Author(s):

Han Guo ◽

Ramakanth Pasunuru ◽

Mohit Bansal

Keyword(s):

Loss Function ◽

Optimal Trajectory ◽

Domain Adaptation ◽

Learning Algorithm ◽

Data Distribution ◽

Distance Measures ◽

Target Domain ◽

Source Domain ◽

Unsupervised Domain Adaptation ◽

Additional Loss

Domain adaptation performance of a learning algorithm on a target domain is a function of its source domain error and a divergence measure between the data distribution of these two domains. We present a study of various distance-based measures in the context of NLP tasks, that characterize the dissimilarity between domains based on sample estimates. We first conduct analysis experiments to show which of these distance measures can best differentiate samples from same versus different domains, and are correlated with empirical results. Next, we develop a DistanceNet model which uses these distance measures, or a mixture of these distance measures, as an additional loss function to be minimized jointly with the task's loss function, so as to achieve better unsupervised domain adaptation. Finally, we extend this model to a novel DistanceNet-Bandit model, which employs a multi-armed bandit controller to dynamically switch between multiple source domains and allow the model to learn an optimal trajectory and mixture of domains for transfer to the low-resource target domain. We conduct experiments on popular sentiment analysis datasets with several diverse domains and show that our DistanceNet model, as well as its dynamic bandit variant, can outperform competitive baselines in the context of unsupervised domain adaptation.

Download Full-text

Unsupervised Domain Adaptation via Structured Prediction Based Selective Pseudo-Labeling

Proceedings of the AAAI Conference on Artificial Intelligence ◽

10.1609/aaai.v34i04.6091 ◽

2020 ◽

Vol 34 (04) ◽

pp. 6243-6250 ◽

Cited By ~ 2

Author(s):

Qian Wang ◽

Toby Breckon

Keyword(s):

Domain Adaptation ◽

Feature Space ◽

Structured Prediction ◽

Target Domain ◽

Source Domain ◽

Domain Specific ◽

Unsupervised Domain Adaptation ◽

Deep Feature ◽

Significant Performance ◽

Error Accumulation

Unsupervised domain adaptation aims to address the problem of classifying unlabeled samples from the target domain whilst labeled samples are only available from the source domain and the data distributions are different in these two domains. As a result, classifiers trained from labeled samples in the source domain suffer from significant performance drop when directly applied to the samples from the target domain. To address this issue, different approaches have been proposed to learn domain-invariant features or domain-specific classifiers. In either case, the lack of labeled samples in the target domain can be an issue which is usually overcome by pseudo-labeling. Inaccurate pseudo-labeling, however, could result in catastrophic error accumulation during learning. In this paper, we propose a novel selective pseudo-labeling strategy based on structured prediction. The idea of structured prediction is inspired by the fact that samples in the target domain are well clustered within the deep feature space so that unsupervised clustering analysis can be used to facilitate accurate pseudo-labeling. Experimental results on four datasets (i.e. Office-Caltech, Office31, ImageCLEF-DA and Office-Home) validate our approach outperforms contemporary state-of-the-art methods.

Download Full-text

Bidirectional Adversarial Training for Semi-Supervised Domain Adaptation

Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence ◽

10.24963/ijcai.2020/130 ◽

2020 ◽

Author(s):

Pin Jiang ◽

Aming Wu ◽

Yahong Han ◽

Yunfeng Shao ◽

Meiyu Qi ◽

...

Keyword(s):

Additional Data ◽

Domain Adaptation ◽

State Of The Art ◽

Powerful Method ◽

Target Domain ◽

Unsupervised Domain Adaptation ◽

Benchmark Datasets ◽

Adversarial Examples ◽

Adversarial Training ◽

Effective Use

Semi-supervised domain adaptation (SSDA) is a novel branch of machine learning that scarce labeled target examples are available, compared with unsupervised domain adaptation. To make effective use of these additional data so as to bridge the domain gap, one possible way is to generate adversarial examples, which are images with additional perturbations, between the two domains and fill the domain gap. Adversarial training has been proven to be a powerful method for this purpose. However, the traditional adversarial training adds noises in arbitrary directions, which is inefficient to migrate between domains, or generate directional noises from the source to target domain and reverse. In this work, we devise a general bidirectional adversarial training method and employ gradient to guide adversarial examples across the domain gap, i.e., the Adaptive Adversarial Training (AAT) for source to target domain and Entropy-penalized Virtual Adversarial Training (E-VAT) for target to source domain. Particularly, we devise a Bidirectional Adversarial Training (BiAT) network to perform diverse adversarial trainings jointly. We evaluate the effectiveness of BiAT on three benchmark datasets and experimental results demonstrate the proposed method achieves the state-of-the-art.

Download Full-text