Correlation alignment with attention mechanism for unsupervised domain adaptation

2021 ◽  
pp. 1-7
Author(s):  
Rong Chen ◽  
Chongguang Ren
Keyword(s):  
Natural Language Processing ◽  
Language Processing ◽  
Transfer Process ◽  
Negative Transfer ◽  
Domain Adaptation ◽  
Attention Mechanism ◽  
Target Domain ◽  
Source Domain ◽  
Second Order Statistics ◽  
Unsupervised Domain Adaptation

Domain adaptation aims to solve the problems of lacking labels. Most existing works of domain adaptation mainly focus on aligning the feature distributions between the source and target domain. However, in the field of Natural Language Processing, some of the words in different domains convey different sentiment. Thus not all features of the source domain should be transferred, and it would cause negative transfer when aligning the untransferable features. To address this issue, we propose a Correlation Alignment with Attention mechanism for unsupervised Domain Adaptation (CAADA) model. In the model, an attention mechanism is introduced into the transfer process for domain adaptation, which can capture the positively transferable features in source and target domain. Moreover, the CORrelation ALignment (CORAL) loss is utilized to minimize the domain discrepancy by aligning the second-order statistics of the positively transferable features extracted by the attention mechanism. Extensive experiments on the Amazon review dataset demonstrate the effectiveness of CAADA method.

scholarly journals Unsupervised Domain Adaptation by Statistics Alignment for Deep Sleep Staging Networks

2021 ◽  
Author(s):  
Jiahao Fan ◽  
Hangyu Zhu ◽  
Xinyu Jiang ◽  
Long Meng ◽  
Cong Fu ◽  
...  
Keyword(s):  
Large Scale ◽  
Domain Adaptation ◽  
Source Model ◽  
Deep Sleep ◽  
Sleep Staging ◽  
Target Domain ◽  
Source Domain ◽  
Unsupervised Domain Adaptation ◽  
Generalization Problem ◽  
Source Models

Deep sleep staging networks have reached top performance on large-scale datasets. However, these models perform poorer when training and testing on small sleep cohorts due to data inefficiency. Transferring well-trained models from large-scale datasets (source domain) to small sleep cohorts (target domain) is a promising solution but still remains challenging due to the domain-shift issue. In this work, an unsupervised domain adaptation approach, domain statistics alignment (DSA), is developed to bridge the gap between the data distribution of source and target domains. DSA adapts the source models on the target domain by modulating the domain-specific statistics of deep features stored in the Batch Normalization (BN) layers. Furthermore, we have extended DSA by introducing cross-domain statistics in each BN layer to perform DSA adaptively (AdaDSA). The proposed methods merely need the well-trained source model without access to the source data, which may be proprietary and inaccessible. DSA and AdaDSA are universally applicable to various deep sleep staging networks that have BN layers. We have validated the proposed methods by extensive experiments on two state-of-the-art deep sleep staging networks, DeepSleepNet+ and U-time. The performance was evaluated by conducting various transfer tasks on six sleep databases, including two large-scale databases, MASS and SHHS, as the source domain, four small sleep databases as the target domain. Thereinto, clinical sleep records acquired in Huashan Hospital, Shanghai, were used. The results show that both DSA and AdaDSA could significantly improve the performance of source models on target domains, providing novel insights into the domain generalization problem in sleep staging tasks.<br>

scholarly journals Multi-Source Domain Adaptation for Text Classification via DistanceNet-Bandits

2020 ◽  
Vol 34 (05) ◽  
pp. 7830-7838 ◽  
Author(s):  
Han Guo ◽  
Ramakanth Pasunuru ◽  
Mohit Bansal
Keyword(s):  
Loss Function ◽  
Optimal Trajectory ◽  
Domain Adaptation ◽  
Learning Algorithm ◽  
Data Distribution ◽  
Distance Measures ◽  
Target Domain ◽  
Source Domain ◽  
Unsupervised Domain Adaptation ◽  
Additional Loss

Domain adaptation performance of a learning algorithm on a target domain is a function of its source domain error and a divergence measure between the data distribution of these two domains. We present a study of various distance-based measures in the context of NLP tasks, that characterize the dissimilarity between domains based on sample estimates. We first conduct analysis experiments to show which of these distance measures can best differentiate samples from same versus different domains, and are correlated with empirical results. Next, we develop a DistanceNet model which uses these distance measures, or a mixture of these distance measures, as an additional loss function to be minimized jointly with the task's loss function, so as to achieve better unsupervised domain adaptation. Finally, we extend this model to a novel DistanceNet-Bandit model, which employs a multi-armed bandit controller to dynamically switch between multiple source domains and allow the model to learn an optimal trajectory and mixture of domains for transfer to the low-resource target domain. We conduct experiments on popular sentiment analysis datasets with several diverse domains and show that our DistanceNet model, as well as its dynamic bandit variant, can outperform competitive baselines in the context of unsupervised domain adaptation.

scholarly journals Unsupervised Domain Adaptation via Structured Prediction Based Selective Pseudo-Labeling

2020 ◽  
Vol 34 (04) ◽  
pp. 6243-6250 ◽  
Author(s):  
Qian Wang ◽  
Toby Breckon
Keyword(s):  
Domain Adaptation ◽  
Feature Space ◽  
Structured Prediction ◽  
Target Domain ◽  
Source Domain ◽  
Domain Specific ◽  
Unsupervised Domain Adaptation ◽  
Deep Feature ◽  
Significant Performance ◽  
Error Accumulation

Unsupervised domain adaptation aims to address the problem of classifying unlabeled samples from the target domain whilst labeled samples are only available from the source domain and the data distributions are different in these two domains. As a result, classifiers trained from labeled samples in the source domain suffer from significant performance drop when directly applied to the samples from the target domain. To address this issue, different approaches have been proposed to learn domain-invariant features or domain-specific classifiers. In either case, the lack of labeled samples in the target domain can be an issue which is usually overcome by pseudo-labeling. Inaccurate pseudo-labeling, however, could result in catastrophic error accumulation during learning. In this paper, we propose a novel selective pseudo-labeling strategy based on structured prediction. The idea of structured prediction is inspired by the fact that samples in the target domain are well clustered within the deep feature space so that unsupervised clustering analysis can be used to facilitate accurate pseudo-labeling. Experimental results on four datasets (i.e. Office-Caltech, Office31, ImageCLEF-DA and Office-Home) validate our approach outperforms contemporary state-of-the-art methods.

scholarly journals Unsupervised Domain Adaptation Method Based on Discriminant Sample Selection

2020 ◽  
Vol 38 (4) ◽  
pp. 828-837
Author(s):  
Linlin Wu ◽  
Guohua Peng ◽  
Weidong Yan
Keyword(s):  
Density Estimation ◽  
Classification Accuracy ◽  
Domain Adaptation ◽  
Sample Selection ◽  
Previous Method ◽  
Probability Density Estimation ◽  
Target Domain ◽  
Source Domain ◽  
Unsupervised Domain Adaptation ◽  
Adaptation Method

In order to solve the problem that low classification accuracy caused by the different distribution of training set and test set, an unsupervised domain adaptation method based on discriminant sample selection (DSS) is proposed. DSS projects the samples of different domains onto a same subspace to reduce the distribution discrepancy between the source domain and the target domain, and weights the source domain instances to make the samples more discriminant. Different from the previous method based on the probability density estimation of samples, DSS tries to obtain the sample weights by solving a quadratic programming problem, which avoids the distribution estimation of samples and can be applied to any fields without suffering from the dimensional trouble caused by high-dimensional density estimation. Finally, DSS congregates the same classes by minimizing the intra-class distance. Experimental results show that the proposed method improves the classification accuracy and robustness.

scholarly journals Unsupervised Domain Adaptation by Statistics Alignment for Deep Sleep Staging Networks

2021 ◽  
Author(s):  
Jiahao Fan ◽  
Hangyu Zhu ◽  
Xinyu Jiang ◽  
Long Meng ◽  
Cong Fu ◽  
...  
Keyword(s):  
Large Scale ◽  
Domain Adaptation ◽  
Source Model ◽  
Deep Sleep ◽  
Sleep Staging ◽  
Target Domain ◽  
Source Domain ◽  
Unsupervised Domain Adaptation ◽  
Generalization Problem ◽  
Source Models

Deep sleep staging networks have reached top performance on large-scale datasets. However, these models perform poorer when training and testing on small sleep cohorts due to data inefficiency. Transferring well-trained models from large-scale datasets (source domain) to small sleep cohorts (target domain) is a promising solution but still remains challenging due to the domain-shift issue. In this work, an unsupervised domain adaptation approach, domain statistics alignment (DSA), is developed to bridge the gap between the data distribution of source and target domains. DSA adapts the source models on the target domain by modulating the domain-specific statistics of deep features stored in the Batch Normalization (BN) layers. Furthermore, we have extended DSA by introducing cross-domain statistics in each BN layer to perform DSA adaptively (AdaDSA). The proposed methods merely need the well-trained source model without access to the source data, which may be proprietary and inaccessible. DSA and AdaDSA are universally applicable to various deep sleep staging networks that have BN layers. We have validated the proposed methods by extensive experiments on two state-of-the-art deep sleep staging networks, DeepSleepNet+ and U-time. The performance was evaluated by conducting various transfer tasks on six sleep databases, including two large-scale databases, MASS and SHHS, as the source domain, four small sleep databases as the target domain. Thereinto, clinical sleep records acquired in Huashan Hospital, Shanghai, were used. The results show that both DSA and AdaDSA could significantly improve the performance of source models on target domains, providing novel insights into the domain generalization problem in sleep staging tasks.<br>

scholarly journals A COMPARISON OF TWO STRATEGIES FOR AVOIDING NEGATIVE TRANSFER IN DOMAIN ADAPTATION BASED ON LOGISTIC REGRESSION

2018 ◽  
Vol XLII-2 ◽  
pp. 845-852 ◽  
Author(s):  
A. Paul ◽  
K. Vogt ◽  
F. Rottensteiner ◽  
J. Ostermann ◽  
C. Heipke
Keyword(s):  
Negative Transfer ◽  
Domain Adaptation ◽  
Classification Performance ◽  
Target Domain ◽  
Maximum Mean Discrepancy ◽  
Source Domain ◽  
Training Samples ◽  
Benchmark Datasets ◽  
Consistent Performance ◽  
Class Labels

In this paper we deal with the problem of measuring the similarity between training and tests datasets in the context of transfer learning (TL) for image classification. TL tries to transfer knowledge from a source domain, where labelled training samples are abundant but the data may follow a different distribution, to a target domain, where labelled training samples are scarce or even unavailable, assuming that the domains are related. Thus, the requirements w.r.t. the availability of labelled training samples in the target domain are reduced. In particular, if no labelled target data are available, it is inherently difficult to find a robust measure of relatedness between the source and target domains. This is of crucial importance for the performance of TL, because the knowledge transfer between unrelated data may lead to negative transfer, i.e. to a decrease of classification performance after transfer. We address the problem of measuring the relatedness between source and target datasets and investigate three different strategies to predict and, consequently, to avoid negative transfer in this paper. The first strategy is based on circular validation. The second strategy relies on the Maximum Mean Discrepancy (MMD) similarity metric, whereas the third one is an extension of MMD which incorporates the knowledge about the class labels in the source domain. Our method is evaluated using two different benchmark datasets. The experiments highlight the strengths and weaknesses of the investigated methods. We also show that it is possible to reduce the amount of negative transfer using these strategies for a TL method and to generate a consistent performance improvement over the whole dataset.

scholarly journals Bayesian Uncertainty Matching for Unsupervised Domain Adaptation

2019 ◽  
Author(s):  
Jun Wen ◽  
Nenggan Zheng ◽  
Junsong Yuan ◽  
Zhefeng Gong ◽  
Changyou Chen
Keyword(s):  
Negative Transfer ◽  
Domain Adaptation ◽  
Target Domain ◽  
Bayesian Neural Network ◽  
Prediction Uncertainty ◽  
Unsupervised Domain Adaptation ◽  
Distribution Matching ◽  
Benchmark Datasets ◽  
Label Distribution ◽  
Target Data

Domain adaptation is an important technique to alleviate performance degradation caused by domain shift, e.g., when training and test data come from different domains. Most existing deep adaptation methods focus on reducing domain shift by matching marginal feature distributions through deep transformations on the input features, due to the unavailability of target domain labels. We show that domain shift may still exist via label distribution shift at the classifier, thus deteriorating model performances. To alleviate this issue, we propose an approximate joint distribution matching scheme by exploiting prediction uncertainty. Specifically, we use a Bayesian neural network to quantify prediction uncertainty of a classifier. By imposing distribution matching on both features and labels (via uncertainty), label distribution mismatching in source and target data is effectively alleviated, encouraging the classifier to produce consistent predictions across domains. We also propose a few techniques to improve our method by adaptively reweighting domain adaptation loss to achieve nontrivial distribution matching and stable training. Comparisons with state of the art unsupervised domain adaptation methods on three popular benchmark datasets demonstrate the superiority of our approach, especially on the effectiveness of alleviating negative transfer.

scholarly journals Transferable Curriculum for Weakly-Supervised Domain Adaptation

2019 ◽  
Vol 33 ◽  
pp. 4951-4958 ◽  
Author(s):  
Yang Shu ◽  
Zhangjie Cao ◽  
Mingsheng Long ◽  
Jianmin Wang
Keyword(s):  
Negative Transfer ◽  
Domain Adaptation ◽  
State Of The Art ◽  
Domain Engineering ◽  
Target Domain ◽  
High Quality ◽  
Source Domain ◽  
Target Task ◽  
Deep Networks ◽  
Weakly Supervised

Domain adaptation improves a target task by knowledge transfer from a source domain with rich annotations. It is not uncommon that “source-domain engineering” becomes a cumbersome process in domain adaptation: the high-quality source domains highly related to the target domain are hardly available. Thus, weakly-supervised domain adaptation has been introduced to address this difficulty, where we can tolerate the source domain with noises in labels, features, or both. As such, for a particular target task, we simply collect the source domain with coarse labeling or corrupted data. In this paper, we try to address two entangled challenges of weaklysupervised domain adaptation: sample noises of the source domain and distribution shift across domains. To disentangle these challenges, a Transferable Curriculum Learning (TCL) approach is proposed to train the deep networks, guided by a transferable curriculum informing which of the source examples are noiseless and transferable. The approach enhances positive transfer from clean source examples to the target and mitigates negative transfer of noisy source examples. A thorough evaluation shows that our approach significantly outperforms the state-of-the-art on weakly-supervised domain adaptation tasks.

scholarly journals DC-FUDA: Improving Deep Clustering via Fully Unsupervised Domain Adaptation

2021 ◽  
Author(s):  
Zhimeng Yang ◽  
Zirui Wu ◽  
Ming Zeng ◽  
Yazhou Ren ◽  
Xiaorong Pu ◽  
...  
Keyword(s):  
Domain Adaptation ◽  
State Of The Art ◽  
Adaptive Threshold ◽  
Generation Process ◽  
Target Domain ◽  
Source Domain ◽  
Unsupervised Domain Adaptation ◽  
Adequate Amount ◽  
Traditional Approaches ◽  
Low Entropy

<div>By transferring knowledge from a source domain, the performance of deep clustering on an unlabeled target domain can be improved. When achieving this, traditional approaches make the assumption that adequate amount of labeled data is available in a source domain. However, this assumption is usually unrealistic in practice. The source domain should be carefully selected to share some characteristics with the target domain, and it can not be guaranteed that rich labeled samples are always available in the selected source domain.</div><div>We propose a novel framework to improve deep clustering by transferring knowledge from a source domain without any labeled data. To select reliable instances in the source domain for transferring, we propose a novel adaptive threshold algorithm to select low entropy instances. To transfer important features of the selected instances, we propose a feature-level domain adaptation network (FeatureDA) which cancels unstable generation process. With extensive experiments, we validate that our method effectively improves deep clustering, without using any labeled data in the source domain. Besides, without using any labeled data in the source domain, our method achieves competitive results, compared to the state-of-the-art methods using labeled data in the source domain.</div>

scholarly journals Clarinet: A One-step Approach Towards Budget-friendly Unsupervised Domain Adaptation

2020 ◽  
Author(s):  
Yiyang Zhang ◽  
Feng Liu ◽  
Zhen Fang ◽  
Bo Yuan ◽  
Guangquan Zhang ◽  
...  
Keyword(s):  
Domain Adaptation ◽  
Unlabeled Data ◽  
Target Domain ◽  
Source Domain ◽  
Label Data ◽  
Unsupervised Domain Adaptation ◽  
Adversarial Network ◽  
True Label ◽  
Source Data ◽  
One Step

In unsupervised domain adaptation (UDA), classifiers for the target domain are trained with massive true-label data from the source domain and unlabeled data from the target domain. However, it may be difficult to collect fully-true-label data in a source domain given limited budget. To mitigate this problem, we consider a novel problem setting where the classifier for the target domain has to be trained with complementary-label data from the source domain and unlabeled data from the target domain named budget-friendly UDA (BFUDA). The key benefit is that it is much less costly to collect complementary-label source data (required by BFUDA) than collecting the true-label source data (required by ordinary UDA). To this end, complementary label adversarial network (CLARINET) is proposed to solve the BFUDA problem. CLARINET maintains two deep networks simultaneously, where one focuses on classifying complementary-label source data and the other takes care of the source-to-target distributional adaptation. Experiments show that CLARINET significantly outperforms a series of competent baselines.

Sign in / Sign up

Export Citation Format

Share Document