scholarly journals Bayesian Uncertainty Matching for Unsupervised Domain Adaptation

2019 ◽  
Author(s):  
Jun Wen ◽  
Nenggan Zheng ◽  
Junsong Yuan ◽  
Zhefeng Gong ◽  
Changyou Chen
Keyword(s):  
Negative Transfer ◽  
Domain Adaptation ◽  
Target Domain ◽  
Bayesian Neural Network ◽  
Prediction Uncertainty ◽  
Unsupervised Domain Adaptation ◽  
Distribution Matching ◽  
Benchmark Datasets ◽  
Label Distribution ◽  
Target Data

Domain adaptation is an important technique to alleviate performance degradation caused by domain shift, e.g., when training and test data come from different domains. Most existing deep adaptation methods focus on reducing domain shift by matching marginal feature distributions through deep transformations on the input features, due to the unavailability of target domain labels. We show that domain shift may still exist via label distribution shift at the classifier, thus deteriorating model performances. To alleviate this issue, we propose an approximate joint distribution matching scheme by exploiting prediction uncertainty. Specifically, we use a Bayesian neural network to quantify prediction uncertainty of a classifier. By imposing distribution matching on both features and labels (via uncertainty), label distribution mismatching in source and target data is effectively alleviated, encouraging the classifier to produce consistent predictions across domains. We also propose a few techniques to improve our method by adaptively reweighting domain adaptation loss to achieve nontrivial distribution matching and stable training. Comparisons with state of the art unsupervised domain adaptation methods on three popular benchmark datasets demonstrate the superiority of our approach, especially on the effectiveness of alleviating negative transfer.

scholarly journals Exploiting Local Feature Patterns for Unsupervised Domain Adaptation

2019 ◽  
Vol 33 ◽  
pp. 5401-5408 ◽  
Author(s):  
Jun Wen ◽  
Risheng Liu ◽  
Nenggan Zheng ◽  
Qian Zheng ◽  
Zhefeng Gong ◽  
...  
Keyword(s):  
Negative Transfer ◽  
Domain Adaptation ◽  
Local Feature ◽  
Fine Grained ◽  
Unsupervised Domain Adaptation ◽  
Distribution Matching ◽  
Benchmark Datasets ◽  
Invariant Representations ◽  
Multi Mode ◽  
Feature Alignment

Unsupervised domain adaptation methods aim to alleviate performance degradation caused by domain-shift by learning domain-invariant representations. Existing deep domain adaptation methods focus on holistic feature alignment by matching source and target holistic feature distributions, without considering local features and their multi-mode statistics. We show that the learned local feature patterns are more generic and transferable and a further local feature distribution matching enables fine-grained feature alignment. In this paper, we present a method for learning domain-invariant local feature patterns and jointly aligning holistic and local feature statistics. Comparisons to the state-of-the-art unsupervised domain adaptation methods on two popular benchmark datasets demonstrate the superiority of our approach and its effectiveness on alleviating negative transfer.

Correlation alignment with attention mechanism for unsupervised domain adaptation

2021 ◽  
pp. 1-7
Author(s):  
Rong Chen ◽  
Chongguang Ren
Keyword(s):  
Natural Language Processing ◽  
Language Processing ◽  
Transfer Process ◽  
Negative Transfer ◽  
Domain Adaptation ◽  
Attention Mechanism ◽  
Target Domain ◽  
Source Domain ◽  
Second Order Statistics ◽  
Unsupervised Domain Adaptation

Domain adaptation aims to solve the problems of lacking labels. Most existing works of domain adaptation mainly focus on aligning the feature distributions between the source and target domain. However, in the field of Natural Language Processing, some of the words in different domains convey different sentiment. Thus not all features of the source domain should be transferred, and it would cause negative transfer when aligning the untransferable features. To address this issue, we propose a Correlation Alignment with Attention mechanism for unsupervised Domain Adaptation (CAADA) model. In the model, an attention mechanism is introduced into the transfer process for domain adaptation, which can capture the positively transferable features in source and target domain. Moreover, the CORrelation ALignment (CORAL) loss is utilized to minimize the domain discrepancy by aligning the second-order statistics of the positively transferable features extracted by the attention mechanism. Extensive experiments on the Amazon review dataset demonstrate the effectiveness of CAADA method.

scholarly journals Bidirectional Adversarial Training for Semi-Supervised Domain Adaptation

2020 ◽  
Author(s):  
Pin Jiang ◽  
Aming Wu ◽  
Yahong Han ◽  
Yunfeng Shao ◽  
Meiyu Qi ◽  
...  
Keyword(s):  
Additional Data ◽  
Domain Adaptation ◽  
State Of The Art ◽  
Powerful Method ◽  
Target Domain ◽  
Unsupervised Domain Adaptation ◽  
Benchmark Datasets ◽  
Adversarial Examples ◽  
Adversarial Training ◽  
Effective Use

Semi-supervised domain adaptation (SSDA) is a novel branch of machine learning that scarce labeled target examples are available, compared with unsupervised domain adaptation. To make effective use of these additional data so as to bridge the domain gap, one possible way is to generate adversarial examples, which are images with additional perturbations, between the two domains and fill the domain gap. Adversarial training has been proven to be a powerful method for this purpose. However, the traditional adversarial training adds noises in arbitrary directions, which is inefficient to migrate between domains, or generate directional noises from the source to target domain and reverse. In this work, we devise a general bidirectional adversarial training method and employ gradient to guide adversarial examples across the domain gap, i.e., the Adaptive Adversarial Training (AAT) for source to target domain and Entropy-penalized Virtual Adversarial Training (E-VAT) for target to source domain. Particularly, we devise a Bidirectional Adversarial Training (BiAT) network to perform diverse adversarial trainings jointly. We evaluate the effectiveness of BiAT on three benchmark datasets and experimental results demonstrate the proposed method achieves the state-of-the-art.

scholarly journals Source-free Domain Adaptation via Avatar Prototype Generation and Adaptation

2021 ◽  
Author(s):  
Zhen Qiu ◽  
Yifan Zhang ◽  
Hongbin Lin ◽  
Shuaicheng Niu ◽  
Yanxia Liu ◽  
...  
Keyword(s):  
Data Privacy ◽  
Domain Adaptation ◽  
Adaptation Strategy ◽  
Source Model ◽  
Target Domain ◽  
Benchmark Datasets ◽  
Classification Boundary ◽  
Hidden Knowledge ◽  
Two Stages ◽  
Target Data

We study a practical domain adaptation task, called source-free unsupervised domain adaptation (UDA) problem, in which we cannot access source domain data due to data privacy issues but only a pre-trained source model and unlabeled target data are available. This task, however, is very difficult due to one key challenge: the lack of source data and target domain labels makes model adaptation very challenging. To address this, we propose to mine the hidden knowledge in the source model and exploit it to generate source avatar prototypes (i.e. representative features for each source class) as well as target pseudo labels for domain alignment. To this end, we propose a Contrastive Prototype Generation and Adaptation (CPGA) method. Specifically, CPGA consists of two stages: (1) prototype generation: by exploring the classification boundary information of the source model, we train a prototype generator to generate avatar prototypes via contrastive learning. (2) prototype adaptation: based on the generated source prototypes and target pseudo labels, we develop a new robust contrastive prototype adaptation strategy to align each pseudo-labeled target data to the corresponding source prototypes. Extensive experiments on three UDA benchmark datasets demonstrate the effectiveness and superiority of the proposed method.

scholarly journals A COMPARISON OF TWO STRATEGIES FOR AVOIDING NEGATIVE TRANSFER IN DOMAIN ADAPTATION BASED ON LOGISTIC REGRESSION

2018 ◽  
Vol XLII-2 ◽  
pp. 845-852 ◽  
Author(s):  
A. Paul ◽  
K. Vogt ◽  
F. Rottensteiner ◽  
J. Ostermann ◽  
C. Heipke
Keyword(s):  
Negative Transfer ◽  
Domain Adaptation ◽  
Classification Performance ◽  
Target Domain ◽  
Maximum Mean Discrepancy ◽  
Source Domain ◽  
Training Samples ◽  
Benchmark Datasets ◽  
Consistent Performance ◽  
Class Labels

In this paper we deal with the problem of measuring the similarity between training and tests datasets in the context of transfer learning (TL) for image classification. TL tries to transfer knowledge from a source domain, where labelled training samples are abundant but the data may follow a different distribution, to a target domain, where labelled training samples are scarce or even unavailable, assuming that the domains are related. Thus, the requirements w.r.t. the availability of labelled training samples in the target domain are reduced. In particular, if no labelled target data are available, it is inherently difficult to find a robust measure of relatedness between the source and target domains. This is of crucial importance for the performance of TL, because the knowledge transfer between unrelated data may lead to negative transfer, i.e. to a decrease of classification performance after transfer. We address the problem of measuring the relatedness between source and target datasets and investigate three different strategies to predict and, consequently, to avoid negative transfer in this paper. The first strategy is based on circular validation. The second strategy relies on the Maximum Mean Discrepancy (MMD) similarity metric, whereas the third one is an extension of MMD which incorporates the knowledge about the class labels in the source domain. Our method is evaluated using two different benchmark datasets. The experiments highlight the strengths and weaknesses of the investigated methods. We also show that it is possible to reduce the amount of negative transfer using these strategies for a TL method and to generate a consistent performance improvement over the whole dataset.

scholarly journals Unsupervised Domain Adaptation on Reading Comprehension

2020 ◽  
Vol 34 (05) ◽  
pp. 7480-7487
Author(s):  
Yu Cao ◽  
Meng Fang ◽  
Baosheng Yu ◽  
Joey Tianyi Zhou
Keyword(s):  
Reading Comprehension ◽  
Large Scale ◽  
Domain Adaptation ◽  
Generalization Capability ◽  
Target Domain ◽  
Adversarial Learning ◽  
Unsupervised Domain Adaptation ◽  
Comparable Performance ◽  
Benchmark Datasets ◽  
Contextual Representation

Reading comprehension (RC) has been studied in a variety of datasets with the boosted performance brought by deep neural networks. However, the generalization capability of these models across different domains remains unclear. To alleviate the problem, we investigate unsupervised domain adaptation on RC, wherein a model is trained on the labeled source domain and to be applied to the target domain with only unlabeled samples. We first show that even with the powerful BERT contextual representation, a model can not generalize well from one domain to another. To solve this, we provide a novel conditional adversarial self-training method (CASe). Specifically, our approach leverages a BERT model fine-tuned on the source dataset along with the confidence filtering to generate reliable pseudo-labeled samples in the target domain for self-training. On the other hand, it further reduces domain distribution discrepancy through conditional adversarial learning across domains. Extensive experiments show our approach achieves comparable performance to supervised models on multiple large-scale benchmark datasets.

scholarly journals Differentially Private Correlation Alignment for Domain Adaptation

2021 ◽  
Author(s):  
Kaizhong Jin ◽  
Xiang Cheng ◽  
Jiaxi Yang ◽  
Kaiyuan Shen
Keyword(s):  
Covariance Matrix ◽  
Domain Adaptation ◽  
Estimation Method ◽  
Training Data ◽  
Target Domain ◽  
Source Domain ◽  
Feature Spaces ◽  
Benchmark Datasets ◽  
Low Dimensional ◽  
Target Data

Domain adaptation solves a learning problem in a target domain by utilizing the training data in a different but related source domain. As a simple and efficient method for domain adaptation, correlation alignment transforms the distribution of the source domain by utilizing the covariance matrix of the target domain, such that a model trained on the transformed source data can be applied to the target data. However, when source and target domains come from different institutes, exchanging information between the two domains might pose a potential privacy risk. In this paper, for the first time, we propose a differentially private correlation alignment approach for domain adaptation called PRIMA, which can provide privacy guarantees for both the source and target data. In PRIMA, to relieve the performance degradation caused by perturbing the covariance matrix in high dimensional setting, we present a random subspace ensemble based covariance estimation method which splits the feature spaces of source and target data into several low dimensional subspaces. Moreover, since perturbing the covariance matrix may destroy its positive semi-definiteness, we develop a shrinking based method for the recovery of positive semi-definiteness of the covariance matrix. Experimental results on standard benchmark datasets confirm the effectiveness of our approach.

scholarly journals Aligning Domain-Specific Distribution and Classifier for Cross-Domain Classification from Multiple Sources

2019 ◽  
Vol 33 ◽  
pp. 5989-5996 ◽  
Author(s):  
Yongchun Zhu ◽  
Fuzhen Zhuang ◽  
Deqing Wang
Keyword(s):  
Domain Adaptation ◽  
Feature Space ◽  
Multiple Sources ◽  
Target Domain ◽  
Domain Specific ◽  
Unsupervised Domain Adaptation ◽  
Specific Distribution ◽  
Benchmark Datasets ◽  
Invariant Representations ◽  
Decision Boundaries

While Unsupervised Domain Adaptation (UDA) algorithms, i.e., there are only labeled data from source domains, have been actively studied in recent years, most algorithms and theoretical results focus on Single-source Unsupervised Domain Adaptation (SUDA). However, in the practical scenario, labeled data can be typically collected from multiple diverse sources, and they might be different not only from the target domain but also from each other. Thus, domain adapters from multiple sources should not be modeled in the same way. Recent deep learning based Multi-source Unsupervised Domain Adaptation (MUDA) algorithms focus on extracting common domain-invariant representations for all domains by aligning distribution of all pairs of source and target domains in a common feature space. However, it is often very hard to extract the same domain-invariant representations for all domains in MUDA. In addition, these methods match distributions without considering domain-specific decision boundaries between classes. To solve these problems, we propose a new framework with two alignment stages for MUDA which not only respectively aligns the distributions of each pair of source and target domains in multiple specific feature spaces, but also aligns the outputs of classifiers by utilizing the domainspecific decision boundaries. Extensive experiments demonstrate that our method can achieve remarkable results on popular benchmark datasets for image classification.

scholarly journals Unsupervised Domain Adaptation by Matching Distributions Based on the Maximum Mean Discrepancy via Unilateral Transformations

2019 ◽  
Vol 33 ◽  
pp. 4106-4113 ◽  
Author(s):  
Atsutoshi Kumagai ◽  
Tomoharu Iwata
Keyword(s):  
Domain Adaptation ◽  
Feature Space ◽  
Classification Performance ◽  
Target Domain ◽  
Target Feature ◽  
Maximum Mean Discrepancy ◽  
Source Domain ◽  
Unsupervised Domain Adaptation ◽  
Real World Datasets ◽  
Target Data

We propose a simple yet effective method for unsupervised domain adaptation. When training and test distributions are different, standard supervised learning methods perform poorly. Semi-supervised domain adaptation methods have been developed for the case where labeled data in the target domain are available. However, the target data are often unlabeled in practice. Therefore, unsupervised domain adaptation, which does not require labels for target data, is receiving a lot of attention. The proposed method minimizes the discrepancy between the source and target distributions of input features by transforming the feature space of the source domain. Since such unilateral transformations transfer knowledge in the source domain to the target one without reducing dimensionality, the proposed method can effectively perform domain adaptation without losing information to be transfered. With the proposed method, it is assumed that the transformed features and the original features differ by a small residual to preserve the relationship between features and labels. This transformation is learned by aligning the higher-order moments of the source and target feature distributions based on the maximum mean discrepancy, which enables to compare two distributions without density estimation. Once the transformation is found, we learn supervised models by using the transformed source data and their labels. We use two real-world datasets to demonstrate experimentally that the proposed method achieves better classification performance than existing methods for unsupervised domain adaptation.

scholarly journals Improving Domain-Specific Classification by Collaborative Learning with Adaptation Networks

2019 ◽  
Vol 33 ◽  
pp. 5450-5457 ◽  
Author(s):  
Si Wu ◽  
Jian Zhong ◽  
Wenming Cao ◽  
Rui Li ◽  
Zhiwen Yu ◽  
...  
Keyword(s):  
Domain Adaptation ◽  
Visual Adaptation ◽  
Generalization Capability ◽  
Target Domain ◽  
Domain Specific ◽  
Unsupervised Domain Adaptation ◽  
Source Data ◽  
Data Points ◽  
Invariant Representations ◽  
Target Data

For unsupervised domain adaptation, the process of learning domain-invariant representations could be dominated by the labeled source data, such that the specific characteristics of the target domain may be ignored. In order to improve the performance in inferring target labels, we propose a targetspecific network which is capable of learning collaboratively with a domain adaptation network, instead of directly minimizing domain discrepancy. A clustering regularization is also utilized to improve the generalization capability of the target-specific network by forcing target data points to be close to accumulated class centers. As this network learns and specializes to the target domain, its performance in inferring target labels improves, which in turn facilitates the learning process of the adaptation network. Therefore, there is a mutually beneficial relationship between these two networks. We perform extensive experiments on multiple digit and object datasets, and the effectiveness and superiority of the proposed approach is presented and verified on multiple visual adaptation benchmarks, e.g., we improve the state-ofthe-art on the task of MNIST→SVHN from 76.5% to 84.9% without specific augmentation.

Sign in / Sign up

Export Citation Format

Share Document