scholarly journals Source-free Domain Adaptation via Avatar Prototype Generation and Adaptation

2021 ◽  
Author(s):  
Zhen Qiu ◽  
Yifan Zhang ◽  
Hongbin Lin ◽  
Shuaicheng Niu ◽  
Yanxia Liu ◽  
...  
Keyword(s):  
Data Privacy ◽  
Domain Adaptation ◽  
Adaptation Strategy ◽  
Source Model ◽  
Target Domain ◽  
Benchmark Datasets ◽  
Classification Boundary ◽  
Hidden Knowledge ◽  
Two Stages ◽  
Target Data

We study a practical domain adaptation task, called source-free unsupervised domain adaptation (UDA) problem, in which we cannot access source domain data due to data privacy issues but only a pre-trained source model and unlabeled target data are available. This task, however, is very difficult due to one key challenge: the lack of source data and target domain labels makes model adaptation very challenging. To address this, we propose to mine the hidden knowledge in the source model and exploit it to generate source avatar prototypes (i.e. representative features for each source class) as well as target pseudo labels for domain alignment. To this end, we propose a Contrastive Prototype Generation and Adaptation (CPGA) method. Specifically, CPGA consists of two stages: (1) prototype generation: by exploring the classification boundary information of the source model, we train a prototype generator to generate avatar prototypes via contrastive learning. (2) prototype adaptation: based on the generated source prototypes and target pseudo labels, we develop a new robust contrastive prototype adaptation strategy to align each pseudo-labeled target data to the corresponding source prototypes. Extensive experiments on three UDA benchmark datasets demonstrate the effectiveness and superiority of the proposed method.

scholarly journals Domain Adaptation for Semantic Parsing

2020 ◽  
Author(s):  
Zechang Li ◽  
Yuxuan Lai ◽  
Yansong Feng ◽  
Dongyan Zhao
Keyword(s):  
Domain Adaptation ◽  
Specific Information ◽  
Semantic Parsing ◽  
Target Domain ◽  
Source Domain ◽  
Domain Specific ◽  
The Difference ◽  
Two Stages ◽  
Coarse To Fine ◽  
Target Data

Recently, semantic parsing has attracted much attention in the community. Although many neural modeling efforts have greatly improved the performance, it still suffers from the data scarcity issue. In this paper, we propose a novel semantic parser for domain adaptation, where we have much fewer annotated data in the target domain compared to the source domain. Our semantic parser benefits from a two-stage coarse-to-fine framework, thus can provide different and accurate treatments for the two stages, i.e., focusing on domain invariant and domain specific information, respectively. In the coarse stage, our novel domain discrimination component and domain relevance attention encourage the model to learn transferable domain general structures. In the fine stage, the model is guided to concentrate on domain related details. Experiments on a benchmark dataset show that our method consistently outperforms several popular domain adaptation strategies. Additionally, we show that our model can well exploit limited target data to capture the difference between the source and target domain, even when the target domain has far fewer training instances.

scholarly journals Bayesian Uncertainty Matching for Unsupervised Domain Adaptation

2019 ◽  
Author(s):  
Jun Wen ◽  
Nenggan Zheng ◽  
Junsong Yuan ◽  
Zhefeng Gong ◽  
Changyou Chen
Keyword(s):  
Negative Transfer ◽  
Domain Adaptation ◽  
Target Domain ◽  
Bayesian Neural Network ◽  
Prediction Uncertainty ◽  
Unsupervised Domain Adaptation ◽  
Distribution Matching ◽  
Benchmark Datasets ◽  
Label Distribution ◽  
Target Data

Domain adaptation is an important technique to alleviate performance degradation caused by domain shift, e.g., when training and test data come from different domains. Most existing deep adaptation methods focus on reducing domain shift by matching marginal feature distributions through deep transformations on the input features, due to the unavailability of target domain labels. We show that domain shift may still exist via label distribution shift at the classifier, thus deteriorating model performances. To alleviate this issue, we propose an approximate joint distribution matching scheme by exploiting prediction uncertainty. Specifically, we use a Bayesian neural network to quantify prediction uncertainty of a classifier. By imposing distribution matching on both features and labels (via uncertainty), label distribution mismatching in source and target data is effectively alleviated, encouraging the classifier to produce consistent predictions across domains. We also propose a few techniques to improve our method by adaptively reweighting domain adaptation loss to achieve nontrivial distribution matching and stable training. Comparisons with state of the art unsupervised domain adaptation methods on three popular benchmark datasets demonstrate the superiority of our approach, especially on the effectiveness of alleviating negative transfer.

scholarly journals Differentially Private Correlation Alignment for Domain Adaptation

2021 ◽  
Author(s):  
Kaizhong Jin ◽  
Xiang Cheng ◽  
Jiaxi Yang ◽  
Kaiyuan Shen
Keyword(s):  
Covariance Matrix ◽  
Domain Adaptation ◽  
Estimation Method ◽  
Training Data ◽  
Target Domain ◽  
Source Domain ◽  
Feature Spaces ◽  
Benchmark Datasets ◽  
Low Dimensional ◽  
Target Data

Domain adaptation solves a learning problem in a target domain by utilizing the training data in a different but related source domain. As a simple and efficient method for domain adaptation, correlation alignment transforms the distribution of the source domain by utilizing the covariance matrix of the target domain, such that a model trained on the transformed source data can be applied to the target data. However, when source and target domains come from different institutes, exchanging information between the two domains might pose a potential privacy risk. In this paper, for the first time, we propose a differentially private correlation alignment approach for domain adaptation called PRIMA, which can provide privacy guarantees for both the source and target data. In PRIMA, to relieve the performance degradation caused by perturbing the covariance matrix in high dimensional setting, we present a random subspace ensemble based covariance estimation method which splits the feature spaces of source and target data into several low dimensional subspaces. Moreover, since perturbing the covariance matrix may destroy its positive semi-definiteness, we develop a shrinking based method for the recovery of positive semi-definiteness of the covariance matrix. Experimental results on standard benchmark datasets confirm the effectiveness of our approach.

scholarly journals TIDOT: A Teacher Imitation Learning Approach for Domain Adaptation with Optimal Transport

2021 ◽  
Author(s):  
Tuan Nguyen ◽  
Trung Le ◽  
Nhan Dam ◽  
Quan Hung Tran ◽  
Truyen Nguyen ◽  
...  
Keyword(s):  
Optimal Transport ◽  
Domain Adaptation ◽  
Empirical Studies ◽  
Joint Space ◽  
Imitation Learning ◽  
Target Domain ◽  
Cooperative Agents ◽  
Benchmark Datasets ◽  
Novel Model ◽  
Target Data

Using the principle of imitation learning and the theory of optimal transport we propose in this paper a novel model for unsupervised domain adaptation named Teacher Imitation Domain Adaptation with Optimal Transport (TIDOT). Our model includes two cooperative agents: a teacher and a student. The former agent is trained to be an expert on labeled data in the source domain, whilst the latter one aims to work with unlabeled data in the target domain. More specifically, optimal transport is applied to quantify the total of the distance between embedded distributions of the source and target data in the joint space, and the distance between predictive distributions of both agents, thus by minimizing this quantity TIDOT could mitigate not only the data shift but also the label shift. Comprehensive empirical studies show that TIDOT outperforms existing state-of-the-art performance on benchmark datasets.

scholarly journals TSAL: Two steps Adversarial learning based domain adaptation

2021 ◽  
Vol 34 (1) ◽  
Author(s):  
Haidi Hasan Badr ◽  
Nayer Mahmoud Wanas ◽  
Magda Fayek
Keyword(s):  
Network Architecture ◽  
Domain Adaptation ◽  
Data Availability ◽  
Fine Tuning ◽  
Target Domain ◽  
Adversarial Learning ◽  
Prior Art ◽  
Real World Problem ◽  
Wide Range ◽  
Benchmark Datasets

Since labeled data availability differs greatly across domains, Domain Adaptation focuses on learning in new and unfamiliar domains by reducing distribution divergence. Recent research suggests that the adversarial learning approach could be a promising way to achieve the domain adaptation objective. Adversarial learning is a strategy for learning domain-transferable features in robust deep networks. This paper introduces the TSAL paradigm, a two-step adversarial learning framework. It addresses the real-world problem of text classification, where source domain(s) has labeled data but target domain (s) has only unlabeled data. TSAL utilizes joint adversarial learning with class information and domain alignment deep network architecture to learn both domain-invariant and domain-specific features extractors. It consists of two training steps that are similar to the paradigm, in which pre-trained model weights are used as initialization for training with new data. TSAL’s two training phases, however, are based on the same data, not different data, as is the case with fine-tuning. Furthermore, TSAL only uses the learned domain-invariant feature extractor from the first training as an initialization for its peer in subsequent training. By doubling the training, TSAL can emphasize the leverage of the small unlabeled target domain and learn effectively what to share between various domains. A detailed analysis of many benchmark datasets reveals that our model consistently outperforms the prior art across a wide range of dataset distributions.

scholarly journals Unsupervised Domain Adaptation by Statistics Alignment for Deep Sleep Staging Networks

2021 ◽  
Author(s):  
Jiahao Fan ◽  
Hangyu Zhu ◽  
Xinyu Jiang ◽  
Long Meng ◽  
Cong Fu ◽  
...  
Keyword(s):  
Large Scale ◽  
Domain Adaptation ◽  
Source Model ◽  
Deep Sleep ◽  
Sleep Staging ◽  
Target Domain ◽  
Source Domain ◽  
Unsupervised Domain Adaptation ◽  
Generalization Problem ◽  
Source Models

Deep sleep staging networks have reached top performance on large-scale datasets. However, these models perform poorer when training and testing on small sleep cohorts due to data inefficiency. Transferring well-trained models from large-scale datasets (source domain) to small sleep cohorts (target domain) is a promising solution but still remains challenging due to the domain-shift issue. In this work, an unsupervised domain adaptation approach, domain statistics alignment (DSA), is developed to bridge the gap between the data distribution of source and target domains. DSA adapts the source models on the target domain by modulating the domain-specific statistics of deep features stored in the Batch Normalization (BN) layers. Furthermore, we have extended DSA by introducing cross-domain statistics in each BN layer to perform DSA adaptively (AdaDSA). The proposed methods merely need the well-trained source model without access to the source data, which may be proprietary and inaccessible. DSA and AdaDSA are universally applicable to various deep sleep staging networks that have BN layers. We have validated the proposed methods by extensive experiments on two state-of-the-art deep sleep staging networks, DeepSleepNet+ and U-time. The performance was evaluated by conducting various transfer tasks on six sleep databases, including two large-scale databases, MASS and SHHS, as the source domain, four small sleep databases as the target domain. Thereinto, clinical sleep records acquired in Huashan Hospital, Shanghai, were used. The results show that both DSA and AdaDSA could significantly improve the performance of source models on target domains, providing novel insights into the domain generalization problem in sleep staging tasks.<br>

scholarly journals Learning Discriminative Correlation Subspace for Heterogeneous Domain Adaptation

2017 ◽  
Author(s):  
Yuguang Yan ◽  
Wen Li ◽  
Michael Ng ◽  
Mingkui Tan ◽  
Hanrui Wu ◽  
...  
Keyword(s):  
Optimization Problem ◽  
Domain Adaptation ◽  
Data Sets ◽  
Target Domain ◽  
Real World Data ◽  
Discriminative Ability ◽  
Convex Optimization Problem ◽  
Alternating Direction ◽  
Feature Spaces ◽  
Target Data

Domain adaptation aims to reduce the effort on collecting and annotating target data by leveraging knowledge from a different source domain. The domain adaptation problem will become extremely challenging when the feature spaces of the source and target domains are different, which is also known as the heterogeneous domain adaptation (HDA) problem. In this paper, we propose a novel HDA method to find the optimal discriminative correlation subspace for the source and target data. The discriminative correlation subspace is inherited from the canonical correlation subspace between the source and target data, and is further optimized to maximize the discriminative ability for the target domain classifier. We formulate a joint objective in order to simultaneously learn the discriminative correlation subspace and the target domain classifier. We then apply an alternating direction method of multiplier (ADMM) algorithm to address the resulting non-convex optimization problem. Comprehensive experiments on two real-world data sets demonstrate the effectiveness of the proposed method compared to the state-of-the-art methods.

ITERATIVE SELF-LABELING DOMAIN ADAPTATION FOR LINEAR STRUCTURED IMAGE CLASSIFICATION

2013 ◽  
Vol 22 (05) ◽  
pp. 1360005 ◽  
Author(s):  
AMAURY HABRARD ◽  
JEAN-PHILIPPE PEYRACHE ◽  
MARC SEBBAN
Keyword(s):  
Image Classification ◽  
Domain Adaptation ◽  
Research Area ◽  
Target Domain ◽  
Sparse Models ◽  
Similarity Functions ◽  
Generalization Bounds ◽  
Source Data ◽  
New Research ◽  
Target Data

A strong assumption to derive generalization guarantees in the standard PAC framework is that training (or source) data and test (or target) data are drawn according to the same distribution. Because of the presence of possibly outdated data in the training set, or the use of biased collections, this assumption is often violated in real-world applications leading to different source and target distributions. To go around this problem, a new research area known as Domain Adaptation (DA) has recently been introduced giving rise to many adaptation algorithms and theoretical results in the form of generalization bounds. This paper deals with self-labeling DA whose goal is to iteratively incorporate semi-labeled target data in the learning set to progressively adapt the classifier from the source to the target domain. The contribution of this work is three-fold: First, we provide the minimum and necessary theoretical conditions for a self-labeling DA algorithm to perform an actual domain adaptation. Second, following these theoretical recommendations, we design a new iterative DA algorithm, called GESIDA, able to deal with structured data. This algorithm makes use of the new theory of learning with (ε,γ,τ)-good similarity functions introduced by Balcan et al., which does not require the use of a valid kernel to learn well and allows us to induce sparse models. Finally, we apply our algorithm on a structured image classification task and show that self-labeling domain adaptation is a new original way to deal with scaling and rotation problems.

scholarly journals Domain Adaptation for Document Image Binarization via Domain Classification

2021 ◽  
Author(s):  
Carlos Garrido-Munoz ◽  
Adrián Sánchez-Hernández ◽  
Francisco J. Castellanos ◽  
Jorge Calvo-Zaragoza
Keyword(s):  
Domain Adaptation ◽  
Adaptation Strategy ◽  
Document Image ◽  
Target Domain ◽  
Current State ◽  
Document Image Binarization ◽  
Initial Hypothesis ◽  
Source Models ◽  
Target Set ◽  
Multiple Domains

Binarization represents a key role in many document image analysis workflows. The current state of the art considers the use of supervised learning, and specifically deep neural networks. However, it is very difficult for the same model to work successfully in a number of document styles, since the set of potential domains is very heterogeneous. We study a multi-source domain adaptation strategy for binarization. Within this scenario, we look into a novel hypothesis where a specialized binarization model must be selected to be used over a target domain, instead of a single model that tries to generalize across multiple domains. The problem then boils down to, given several specialized models and a new target set, deciding which model to use. We propose here a simple way to address this question by using a domain classifier, that estimates which of the source models must be considered to binarize the new target domain. Our experiments on several datasets, including different text styles and music scores, show that our initial hypothesis is quite promising, yet the way to deal with the decision of which model to use still shows great room for improvement.

scholarly journals Self-Ensembling Attention Networks: Addressing Domain Shift for Semantic Segmentation

2019 ◽  
Vol 33 ◽  
pp. 5581-5588 ◽  
Author(s):  
Yonghao Xu ◽  
Bo Du ◽  
Lefei Zhang ◽  
Qian Zhang ◽  
Guoli Wang ◽  
...  
Keyword(s):  
Domain Adaptation ◽  
State Of The Art ◽  
Semantic Segmentation ◽  
Great Success ◽  
Learning Models ◽  
Target Domain ◽  
Attention Networks ◽  
Source Domain ◽  
Benchmark Datasets ◽  
Different Levels

Recent years have witnessed the great success of deep learning models in semantic segmentation. Nevertheless, these models may not generalize well to unseen image domains due to the phenomenon of domain shift. Since pixel-level annotations are laborious to collect, developing algorithms which can adapt labeled data from source domain to target domain is of great significance. To this end, we propose self-ensembling attention networks to reduce the domain gap between different datasets. To the best of our knowledge, the proposed method is the first attempt to introduce selfensembling model to domain adaptation for semantic segmentation, which provides a different view on how to learn domain-invariant features. Besides, since different regions in the image usually correspond to different levels of domain gap, we introduce the attention mechanism into the proposed framework to generate attention-aware features, which are further utilized to guide the calculation of consistency loss in the target domain. Experiments on two benchmark datasets demonstrate that the proposed framework can yield competitive performance compared with the state of the art methods.

Sign in / Sign up

Export Citation Format

Share Document