scholarly journals Transfer Learning and Deep Domain Adaptation

2020 ◽  
Author(s):  
Wen Xu ◽  
Jing He ◽  
Yanfeng Shu
Keyword(s):  
Machine Learning ◽  
Neural Networks ◽  
Transfer Learning ◽  
Real World ◽  
Deep Neural Networks ◽  
Domain Adaptation ◽  
Fine Tuning ◽  
Real World Applications ◽  
Comprehensive Survey ◽  
Sample Reconstruction

Transfer learning is an emerging technique in machine learning, by which we can solve a new task with the knowledge obtained from an old task in order to address the lack of labeled data. In particular deep domain adaptation (a branch of transfer learning) gets the most attention in recently published articles. The intuition behind this is that deep neural networks usually have a large capacity to learn representation from one dataset and part of the information can be further used for a new task. In this research, we firstly present the complete scenarios of transfer learning according to the domains and tasks. Secondly, we conduct a comprehensive survey related to deep domain adaptation and categorize the recent advances into three types based on implementing approaches: fine-tuning networks, adversarial domain adaptation, and sample-reconstruction approaches. Thirdly, we discuss the details of these methods and introduce some typical real-world applications. Finally, we conclude our work and explore some potential issues to be further addressed.

Chapter 15. Human-Centered Concept Explanations for Neural Networks

2021 ◽  
Author(s):  
Chih-Kuan Yeh ◽  
Been Kim ◽  
Pradeep Ravikumar
Keyword(s):  
Machine Learning ◽  
Neural Networks ◽  
Case Studies ◽  
Real World ◽  
Deep Neural Networks ◽  
Learning Models ◽  
Real World Applications ◽  
The Right ◽  
Concept Activation ◽  
Machine Learning Models

Understanding complex machine learning models such as deep neural networks with explanations is crucial in various applications. Many explanations stem from the model perspective, and may not necessarily effectively communicate why the model is making its predictions at the right level of abstraction. For example, providing importance weights to individual pixels in an image can only express which parts of that particular image is important to the model, but humans may prefer an explanation which explains the prediction by concept-based thinking. In this work, we review the emerging area of concept based explanations. We start by introducing concept explanations including the class of Concept Activation Vectors (CAV) which characterize concepts using vectors in appropriate spaces of neural activations, and discuss different properties of useful concepts, and approaches to measure the usefulness of concept vectors. We then discuss approaches to automatically extract concepts, and approaches to address some of their caveats. Finally, we discuss some case studies that showcase the utility of such concept-based explanations in synthetic settings and real world applications.

scholarly journals Domain Adaptation and Transfer Learning in StochasticNets

2015 ◽  
Vol 1 (1) ◽  
Author(s):  
Mohammad Javad Shafiee ◽  
Parthipan Siva ◽  
Paul Fieguth ◽  
Alexander Wong
Keyword(s):  
Machine Learning ◽  
Neural Networks ◽  
Transfer Learning ◽  
Deep Neural Networks ◽  
Domain Adaptation ◽  
Experimental Results ◽  
Training Data ◽  
Learning Research ◽  
Sparse Connectivity

<p>Transfer learning is a recent field of machine learning research that<br />aims to resolve the challenge of dealing with insufficient training<br />data in the domain of interest. This is a particular issue with traditional<br />deep neural networks where a large amount of training<br />data is needed. Recently, StochasticNets was proposed to take<br />advantage of sparse connectivity in order to decrease the number<br />of parameters that needs to be learned, which in turn may relax<br />training data size requirements. In this paper, we study the efficacy<br />of transfer learning on StochasticNet frameworks. Experimental results<br />show 7% improvement on StochasticNet performance when<br />the transfer learning is applied in training step.</p>

scholarly journals Natural Images Allow Universal Adversarial Attacks on Medical Image Classification Using Deep Neural Networks with Transfer Learning

2021 ◽  
Author(s):  
Akinori Minagi ◽  
Hokuto Hirano ◽  
Kazuhiro Takemoto
Keyword(s):  
Neural Networks ◽  
Image Classification ◽  
Transfer Learning ◽  
Medical Image ◽  
Deep Neural Networks ◽  
Disease Diagnosis ◽  
Natural Images ◽  
Fine Tuning ◽  
Security And Privacy ◽  
Medical Image Classification

Abstract Transfer learning from natural images is well used in deep neural networks (DNNs) for medical image classification to achieve computer-aided clinical diagnosis. Although the adversarial vulnerability of DNNs hinders practical applications owing to the high stakes of diagnosis, adversarial attacks are expected to be limited because training data — which are often required for adversarial attacks — are generally unavailable in terms of security and privacy preservation. Nevertheless, we hypothesized that adversarial attacks are also possible using natural images because pre-trained models do not change significantly after fine-tuning. We focused on three representative DNN-based medical image classification tasks (i.e., skin cancer, referable diabetic retinopathy, and pneumonia classifications) and investigated whether medical DNN models with transfer learning are vulnerable to universal adversarial perturbations (UAPs), generated using natural images. UAPs from natural images are useful for both non-targeted and targeted attacks. The performance of UAPs from natural images was significantly higher than that of random controls, although slightly lower than that of UAPs from training images. Vulnerability to UAPs from natural images was observed between different natural image datasets and between different model architectures. The use of transfer learning causes a security hole, which decreases the reliability and safety of computer-based disease diagnosis. Model training from random initialization (without transfer learning) reduced the performance of UAPs from natural images; however, it did not completely avoid vulnerability to UAPs. The vulnerability of UAPs from natural images will become a remarkable security threat.

scholarly journals Sentiment Analysis of Short Russian Texts Using BERT and Word2Vec Embeddings

2021 ◽  
Author(s):  
Ekaterina Popova ◽  
Vladimir Spitsyn
Keyword(s):  
Machine Learning ◽  
Neural Networks ◽  
Social Networks ◽  
Sentiment Analysis ◽  
Deep Neural Networks ◽  
Functional Method ◽  
Fine Tuning ◽  
Machine Learning Techniques ◽  
Word Embeddings ◽  
Learning Techniques

This article is devoted to modern approaches for sentiment analysis of short Russian texts from social networks using deep neural networks. Sentiment analysis is the process of detecting, extracting, and classifying opinions, sentiments, and attitudes concerning different topics expressed in texts. The importance of this topic is linked to the growth and popularity of social networks, online recommendation services, news portals, and blogs, all of which contain a significant number of people's opinions on a variety of topics. In this paper, we propose machine-learning techniques with BERT and Word2Vec embeddings for tweets sentiment analysis. Two approaches were explored: (a) a method, of word embeddings extraction and using the DNN classifier; (b) refinement of the pre-trained BERT model. As a result, the fine- tuning BERT outperformed the functional method to solving the problem.

scholarly journals Adversarial Perturbation Defense on Deep Neural Networks

2022 ◽  
Vol 54 (8) ◽  
pp. 1-36
Author(s):  
Xingwei Zhang ◽  
Xiaolong Zheng ◽  
Wenji Mao
Keyword(s):  
Machine Learning ◽  
Neural Networks ◽  
High Probability ◽  
Deep Neural Networks ◽  
State Of The Art ◽  
Learning Systems ◽  
Small Perturbations ◽  
Image Objects ◽  
Comprehensive Survey ◽  
Human Eyes

Deep neural networks (DNNs) have been verified to be easily attacked by well-designed adversarial perturbations. Image objects with small perturbations that are imperceptible to human eyes can induce DNN-based image class classifiers towards making erroneous predictions with high probability. Adversarial perturbations can also fool real-world machine learning systems and transfer between different architectures and datasets. Recently, defense methods against adversarial perturbations have become a hot topic and attracted much attention. A large number of works have been put forward to defend against adversarial perturbations, enhancing DNN robustness against potential attacks, or interpreting the origin of adversarial perturbations. In this article, we provide a comprehensive survey on classical and state-of-the-art defense methods by illuminating their main concepts, in-depth algorithms, and fundamental hypotheses regarding the origin of adversarial perturbations. In addition, we further discuss potential directions of this domain for future researchers.

scholarly journals Hardware-friendly Deep Learning by Network Quantization and Binarization

2021 ◽  
Author(s):  
Haotong Qin
Keyword(s):  
Neural Networks ◽  
Deep Learning ◽  
Real World ◽  
Deep Neural Networks ◽  
Efficient Approach ◽  
Resource Limited ◽  
Complex Scenes ◽  
Flexible Hardware ◽  
Real World Applications

Quantization is emerging as an efficient approach to promote hardware-friendly deep learning and run deep neural networks on resource-limited hardware. However, it still causes a significant decrease to the network in accuracy. We summarize challenges of quantization into two categories: Quantization for Diverse Architectures and Quantization on Complex Scenes. Our studies focus mainly on applying quantization on various architectures and scenes and pushing the limit of quantization to extremely compress and accelerate networks. The comprehensive research on quantization will achieve more powerful, more efficient, and more flexible hardware-friendly deep learning, and make it better suited to more real-world applications.

scholarly journals Natural Images Allow Universal Adversarial Attacks on Medical Image Classification Using Deep Neural Networks With Transfer Learning

2021 ◽  
Author(s):  
Akinori Minagi ◽  
Hokuto Hirano ◽  
Kazuhiro Takemoto
Keyword(s):  
Neural Networks ◽  
Image Classification ◽  
Transfer Learning ◽  
Medical Image ◽  
Deep Neural Networks ◽  
Disease Diagnosis ◽  
Natural Images ◽  
Fine Tuning ◽  
Security And Privacy ◽  
Medical Image Classification

Abstract Background. Transfer learning from natural images is well used in deep neural networks (DNNs) for medical image classification to achieve computer-aided clinical diagnosis. Although the adversarial vulnerability of DNNs hinders practical applications owing to the high stakes of diagnosis, adversarial attacks are expected to be limited because training data — which are often required for adversarial attacks — are generally unavailable in terms of security and privacy preservation. Nevertheless, we hypothesized that adversarial attacks are also possible using natural images because pre-trained models do not change significantly after fine-tuning.Methods. We considered three representative DNN-based medical image classification tasks (i.e., skin cancer, referable diabetic retinopathy, and pneumonia classifications) to investigate whether medical DNN models with transfer learning are vulnerable to universal adversarial perturbations (UAPs), generated using natural images.Results. UAPs from natural images are useful for both non-targeted and targeted attacks. The performance of UAPs from natural images was significantly higher than that of random controls, although slightly lower than that of UAPs from training images. Vulnerability to UAPs from natural images was observed between different natural image datasets and between different model architectures.Conclusion. The use of transfer learning causes a security hole, which decreases the reliability and safety of computer-based disease diagnosis. Model training from random initialization (without transfer learning) reduced the performance of UAPs from natural images; however, it did not completely avoid vulnerability to UAPs. The vulnerability of UAPs from natural images will become a remarkable security threat.

scholarly journals Towards Understanding Deep Learning from Noisy Labels with Small-Loss Criterion

2021 ◽  
Author(s):  
Xian-Jin Gui ◽  
Wei Wang ◽  
Zhang-Hao Tian
Keyword(s):  
Neural Networks ◽  
Deep Learning ◽  
Real World ◽  
Deep Neural Networks ◽  
Theoretical Explanation ◽  
Small Loss ◽  
The Past ◽  
Real World Applications ◽  
Theoretical Analyses ◽  
Noisy Labels

Deep neural networks need large amounts of labeled data to achieve good performance. In real-world applications, labels are usually collected from non-experts such as crowdsourcing to save cost and thus are noisy. In the past few years, deep learning methods for dealing with noisy labels have been developed, many of which are based on the small-loss criterion. However, there are few theoretical analyses to explain why these methods could learn well from noisy labels. In this paper, we theoretically explain why the widely-used small-loss criterion works. Based on the explanation, we reformalize the vanilla small-loss criterion to better tackle noisy labels. The experimental results verify our theoretical explanation and also demonstrate the effectiveness of the reformalization.

Sign in / Sign up

Export Citation Format

Share Document