scholarly journals Robust Regularization with Adversarial Labelling of Perturbed Samples

2021 ◽  
Author(s):  
Xiaohui Guo ◽  
Richong Zhang ◽  
Yaowei Zheng ◽  
Yongyi Mao
Keyword(s):  
Analytic Solution ◽  
Predictive Accuracy ◽  
State Of The Art ◽  
Maximization Problem ◽  
Risk Minimization ◽  
Regularization Scheme ◽  
Training Scheme ◽  
Input Sample ◽  
Adversarial Training ◽  
The Alps

Recent researches have suggested that the predictive accuracy of neural network may contend with its adversarial robustness. This presents challenges in designing effective regularization schemes that also provide strong adversarial robustness. Revisiting Vicinal Risk Minimization (VRM) as a unifying regularization principle, we propose Adversarial Labelling of Perturbed Samples (ALPS) as a regularization scheme that aims at improving the generalization ability and adversarial robustness of the trained model. ALPS trains neural networks with synthetic samples formed by perturbing each authentic input sample towards another one along with an adversarially assigned label. The ALPS regularization objective is formulated as a min-max problem, in which the outer problem is minimizing an upper-bound of the VRM loss, and the inner problem is L1-ball constrained adversarial labelling on perturbed sample. The analytic solution to the induced inner maximization problem is elegantly derived, which enables computational efficiency. Experiments on the SVHN, CIFAR-10, CIFAR-100 and Tiny-ImageNet datasets show that the ALPS has a state-of-the-art regularization performance while also serving as an effective adversarial training scheme.

scholarly journals Regularized Training and Tight Certification for Randomized Smoothed Classifier with Provable Robustness

2020 ◽  
Vol 34 (04) ◽  
pp. 3858-3865
Author(s):  
Huijie Feng ◽  
Chunpeng Wu ◽  
Guoyang Chen ◽  
Weifeng Zhang ◽  
Yang Ning
Keyword(s):  
Neural Network ◽  
High Probability ◽  
Deep Neural Network ◽  
State Of The Art ◽  
Computationally Efficient ◽  
Base Classifier ◽  
Training Scheme ◽  
Adversarial Training ◽  
Gaussian Perturbation ◽  
Probabilistic Robustness

Recently smoothing deep neural network based classifiers via isotropic Gaussian perturbation is shown to be an effective and scalable way to provide state-of-the-art probabilistic robustness guarantee against ℓ2 norm bounded adversarial perturbations. However, how to train a good base classifier that is accurate and robust when smoothed has not been fully investigated. In this work, we derive a new regularized risk, in which the regularizer can adaptively encourage the accuracy and robustness of the smoothed counterpart when training the base classifier. It is computationally efficient and can be implemented in parallel with other empirical defense methods. We discuss how to implement it under both standard (non-adversarial) and adversarial training scheme. At the same time, we also design a new certification algorithm, which can leverage the regularization effect to provide tighter robustness lower bound that holds with high probability. Our extensive experimentation demonstrates the effectiveness of the proposed training and certification approaches on CIFAR-10 and ImageNet datasets.

scholarly journals Robust CNN Compression Framework for Security-Sensitive Embedded Systems

2021 ◽  
Vol 11 (3) ◽  
pp. 1093
Author(s):  
Jeonghyun Lee ◽  
Sangkyun Lee
Keyword(s):  
Embedded Systems ◽  
Optimization Problem ◽  
State Of The Art ◽  
Classification Problems ◽  
Proximal Gradient Method ◽  
Knowledge Distillation ◽  
New Type ◽  
Adversarial Examples ◽  
Adversarial Training ◽  
Memory Efficient

Convolutional neural networks (CNNs) have achieved tremendous success in solving complex classification problems. Motivated by this success, there have been proposed various compression methods for downsizing the CNNs to deploy them on resource-constrained embedded systems. However, a new type of vulnerability of compressed CNNs known as the adversarial examples has been discovered recently, which is critical for security-sensitive systems because the adversarial examples can cause malfunction of CNNs and can be crafted easily in many cases. In this paper, we proposed a compression framework to produce compressed CNNs robust against such adversarial examples. To achieve the goal, our framework uses both pruning and knowledge distillation with adversarial training. We formulate our framework as an optimization problem and provide a solution algorithm based on the proximal gradient method, which is more memory-efficient than the popular ADMM-based compression approaches. In experiments, we show that our framework can improve the trade-off between adversarial robustness and compression rate compared to the existing state-of-the-art adversarial pruning approach.

scholarly journals Entry and Exit Decision Problem with Implementation Delay

2008 ◽  
Vol 45 (04) ◽  
pp. 1039-1059 ◽  
Author(s):  
Marius Costeniuc ◽  
Michaela Schnetzer ◽  
Luca Taschini
Keyword(s):  
Decision Problem ◽  
Analytic Solution ◽  
Time Lag ◽  
Probabilistic Approach ◽  
Stopping Time ◽  
Stopping Times ◽  
Maximization Problem ◽  
Entry And Exit ◽  
Brownian Motion With Drift ◽  
Implementation Delay

We study investment and disinvestment decisions in situations where there is a time lagd> 0 from the timetwhen the decision is taken to the timet+dwhen the decision is implemented. In this paper we apply the probabilistic approach to the combined entry and exit decisions under the Parisian implementation delay. In particular, we prove the independence between Parisian stopping times and a general Brownian motion with drift stopped at the stopping time. Relying on this result, we solve the constrained maximization problem, obtaining an analytic solution to the optimal ‘starting’ and ‘stopping’ levels. We compare our results with the instantaneous entry and exit situation, and show that an increase in the uncertainty of the underlying process hastens the decision to invest or disinvest, extending a result of Bar-Ilan and Strange (1996).

Discovery of polynomial equations for regression

2004 ◽  
Vol 1 (1) ◽  
pp. 131-142
Author(s):  
Ljupčo Todorovski ◽  
Sašo Džeroski ◽  
Peter Ljubič
Keyword(s):  
Efficient Method ◽  
Regression Models ◽  
Predictive Accuracy ◽  
State Of The Art ◽  
Numerical Data ◽  
Predictive Performance ◽  
Polynomial Equations ◽  
Regression Methods ◽  
Piecewise Regression ◽  
Standard Regression

Both equation discovery and regression methods aim at inducing models of numerical data. While the equation discovery methods are usually evaluated in terms of comprehensibility of the induced model, the emphasis of the regression methods evaluation is on their predictive accuracy. In this paper, we present Ciper, an efficient method for discovery of polynomial equations and empirically evaluate its predictive performance on standard regression tasks. The evaluation shows that polynomials compare favorably to linear and piecewise regression models, induced by the existing state-of-the-art regression methods, in terms of degree of fit and complexity.

scholarly journals An efficient training scheme for supermodels

2017 ◽  
Vol 8 (2) ◽  
pp. 429-438 ◽  
Author(s):  
Francine J. Schevenhoven ◽  
Frank M. Selten
Keyword(s):  
Climate Models ◽  
State Of The Art ◽  
Atmospheric Model ◽  
High Dimensional ◽  
Model Errors ◽  
Global Atmospheric Model ◽  
Training Scheme ◽  
Skill Scores ◽  
Weather And Climate ◽  
The Individual

Abstract. Weather and climate models have improved steadily over time as witnessed by objective skill scores, although significant model errors remain. Given these imperfect models, predictions might be improved by combining them dynamically into a so-called supermodel. In this paper a new training scheme to construct such a supermodel is explored using a technique called cross pollination in time (CPT). In the CPT approach the models exchange states during the prediction. The number of possible predictions grows quickly with time, and a strategy to retain only a small number of predictions, called pruning, needs to be developed. The method is explored using low-order dynamical systems and applied to a global atmospheric model. The results indicate that the CPT training is efficient and leads to a supermodel with improved forecast quality as compared to the individual models. Due to its computational efficiency, the technique is suited for application to state-of-the art high-dimensional weather and climate models.

scholarly journals Adversarial Training for Community Question Answer Selection Based on Multi-Scale Matching

2019 ◽  
Vol 33 ◽  
pp. 395-402
Author(s):  
Xiao Yang ◽  
Madian Khabsa ◽  
Miaosen Wang ◽  
Wei Wang ◽  
Ahmed Hassan Awadallah ◽  
...  
Keyword(s):  
Question Answering ◽  
State Of The Art ◽  
Classification Problem ◽  
Classification Model ◽  
Training Set ◽  
Community Based ◽  
Multi Scale ◽  
Adversarial Training ◽  
Source Of Information ◽  
Different Levels

Community-based question answering (CQA) websites represent an important source of information. As a result, the problem of matching the most valuable answers to their corresponding questions has become an increasingly popular research topic. We frame this task as a binary (relevant/irrelevant) classification problem, and present an adversarial training framework to alleviate label imbalance issue. We employ a generative model to iteratively sample a subset of challenging negative samples to fool our classification model. Both models are alternatively optimized using REINFORCE algorithm. The proposed method is completely different from previous ones, where negative samples in training set are directly used or uniformly down-sampled. Further, we propose using Multi-scale Matching which explicitly inspects the correlation between words and ngrams of different levels of granularity. We evaluate the proposed method on SemEval 2016 and SemEval 2017 datasets and achieves state-of-the-art or similar performance.

scholarly journals Distributionally Robust Counterfactual Risk Minimization

2020 ◽  
Vol 34 (04) ◽  
pp. 3850-3857
Author(s):  
Louis Faury ◽  
Ugo Tanielian ◽  
Elvis Dohmatob ◽  
Elena Smirnova ◽  
Flavian Vasile
Keyword(s):  
Decision Making ◽  
Robust Optimization ◽  
Model Uncertainty ◽  
State Of The Art ◽  
Risk Minimization ◽  
Practical Applications ◽  
Uncertainty Measures ◽  
Leibler Divergence ◽  
Benchmark Datasets ◽  
Distributionally Robust

This manuscript introduces the idea of using Distributionally Robust Optimization (DRO) for the Counterfactual Risk Minimization (CRM) problem. Tapping into a rich existing literature, we show that DRO is a principled tool for counterfactual decision making. We also show that well-established solutions to the CRM problem like sample variance penalization schemes are special instances of a more general DRO problem. In this unifying framework, a variety of distributionally robust counterfactual risk estimators can be constructed using various probability distances and divergences as uncertainty measures. We propose the use of Kullback-Leibler divergence as an alternative way to model uncertainty in CRM and derive a new robust counterfactual objective. In our experiments, we show that this approach outperforms the state-of-the-art on four benchmark datasets, validating the relevance of using other uncertainty measures in practical applications.

scholarly journals Entry and Exit Decision Problem with Implementation Delay

2008 ◽  
Vol 45 (4) ◽  
pp. 1039-1059 ◽  
Author(s):  
Marius Costeniuc ◽  
Michaela Schnetzer ◽  
Luca Taschini
Keyword(s):  
Decision Problem ◽  
Analytic Solution ◽  
Time Lag ◽  
Probabilistic Approach ◽  
Stopping Time ◽  
Stopping Times ◽  
Maximization Problem ◽  
Entry And Exit ◽  
Brownian Motion With Drift ◽  
Implementation Delay

We study investment and disinvestment decisions in situations where there is a time lagd> 0 from the timetwhen the decision is taken to the timet+dwhen the decision is implemented. In this paper we apply the probabilistic approach to the combined entry and exit decisions under the Parisian implementation delay. In particular, we prove the independence between Parisian stopping times and a general Brownian motion with drift stopped at the stopping time. Relying on this result, we solve the constrained maximization problem, obtaining an analytic solution to the optimal ‘starting’ and ‘stopping’ levels. We compare our results with the instantaneous entry and exit situation, and show that an increase in the uncertainty of the underlying process hastens the decision to invest or disinvest, extending a result of Bar-Ilan and Strange (1996).

scholarly journals Generating Adversarial Examples with Adversarial Networks

2018 ◽  
Author(s):  
Chaowei Xiao ◽  
Bo Li ◽  
Jun-yan Zhu ◽  
Warren He ◽  
Mingyan Liu ◽  
...  
Keyword(s):  
Deep Neural Networks ◽  
State Of The Art ◽  
Black Box ◽  
Generative Adversarial Networks ◽  
Perceptual Quality ◽  
Small Magnitude ◽  
Adversarial Networks ◽  
Original Target ◽  
Adversarial Examples ◽  
Adversarial Training

Deep neural networks (DNNs) have been found to be vulnerable to adversarial examples resulting from adding small-magnitude perturbations to inputs. Such adversarial examples can mislead DNNs to produce adversary-selected results. Different attack strategies have been proposed to generate adversarial examples, but how to produce them with high perceptual quality and more efficiently requires more research efforts. In this paper, we propose AdvGAN to generate adversarial exam- ples with generative adversarial networks (GANs), which can learn and approximate the distribution of original instances. For AdvGAN, once the generator is trained, it can generate perturbations efficiently for any instance, so as to potentially accelerate adversarial training as defenses. We apply Adv- GAN in both semi-whitebox and black-box attack settings. In semi-whitebox attacks, there is no need to access the original target model after the generator is trained, in contrast to traditional white-box attacks. In black-box attacks, we dynamically train a distilled model for the black-box model and optimize the generator accordingly. Adversarial examples generated by AdvGAN on different target models have high attack success rate under state-of-the-art defenses compared to other attacks. Our attack has placed the first with 92.76% accuracy on a public MNIST black-box attack challenge.

scholarly journals CAGAN: Consistent Adversarial Training Enhanced GANs

2018 ◽  
Author(s):  
Yao Ni ◽  
Dandan Song ◽  
Xi Zhang ◽  
Hao Wu ◽  
Lejian Liao
Keyword(s):  
Neural Network ◽  
Parameter Space ◽  
Supervised Classification ◽  
State Of The Art ◽  
Generative Adversarial Networks ◽  
Image Generation ◽  
Real Samples ◽  
Adversarial Networks ◽  
Novel Approach ◽  
Adversarial Training

Generative adversarial networks (GANs) have shown impressive results, however, the generator and the discriminator are optimized in finite parameter space which means their performance still need to be improved. In this paper, we propose a novel approach of adversarial training between one generator and an exponential number of critics which are sampled from the original discriminative neural network via dropout. As discrepancy between outputs of different sub-networks of a same sample can measure the consistency of these critics, we encourage the critics to be consistent to real samples and inconsistent to generated samples during training, while the generator is trained to generate consistent samples for different critics. Experimental results demonstrate that our method can obtain state-of-the-art Inception scores of 9.17 and 10.02 on supervised CIFAR-10 and unsupervised STL-10 image generation tasks, respectively, as well as achieve competitive semi-supervised classification results on several benchmarks. Importantly, we demonstrate that our method can maintain stability in training and alleviate mode collapse.

Sign in / Sign up

Export Citation Format

Share Document