scholarly journals Feature-Guided Black-Box Safety Testing of Deep Neural Networks

2018 ◽  
pp. 408-426 ◽  
Author(s):  
Matthew Wicker ◽  
Xiaowei Huang ◽  
Marta Kwiatkowska
Keyword(s):  
Neural Networks ◽  
Deep Neural Networks ◽  
Black Box ◽  
Safety Testing

scholarly journals SADA: Semantic Adversarial Diagnostic Attacks for Autonomous Applications

2020 ◽  
Vol 34 (07) ◽  
pp. 10901-10908 ◽  
Author(s):  
Abdullah Hamdi ◽  
Matthias Mueller ◽  
Bernard Ghanem
Keyword(s):  
Neural Networks ◽  
Recent Work ◽  
Autonomous Navigation ◽  
General Framework ◽  
Deep Neural Networks ◽  
Autonomous Driving ◽  
Black Box ◽  
Semantic Meaning ◽  
Safety Critical ◽  
Adversarial Attack

One major factor impeding more widespread adoption of deep neural networks (DNNs) is their lack of robustness, which is essential for safety-critical applications such as autonomous driving. This has motivated much recent work on adversarial attacks for DNNs, which mostly focus on pixel-level perturbations void of semantic meaning. In contrast, we present a general framework for adversarial attacks on trained agents, which covers semantic perturbations to the environment of the agent performing the task as well as pixel-level attacks. To do this, we re-frame the adversarial attack problem as learning a distribution of parameters that always fools the agent. In the semantic case, our proposed adversary (denoted as BBGAN) is trained to sample parameters that describe the environment with which the black-box agent interacts, such that the agent performs its dedicated task poorly in this environment. We apply BBGAN on three different tasks, primarily targeting aspects of autonomous navigation: object detection, self-driving, and autonomous UAV racing. On these tasks, BBGAN can generate failure cases that consistently fool a trained agent.

scholarly journals Analysis of Explainers of Black Box Deep Neural Networks for Computer Vision: A Survey

2021 ◽  
Vol 3 (4) ◽  
pp. 966-989
Author(s):  
Vanessa Buhrmester ◽  
David Münch ◽  
Michael Arens
Keyword(s):  
Neural Networks ◽  
Computer Vision ◽  
Deep Neural Networks ◽  
State Of The Art ◽  
Black Box ◽  
Complex Data ◽  
Comprehensive Overview ◽  
Nonlinear Structure ◽  
Black Boxes ◽  
Insight Into

Deep Learning is a state-of-the-art technique to make inference on extensive or complex data. As a black box model due to their multilayer nonlinear structure, Deep Neural Networks are often criticized as being non-transparent and their predictions not traceable by humans. Furthermore, the models learn from artificially generated datasets, which often do not reflect reality. By basing decision-making algorithms on Deep Neural Networks, prejudice and unfairness may be promoted unknowingly due to a lack of transparency. Hence, several so-called explanators, or explainers, have been developed. Explainers try to give insight into the inner structure of machine learning black boxes by analyzing the connection between the input and output. In this survey, we present the mechanisms and properties of explaining systems for Deep Neural Networks for Computer Vision tasks. We give a comprehensive overview about the taxonomy of related studies and compare several survey papers that deal with explainability in general. We work out the drawbacks and gaps and summarize further research ideas.

scholarly journals Optimizing for Interpretability in Deep Neural Networks with Tree Regularization

2021 ◽  
Vol 72 ◽  
pp. 1-37
Author(s):  
Mike Wu ◽  
Sonali Parbhoo ◽  
Michael C. Hughes ◽  
Volker Roth ◽  
Finale Doshi-Velez
Keyword(s):  
Neural Networks ◽  
Predictive Power ◽  
Deep Neural Networks ◽  
Large Body ◽  
Black Box ◽  
New Family ◽  
Deep Model ◽  
Real World Applications ◽  
Adversarial Examples ◽  
Key Barrier

Deep models have advanced prediction in many domains, but their lack of interpretability  remains a key barrier to the adoption in many real world applications. There exists a large  body of work aiming to help humans understand these black box functions to varying levels  of granularity – for example, through distillation, gradients, or adversarial examples. These  methods however, all tackle interpretability as a separate process after training. In this  work, we take a different approach and explicitly regularize deep models so that they are  well-approximated by processes that humans can step through in little time. Specifically,  we train several families of deep neural networks to resemble compact, axis-aligned decision  trees without significant compromises in accuracy. The resulting axis-aligned decision  functions uniquely make tree regularized models easy for humans to interpret. Moreover,  for situations in which a single, global tree is a poor estimator, we introduce a regional tree regularizer that encourages the deep model to resemble a compact, axis-aligned decision  tree in predefined, human-interpretable contexts. Using intuitive toy examples, benchmark  image datasets, and medical tasks for patients in critical care and with HIV, we demonstrate  that this new family of tree regularizers yield models that are easier for humans to simulate  than L1 or L2 penalties without sacrificing predictive power. 

Agency MBS Prepayment Model Using Neural Networks

2019 ◽  
Keyword(s):  
Risk Factors ◽  
Neural Networks ◽  
Regime Shift ◽  
Deep Neural Networks ◽  
Black Box ◽  
Trial And Error ◽  
Fixed Rate ◽  
Pool Level ◽  
Highly Nonlinear ◽  
The Neural Networks

The authors apply deep neural networks, a type of machine learning method, to model agency mortgage-backed security (MBS) 30-year, fixed-rate pool prepayment behaviors. The neural networks model (NNM) is able to produce highly accurate model fits to the historical prepayment patterns as well as accurate sensitivities to economic and pool-level risk factors. These results are comparable with model results and intuitions obtained from a traditional agency pool-level prepayment model that is in production and was built via many iterations of trial and error over many months and years. This example shows NNM can process large datasets efficiently, capture very complex prepayment patterns, and model large group of risk factors that are highly nonlinear and interactive. The authors also examine various potential shortcomings of this approach, including nontransparency/“black-box” issues, model overfitting, and regime shift issues.

PIPxResNet: Penalty Induced Prototype-Based eXplainable Residual Neural Network for Heartbeat Classification

2021 ◽  
Author(s):  
Deepankar Nankani ◽  
Rashmi Dutta Baruah
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Deep Neural Networks ◽  
Early Stage ◽  
Black Box ◽  
Training Dataset ◽  
Medical Attention ◽  
Generative Adversarial Network ◽  
Heartbeat Classification ◽  
General Physicians

Abstract Early stage heartbeat classification using the electrocardiogram signals can prevent cardiovascular diseases that causes millions of deaths annually around the world. In the past, researchers have used deep neural networks to achieve significant performance for heartbeat classification but their black-box nature and prediction rationale limits real-world deployment. We propose a Penalty Induced Prototype based eXplainable Residual Neural Network (PIPxResNet) that addresses the black-box nature of deep neural networks. PIPxResNet encodes the temporal variations of heartbeats by employing pretrained residual neural network following the concept of task transfer learning. The algorithm further extracts prototypes that are most representative of the training dataset that explain model predictions to general physicians, making them clinically relevant. The prototypes of a particular class having close resemblance to other class prototypes are penalised and their contribution towards corresponding class is reduced. In addition, the classification performance is improved by synthesising regular and irregular heartbeats using a deep convolution conditional generative adversarial network. The proposed method can easily be adopted to other domains that requires explanations for the classification tasks. The PIPxResNet performs at par with existing state-of-the-art algorithms without compromising individual class performance when tested on four publicly available annotated datasets. The proposed model is capable to perform automated screening and provide medical attention by simulating a clinical decision support system for general physicians.

scholarly journals Exposing Vulnerabilities of Deepfake Detection Systems with Robust Attacks

2021 ◽  
Author(s):  
Shehzeen Hussain ◽  
Paarth Neekhara ◽  
Brian Dolhansky ◽  
Joanna Bitton ◽  
Cristian Canton Ferrer ◽  
...  
Keyword(s):  
Neural Networks ◽  
Video Compression ◽  
Real World ◽  
Deep Neural Networks ◽  
Black Box ◽  
Detection Methods ◽  
Detection Systems ◽  
Attack Scenario ◽  
Video Manipulation ◽  
Image And Video Compression

Recent advances in video manipulation techniques have made the generation of fake videos more accessible than ever before. Manipulated videos can fuel disinformation and reduce trust in media. Therefore detection of fake videos has garnered immense interest in academia and industry. Recently developed Deepfake detection methods rely on Deep Neural Networks (DNNs) to distinguish AI-generated fake videos from real videos. In this work, we demonstrate that it is possible to bypass such detectors by adversarially modifying fake videos synthesized using existing Deepfake generation methods. We further demonstrate that our adversarial perturbations are robust to image and video compression codecs, making them a real-world threat. We present pipelines in both white-box and black-box attack scenarios that can fool DNN based Deepfake detectors into classifying fake videos as real. Finally, we study the extent to which adversarial perturbations transfer across different Deepfake detectors and create more accessible attacks using universal adversarial perturbations that pose a very feasible attack scenario since they can be easily shared amongst attackers.

Sign in / Sign up

Export Citation Format

Share Document