Theoretical Investigation of Generalization Bound for Residual Networks

This paper presents a framework for norm-based capacity control with respect to an lp,q-norm in weight-normalized Residual Neural Networks (ResNets). We first formulate the representation of each residual block. For the regression problem, we analyze the Rademacher Complexity of the ResNets family. We also establish a tighter generalization upper bound for weight-normalized ResNets. in a more general sight. Using the lp,q-norm weight normalization in which 1/p+1/q >=1, we discuss the properties of a width-independent capacity control, which only relies on the depth according to a square root term. Several comparisons suggest that our result is tighter than previous work. Parallel results for Deep Neural Networks (DNN) and Convolutional Neural Networks (CNN) are included by introducing the lp,q-norm weight normalization for DNN and the lp,q-norm kernel normalization for CNN. Numerical experiments also verify that ResNet structures contribute to better generalization properties.

Download Full-text

Understanding Local Robustness of Deep Neural Networks under Natural Variations

Fundamental Approaches to Software Engineering - Lecture Notes in Computer Science ◽

10.1007/978-3-030-71500-7_16 ◽

2021 ◽

pp. 313-337

Author(s):

Ziyuan Zhong ◽

Yuchi Tian ◽

Baishakhi Ray

Keyword(s):

Neural Networks ◽

Image Classification ◽

Deep Neural Networks ◽

Autonomous Driving ◽

Regression Problem ◽

Rigorous Testing ◽

Wide Range ◽

Natural Variants ◽

Norm Bound ◽

Commercial Applications

AbstractDeep Neural Networks (DNNs) are being deployed in a wide range of settings today, from safety-critical applications like autonomous driving to commercial applications involving image classifications. However, recent research has shown that DNNs can be brittle to even slight variations of the input data. Therefore, rigorous testing of DNNs has gained widespread attention.While DNN robustness under norm-bound perturbation got significant attention over the past few years, our knowledge is still limited when natural variants of the input images come. These natural variants, e.g., a rotated or a rainy version of the original input, are especially concerning as they can occur naturally in the field without any active adversary and may lead to undesirable consequences. Thus, it is important to identify the inputs whose small variations may lead to erroneous DNN behaviors. The very few studies that looked at DNN’s robustness under natural variants, however, focus on estimating the overall robustness of DNNs across all the test data rather than localizing such error-producing points. This work aims to bridge this gap.To this end, we study the local per-input robustness properties of the DNNs and leverage those properties to build a white-box (DeepRobust-W) and a black-box (DeepRobust-B) tool to automatically identify the non-robust points. Our evaluation of these methods on three DNN models spanning three widely used image classification datasets shows that they are effective in flagging points of poor robustness. In particular, DeepRobust-W and DeepRobust-B are able to achieve an F1 score of up to 91.4% and 99.1%, respectively. We further show that DeepRobust-W can be applied to a regression problem in a domain beyond image classification. Our evaluation on three self-driving car models demonstrates that DeepRobust-W is effective in identifying points of poor robustness with F1 score up to 78.9%.

Download Full-text

Artificial Intelligence in Colorectal Cancer Diagnosis Using Clinical Data: Non-Invasive Approach

Diagnostics ◽

10.3390/diagnostics11030514 ◽

2021 ◽

Vol 11 (3) ◽

pp. 514

Author(s):

Noémi Lorenzovici ◽

Eva-H. Dulf ◽

Teodora Mocan ◽

Lucian Mocan

Keyword(s):

Colorectal Cancer ◽

Machine Learning ◽

Neural Networks ◽

Cancer Diagnosis ◽

Deep Neural Networks ◽

Living Environment ◽

Regression Problem ◽

Invasive Approach ◽

Diagnosis System ◽

Computer Aided

Colorectal cancer is the third most common and second most lethal tumor globally, causing 900,000 deaths annually. In this research, a computer aided diagnosis system was designed that detects colorectal cancer, using an innovative dataset composing of both numeric (blood and urine analysis) and qualitative data (living environment of the patient, tumor position, T, N, M, Dukes classification, associated pathology, technical approach, complications, incidents, ultrasonography-dimensions as well as localization). The intelligent computer aided colorectal cancer diagnosis system was designed using different machine learning techniques, such as classification and shallow and deep neural networks. The maximum accuracy obtained from solving the binary classification problem with traditional machine learning algorithms was 77.8%. However, the regression problem solved with deep neural networks yielded with significantly better performance in terms of mean squared error minimization, reaching the value of 0.0000529.

Download Full-text

Reachability Analysis of Deep Neural Networks with Provable Guarantees

Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence ◽

10.24963/ijcai.2018/368 ◽

2018 ◽

Cited By ~ 24

Author(s):

Wenjie Ruan ◽

Xiaowei Huang ◽

Marta Kwiatkowska

Keyword(s):

Neural Networks ◽

Upper Bound ◽

Deep Neural Networks ◽

State Of The Art ◽

Range Analysis ◽

Reachability Problem ◽

Analysis Problem ◽

Lipschitz Continuous ◽

Verification Problem ◽

Novel Algorithm

Verifying correctness for deep neural networks (DNNs) is challenging. We study a generic reachability problem for feed-forward DNNs which, for a given set of inputs to the network and a Lipschitz-continuous function over its outputs computes the lower and upper bound on the function values. Because the network and the function are Lipschitz continuous, all values in the interval between the lower and upper bound are reachable. We show how to obtain the safety verification problem, the output range analysis problem and a robustness measure by instantiating the reachability problem. We present a novel algorithm based on adaptive nested optimisation to solve the reachability problem. The technique has been implemented and evaluated on a range of DNNs, demonstrating its efficiency, scalability and ability to handle a broader class of networks than state-of-the-art verification approaches.

Download Full-text