Generalized Batch Normalization: Towards Accelerating Deep Neural Networks

Utilizing recently introduced concepts from statistics and quantitative risk management, we present a general variant of Batch Normalization (BN) that offers accelerated convergence of Neural Network training compared to conventional BN. In general, we show that mean and standard deviation are not always the most appropriate choice for the centering and scaling procedure within the BN transformation, particularly if ReLU follows the normalization step. We present a Generalized Batch Normalization (GBN) transformation, which can utilize a variety of alternative deviation measures for scaling and statistics for centering, choices which naturally arise from the theory of generalized deviation measures and risk theory in general. When used in conjunction with the ReLU non-linearity, the underlying risk theory suggests natural, arguably optimal choices for the deviation measure and statistic. Utilizing the suggested deviation measure and statistic, we show experimentally that training is accelerated more so than with conventional BN, often with improved error rate as well. Overall, we propose a more flexible BN transformation supported by a complimentary theoretical framework that can potentially guide design choices.

Teoria de Medidas de Risco: uma revisão abrangente

A fundamental aspect of proper risk management is the measurement, especially forecasting of risk measures. Measures such as variance, volatility and Value at Risk had been considered valid because of their practical intuition. However, a solid theoretical framework it is important to ensure better properties for risk measures. Such background is the risk measures theory. This paper presents a comprehensive literature review on risk measures theory, focusing in basic theory and extensions to this fundamental outline. The paper is structured in order to cover the main risk measures classes from literature, which are coherent risk measures, convex risk measures, spectral and distortion risk measures and generalized deviation measures.

A Scheme for Transformation of Tolerance Specifications to Generalized Deviation Space for Use in Tolerance Synthesis and Analysis

Traditionally tolerances for manufactured parts are specified using symbolic schemes as per ASME or ISO standards. To use these tolerance specifications in computerized tolerance synthesis and analysis, we need information models to represent the tolerances. Tolerance specifications could be modeled as a class with its attributes and methods [ROY01]. Tolerances impose restrictions on the possible deviation of features from its nominal size/shape. These variations of shape/size of a feature could be modeled as deviation of a set of generalized coordinates defined at some convenient point on the feature [BAL98]. In this paper, we present a method for converting tolerance specifications as per MMC (Maximum Material Condition) / LMC (Least Material Condition) / RFS (Regardless of Feature Size) material conditions for standard mating features (planar, cylindrical, and spherical) into a set of inequalities in a deviation space for representation of deviation of a feature from it’s nominal shape. We have used the virtual condition boundaries (VCB) as well as tolerance zones (as the case may be) for these mappings. For the planar feature, these relations are linear and the bounded space is diamond shaped. For the other cases, the mapping is a set of nonlinear inequalities. The mapping transforms the tolerance specifications into a generalized coordinate frame as a set of inequalities. These are useful in tolerance synthesis, and analysis as well as in assemblability analysis in the generalized coordinate system (deviation space). In this paper, we also illustrate the mapping procedures with an example.