Effects of depth, width, and initialization: A convergence analysis of layer-wise training for deep linear neural networks

Deep neural networks have been used in various machine learning applications and achieved tremendous empirical successes. However, training deep neural networks is a challenging task. Many alternatives have been proposed in place of end-to-end back-propagation. Layer-wise training is one of them, which trains a single layer at a time, rather than trains the whole layers simultaneously. In this paper, we study a layer-wise training using a block coordinate gradient descent (BCGD) for deep linear networks. We establish a general convergence analysis of BCGD and found the optimal learning rate, which results in the fastest decrease in the loss. We identify the effects of depth, width, and initialization. When the orthogonal-like initialization is employed, we show that the width of intermediate layers plays no role in gradient-based training beyond a certain threshold. Besides, we found that the use of deep networks could drastically accelerate convergence when it is compared to those of a depth 1 network, even when the computational cost is considered. Numerical examples are provided to justify our theoretical findings and demonstrate the performance of layer-wise training by BCGD.

Deep Learning based Handwriting Recognition with Adversarial Feature Deformation and Regularization

Due to the complex and irregular shapes of handwritten text, it is challenging to spot and recognize the handwritten words. In low-resource scripts, retrieval of words is a difficult and laborious task. The need for increasing the number of samples and introducing variations in the extended training datasets occur with the use of deep learning and neural network models. All possible variations and occurrences cannot be covered in an efficient manner with the use of the existing preprocessing strategies and theories. A scalable and elastic methodology for wrapping the extracted features is presented with the introduction of an adversarial feature deformation and regularization module in this paper. In the original deep learning framework, this module is introduced between the intermediate layers while training in an alternative manner. When compared to the conventional models, highly informative features are learnt in an efficient manner with the help of this setup. Extensive word datasets are used for testing the proposed model, which is built on popular frameworks available for word recognition and spotting, while enhancing them with the proposed module. While varying the training data size, the results are recorded and compared with the conventional models. Improvement in the mAP scores, word-error rate and low data regime is observed from the results of comparison.

A Face Recognition Algorithm Based on Intermediate Layers Connected by the CNN

Face recognition is difficult due to the higher dimension of face image features and fewer training samples. Firstly, in order to improve the performance of feature extraction, we inventively construct a double hierarchical network structure convolution neural network (CNN) model. The front-end network adopts a relatively simple network model to achieve rough feature extraction from input images and obtain multiple suspect face candidate windows. The back-end network uses a relatively complex network model to filter the best detection window and return the face size and position by nonmaximum suppression. Then, in order to fully extract the face features in the optimal window, a face recognition algorithm based on intermediate layers connected by the deep CNN is proposed in this paper. Based on AlexNet, the front, intermediate and end convolution layers are combined by deep connection. Then, the feature vector describing the face image is obtained by the operation of the pooling layer and the full connection layer. Finally, the auxiliary classifier training method is used to train the model to ensure the effectiveness of the features of the intermediate layer. Experimental results based on open face database show that the recognition accuracy of the proposed algorithm is higher than that of other face recognition algorithms compared in this paper.

Intellectualisation of the decision support system by a multiagent robotic system based on neural network technology

The article considers the process of forming management decisions for the correct operation of a multi-agent robotic system (grouping of unmanned aerial vehicles (UAVs)) in a rapidly changing conditions. A neural network (NN) is used as a regulator, which is a perceptron with two intermediate layers, one is input and the other one is output. The training of the constructed neural network is carried out according to an algorithm that combines the ideas of the conjugate gradient method with quasi-Newtonian methods, and in particular, uses the approach implemented in the Levenberg-Marquardt algorithm. As input parameters for building a control system, various types of situations are used in which the UAV leader has to make decisions. Such initial data are a three-dimensional data structure, which is a two-dimensional array, each element of which is a vector. One vector reflects all available information about the situation in certain geographic coordinates (weather conditions, data on other damaging factors affecting the ability of the UAV to operate in a certain area), and the other describes the current state of the UAV (power reserve, weight, operability of components and assemblies, term of next service, etc.). The obtained information is put down into the input layer of the NN by numerical equivalents. The paper proposes a situational work model of a UAV group in special conditions.

PROBABILITY METHODS OF ANALYSIS OF ASSESSMENT OF THE CONDITION OF ROAD CLOTHES IN USE

This article is a continuation of the analysis of methods and criteria for assessing the condition of non-rigid pavement, which contains in its structural layers such hidden defects as cracks, material disintegration, violation of the structure of intermediate layers of monolithic material and others. The variety of models for assessing the condition of the pavement with destruction is explained by: variability of soil-geological, climatic conditions of operation of pavement; the variety of physical and mechanical characteristics of the materials of the structural layers of the pavement and the soil of the ground; heterogeneity of geometric parameters of pavements in the longitudinal and transverse profiles. According to many scientists to assess the condition of the pavement is a necessary condition for the use of methods of probabilistic analysis. Methods of designing and assessing the condition of the pavement structure must take into account the principles of reliability, including probabilistic methods of reliability analysis. Most of the considered probabilistic models for assessing the condition of pavement are focused on the design of new construction, which allows not to take into account the patterns of changes in physical and mechanical properties of materials of structural layers and their geometric parameters during operation. However, the most relevant is to assess the condition of pavement with damage, including hidden cracks. Much of the article is devoted to the analysis of this area of research. According to the results of the analysis, it is concluded that probabilistic methods allow to take into account the heterogeneity of parameters that characterize the stress-strain state of the pavement structure and can be used in models to assess the condition of pavement with cracks in layers of monolithic materials.

Model-based Damage Detection through Physics Guided Learning

Data-driven learning approaches have gained a lot of interest in evaluating and validating complex dynamic systems. One of the main challenges for developing a reliable learning model is the lack of training data covering a large range of various operational conditions. Extensive training data can be generated using a physics-based simulation model. On the other hand, the learning algorithm should be still tested with experimental data obtained from the actual system. Modeling errors may lead to a statistical divergence between the simulation training data and the experimental testing data, causing poor performance, especially for domain-agnostic black-box learning methods. To close the gap between the simulation and experimental domains, this paper proposes a physics-guided learning approach that combines the power of the neural network with domain-specific physics knowledge. Specifically, the proposed architecture integrates physical parameters extracted from the physics-based simulation model into the intermediate layers of the neural network to constrain the learning process. To demonstrate the effectiveness of the proposed approach, the architecture is adopted to a damage classification problem for a three-story structure. Our results show that the accuracy for localizing the damage correctly based on experimental data improves significantly over black-box models, especially under large modeling errors. In this paper, we also use the physics-based intermediate layers to analyze the interpretability of the classification results.

ResMem-Net: memory based deep CNN for image memorability estimation

Image memorability is a very hard problem in image processing due to its subjective nature. But due to the introduction of Deep Learning and the large availability of data and GPUs, great strides have been made in predicting the memorability of an image. In this paper, we propose a novel deep learning architecture called ResMem-Net that is a hybrid of LSTM and CNN that uses information from the hidden layers of the CNN to compute the memorability score of an image. The intermediate layers are important for predicting the output because they contain information about the intrinsic properties of the image. The proposed architecture automatically learns visual emotions and saliency, shown by the heatmaps generated using the GradRAM technique. We have also used the heatmaps and results to analyze and answer one of the most important questions in image memorability: “What makes an image memorable?”. The model is trained and evaluated using the publicly available Large-scale Image Memorability dataset (LaMem) from MIT. The results show that the model achieves a rank correlation of 0.679 and a mean squared error of 0.011, which is better than the current state-of-the-art models and is close to human consistency (p = 0.68). The proposed architecture also has a significantly low number of parameters compared to the state-of-the-art architecture, making it memory efficient and suitable for production.

Two-stage PVD method for protective coating formation

The CVD methods are typically used for the formation of aluminum oxide coatings since aluminum oxide is a dielectric. The adhesion between the protective coating and the substrate material is normally improved by growing thin intermediate layers based on titanium oxides and nitrides. These intermediate layers are mainly formed using the PVD methods. In this paper, we propose a two-stage PVD method for forming a layered structure on the titanium substrate. The formation of intermediate layers was carried out by the magnetron method (first stage), and the main protective layer was deposited at the second stage using a fore-vacuum electron source. The dense beam plasma generated during the electron beam transport in a fore-vacuum gas medium compensates for the negative electrical charge accumulating on the surface of the aluminum oxide target and facilitates its effective evaporation. The electrical properties of the intermediate layers and the resulting layered coatings have been investigated, including the tangent of dielectric loss angle, the real and imaginary parts of the conductivity and the dielectric constant dependencies on frequency.