Multilevel Initialization for Layer-Parallel Deep Neural Network Training

This paper investigates multilevel initializa- tion strategies for training very deep neural networks with a layer-parallel multigrid solver. The scheme is based on a continuous interpretation of the training problem as an optimal control problem, in which neu- ral networks are represented as discretizations of time- dependent ordinary differential equations. A key goal is to develop a method able to intelligently initialize the network parameters for the very deep networks en- abled by scalable layer-parallel training. To do this, we apply a uniform refinement strategy across the time domain, that is equivalent to refining in the layer di- mension. This refinement algorithm builds good ini- tializations for deep networks with network parameters coming from the coarser trained networks. The effec- tiveness of multilevel strategies (called nested iteration) for training is investigated using the Peaks and Indian Pines classification data sets. In both cases, the vali- dation accuracy achieved by nested iteration is higher than non-nested training. Moreover, run time to achieve the same validation accuracy is reduced. For instance, the Indian Pines example takes around 25% less time to train with the nested iteration algorithm. Finally, using the Peaks problem, we present preliminary anec- dotal evidence that the initialization strategy provides a regularizing effect on the training process, reducing sensitivity to hyperparameters and randomness in ini- tial network parameters.

An Analytical Solution to Dujmovic’s Iterative OWA

Iterative OWA (ItOWA) as proposed by Dujmovic, is a two-stage procedure for computing the weighting vector by a double nested iteration: (i) weights at step h are computed as limit to infinity of a matrix power, (ii) the result is used to start the computation at step h + 1, until the OWA operator arity n is reached. Thereafter Dujmovic suggested a computational solution based on the conjecture that the limit exists, and numerical simulations have being supported the hypothesis that the conjecture is correct. In this paper, we prove that the limit actually exists and we provide an analytical solution to the procedure, so the weighting vector can be computed directly instead of an iterative time-consuming procedure. This theoretical result enables a faster computation of the weighting vector and characterization in terms of weights values, attitudinal character and entropy.