Understanding approximate Fisher information for fast convergence of natural gradient descent in wide neural networks*

Natural gradient descent (NGD) helps to accelerate the convergence of gradient descent dynamics, but it requires approximations in large-scale deep neural networks because of its high computational cost. Empirical studies have confirmed that some NGD methods with approximate Fisher information converge sufficiently fast in practice. Nevertheless, it remains unclear from the theoretical perspective why and under what conditions such heuristic approximations work well. In this work, we reveal that, under specific conditions, NGD with approximate Fisher information achieves the same fast convergence to global minima as exact NGD. We consider deep neural networks in the infinite-width limit, and analyze the asymptotic training dynamics of NGD in function space via the neural tangent kernel. In the function space, the training dynamics with the approximate Fisher information are identical to those with the exact Fisher information, and they converge quickly. The fast convergence holds in layer-wise approximations; for instance, in block diagonal approximation where each block corresponds to a layer as well as in block tri-diagonal and K-FAC approximations. We also find that a unit-wise approximation achieves the same fast convergence under some assumptions. All of these different approximations have an isotropic gradient in the function space, and this plays a fundamental role in achieving the same convergence properties in training. Thus, the current study gives a novel and unified theoretical foundation with which to understand NGD methods in deep learning.

An Optical Flow Based Left-Invariant Metric for Natural Gradient Descent in Affine Image Registration

Accurate spatial alignment is essential for any population neuroimaging study, and affine (12 parameter linear/translation) or rigid (6 parameter rotation/translation) alignments play a major role. Here we consider intensity based alignment of neuroimages using gradient based optimization, which is a problem that continues to be important in many other areas of medical imaging and computer vision in general. A key challenge is robustness. Optimization often fails when transformations have components with different characteristic scales, such as linear versus translation parameters. Hand tuning or other scaling approaches have been used, but efficient automatic methods are essential for generalizing to new imaging modalities, to specimens of different sizes, and to big datasets where manual approaches are not feasible. To address this we develop a left invariant metric on these two matrix groups, based on the norm squared of optical flow induced on a template image. This metric is used in a natural gradient descent algorithm, where gradients (covectors) are converted to perturbations (vectors) by applying the inverse of the metric to define a search direction in which to update parameters. Using a publicly available magnetic resonance neuroimage database, we show that this approach outperforms several other gradient descent optimization strategies. Due to left invariance, our metric needs to only be computed once during optimization, and can therefore be implemented with negligible computation time.

MA Cation-Induced Diffusional Growth of Low-Bandgap FA-Cs Perovskites Driven by Natural Gradient Annealing

Low-bandgap formamidinium-cesium (FA-Cs) perovskites of FA1-xCsxPbI3 (x<0.1) are promising candidates for efficient and robust perovskite solar cells, but their black-phase crystallization is very sensitive to annealing temperature. Unfortunately, the low heat conductivity of the glass substrate builds up a temperature gradient within from bottom to top and makes the initial annealing temperature of the perovskite film lower than the black-phase crystallization point (~150°C). Herein, we take advantage of such temperature gradient for the diffusional growth of high-quality FA-Cs perovskites by introducing a thermally unstable MA+ cation, which would firstly form α-phase FA-MA-Cs mixed perovskites with low formation energy at the hot bottom of the perovskite films in the early annealing stage. The natural gradient annealing temperature and the thermally unstable MA+ cation then lead to the bottom-to-top diffusional growth of highly orientated α-phase FA-Cs perovskite, which exhibits 10-fold of enhanced crystallinity and reduced trap density (~3.85×1015 cm−3). Eventually, such FA-Cs perovskite films were fabricated into stable solar cell devices with champion efficiency up to 23.11%, among the highest efficiency of MA-free perovskite solar cells.

PEMISAHAN SINYAL BUNYI DARI MICROPHONE ARRAY DENGAN MENGGUNAKAN METODE BLIND SOURCE SEPARATION - INDEPENDENT COMPONENT ANALYSIS UNTUK MEMANTAU KONDISI POROS RETAK

Abstrak Tulisan ini mempresentasikan tentang pemisahan sinyal bunyi untuk memantau kondisi poros dengan menggunakan metode Blind Source Separation (BSS) - Independent Component Analysis (ICA). Pada penelitian ini, bunyi poros retak yang sementara berputar direkam melalui susunan mikrofon (microphone array) sebagai sensornya. Tiap-tiap mikrofon menerima sinyal dari poros tersebut, sehingga sinyal output dari tiap mikrofon merupakan sinyal campuran. BSS merupakan teknik memisahkan sinyal campuran berdasarkan analisa kebebasan statistik ICA sumber bunyi. Dengan memperhatikan jarak dan sudut datang antara mikrofon dengan poros maka tiap mikrofon menerima sinyal berbeda pula. Sinyal campuran dari tiap mikrofon akan diestimasi untuk memantau kondisi poros berdasarkan analisa pola bunyi. Pada penelitian ini pemisahan sinyal dilakukan pada time-domain dengan algoritma natural gradient. Berdasarkan hasil penelitian diperoleh metode pemisahan sinyal terbaik adalah metode pemisahan sinyal dalam kawasan waktu (TDICA) jauh lebih baik dari metode FDICA karena nilai MSE melalui TDICA jauh lebih kecil.

Serialized gradient chromatography for the digital blending of colored fibers and spinning of gradient colored yarn

Gradient colored yarns are manufactured by controlling the blending ratios of three-primary-colored fibers in a slight distribution of gradients along the yarn length, thereby resulting in a continuous natural variation in mixed colors of the fibers throughout the whole yarn. The spinning of gradient colored yarns still remains a challenge, which requires the reliance on digital blending theory of colored fibers and the supporting of multi-channel computer numerical control (CNC) spinning technique. This paper constructed a three-primary-colored fiber gridded color mixing model and its mass mixing matrix and color mixing chromatography matrix by mass discretization and coupling pairing with a 10% gradient for the three-primary-colored fibers. With the aim of continuous natural gradient of mixed colors, the blending ratio gradient path of three-primary-colored fibers was planned based on the mass mixing matrix, and a method of regulating the gradient of color difference between adjacent color segments was proposed. In order to realize the natural gradient of color of the forming yarn, the spinning mechanism of gradient colored yarn was established based on three-channel CNC spinning mechanism and the time-series yarn simulation model, and the time-series spinning processing parameters of three-channel CNC spinning machine were devised. Four gradient colored yarns with different gradient paths were designed and prepared, the linear density, twist, unevenness, surface hairiness, and tensile strength of the spun yarns were measured, compared, and analyzed, and knitted fabrics with color gradient effect were fabricated by small circular knitting machine to obtain continuous and natural color transition with a dreamy and mysterious color effect.

A new adaptive variable step size natural gradient BSS algorithm

The traditional blind source separation (BSS) algorithm is mainly used to deal with signal separation under the noiseless model, but it does not apply to data with the low signal to noise ratio (SNR). To solve the problem, an adaptive variable step size natural gradient BSS algorithm based on an improved wavelet threshold is proposed in this paper. Firstly, an improved wavelet threshold method is used to reduce the noise of the signal. Secondly, the wavelet coefficient layer with obvious periodicity is denoised using a morphological component analysis (MCA) algorithm, and the processed wavelet coefficients are recombined to obtain the ideal model. Thirdly, the recombined signal is pre-whitened, and a new separation matrix update formula of natural gradient algorithm is constructed by defining a new separation degree estimation function. Finally, the adaptive variable step size natural gradient blind source algorithm is used to separate the noise reduction signal. The results show that the algorithm can not only adaptively adjust the step size according to different signals, but also improve the convergence speed, stability and separation accuracy.