Faster Distributed Deep Net Training: Computation and Communication Decoupled Stochastic Gradient Descent

With the increase in the amount of data and the expansion of model scale, distributed parallel training becomes an important and successful technique to address the optimization challenges. Nevertheless, although distributed stochastic gradient descent (SGD) algorithms can achieve a linear iteration speedup, they are limited significantly in practice by the communication cost, making it difficult to achieve a linear time speedup. In this paper, we propose a computation and communication decoupled stochastic gradient descent (CoCoD-SGD) algorithm to run computation and communication in parallel to reduce the communication cost. We prove that CoCoD-SGD has a linear iteration speedup with respect to the total computation capability of the hardware resources. In addition, it has a lower communication complexity and better time speedup comparing with traditional distributed SGD algorithms. Experiments on deep neural network training demonstrate the significant improvements of CoCoD-SGD: when training ResNet18 and VGG16 with 16 Geforce GTX 1080Ti GPUs, CoCoD-SGD is up to 2-3 x faster than traditional synchronous SGD.

A CLASS OF S-STEP NON-LINEAR ITERATION SCHEME BASED ON PROJECTION METHOD FOR GAUSS METHOD

Various iteration schemes are proposed by various authors to solve nonlinear equations arising in the implementation of implicit Runge-Kutta methods. In this paper, a class of s-step non-linear scheme based on projection method is proposed to accelerate the convergence rate of those linear iteration schemes. In this scheme, sequence of numerical solutions is updated after each sub-step is completed. For 2-stage Gauss method, upper bound for the spectral radius of its iteration matrix was obtained in the left half complex plane. This result is extended to 3-stage and 4-stage Gauss methods by transforming the coefficient matrix and the iteration matrix to a block diagonal form. Finally, some numerical experiments are carried out to confirm the obtained theoretical results.

Numerical simulations of the energy-stable scheme for Swift-Hohenberg equation

A collocation Fourier scheme for Swift-Hohenberg equation based on the convex splitting idea is implemented. To ensure an efficient numerical computation, we propose a general framework with linear iteration algorithm to solve the non-linear coupled equations which arise with the semi-implicit scheme. Following the contraction mapping theorem, we present a detailed convergence analysis for the linear iteration algorithm. Various numerical simulations, including verification of accuracy, dissipative property of discrete energy and pattern formation, are presented to demonstrate the efficiency and the robustness of proposed method.