Stronger convergence results for deep residual networks: network width scales linearly with training data size

Deep neural networks are highly expressive machine learning models with the ability to interpolate arbitrary datasets. Deep nets are typically optimized via first-order methods, and the optimization process crucially depends on the characteristics of the network as well as the dataset. This work sheds light on the relation between the network size and the properties of the dataset with an emphasis on deep residual networks (ResNets). Our contribution is that if the network Jacobian is full rank, gradient descent for the quadratic loss and smooth activation converges to the global minima even if the network width $m$ of the ResNet scales linearly with the sample size $n$ and logarithmically with the network depth $H$. Consequently, our work is able to provide a theoretical guarantee for the convergence of deep neural networks in the $m=\varOmega (n,\log H)$ regime.

Coarse fundamental groups and box spaces

We use a coarse version of the fundamental group first introduced by Barcelo, Kramer, Laubenbacher and Weaver to show that box spaces of finitely presented groups detect the normal subgroups used to construct the box space, up to isomorphism. As a consequence, we have that two finitely presented groups admit coarsely equivalent box spaces if and only if they are commensurable via normal subgroups. We also provide an example of two filtrations (Ni) and (Mi) of a free group F such that Mi > Ni for all i with [Mi:Ni] uniformly bounded, but with $\squ _{(N_i)}F$ not coarsely equivalent to $\squ _{(M_i)}F$. Finally, we give some applications of the main theorem for rank gradient and the first ℓ2 Betti number, and show that the main theorem can be used to construct infinitely many coarse equivalence classes of box spaces with various properties.

Groups with positive rank gradient and their actions

We show that given a finitely generated LERF group G with positive rank gradient, and finitely generated subgroups A, B ≤ G of infinite index, one can find a finite index subgroup B0 of B such that [G : 〈A ∪ B0〉] = ∞. This generalizes a theorem of Olshanskii on free groups. We conclude that a finite product of finitely generated subgroups of infinite index does not cover G. We construct a transitive virtually faithful action of G such that the orbits of finitely generated subgroups of infinite index are finite. Some of the results extend to profinite groups with positive rank gradient.