Understanding deep learning (still) requires rethinking generalization

Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small gap between training and test performance. Conventional wisdom attributes small generalization error either to properties of the model family or to the regularization techniques used during training. Through extensive systematic experiments, we show how these traditional approaches fail to explain why large neural networks generalize well in practice. Specifically, our experiments establish that state-of-the-art convolutional networks for image classification trained with stochastic gradient methods easily fit a random labeling of the training data. This phenomenon is qualitatively unaffected by explicit regularization and occurs even if we replace the true images by completely unstructured random noise. We corroborate these experimental findings with a theoretical construction showing that simple depth two neural networks already have perfect finite sample expressivity as soon as the number of parameters exceeds the number of data points as it usually does in practice. We interpret our experimental findings by comparison with traditional models. We supplement this republication with a new section at the end summarizing recent progresses in the field since the original version of this paper.

Error rates in spatial classification of Gaussian data with random labeling

In spatial classification it is usually assumed that features observations given labels are independently distributed. We have retracted this assumption by proposing stationary Gaussian random field model for features observations. The label are assumed to follow Disrete Random Field (DRF) model. Formula for exact error rate based on Bayes discriminant function (BDF) is derived. In the case of partial parametric uncertainty (mean parameters and variance are unknown), the approximation of the expected error rate associated with plug-in BDF is also derived. The dependence of considered error rates on the values of range and clustering parameters is investigated numerically for training locations being second-order neighbors to location of observation to be classified.

Some Monotone Almost Balanced Knockout Tournament Plans

Knockout tournaments are often used in sports (or experiments where preferences are registered by comparisons instead of measurements) to determine the champion of an event. A knockout tournament plan (KTP) for n players is a rooted binary tree with n leaves to be labeled by the n players. Each subtree of two leaves represents a match between the two players labeling the two leaves; the winner of the match then moves on to label the root of the subtree. While there are many KTPs to choose from for a given number of players, in the real world an almost balanced KTP is usually chosen. One reason could be the perception that a balanced KTP is “fair” to the players, in the sense that, given a random labeling of leaves by players, a stronger player has a better chance to win the tournament. Surprisingly, it has been shown that not all KTPs have this property, and it is difficult to prove this property for any general class of KTPs. So far the property has been shown to hold only for balanced KTPs. In this paper we extend it to some classes of almost balanced KTPs.

Extremes of urine osmolality: lack of effect on red blood cell survival

Rats were allowed a third of normal water intake for 20 days, and food consumption decreased. The reticulocyte count indicated a suppression of erythropoiesis. Urine osmolality increased from 2,000 mosmol/kg to 3,390 mosmol/kg. Random hemolysis and senescence of a cohort of red blood cell (RBC) previously labeled with [2-14C]glycine was monitored via the production of 14CO. Neither hemolysis nor senescence was affected. Following water restriction, the polydipsic rats generated a hypotonic urine. Urine osmolality decreased to 1,300 mosmol/kg for at least 6 days; a reticulocytosis occurred, but RBC survival was unaffected. These results contradict those previously reported, which suggest that RBC survival is influenced by the osmotic stress imposed on the RBC by extremes of urine tonicity. This discrepancy, we conclude, is due to differences in the methods employed for measuring RBC survival. The random-labeling technique employed previously assumes a steady state between RBC production and destruction. The cohort-labeling technique used here measures hemolysis and senescence independent of changes in RBC production, which is known to be depressed by fasting.

MITOTIC AND NONMITOTIC MULTIPLE-LAYERED PERFUSION CULTURES

Cell types in addition to those previously described (Kruse et al. 1963. J. Nat. Cancer Inst. 31:109; Kruse and Miedema. 1965. J. Cell Biol. 27:273) were found to form multiple-layered cultures by perfusion-culture technique. Dense populations containing 43 x 106 embryonic rat muscle (NF-ER) cells, 23 x 106 diploid human tonsillar (NF-JAM) cells, 77 x 106 human pleural effusion isolate (RPMI 2650) cells, 35 x 106 embryonic diploid human lung (Flow 2000) cells, 21 x 106 bovine lung (FB4BM) cells, 108 x 106 bat lung (Tb1Lu) cells, and 81 x 106 SV-40 virus-transformed embryonic diploid human lung (WI-38VA13A) cells were obtained in 6–14 days from dilute inocula in T-60 or T-75 flasks; these were equivalent to about 4, 3, 3, 4, 2, 4, and eight monolayers, respectively. Perfusion of an NF-ER culture for 6 wk with medium plus 10% whole calf serum yielded a cell density equivalent to 12 monolayers (140 x 106 cells per T-75 flask). This culture exhibited random labeling of nuclei from bottom to top after pulsing for 90 min with thymidine-3H. Medium plus 0.1% serum maintained NF-JAM cultures at constant viable cell numbers with virtual absence of thymidine-3H labeling. Similar results were obtained with WI-38 cultures, but WI-38VA13A cells continued active DNA synthesis and mitosis in medium with 0.1% serum to form 16–20 layers of cells (191–239 x 106 cells per T-75 flask) in 27 days. WI-38VA13A cells ceased proliferation and became nonviable rapidly in serumless medium.