Avoiding Overfitting: A Survey on Regularization Methods for Convolutional Neural Networks

Several image processing tasks, such as image classification and object detection, have been significantly improved using Convolutional Neural Networks (CNN). Like ResNet and EfficientNet, many architectures have achieved outstanding results in at least one dataset by the time of their creation. A critical factor in training concerns the network’s regularization, which prevents the structure from overfitting. This work analyzes several regularization methods developed in the last few years, showing significant improvements for different CNN models. The works are classified into three main areas: the first one is called “data augmentation”, where all the techniques focus on performing changes in the input data. The second, named “internal changes”, which aims to describe procedures to modify the feature maps generated by the neural network or the kernels. The last one, called “label”, concerns transforming the labels of a given input. This work presents two main differences comparing to other available surveys about regularization: (i) the first concerns the papers gathered in the manuscript, which are not older than five years, and (ii) the second distinction is about reproducibility, i.e., all works refered here have their code available in public repositories or they have been directly implemented in some framework, such as TensorFlow or Torch.

Restoration of Hardware Function of Spectral Colorimeters Using Regularization Methods

The problem of determining the spectral characteristic of a controlled sample under conditions of limited a priori information using regularization methods is considered in the paper. A change in the state of the surface of optical elements significantly increases the light scattering, so it is necessary regularly to take into account the amount of scattered light in the light flux reflected from the surface and the measured and comparative samples. The conversion of the light flux into the electrical signal of the photodetector can also occur non-linearly. This requires the development of such measurement method that considers both the scattered light and various non-linearities of the measuring circuit. It is known that the mathematical model of measurement is described by the Fredholm integral equation of the first kind, its solution under the accepted assumptions is recommended to be sought in the form of a matrix equation using a recurring procedure. With regard to the fact that the estimation of the initial data errors in the equation is associated with certain difficulties, in the case under consideration, it is advisable to determine the regularization parameter based on the method of quasi-optimality. A characteristic disadvantage of the known analytical and experimental methods for determining the hardware function of a spectral device is that they do not take into account its change during operation. Since the actual hardware function of the device usually differs from the Gaussian curve, the use of hardware functions in the form of analytical dependencies does not always give the desired result, and for experimental methods, special equipment with a quasi-monochromatic radiation source is required. An algorithm for restoring the hardware function of a spectral device based on regular methods for solving ill-posed problems is proposed. The estimation of the matrix operator of the hardware function is proposed to be obtained on the basis of explicit least squares estimation algorithms. The expediency of choosing a value of the regularization parameter that minimizes the accepted characteristic of the accuracy of the solution is indicated.

Interval forecasting of time series using orderstatistics

In the traditional approach of obtaining time series forecasts based on the selected model, the model parameters are first estimated, then a point forecast using the obtained estimatesis made and then an interval forecast with a given probability is made. In the article the authors propose a nonparametric method for obtaining a single-stage interval forecasting of a time series based on constructing predictive and target variables sets using robust statistics and obtaining the forecast boundaries by constructing linear regression models. The predictive algorithm is based on the problems of estimating the parameters of linear multiple regression using a model regularization methods. The results of forecasting prove the expediency and effectiveness of the proposed method.

An Experimental Study on State Representation Extraction for Vision-Based Deep Reinforcement Learning

Scaling end-to-end learning to control robots with vision inputs is a challenging problem in the field of deep reinforcement learning (DRL). While achieving remarkable success in complex sequential tasks, vision-based DRL remains extremely data-inefficient, especially when dealing with high-dimensional pixels inputs. Many recent studies have tried to leverage state representation learning (SRL) to break through such a barrier. Some of them could even help the agent learn from pixels as efficiently as from states. Reproducing existing work, accurately judging the improvements offered by novel methods, and applying these approaches to new tasks are vital for sustaining this progress. However, the demands of these three aspects are seldom straightforward. Without significant criteria and tighter standardization of experimental reporting, it is difficult to determine whether improvements over the previous methods are meaningful. For this reason, we conducted ablation studies on hyperparameters, embedding network architecture, embedded dimension, regularization methods, sample quality and SRL methods to compare and analyze their effects on representation learning and reinforcement learning systematically. Three evaluation metrics are summarized, including five baseline algorithms (including both value-based and policy-based methods) and eight tasks are adopted to avoid the particularity of each experiment setting. We highlight the variability in reported methods and suggest guidelines to make future results in SRL more reproducible and stable based on a wide number of experimental analyses. We aim to spur discussion about how to assure continued progress in the field by minimizing wasted effort stemming from results that are non-reproducible and easily misinterpreted.

Flexible Krylov Methods for Edge Enhancement in Imaging

Many successful variational regularization methods employed to solve linear inverse problems in imaging applications (such as image deblurring, image inpainting, and computed tomography) aim at enhancing edges in the solution, and often involve non-smooth regularization terms (e.g., total variation). Such regularization methods can be treated as iteratively reweighted least squares problems (IRLS), which are usually solved by the repeated application of a Krylov projection method. This approach gives rise to an inner–outer iterative scheme where the outer iterations update the weights and the inner iterations solve a least squares problem with fixed weights. Recently, flexible or generalized Krylov solvers, which avoid inner–outer iterations by incorporating iteration-dependent weights within a single approximation subspace for the solution, have been devised to efficiently handle IRLS problems. Indeed, substantial computational savings are generally possible by avoiding the repeated application of a traditional Krylov solver. This paper aims to extend the available flexible Krylov algorithms in order to handle a variety of edge-enhancing regularization terms, with computationally convenient adaptive regularization parameter choice. In order to tackle both square and rectangular linear systems, flexible Krylov methods based on the so-called flexible Golub–Kahan decomposition are considered. Some theoretical results are presented (including a convergence proof) and numerical comparisons with other edge-enhancing solvers show that the new methods compute solutions of similar or better quality, with increased speedup.

Regularization methods for ill-posed problems of general physics

On the example of a specific physical problem of noise reduction associated with losses, dark counts, and background radiation, a summary of methods for regularizing ill-posed problems is given in the statistics of photocounts of quantum light. The mathematical formulation of the problem is presented by an operator equation of the first kind. The operator is generated by a matrix with countable elements. In the sense of Hadamard, the problem of reconstructing the number of photons of quantum light is due to the compactness of the operator of the mathematical model. A rigorous definition of a regularizing operator (regularizer) is given. The problem of stable approximation to the exact solution of the operator equation with inaccurately given initial data can be overcome by one of the most well-known regularization methods, the theoretical foundations of which were laid in the works of A.N. Tikhonov. The selection of an important class of regularizing algorithms is based on the construction of a parametric family of functions that are Borel measurable on the semiaxis and satisfy some additional conditions. The set of regularizers in this family includes most of the known regularization methods. The main ones are given in the work.

An Insightful Overview of the Wiener Filter for System Identification

Efficiently solving a system identification problem represents an important step in numerous important applications. In this framework, some of the most popular solutions rely on the Wiener filter, which is widely used in practice. Moreover, it also represents a benchmark for other related optimization problems. In this paper, new insights into the regularization of the Wiener filter are provided, which is a must in real-world scenarios. A proper regularization technique is of great importance, especially in challenging conditions, e.g., when operating in noisy environments and/or when only a low quantity of data is available for the estimation of the statistics. Different regularization methods are investigated in this paper, including several new solutions that fit very well for the identification of sparse and low-rank systems. Experimental results support the theoretical developments and indicate the efficiency of the proposed techniques.

Learning Domain Invariant Representation via Self-Rugularization

Unsupervised domain adaptation often gives impressive solutions to handle domain shift of data. Most of current approaches assume that unlabeled target data to train is abundant. This assumption is not always true in practices. To tackle this issue, we propose a general solution to solve the domain gap minimization problem without any target data. Our method consists of two regularization steps. The first step is a pixel regularization by arbitrary style transfer. Recently, some methods bring style transfer algorithms to domain adaptation and domain generalization process. They use style transfer algorithms to remove texture bias in source domain data. We also use style transfer algorithms for removing texture bias, but our method depends on neither domain adaptation nor domain generalization paradigm. The second regularization step is a feature regularization by feature alignment. Adding a feature alignment loss term to the model loss, the model learns domain invariant representation more efficiently. We evaluate our regularization methods from several experiments both on small dataset and large dataset. From the experiments, we show that our model can learn domain invariant representation as much as unsupervised domain adaptation methods.