Interval universal approximation for neural networks

To verify safety and robustness of neural networks, researchers have successfully applied abstract interpretation , primarily using the interval abstract domain. In this paper, we study the theoretical power and limits of the interval domain for neural-network verification. First, we introduce the interval universal approximation (IUA) theorem. IUA shows that neural networks not only can approximate any continuous function f (universal approximation) as we have known for decades, but we can find a neural network, using any well-behaved activation function, whose interval bounds are an arbitrarily close approximation of the set semantics of f (the result of applying f to a set of inputs). We call this notion of approximation interval approximation . Our theorem generalizes the recent result of Baader et al. from ReLUs to a rich class of activation functions that we call squashable functions . Additionally, the IUA theorem implies that we can always construct provably robust neural networks under ℓ ∞ -norm using almost any practical activation function. Second, we study the computational complexity of constructing neural networks that are amenable to precise interval analysis. This is a crucial question, as our constructive proof of IUA is exponential in the size of the approximation domain. We boil this question down to the problem of approximating the range of a neural network with squashable activation functions. We show that the range approximation problem (RA) is a Δ 2 -intermediate problem, which is strictly harder than NP -complete problems, assuming coNP ⊄ NP . As a result, IUA is an inherently hard problem : No matter what abstract domain or computational tools we consider to achieve interval approximation, there is no efficient construction of such a universal approximator. This implies that it is hard to construct a provably robust network, even if we have a robust network to start with.

Superpixel Weighted Low-rank and Sparse Approximation for Hyperspectral Unmixing

We propose a superpixel weighted low-rank and sparse unmixing (SWLRSU) method for sparse unmixing. The proposed method consists of two steps. In the first step, we segment hyperspectral image into superpixels which are defined as the homogeneous regions with different shape and sizes according to the spatial structure. Then, an efficient method is proposed to obtain a spatial weight term using superpixels to capture the spatial structure of hyperspectral data. In the second step, we solve a superpixel guided low-rank and spatially weighted sparse approximation problem in which spatial weight term obtained in the first step is used as a weight term in sparsity promoting norm. This formulation exploits the spatial correlation of the pixels in the hyperspectral image efficiently, which yields satisfactory unmixing results. The experiments are conducted on simulated and real data sets to show the effectiveness of the proposed method.

Superpixel Weighted Low-rank and Sparse Approximation for Hyperspectral Unmixing

We propose a superpixel weighted low-rank and sparse unmixing (SWLRSU) method for sparse unmixing. The proposed method consists of two steps. In the first step, we segment hyperspectral image into superpixels which are defined as the homogeneous regions with different shape and sizes according to the spatial structure. Then, an efficient method is proposed to obtain a spatial weight term using superpixels to capture the spatial structure of hyperspectral data. In the second step, we solve a superpixel guided low-rank and spatially weighted sparse approximation problem in which spatial weight term obtained in the first step is used as a weight term in sparsity promoting norm. This formulation exploits the spatial correlation of the pixels in the hyperspectral image efficiently, which yields satisfactory unmixing results. The experiments are conducted on simulated and real data sets to show the effectiveness of the proposed method.

Improving Sample Average Approximation Using Distributional Robustness

Sample average approximation is a popular approach to solving stochastic optimization problems. It has been widely observed that some form of robustification of these problems often improves the out-of-sample performance of the solution estimators. In estimation problems, this improvement boils down to a trade-off between the opposing effects of bias and shrinkage. This paper aims to characterize the features of more general optimization problems that exhibit this behaviour when a distributionally robust version of the sample average approximation problem is used. The paper restricts attention to quadratic problems for which sample average approximation solutions are unbiased and shows that expected out-of-sample performance can be calculated for small amounts of robustification and depends on the type of distributionally robust model used and properties of the underlying ground-truth probability distribution of random variables. The paper was written as part of a New Zealand funded research project that aimed to improve stochastic optimization methods in the electric power industry. The authors of the paper have worked together in this domain for the past 25 years.

Fast approximation by periodic kernel-based lattice-point interpolation with application in uncertainty quantification

This paper deals with the kernel-based approximation of a multivariate periodic function by interpolation at the points of an integration lattice—a setting that, as pointed out by Zeng et al. (Monte Carlo and Quasi-Monte Carlo Methods 2004, Springer, New York, 2006) and Zeng et al. (Constr. Approx. 30: 529–555, 2009), allows fast evaluation by fast Fourier transform, so avoiding the need for a linear solver. The main contribution of the paper is the application to the approximation problem for uncertainty quantification of elliptic partial differential equations, with the diffusion coefficient given by a random field that is periodic in the stochastic variables, in the model proposed recently by Kaarnioja et al. (SIAM J Numer Anal 58(2): 1068–1091, 2020). The paper gives a full error analysis, and full details of the construction of lattices needed to ensure a good (but inevitably not optimal) rate of convergence and an error bound independent of dimension. Numerical experiments support the theory.

CONDITIONS OF MONOTONE APPROXIMATION OF RAMSEY CURVES AND THEIR MODIFICATIONS

The algorithm for approximating the experimental data of the Ramsey curve and its modifications has been developed, which provides a monotonic increase of the approximating function in the interval [0;\infty) and an existence of a given number of inflection points. The Ramsey curve belongs to the family of logistic curves that are widely used in modeling of limited increasing processes in various subject fields. The classical Ramsey curve has two parameters and has a left constant asymmetry. It is also known that its three-parameter modification provides the possibility of displacement along the axes of ordinate. The extensive practical use of the Ramsey curve with both two and more parameters for approximating experimental dependences is restrained by the frequent loss by this curve of the logistic shape when approximating without additional restrictions on the relationships between its parameters. The article discusses modifications of the Ramsey curve with three and five parameters. The first and second derivatives of the studied modifications of the Ramsey function have a special structure. They are products of polynomial and exponential functions. This allows using Sturm's theorem on the number of polynomial roots in a given interval to control the shape of the approximating curve. It has been shown that with an increase in the number of parameters for the modified curve, the number of possible combinations of restrictions on the values of the parameters ensuring the preservation of its like shape increases significantly. The solution to the approximation problem in this case consists of solving a sequence of conditional global optimization problems with different constraints and choosing a solution that provides the smallest approximation error. Also, the studies of the accuracy of estimating the parameters of the Ramsey curve in accordance with the accuracy of the experimental data have been carried out. In order to simulate the presence of measurement errors, the values of a normally distributed random variable with a mathematical expectation equal to zero and different values of the standard deviation for different series of computational experiments were added to the values of the deterministic sequence. Computational experiments have shown a significant sensitivity of the values of the Ramsey function parameters to the measurement accuracy of experimental data.

Two-stage spline-approximation in linear structure routing

In the article, computer design of routes of linear structures is considered as a spline approximation problem. A fundamental feature of the corresponding design tasks is that the plan and longitudinal profile of the route consist of elements of a given type. Depending on the type of linear structure, line segments, arcs of circles, parabolas of the second degree, clothoids, etc. are used. In any case, the design result is a curve consisting of the required sequence of elements of a given type. At the points of conjugation, the elements have a common tangent, and in the most difficult case, a common curvature. Such curves are usually called splines. In contrast to other applications of splines in the design of routes of linear structures, it is necessary to take into account numerous restrictions on the parameters of spline elements arising from the need to comply with technical standards in order to ensure the normal operation of the future structure. Technical constraints are formalized as a system of inequalities. The main distinguishing feature of the considered design problems is that the number of elements of the required spline is usually unknown and must be determined in the process of solving the problem. This circumstance fundamentally complicates the problem and does not allow using mathematical models and nonlinear programming algorithms to solve it, since the dimension of the problem is unknown. The article proposes a two-stage scheme for spline approximation of a plane curve. The curve is given by a sequence of points, and the number of spline elements is unknown. At the first stage, the number of spline elements and an approximate solution to the approximation problem are determined. The method of dynamic programming with minimization of the sum of squares of deviations at the initial points is used. At the second stage, the parameters of the spline element are optimized. The algorithms of nonlinear programming are used. They were developed taking into account the peculiarities of the system of constraints. Moreover, at each iteration of the optimization process for the corresponding set of active constraints, a basis is constructed in the null space of the constraint matrix and in the subspace – its complement. This makes it possible to find the direction of descent and solve the problem of excluding constraints from the active set without solving systems of linear equations. As an objective function, along with the traditionally used sum of squares of the deviations of the initial points from the spline, the article proposes other functions taking into account the specificity of a particular project task.

Regression and Evaluation on a Forward Interpolated Version of the Great Circle Arcs–Based Distortion Metric of Map Projections

We studied the numerical approximation problem of distortion in map projections. Most widely used differential methods calculate area distortion and maximum angular distortion using partial derivatives of forward equations of map projections. However, in certain map projections, partial derivatives are difficult to calculate because of the complicated forms of forward equations, e.g., equations with iterations, integrations, or multi-way branches. As an alternative, the spherical great circle arcs–based metric employs the inverse equations of map projections to transform sample points from the projection plane to the spherical surface, and then calculates a differential-independent distortion metric for the map projections. We introduce a novel forward interpolated version of the previous spherical great circle arcs–based metric, solely dependent on the forward equations of map projections. In our proposed numerical solution, a rational function–based regression is also devised and applied to our metric to obtain an approximate metric of angular distortion. The statistical and graphical results indicate that the errors of the proposed metric are fairly low, and a good numerical estimation with high correlation to the differential-based metric can be achieved.

Monotonicity properties and solvability of dominated best approximation problem in Orlicz spaces equipped with s-norms

In the paper, Wisła (J Math Anal Appl 483(2):123659, 2020, 10.1016/j.jmaa.2019.123659), it was proved that the classical Orlicz norm, Luxemburg norm and (introduced in 2009) p-Amemiya norm are, in fact, special cases of the s-norms defined by the formula $$\left\| x\right\| _{\Phi ,s}=\inf _{k>0}\frac{1}{k}s\left( \int _T \Phi (kx)d\mu \right) $$ x Φ , s = inf k > 0 1 k s ∫ T Φ ( k x ) d μ , where s and $$\Phi $$ Φ are an outer and Orlicz function respectively and x is a measurable real-valued function over a $$\sigma $$ σ -finite measure space $$(T,\Sigma ,\mu )$$ ( T , Σ , μ ) . In this paper the strict monotonicity, lower and upper uniform monotonicity and uniform monotonicity of Orlicz spaces equipped with the s-norm are studied. Criteria for these properties are given. In particular, it is proved that all of these monotonicity properties (except strict monotonicity) are equivalent, provided the outer function s is strictly increasing or the measure $$\mu $$ μ is atomless. Finally, some applications of the obtained results to the best dominated approximation problems are presented.