Interval universal approximation for neural networks

To verify safety and robustness of neural networks, researchers have successfully applied abstract interpretation , primarily using the interval abstract domain. In this paper, we study the theoretical power and limits of the interval domain for neural-network verification. First, we introduce the interval universal approximation (IUA) theorem. IUA shows that neural networks not only can approximate any continuous function f (universal approximation) as we have known for decades, but we can find a neural network, using any well-behaved activation function, whose interval bounds are an arbitrarily close approximation of the set semantics of f (the result of applying f to a set of inputs). We call this notion of approximation interval approximation . Our theorem generalizes the recent result of Baader et al. from ReLUs to a rich class of activation functions that we call squashable functions . Additionally, the IUA theorem implies that we can always construct provably robust neural networks under ℓ ∞ -norm using almost any practical activation function. Second, we study the computational complexity of constructing neural networks that are amenable to precise interval analysis. This is a crucial question, as our constructive proof of IUA is exponential in the size of the approximation domain. We boil this question down to the problem of approximating the range of a neural network with squashable activation functions. We show that the range approximation problem (RA) is a Δ 2 -intermediate problem, which is strictly harder than NP -complete problems, assuming coNP ⊄ NP . As a result, IUA is an inherently hard problem : No matter what abstract domain or computational tools we consider to achieve interval approximation, there is no efficient construction of such a universal approximator. This implies that it is hard to construct a provably robust network, even if we have a robust network to start with.

Ordered Weighted Averaging (OWA), Decision Making Under Uncertainty, and Deep Learning: How Is This All Related?

Among many research areas to which Ron Yager contributed are decision making under uncertainty (in particular, under interval and fuzzy uncertainty) and aggregation – where he proposed, analyzed, and utilized the use of Ordered Weighted Averaging (OWA). The OWA algorithm itself provides only a specific type of data aggregation. However, it turns out that if we allows several OWA stages one after another, we get a scheme with a universal approximation property – moreover, a scheme which is perfectly equivalent to deep neural networks. In this sense, Ron Yager can be viewed as a (grand)father of deep learning. We also show that the existing schemes for decision making under uncertainty are also naturally interpretable in OWA terms.

Universal Approximation Theorem for Tessarine-Valued Neural Networks

The universal approximation theorem ensures that any continuous real-valued function defined on a compact subset can be approximated with arbitrary precision by a single hidden layer neural network. In this paper, we show that the universal approximation theorem also holds for tessarine-valued neural networks. Precisely, any continuous tessarine-valued function can be approximated with arbitrary precision by a single hidden layer tessarine-valued neural network with split activation functions in the hidden layer. A simple numerical example, confirming the theoretical result and revealing the superior performance of a tessarine-valued neural network over a real-valued model for interpolating a vector-valued function, is presented in the paper.

Universal Taylor Series in Several Variables Depending on Parameters

We establish generic existence of Universal Taylor Series on products $$\varOmega = \prod \varOmega _i$$ Ω = ∏ Ω i of planar simply connected domains $$\varOmega _i$$ Ω i where the universal approximation holds on products K of planar compact sets with connected complements provided $$K \cap \varOmega = \emptyset $$ K ∩ Ω = ∅ . These classes are with respect to one or several centers of expansion and the universal approximation is at the level of functions or at the level of all derivatives. Also, the universal functions can be smooth up to the boundary, provided that $$K \cap \overline{\varOmega } = \emptyset $$ K ∩ Ω ¯ = ∅ and $$\{\infty \} \cup [{\mathbb {C}} {\setminus } \overline{\varOmega }_i]$$ { ∞ } ∪ [ C \ Ω ¯ i ] is connected for all i. All previous kinds of universal series may depend on some parameters; then the approximable functions may depend on the same parameters, as it is shown in the present paper.