On the Complexity of Computing and Learning with Multiplicative Neural Networks

2002 ◽  
Vol 14 (2) ◽  
pp. 241-301 ◽  
Author(s):  
Michael Schmitt
Keyword(s):  
Neural Networks ◽  
Lower Bounds ◽  
Higher Order ◽  
Upper And Lower Bounds ◽  
Biological Behavior ◽  
Vc Dimension ◽  
Neuron Models ◽  
Learning Capabilities ◽  
Order Of Magnitude ◽  
The Impact

In a great variety of neuron models, neural inputs are combined using the summing operation. We introduce the concept of multiplicative neural networks that contain units that multiply their inputs instead of summing them and thus allow inputs to interact nonlinearly. The class of multiplicative neural networks comprises such widely known and well-studied network types as higher-order networks and product unit networks. We investigate the complexity of computing and learning for multiplicative neural networks. In particular, we derive upper and lower bounds on the Vapnik-Chervonenkis (VC) dimension and the pseudo-dimension for various types of networks with multiplicative units. As the most general case, we consider feedforward networks consisting of product and sigmoidal units, showing that their pseudo-dimension is bounded from above by a polynomial with the same order of magnitude as the currently best-known bound for purely sigmoidal networks. Moreover, we show that this bound holds even when the unit type, product or sigmoidal, may be learned. Crucial for these results are calculations of solution set components bounds for new network classes. As to lower bounds, we construct product unit networks of fixed depth with super-linear VC dimension. For sigmoidal networks of higher order, we establish polynomial bounds that, in contrast to previous results, do not involve any restriction of the network order. We further consider various classes of higher-order units, also known as sigma-pi units, that are characterized by connectivity constraints. In terms of these, we derive some asymptotically tight bounds. Multiplication plays an important role in both neural modeling of biological behavior and computing and learning with artificial neural networks. We briefly survey research in biology and in applications where multiplication is considered an essential computational element. The results we present here provide new tools for assessing the impact of multiplication on the computational power and the learning capabilities of neural networks.

scholarly journals The VC Dimension of Metric Balls under Fréchet and Hausdorff Distances

2021 ◽  
Author(s):  
Anne Driemel ◽  
André Nusser ◽  
Jeff M. Phillips ◽  
Ioannis Psarros
Keyword(s):  
Density Estimation ◽  
Lower Bounds ◽  
Upper And Lower Bounds ◽  
Similarity Metrics ◽  
Vc Dimension ◽  
Set Systems ◽  
Polygonal Curves ◽  
The Individual ◽  
Curve Similarity ◽  
Metric Balls

AbstractThe Vapnik–Chervonenkis dimension provides a notion of complexity for systems of sets. If the VC dimension is small, then knowing this can drastically simplify fundamental computational tasks such as classification, range counting, and density estimation through the use of sampling bounds. We analyze set systems where the ground set X is a set of polygonal curves in $$\mathbb {R}^d$$ R d and the sets $$\mathcal {R}$$ R are metric balls defined by curve similarity metrics, such as the Fréchet distance and the Hausdorff distance, as well as their discrete counterparts. We derive upper and lower bounds on the VC dimension that imply useful sampling bounds in the setting that the number of curves is large, but the complexity of the individual curves is small. Our upper and lower bounds are either near-quadratic or near-linear in the complexity of the curves that define the ranges and they are logarithmic in the complexity of the curves that define the ground set.

Descartes' Rule of Signs for Radial Basis Function Neural Networks

2002 ◽  
Vol 14 (12) ◽  
pp. 2997-3011 ◽  
Author(s):  
Michael Schmitt
Keyword(s):  
Neural Networks ◽  
Radial Basis Function ◽  
Lower Bounds ◽  
Basis Function ◽  
Rbf Neural Networks ◽  
Vc Dimension ◽  
Network Parameters ◽  
Number Of Zeros ◽  
Radial Basis

We establish versions of Descartes' rule of signs for radial basis function (RBF) neural networks. The RBF rules of signs provide tight bounds for the number of zeros of univariate networks with certain parameter restrictions. Moreover, they can be used to infer that the Vapnik-Chervonenkis (VC) dimension and pseudodimension of these networks are no more than linear. This contrasts with previous work showing that RBF neural networks with two or more input nodes have superlinear VC dimension. The rules also give rise to lower bounds for network sizes, thus demonstrating the relevance of network parameters for the complexity of computing with RBF neural networks.

SIMULATIONS OF UNARY ONE-WAY MULTI-HEAD FINITE AUTOMATA

2014 ◽  
Vol 25 (07) ◽  
pp. 877-896 ◽  
Author(s):  
MARTIN KUTRIB ◽  
ANDREAS MALCHER ◽  
MATTHIAS WENDLANDT
Keyword(s):  
Lower Bound ◽  
Lower Bounds ◽  
Finite Automaton ◽  
Finite Automata ◽  
Upper And Lower Bounds ◽  
Unary Languages ◽  
Order Of Magnitude ◽  
Simulation Results ◽  
Special Case ◽  
Nondeterministic Finite Automaton

We investigate the descriptional complexity of deterministic one-way multi-head finite automata accepting unary languages. It is known that in this case the languages accepted are regular. Thus, we study the increase of the number of states when an n-state k-head finite automaton is simulated by a classical (one-head) deterministic or nondeterministic finite automaton. In the former case upper and lower bounds that are tight in the order of magnitude are shown. For the latter case we obtain an upper bound of O(n2k) and a lower bound of Ω(nk) states. We investigate also the costs for the conversion of one-head nondeterministic finite automata to deterministic k-head finite automata, that is, we trade nondeterminism for heads. In addition, we study how the conversion costs vary in the special case of finite and, in particular, of singleton unary lanuages. Finally, as an application of the simulation results, we show that decidability problems for unary deterministic k-head finite automata such as emptiness or equivalence are LOGSPACE-complete.

scholarly journals ON THE ORDER OF MAGNITUDE OF SUMS OF NEGATIVE POWERS OF INTEGRATED PROCESSES

2012 ◽  
Vol 29 (3) ◽  
pp. 642-658 ◽  
Author(s):  
Benedikt M. Pötscher
Keyword(s):  
Lower Bounds ◽  
Random Variables ◽  
Upper Bounds ◽  
Upper And Lower Bounds ◽  
Integrated Process ◽  
Order Of Magnitude ◽  
Image Position ◽  
Integrated Processes

Upper and lower bounds on the order of magnitude of $\sum\nolimits_{t = 1}^n {\lefttnq#x007C; {x_t } \righttnq#x007C;^{ - \alpha } } $, where xt is an integrated process, are obtained. Furthermore, upper bounds for the order of magnitude of the related quantity $\sum\nolimits_{t = 1}^n {v_t } \lefttnq#x007C; {x_t } \righttnq#x007C;^{ - \alpha } $, where vt are random variables satisfying certain conditions, are also derived.

scholarly journals LS-category of moment-angle manifolds and higher order Massey products

2021 ◽  
Vol 33 (5) ◽  
pp. 1179-1205
Author(s):  
Piotr Beben ◽  
Jelena Grbić
Keyword(s):  
Simplicial Complex ◽  
Structural Properties ◽  
Lower Bounds ◽  
Higher Order ◽  
Hyperbolic Manifolds ◽  
Upper And Lower Bounds ◽  
Connected Sum ◽  
Massey Products ◽  
Higher Dimensional ◽  
Lusternik Schnirelmann

Abstract Using the combinatorics of the underlying simplicial complex K, we give various upper and lower bounds for the Lusternik–Schnirelmann (LS) category of moment-angle complexes 𝒵 K {\mathcal{Z}_{K}} . We describe families of simplicial complexes and combinatorial operations which allow for a systematic description of the LS-category. In particular, we characterize the LS-category of moment-angle complexes 𝒵 K {\mathcal{Z}_{K}} over triangulated d-manifolds K for d ≤ 2 {d\leq 2} , as well as higher-dimensional spheres built up via connected sum, join, and vertex doubling operations. We show that the LS-category closely relates to vanishing of Massey products in H * ⁢ ( 𝒵 K ) {H^{*}(\mathcal{Z}_{K})} , and through this connection we describe first structural properties of Massey products in moment-angle manifolds. Some of the further applications include calculations of the LS-category and the description of conditions for vanishing of Massey products for moment-angle manifolds over fullerenes, Pogorelov polytopes and k-neighborly complexes, which double as important examples of hyperbolic manifolds.

scholarly journals On non-uniform and global descriptions of the rate of convergence of Asymptotic expansions in the central limit theorem

1986 ◽  
Vol 41 (3) ◽  
pp. 326-335 ◽  
Author(s):  
Peter Hall ◽  
T. Nakata
Keyword(s):  
Central Limit Theorem ◽  
Limit Theorem ◽  
Rate Of Convergence ◽  
Central Limit ◽  
Lower Bounds ◽  
Rates Of Convergence ◽  
Upper And Lower Bounds ◽  
The Central Limit Theorem ◽  
Order Of Magnitude ◽  
Leading Term

AbstractThe leading term approach to rates of convergence is employed to derive non-uniform and global descriptions of the rate of convergence in the central limit theorem. Both upper and lower bounds are obtained, being of the same order of magnitude, modulo terms of order n-r. We are able to derive general results by considering only those expansions with an odd number of terms.

Correction to “Lower Bounds on VC-Dimension of Smoothly Parameterized Function Classes”

1997 ◽  
Vol 9 (4) ◽  
pp. 765-769 ◽  
Author(s):  
Wee Sun Lee ◽  
Peter L. Bartlett ◽  
Robert C. Williamson
Keyword(s):  
Neural Networks ◽  
Lower Bound ◽  
Lower Bounds ◽  
Alternative Proof ◽  
Activation Functions ◽  
Vc Dimension ◽  
Function Classes ◽  
Decision Boundaries

The earlier article gives lower bounds on the VC-dimension of various smoothly parameterized function classes. The results were proved by showing a relationship between the uniqueness of decision boundaries and the VC-dimension of smoothly parameterized function classes. The proof is incorrect; there is no such relationship under the conditions stated in the article. For the case of neural networks with tanh activation functions, we give an alternative proof of a lower bound for the VC-dimension proportional to the number of parameters, which holds even when the magnitude of the parameters is restricted to be arbitrarily small.

scholarly journals Vapnik-Chervonenkis Generalization Bounds for Real Valued Neural Networks

1996 ◽  
Vol 8 (6) ◽  
pp. 1277-1299 ◽  
Author(s):  
Arne Hole
Keyword(s):  
Neural Networks ◽  
Uniform Convergence ◽  
Lower Bounds ◽  
Neural Nets ◽  
Vc Dimension ◽  
Generalization Ability ◽  
Generalization Bounds

We show how lower bounds on the generalization ability of feedforward neural nets with real outputs can be derived within a formalism based directly on the concept of VC dimension and Vapnik's theorem on uniform convergence of estimated probabilities.

The impact of high-order interactions on the rate of synchronous discharge and information transmission in somatosensory cortex

2009 ◽  
Vol 367 (1901) ◽  
pp. 3297-3310 ◽  
Author(s):  
Fernando Montani ◽  
Robin A. A. Ince ◽  
Riccardo Senatore ◽  
Ehsan Arabzadeh ◽  
Mathew E. Diamond ◽  
...  
Keyword(s):  
Neural Networks ◽  
Somatosensory Cortex ◽  
Local Population ◽  
Interaction Model ◽  
Higher Order ◽  
High Order ◽  
Neural Population ◽  
Somatosensory Information ◽  
Representational Capacity ◽  
The Impact

Understanding the operations of neural networks in the brain requires an understanding of whether interactions among neurons can be described by a pairwise interaction model, or whether a higher order interaction model is needed. In this article we consider the rate of synchronous discharge of a local population of neurons, a macroscopic index of the activation of the neural network that can be measured experimentally. We analyse a model based on physics’ maximum entropy principle that evaluates whether the probability of synchronous discharge can be described by interactions up to any given order. When compared with real neural population activity obtained from the rat somatosensory cortex, the model shows that interactions of at least order three or four are necessary to explain the data. We use Shannon information to compute the impact of high-order correlations on the amount of somatosensory information transmitted by the rate of synchronous discharge, and we find that correlations of higher order progressively decrease the information available through the neural population. These results are compatible with the hypothesis that high-order interactions play a role in shaping the dynamics of neural networks, and that they should be taken into account when computing the representational capacity of neural populations.

scholarly journals Low Pseudomoments of the Riemann Zeta Function and Its Powers

2020 ◽  
Author(s):  
Maxim Gerspach
Keyword(s):  
Zeta Function ◽  
Lower Bounds ◽  
Riemann Zeta Function ◽  
Critical Line ◽  
Probabilistic Methods ◽  
Upper And Lower Bounds ◽  
Partial Sum ◽  
The Riemann Zeta Function ◽  
Order Of Magnitude ◽  
Riemann Zeta

Abstract The $2 q$-th pseudomoment $\Psi _{2q,\alpha }(x)$ of the $\alpha $-th power of the Riemann zeta function is defined to be the $2 q$-th moment of the partial sum up to $x$ of $\zeta ^\alpha $ on the critical line. Using probabilistic methods of Harper, we prove upper and lower bounds for these pseudomoments when $q \le \frac{1}{2}$ and $\alpha \ge 1$. Combined with results of Bondarenko et al., these bounds determine the size of all pseudomoments with $q> 0$ and $\alpha \ge 1$ up to powers of $\log \log x$, where $x$ is the length of the partial sum, and it turns out that there are three different ranges with different growth behaviours. In particular, the results give the order of magnitude of $\Psi _{2 q, 1}(x)$ for all $q> 0$.

Sign in / Sign up

Export Citation Format

Share Document