scholarly journals Vapnik-Chervonenkis Generalization Bounds for Real Valued Neural Networks

1996 ◽  
Vol 8 (6) ◽  
pp. 1277-1299 ◽  
Author(s):  
Arne Hole
Keyword(s):  
Neural Networks ◽  
Uniform Convergence ◽  
Lower Bounds ◽  
Neural Nets ◽  
Vc Dimension ◽  
Generalization Ability ◽  
Generalization Bounds

We show how lower bounds on the generalization ability of feedforward neural nets with real outputs can be derived within a formalism based directly on the concept of VC dimension and Vapnik's theorem on uniform convergence of estimated probabilities.

Descartes' Rule of Signs for Radial Basis Function Neural Networks

2002 ◽  
Vol 14 (12) ◽  
pp. 2997-3011 ◽  
Author(s):  
Michael Schmitt
Keyword(s):  
Neural Networks ◽  
Radial Basis Function ◽  
Lower Bounds ◽  
Basis Function ◽  
Rbf Neural Networks ◽  
Vc Dimension ◽  
Network Parameters ◽  
Number Of Zeros ◽  
Radial Basis

We establish versions of Descartes' rule of signs for radial basis function (RBF) neural networks. The RBF rules of signs provide tight bounds for the number of zeros of univariate networks with certain parameter restrictions. Moreover, they can be used to infer that the Vapnik-Chervonenkis (VC) dimension and pseudodimension of these networks are no more than linear. This contrasts with previous work showing that RBF neural networks with two or more input nodes have superlinear VC dimension. The rules also give rise to lower bounds for network sizes, thus demonstrating the relevance of network parameters for the complexity of computing with RBF neural networks.

Orthogonality of Decision Boundaries in Complex-Valued Neural Networks

2004 ◽  
Vol 16 (1) ◽  
pp. 73-97 ◽  
Author(s):  
Tohru Nitta
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Neural Nets ◽  
Decision Boundary ◽  
Generalization Ability ◽  
The Real ◽  
Learning Speed ◽  
Decision Boundaries ◽  
Complex Valued ◽  
Layered Complex

This letter presents some results of an analysis on the decision boundaries of complex-valued neural networks whose weights, threshold values, input and output signals are all complex numbers. The main results may be summarized as follows. (1) A decision boundary of a single complex-valued neuron consists of two hypersurfaces that intersect orthogonally, and divides a decision region into four equal sections. The XOR problem and the detection of symmetry problem that cannot be solved with two-layered real-valued neural networks, can be solved by two-layered complex-valued neural networks with the orthogonal decision boundaries, which reveals a potent computational power of complex-valued neural nets. Furthermore, the fading equalization problem can be successfully solved by the two-layered complex-valued neural network with the highest generalization ability. (2) A decision boundary of a three-layered complex-valued neural network has the orthogonal property as a basic structure, and its two hypersurfaces approach orthogonality as all the net inputs to each hidden neuron grow. In particular, most of the decision boundaries in the three-layered complex-valued neural network inetersect orthogonally when the network is trained using Complex-BP algorithm. As a result, the orthogonality of the decision boundaries improves its generalization ability. (3) The average of the learning speed of the Complex-BP is several times faster than that of the Real-BP. The standard deviation of the learning speed of the Complex-BP is smaller than that of the Real-BP. It seems that the complex-valued neural network and the related algorithm are natural for learning complex-valued patterns for the above reasons.

Correction to “Lower Bounds on VC-Dimension of Smoothly Parameterized Function Classes”

1997 ◽  
Vol 9 (4) ◽  
pp. 765-769 ◽  
Author(s):  
Wee Sun Lee ◽  
Peter L. Bartlett ◽  
Robert C. Williamson
Keyword(s):  
Neural Networks ◽  
Lower Bound ◽  
Lower Bounds ◽  
Alternative Proof ◽  
Activation Functions ◽  
Vc Dimension ◽  
Function Classes ◽  
Decision Boundaries

The earlier article gives lower bounds on the VC-dimension of various smoothly parameterized function classes. The results were proved by showing a relationship between the uniqueness of decision boundaries and the VC-dimension of smoothly parameterized function classes. The proof is incorrect; there is no such relationship under the conditions stated in the article. For the case of neural networks with tanh activation functions, we give an alternative proof of a lower bound for the VC-dimension proportional to the number of parameters, which holds even when the magnitude of the parameters is restricted to be arbitrarily small.

Neural Networks with Local Receptive Fields and Superlinear VC Dimension

2002 ◽  
Vol 14 (4) ◽  
pp. 919-956 ◽  
Author(s):  
Michael Schmitt
Keyword(s):  
Neural Networks ◽  
Receptive Field ◽  
Lower Bounds ◽  
Receptive Fields ◽  
Feedforward Neural Networks ◽  
Rbf Neural Networks ◽  
Vc Dimension ◽  
Difference Of Gaussians ◽  
Hidden Layer ◽  
Fixed Input

Local receptive field neurons comprise such well-known and widely used unit types as radial basis function (RBF) neurons and neurons with center-surround receptive field. We study the Vapnik-Chervonenkis (VC) dimension of feedforward neural networks with one hidden layer of these units. For several variants of local receptive field neurons, we show that the VC dimension of these networks is superlinear. In particular, we establish the bound Ω (w log k) for any reasonably sized network with W parameters and k hidden nodes. This bound is shown to hold for discrete center-surround receptive field neurons, which are physiologically relevant models of cells in the mammalian visual system, for neurons computing a difference of gaussians, which are popular in computational vision, and for standard RBF neurons, a major alternative to sigmoidal neurons in artificial neural networks. The result for RBF neural networks is of particular interest since it answers a question that has been open for several years. The results also give rise to lower bounds for networks with fixed input dimension. Regarding constants, all bounds are larger than those known thus far for similar architectures with sigmoidal neurons. The superlinear lower bounds contrast with linear upper bounds for single local receptive field neurons also derived here.

On the Complexity of Computing and Learning with Multiplicative Neural Networks

2002 ◽  
Vol 14 (2) ◽  
pp. 241-301 ◽  
Author(s):  
Michael Schmitt
Keyword(s):  
Neural Networks ◽  
Lower Bounds ◽  
Higher Order ◽  
Upper And Lower Bounds ◽  
Biological Behavior ◽  
Vc Dimension ◽  
Neuron Models ◽  
Learning Capabilities ◽  
Order Of Magnitude ◽  
The Impact

In a great variety of neuron models, neural inputs are combined using the summing operation. We introduce the concept of multiplicative neural networks that contain units that multiply their inputs instead of summing them and thus allow inputs to interact nonlinearly. The class of multiplicative neural networks comprises such widely known and well-studied network types as higher-order networks and product unit networks. We investigate the complexity of computing and learning for multiplicative neural networks. In particular, we derive upper and lower bounds on the Vapnik-Chervonenkis (VC) dimension and the pseudo-dimension for various types of networks with multiplicative units. As the most general case, we consider feedforward networks consisting of product and sigmoidal units, showing that their pseudo-dimension is bounded from above by a polynomial with the same order of magnitude as the currently best-known bound for purely sigmoidal networks. Moreover, we show that this bound holds even when the unit type, product or sigmoidal, may be learned. Crucial for these results are calculations of solution set components bounds for new network classes. As to lower bounds, we construct product unit networks of fixed depth with super-linear VC dimension. For sigmoidal networks of higher order, we establish polynomial bounds that, in contrast to previous results, do not involve any restriction of the network order. We further consider various classes of higher-order units, also known as sigma-pi units, that are characterized by connectivity constraints. In terms of these, we derive some asymptotically tight bounds. Multiplication plays an important role in both neural modeling of biological behavior and computing and learning with artificial neural networks. We briefly survey research in biology and in applications where multiplication is considered an essential computational element. The results we present here provide new tools for assessing the impact of multiplication on the computational power and the learning capabilities of neural networks.

Linear Constraints on Weight Representation for Generalized Learning of Multilayer Networks

2001 ◽  
Vol 13 (12) ◽  
pp. 2851-2863 ◽  
Author(s):  
Masaki Ishii ◽  
Itsuo Kumazawa
Keyword(s):  
Neural Networks ◽  
Error Function ◽  
Linear Constraints ◽  
Learning Method ◽  
Vc Dimension ◽  
Multilayer Networks ◽  
Penalty Term ◽  
Generalization Ability ◽  
The Neural Networks ◽  
Multilayer Neural Networks

In this article, we present a technique to improve the generalization ability of multilayer neural networks. The proposed method introduces linear constraints on weight representation based on the invariance natures of training targets. We propose a learning method that introduces effective linear constraints into an error function as a penalty term. Furthermore, introduction of such constraints leads to reduction of the VC dimension of neural networks. We show bounds on the VC dimension of the neural networks with such constraints. Finally, we demonstrate the effectiveness of the proposed method by some experiments.

scholarly journals The VC Dimension of Metric Balls under Fréchet and Hausdorff Distances

2021 ◽  
Author(s):  
Anne Driemel ◽  
André Nusser ◽  
Jeff M. Phillips ◽  
Ioannis Psarros
Keyword(s):  
Density Estimation ◽  
Lower Bounds ◽  
Upper And Lower Bounds ◽  
Similarity Metrics ◽  
Vc Dimension ◽  
Set Systems ◽  
Polygonal Curves ◽  
The Individual ◽  
Curve Similarity ◽  
Metric Balls

AbstractThe Vapnik–Chervonenkis dimension provides a notion of complexity for systems of sets. If the VC dimension is small, then knowing this can drastically simplify fundamental computational tasks such as classification, range counting, and density estimation through the use of sampling bounds. We analyze set systems where the ground set X is a set of polygonal curves in $$\mathbb {R}^d$$ R d and the sets $$\mathcal {R}$$ R are metric balls defined by curve similarity metrics, such as the Fréchet distance and the Hausdorff distance, as well as their discrete counterparts. We derive upper and lower bounds on the VC dimension that imply useful sampling bounds in the setting that the number of curves is large, but the complexity of the individual curves is small. Our upper and lower bounds are either near-quadratic or near-linear in the complexity of the curves that define the ranges and they are logarithmic in the complexity of the curves that define the ground set.

scholarly journals Road Extraction from Unmanned Aerial Vehicle Remote Sensing Images Based on Improved Neural Networks

Sensors ◽  
2019 ◽  
Vol 19 (19) ◽  
pp. 4115 ◽  
Author(s):  
Yuxia Li ◽  
Bo Peng ◽  
Lei He ◽  
Kunlong Fan ◽  
Zhenxu Li ◽  
...  
Keyword(s):  
Neural Network ◽  
Remote Sensing ◽  
Neural Networks ◽  
Unmanned Aerial Vehicle ◽  
Computational Efficiency ◽  
Neural Nets ◽  
Road Extraction ◽  
Remote Sensing Images ◽  
Feature Maps ◽  
Aerial Vehicle

Roads are vital components of infrastructure, the extraction of which has become a topic of significant interest in the field of remote sensing. Because deep learning has been a popular method in image processing and information extraction, researchers have paid more attention to extracting road using neural networks. This article proposes the improvement of neural networks to extract roads from Unmanned Aerial Vehicle (UAV) remote sensing images. D-Linknet was first considered for its high performance; however, the huge scale of the net reduced computational efficiency. With a focus on the low computational efficiency problem of the popular D-LinkNet, this article made some improvements: (1) Replace the initial block with a stem block. (2) Rebuild the entire network based on ResNet units with a new structure, allowing for the construction of an improved neural network D-Linknetplus. (3) Add a 1 × 1 convolution layer before DBlock to reduce the input feature maps, reducing parameters and improving computational efficiency. Add another 1 × 1 convolution layer after DBlock to recover the required number of output channels. Accordingly, another improved neural network B-D-LinknetPlus was built. Comparisons were performed between the neural nets, and the verification were made with the Massachusetts Roads Dataset. The results show improved neural networks are helpful in reducing the network size and developing the precision needed for road extraction.

Sign in / Sign up

Export Citation Format

Share Document