Algebraic Analysis for Nonidentifiable Learning Machines

2001 ◽  
Vol 13 (4) ◽  
pp. 899-933 ◽  
Author(s):  
Sumio Watanabe
Keyword(s):  
Learning Curve ◽  
Geometrical Structure ◽  
Generalization Error ◽  
True Parameter ◽  
Stochastic Complexity ◽  
Algebraic Analysis ◽  
Learning Machines ◽  
Training Samples ◽  
Learning Machine ◽  
Birational Invariant

This article clarifies the relation between the learning curve and the algebraic geometrical structure of a nonidentifiable learning machine such as a multilayer neural network whose true parameter set is an analytic set with singular points. By using a concept in algebraic analysis, we rigorously prove that the Bayesian stochastic complexity or the free energy is asymptotically equal to λ1 logn − (m1 − 1) loglogn + constant, where n is the number of training samples and λ1 and m1 are the rational number and the natural number, which are determined as the birational invariant values of the singularities in the parameter space. Also we show an algorithm to calculate λ1 and m1 based on the resolution of singularities in algebraic geometry. In regular statistical models, 2λ1 is equal to the number of parameters and m1 = 1, whereas in nonregular models, such as multilayer networks, 2λ1 is not larger than the number of parameters and m1 ≥ 1. Since the increase of the stochastic complexity is equal to the learning curve or the generalization error, the nonidentifiable learning machines are better models than the regular ones if Bayesian ensemble learning is applied.

Learning Coefficients of Layered Models When the True Distribution Mismatches the Singularities

2003 ◽  
Vol 15 (5) ◽  
pp. 1013-1033 ◽  
Author(s):  
Sumio Watanabe ◽  
Shun-ichi Amari
Keyword(s):  
Information Matrix ◽  
Bayes Estimation ◽  
Generalization Error ◽  
Parameter Spaces ◽  
Hierarchical Learning ◽  
Learning Machines ◽  
Regular Model ◽  
Training Samples ◽  
Learning Machine ◽  
True Distribution

Hierarchical learning machines such as layered neural networks have singularities in their parameter spaces. At singularities, the Fisher information matrix becomes degenerate, with the result that the conventional learning theory of regular statistical models does not hold. Recently, it was proved that if the parameter of the true distribution is contained in the singularities of the learning machine, the generalization error in Bayes estimation is asymptotically equal toλ/n, where 2λ is smaller than the dimension of the parameter andn is the number of training samples. However, the constantλ strongly depends on the local geometrical structure of singularities; hence, the generalization error is not yet clarified when the true distribution is almost but not completely contained in the singularities. In this article, in order to analyze such cases, we study the Bayes generalization error under the condition that the Kullback distance of the true distribution from the distribution represented by singularities is in proportion to 1/n and show two results. First, if the dimension of the parameter from inputs to hidden units is not larger than three, then there exists a region of true parameters such that the generalization error is larger than that of the corresponding regular model. Second, if the dimension from inputs to hidden units is larger than three, then for arbitrary true distribution, the generalization error is smaller than that of the corresponding regular model.

scholarly journals Convolutional Extreme Learning Machines: A Systematic Review

2021 ◽  
Author(s):  
Iago Richard Rodrigues ◽  
Sebastião Rogério ◽  
Judith Kelner ◽  
Djamel Sadok ◽  
Patricia Takako Endo
Keyword(s):  
Systematic Review ◽  
Deep Learning ◽  
Extreme Learning Machine ◽  
Research Area ◽  
New Paradigm ◽  
Learning Machines ◽  
Starting Point ◽  
Benchmark Datasets ◽  
Good Starting Point ◽  
Learning Machine

Many works have recently identified the need to combine deep learning with extreme learning to strike a performance balance with accuracy especially in the domain of multimedia applications. Considering this new paradigm, namely convolutional extreme learning machine (CELM), we present a systematic review that investigates alternative deep learning architectures that use extreme learning machine (ELM) for a faster training to solve problems based on image analysis. We detail each of the architectures found in the literature, application scenarios, benchmark datasets, main results, advantages, and present the open challenges for CELM. We follow a well structured methodology and establish relevant research questions that guide our findings. We hope that the observation and classification of such works can leverage the CELM research area providing a good starting point to cope with some of the current problems in the image-based computer vision analysis.

Comparison of Several Extreme Learning Machine Algorithm for Modeling Concrete Compressive Strength

2014 ◽  
Vol 548-549 ◽  
pp. 1735-1738 ◽  
Author(s):  
Jian Tang ◽  
Dong Yan ◽  
Li Jie Zhao
Keyword(s):  
Compressive Strength ◽  
Extreme Learning Machine ◽  
Prediction Accuracy ◽  
Partial Least Square ◽  
Least Square ◽  
Extreme Learning Machines ◽  
Concrete Compressive Strength ◽  
Learning Machines ◽  
Learning Machine

Modeling concrete compressive strength is useful to ensure quality of civil engineering. This paper aims to compare several Extreme learning machines (ELMs) based modeling approaches for predicting the concrete compressive strength. Normal ELM algorithm, Partial least square-based extreme learning machines (PLS-ELMs) algorithm and Kernel ELM (KELM) algorithm are used and evaluated. Results indicate that the normal ELMs algorithm has the highest modeling speed, and the KELM has the best prediction accuracy. Every method is validated for modeling concrete compressive strength. The appropriate modeling approach should be selected according different purposes.

scholarly journals Online Regularized and Kernelized Extreme Learning Machines with Forgetting Mechanism

2014 ◽  
Vol 2014 ◽  
pp. 1-11 ◽  
Author(s):  
Xinran Zhou ◽  
Zijian Liu ◽  
Congxu Zhu
Keyword(s):  
Extreme Learning Machine ◽  
Feedforward Neural Networks ◽  
Matrix Inversion ◽  
Time Varying ◽  
Extreme Learning Machines ◽  
Learning Machines ◽  
Matrix Expansion ◽  
Learning Machine ◽  
Hidden Layer ◽  
Forgetting Mechanism

To apply the single hidden-layer feedforward neural networks (SLFN) to identify time-varying system, online regularized extreme learning machine (ELM) with forgetting mechanism (FORELM) and online kernelized ELM with forgetting mechanism (FOKELM) are presented in this paper. The FORELM updates the output weights of SLFN recursively by using Sherman-Morrison formula, and it combines advantages of online sequential ELM with forgetting mechanism (FOS-ELM) and regularized online sequential ELM (ReOS-ELM); that is, it can capture the latest properties of identified system by studying a certain number of the newest samples and also can avoid issue of ill-conditioned matrix inversion by regularization. The FOKELM tackles the problem of matrix expansion of kernel based incremental ELM (KB-IELM) by deleting the oldest sample according to the block matrix inverse formula when samples occur continually. The experimental results show that the proposed FORELM and FOKELM have better stability than FOS-ELM and have higher accuracy than ReOS-ELM in nonstationary environments; moreover, FORELM and FOKELM have time efficiencies superiority over dynamic regression extreme learning machine (DR-ELM) under certain conditions.

scholarly journals Water demand forecasting using extreme learning machines

2016 ◽  
Vol 28 (1) ◽  
pp. 37-52 ◽  
Author(s):  
Mukesh Tiwari ◽  
Jan Adamowski ◽  
Kazimierz Adamowski
Keyword(s):  
Water Demand ◽  
Demand Forecasting ◽  
Urban Water ◽  
Demand Model ◽  
Demand Prediction ◽  
Learning Machines ◽  
Water Demand Forecasting ◽  
Urban Water Demand ◽  
Learning Machine ◽  
Similar Accuracy

AbstractThe capacity of recently-developed extreme learning machine (ELM) modelling approaches in forecasting daily urban water demand from limited data, alone or in concert with wavelet analysis (W) or bootstrap (B) methods (i.e., ELM, ELMW, ELMB), was assessed, and compared to that of equivalent traditional artificial neural network-based models (i.e., ANN, ANNW, ANNB). The urban water demand forecasting models were developed using 3-year water demand and climate datasets for the city of Calgary, Alberta, Canada. While the hybrid ELMBand ANNBmodels provided satisfactory 1-day lead-time forecasts of similar accuracy, the ANNWand ELMWmodels provided greater accuracy, with the ELMWmodel outperforming the ANNWmodel. Significant improvement in peak urban water demand prediction was only achieved with the ELMWmodel. The superiority of the ELMWmodel over both the ANNWor ANNBmodels demonstrated the significant role of wavelet transformation in improving the overall performance of the urban water demand model.

Extremely fast retrieval of volcanic SO2 layer heights from UV satellite data using inverse learning machines

2020 ◽  
Author(s):  
Pascal Hedelt ◽  
MariLiza Koukouli ◽  
Isabelle Taylor ◽  
Dimitris Balis ◽  
Don Grainger ◽  
...  
Keyword(s):  
Spatial Resolution ◽  
Volcanic Eruption ◽  
Accurate Determination ◽  
Volcanic Eruptions ◽  
Learning Machines ◽  
Precise Knowledge ◽  
First Results ◽  
Gas Measurements ◽  
Learning Machine ◽  
Atmospheric Trace Gas

<p>Precise knowledge of the location and height of the volcanic sulfur dioxide (SO<sub>2</sub>) plume is essential for accurate determination of SO<sub>2</sub> emitted by volcanic eruptions. So far, UV based SO<sub>2</sub> plume height retrieval algorithms are very time-consuming and therefore not suitable for near-real-time applications like aviation control. We have therefore developed the Full-Physics Inverse Learning Machine (FP_ILM) algorithm for extremely fast and accurate retrieval of volcanic SO<sub>2</sub> layer heights based on the UV satellite instruments Sentinel-5 Precursor/TROPOMI and MetOp/GOME-2.</p><p>In this presentation, we will present the FP-ILM algorithm and show results of the 2019 Raikoke eruption; a strong volcanic eruption which has emitted a huge ash cloud accompanied by more than 1300 DU of SO<sub>2</sub>, which could be detected  even two months after the end of eruptive event. We will also present first results of the recent Taal volcanic eruption on 13 January 2020 in Indonesia, which has injected a huge ash and SO<sub>2</sub> plume into the upper atmosphere, with plume heights of up to 20km. </p><p>The algorithm is developed in the framework of ESA's  "Sentinel-5p+ Innovation: SO<sub>2</sub> Layer Height project" (S5P+I: SO2 LH),  dedicated to the generation of an SO<sub>2</sub> LH product and its extensive verification with collocated ground- and space-born measurements.</p><p>The high-resolution UV spectrometer GOME-2 aboard the three EPS MetOp-A, -B, and –C satellites perform global daily atmospheric trace-gas measurements with a spatial resolution of  40x40km<sup>2</sup> at an overpass time of 8:30h local time. The UV spectrometer TROPOMI aboard the ESA Sentinel-5P satellite provides a much higher spatial resolution of currently 5.6x3.6km<sup>2</sup> per ground pixel, at an overpass time of 13:30h. In the future, also UV instruments aboard the Sentinel-4 (geostationary) and Sentinel-5 will complement the satellite-based global monitoring of atmospheric trace gases.</p>

Data representations and generalization error in kernel based learning machines

2006 ◽  
Vol 39 (9) ◽  
pp. 1588-1603 ◽  
Author(s):  
Nicola Ancona ◽  
Rosalia Maglietta ◽  
Ettore Stella
Keyword(s):  
Generalization Error ◽  
Learning Machines ◽  
Data Representations

Four Types of Learning Curves

1992 ◽  
Vol 4 (4) ◽  
pp. 605-618 ◽  
Author(s):  
Shun-ichi Amari ◽  
Naotake Fujita ◽  
Shigeru Shinomoto
Keyword(s):  
Learning Curve ◽  
Bayesian Approach ◽  
Finite Measure ◽  
Learning Curves ◽  
Machine Parameter ◽  
Generalization Error ◽  
Incorrect Decision ◽  
Average Probability ◽  
The Bayesian Approach

If machines are learning to make decisions given a number of examples, the generalization error ε(t) is defined as the average probability that an incorrect decision is made for a new example by a machine when trained with t examples. The generalization error decreases as t increases, and the curve ε(t) is called a learning curve. The present paper uses the Bayesian approach to show that given the annealed approximation, learning curves can be classified into four asymptotic types. If the machine is deterministic with noiseless teacher signals, then (1) ε ∼ at-1 when the correct machine parameter is unique, and (2) ε ∼ at-2 when the set of the correct parameters has a finite measure. If the teacher signals are noisy, then (3) ε ∼ at-1/2 for a deterministic machine, and (4) ε ∼ c + at-1 for a stochastic machine.

scholarly journals An Application and Performance Evaluation of Twin Extreme Learning Machine Classifier for Intrusion Detection

2019 ◽  
Vol 8 (9) ◽  
pp. 219-222
Keyword(s):  
Intrusion Detection ◽  
Extreme Learning Machine ◽  
Intrusion Detection System ◽  
Detection System ◽  
Twin Support Vector Machine ◽  
Support Vector ◽  
Detection Techniques ◽  
Learning Machines ◽  
And Performance ◽  
Learning Machine

Network along with Security is most significant in the digitalized environment. It is necessary to secure data from hackers and intruders. A strategy involved in protection of information from hackers will be termed as Intrusion Detection System (IDS).By taking into nature of attack or the usual conduct of user, investigation along with forecasting activities of the clients will be performed by mentioned system.Variousstrategies are utilized for the intrusion detection system. For the purpose of identification of hacking activity, utilization of machine learning based approach might be considered as novel strategy.In this paper, for identification of the hacking activity will be carried out by Twin Extreme Learning Machines (TELM).Employing the concept of Twin Support Vector Machine with the fundamental structure of Extreme Learning Machine is considered in the establishment of Twin Extreme Learning Machine (TELM).Also, its performance and accuracy are compared with the other intrusion detection techniques

Sign in / Sign up

Export Citation Format

Share Document