STRUCTURAL CONNECTIONIST LEARNING WITH COMPLEMENTARY CODING

1992 ◽  
Vol 03 (01) ◽  
pp. 19-30 ◽  
Author(s):  
AKIRA NAMATAME ◽  
YOSHIAKI TSUKAMOTO
Keyword(s):  
Learning Algorithm ◽  
Internal Representation ◽  
Threshold Function ◽  
Sufficient Condition ◽  
Structural Learning ◽  
Similarity Matrix ◽  
Training Set ◽  
Threshold Functions ◽  
Connectionist Networks ◽  
Hidden Layer

We propose a new learning algorithm, structural learning with the complementary coding for concept learning problems. We introduce the new grouping measure that forms the similarity matrix over the training set and show this similarity matrix provides a sufficient condition for the linear separability of the set. Using the sufficient condition one should figure out a suitable composition of linearly separable threshold functions that classify exactly the set of labeled vectors. In the case of the nonlinear separability, the internal representation of connectionist networks, the number of the hidden units and value-space of these units, is pre-determined before learning based on the structure of the similarity matrix. A three-layer neural network is then constructed where each linearly separable threshold function is computed by a linear-threshold unit whose weights are determined by the one-shot learning algorithm that requires a single presentation of the training set. The structural learning algorithm proceeds to capture the connection weights so as to realize the pre-determined internal representation. The pre-structured internal representation, the activation value spaces at the hidden layer, defines intermediate-concepts. The target-concept is then learned as a combination of those intermediate-concepts. The ability to create the pre-structured internal representation based on the grouping measure distinguishes the structural learning from earlier methods such as backpropagation.

ON THE CLASSIFICATION CAPABILITY OF A DYNAMIC THRESHOLD NEURAL NETWORK

1994 ◽  
Vol 05 (02) ◽  
pp. 103-114
Author(s):  
CHENG-CHIN CHIANG ◽  
HSIN-CHIA FU
Keyword(s):  
Neural Network ◽  
Upper Bound ◽  
Learning Algorithm ◽  
Dynamic Threshold ◽  
Training Set ◽  
New Type ◽  
Hidden Layer ◽  
Hidden Neurons ◽  
The Given ◽  
Classification Capability

This paper proposes a new type of neural network called the Dynamic Threshold Neural Network (DTNN) which is theoretically and experimentally superior to a conventional sigmoidal multilayer neural network in classification capability, Given a training set containing 4k+1 patterns in ℜn, to successfully learn this training set, the upper bound on the number of free parameters for a DTNN is (k+1)(n+2)+2(k +1), while the upper bound for a sigmoidal network is 2k(n+1)+(2k+1). We also derive a learning algorithm for the DTNN in a similar way to the derivation of the backprop learning algorithm. In simulations on learning the Two-Spirals problem, our DTNN with 30 neurons in one hidden layer takes only 3200 epochs on average to successfully learn the whole training set, while the single-hidden-layer feedforward sigmoidal neural networks have never been reported to successfully learn the given training set even though more hidden neurons are used.

scholarly journals Monotone Boolean Functions with s Zeros Farthest from Threshold Functions

2005 ◽  
Vol DMTCS Proceedings vol. AE,... (Proceedings) ◽  
Author(s):  
Kazuyuki Amano ◽  
Jun Tarui
Keyword(s):  
Boolean Function ◽  
Boolean Functions ◽  
Learning Algorithm ◽  
Monotone Function ◽  
Threshold Function ◽  
Binary Representation ◽  
Threshold Functions ◽  
Monotone Boolean Function ◽  
Monotone Boolean Functions ◽  
International Audience

International audience Let $T_t$ denote the $t$-threshold function on the $n$-cube: $T_t(x) = 1$ if $|\{i : x_i=1\}| \geq t$, and $0$ otherwise. Define the distance between Boolean functions $g$ and $h$, $d(g,h)$, to be the number of points on which $g$ and $h$ disagree. We consider the following extremal problem: Over a monotone Boolean function $g$ on the $n$-cube with $s$ zeros, what is the maximum of $d(g,T_t)$? We show that the following monotone function $p_s$ maximizes the distance: For $x \in \{0,1\}^n$, $p_s(x)=0$ if and only if $N(x) < s$, where $N(x)$ is the integer whose $n$-bit binary representation is $x$. Our result generalizes the previous work for the case $t=\lceil n/2 \rceil$ and $s=2^{n-1}$ by Blum, Burch, and Langford [BBL98-FOCS98], who considered the problem to analyze the behavior of a learning algorithm for monotone Boolean functions, and the previous work for the same $t$ and $s$ by Amano and Maruoka [AM02-ALT02].

What Size Net Gives Valid Generalization?

1989 ◽  
Vol 1 (1) ◽  
pp. 151-160 ◽  
Author(s):  
Eric B. Baum ◽  
David Haussler
Keyword(s):  
Learning Algorithm ◽  
Network Size ◽  
Threshold Functions ◽  
Linear Threshold Functions ◽  
Linear Threshold ◽  
Training Examples ◽  
Hidden Layer ◽  
Fully Connected ◽  
Future Test ◽  
Valid Generalization

We address the question of when a network can be expected to generalize from m random training examples chosen from some arbitrary probability distribution, assuming that future test examples are drawn from the same distribution. Among our results are the following bounds on appropriate sample vs. network size. Assume 0 < ∊ ≤ 1/8. We show that if m ≥ O(W/∊ log N/∊) random examples can be loaded on a feedforward network of linear threshold functions with N nodes and W weights, so that at least a fraction 1 − ∊/2 of the examples are correctly classified, then one has confidence approaching certainty that the network will correctly classify a fraction 1 − ∊ of future test examples drawn from the same distribution. Conversely, for fully-connected feedforward nets with one hidden layer, any learning algorithm using fewer than Ω(W/∊) random training examples will, for some distributions of examples consistent with an appropriate weight choice, fail at least some fixed fraction of the time to find a weight choice that will correctly classify more than a 1 − ∊ fraction of the future test examples.

scholarly journals Unsupervised learning by competing hidden units

2019 ◽  
Vol 116 (16) ◽  
pp. 7723-7731 ◽  
Author(s):  
Dmitry Krotov ◽  
John J. Hopfield
Keyword(s):  
Learning Algorithm ◽  
Lower Layer ◽  
Learning Rule ◽  
Backpropagation Algorithm ◽  
Feedforward Networks ◽  
Feature Detectors ◽  
End To End ◽  
Hidden Layer ◽  
Full Network ◽  
Global Inhibition

It is widely believed that end-to-end training with the backpropagation algorithm is essential for learning good feature detectors in early layers of artificial neural networks, so that these detectors are useful for the task performed by the higher layers of that neural network. At the same time, the traditional form of backpropagation is biologically implausible. In the present paper we propose an unusual learning rule, which has a degree of biological plausibility and which is motivated by Hebb’s idea that change of the synapse strength should be local—i.e., should depend only on the activities of the pre- and postsynaptic neurons. We design a learning algorithm that utilizes global inhibition in the hidden layer and is capable of learning early feature detectors in a completely unsupervised way. These learned lower-layer feature detectors can be used to train higher-layer weights in a usual supervised way so that the performance of the full network is comparable to the performance of standard feedforward networks trained end-to-end with a backpropagation algorithm on simple tasks.

Study on Deep Structure of Extreme Learning Machine (DS-ELM) for Datasets with Noise

2014 ◽  
Vol 989-994 ◽  
pp. 3679-3682 ◽  
Author(s):  
Meng Meng Ma ◽  
Bo He
Keyword(s):  
Extreme Learning Machine ◽  
Simple Structure ◽  
Learning Algorithm ◽  
Deep Structure ◽  
Noisy Data ◽  
Machine Learning Algorithm ◽  
Feed Forward Neural Networks ◽  
Learning Machine ◽  
Hidden Layer ◽  
Auto Associative Neural Network

Extreme learning machine (ELM), a relatively novel machine learning algorithm for single hidden layer feed-forward neural networks (SLFNs), has been shown competitive performance in simple structure and superior training speed. To improve the effectiveness of ELM for dealing with noisy datasets, a deep structure of ELM, short for DS-ELM, is proposed in this paper. DS-ELM contains three level networks (actually contains three nets ): the first level network is trained by auto-associative neural network (AANN) aim to filter out noise as well as reduce dimension when necessary; the second level network is another AANN net aim to fix the input weights and bias of ELM; and the last level network is ELM. Experiments on four noisy datasets are carried out to examine the new proposed DS-ELM algorithm. And the results show that DS-ELM has higher performance than ELM when dealing with noisy data.

Simulation of Compressor Transient Behavior Through Recurrent Neural Network Models

2005 ◽  
Vol 128 (3) ◽  
pp. 444-454 ◽  
Author(s):  
M. Venturini
Keyword(s):  
Measurement Errors ◽  
Learning Algorithm ◽  
Back Propagation ◽  
Network Models ◽  
Transient Behavior ◽  
Optimal Number ◽  
Measured Data ◽  
Outlet Temperature ◽  
Neural Network Models ◽  
Hidden Layer

In the paper, self-adapting models capable of reproducing time-dependent data with high computational speed are investigated. The considered models are recurrent feed-forward neural networks (RNNs) with one feedback loop in a recursive computational structure, trained by using a back-propagation learning algorithm. The data used for both training and testing the RNNs have been generated by means of a nonlinear physics-based model for compressor dynamic simulation, which was calibrated on a multistage axial-centrifugal small size compressor. The first step of the analysis is the selection of the compressor maneuver to be used for optimizing RNN training. The subsequent step consists in evaluating the most appropriate RNN structure (optimal number of neurons in the hidden layer and number of outputs) and RNN proper delay time. Then, the robustness of the model response towards measurement uncertainty is ascertained, by comparing the performance of RNNs trained on data uncorrupted or corrupted with measurement errors with respect to the simulation of data corrupted with measurement errors. Finally, the best RNN model is tested on field data taken on the axial-centrifugal compressor on which the physics-based model was calibrated, by comparing physics-based model and RNN predictions against measured data. The comparison between RNN predictions and measured data shows that the agreement can be considered acceptable for inlet pressure, outlet pressure and outlet temperature, while errors are significant for inlet mass flow rate.

Convergence of a modified gradient-based learning algorithm with penalty for single-hidden-layer feed-forward networks

2018 ◽  
Vol 32 (7) ◽  
pp. 2445-2456 ◽  
Author(s):  
Jian Wang ◽  
Bingjie Zhang ◽  
Zhaoyang Sang ◽  
Yusong Liu ◽  
Shujun Wu ◽  
...  
Keyword(s):  
Learning Algorithm ◽  
Feed Forward ◽  
Gradient Based ◽  
Hidden Layer

Machine learning to identify lung cancer with tuberculosis from isolated tuberculosis (Preprint)

2021 ◽  
Author(s):  
Zhenhao Li
Keyword(s):  
Machine Learning ◽  
Lung Cancer ◽  
Cancer Patients ◽  
Tumor Markers ◽  
Learning Algorithm ◽  
Laboratory Data ◽  
Pathological Examination ◽  
Training Set ◽  
Lung Cancer Patients ◽  
Tb Pcr

UNSTRUCTURED Tuberculosis (TB) is a precipitating cause of lung cancer. Lung cancer patients coexisting with TB is difficult to differentiate from isolated TB patients. The aim of this study is to develop a prediction model in identifying those two diseases between the comorbidities and TB. In this work, based on the laboratory data from 389 patients, 81 features, including main laboratory examination of blood test, biochemical test, coagulation assay, tumor markers and baseline information, were initially used as integrated markers and then reduced to form a discrimination system consisting of 31 top-ranked indices. Patients diagnosed with TB PCR ＞1mtb/ml as negative samples, lung cancer patients with TB were confirmed by pathological examination and TB PCR ＞1mtb/ml as positive samples. We used Spatially Uniform ReliefF (SURF) algorithm to determine feature importance, and the predictive model was built using machine learning algorithm Random Forest. For cross-validation, the samples were randomly split into four training set and one test set. The selected features are composed of four tumor markers (Scc, Cyfra21-1, CEA, ProGRP and NSE), fifteen blood biochemical indices (GLU, IBIL, K, CL, Ur, NA, TBA, CHOL, SA, TG, A/G, AST, CA, CREA and CRP), six routine blood indices (EO#, EO%, MCV, RDW-S, LY# and MPV) and four coagulation indices (APTT ratio, APTT, PTA, TT ratio). This model presented a robust and stable classification performance, which can easily differentiate the comorbidity group from the isolated TB group with AUC, ACC, sensitivity and specificity of 0.8817, 0.8654, 0.8594 and 0.8656 for the training set, respectively. Overall, this work may provide a novel strategy for identifying the TB patients with lung cancer from routine admission lab examination with advantages of being timely and economical. It also indicated that our model with enough indices may further increase the effectiveness and efficiency of diagnosis.

scholarly journals Self-Paced Robust Learning for Leveraging Clean Labels in Noisy Data

2020 ◽  
Vol 34 (04) ◽  
pp. 6853-6860
Author(s):  
Xuchao Zhang ◽  
Xian Wu ◽  
Fanglan Chen ◽  
Liang Zhao ◽  
Chang-Tien Lu
Keyword(s):  
Real World ◽  
Large Scale ◽  
Learning Algorithm ◽  
Noisy Data ◽  
Training Set ◽  
Robust Learning ◽  
Robust Model ◽  
Small Set ◽  
Real World Datasets ◽  
Theoretical Analyses

The success of training accurate models strongly depends on the availability of a sufficient collection of precisely labeled data. However, real-world datasets contain erroneously labeled data samples that substantially hinder the performance of machine learning models. Meanwhile, well-labeled data is usually expensive to obtain and only a limited amount is available for training. In this paper, we consider the problem of training a robust model by using large-scale noisy data in conjunction with a small set of clean data. To leverage the information contained via the clean labels, we propose a novel self-paced robust learning algorithm (SPRL) that trains the model in a process from more reliable (clean) data instances to less reliable (noisy) ones under the supervision of well-labeled data. The self-paced learning process hedges the risk of selecting corrupted data into the training set. Moreover, theoretical analyses on the convergence of the proposed algorithm are provided under mild assumptions. Extensive experiments on synthetic and real-world datasets demonstrate that our proposed approach can achieve a considerable improvement in effectiveness and robustness to existing methods.

scholarly journals IMPLEMENTASI BACKPROPAGATION NEURAL NETWORK DALAM PRAKIRAAN CUACA DI DAERAH BALI SELATAN

2016 ◽  
Vol 5 (4) ◽  
pp. 126 ◽  
Author(s):  
I MADE DWI UDAYANA PUTRA ◽  
G. K. GANDHIADI ◽  
LUH PUTU IDA HARINI
Keyword(s):  
Neural Network ◽  
Artificial Neural Network ◽  
Learning Algorithm ◽  
Weather Forecast ◽  
Weather Forecasts ◽  
Forecast Models ◽  
Artificial Neural Network Method ◽  
Artificial Neural ◽  
Hidden Layer ◽  
Backpropagation Learning

Weather information has an important role in human life in various fields, such as agriculture, marine, and aviation. The accurate weather forecasts are needed in order to improve the performance of various fields. In this study, use artificial neural network method with backpropagation learning algorithm to create a model of weather forecasting in the area of ??South Bali. The aim of this study is to determine the effect of the number of neurons in the hidden layer and to determine the level of accuracy of the method of artificial neural network with backpropagation learning algorithm in weather forecast models. Weather forecast models in this study use input of the factors that influence the weather, namely air temperature, dew point, wind speed, visibility, and barometric pressure.The results of testing the network with a different number of neurons in the hidden layer of artificial neural network method with backpropagation learning algorithms show that the increase in the number of neurons in the hidden layer is not directly proportional to the value of the accuracy of the weather forecasts, the increase in the number of neurons in the hidden layer does not necessarily increase or decrease value accuracy of weather forecasts we obtain the best accuracy rate of 51.6129% on a network model with three neurons in the hidden layer.

Sign in / Sign up

Export Citation Format

Share Document