scholarly journals A Beginner's Tutorial of Restricted Boltzmann Machines

2020 ◽  
Author(s):  
Yiping Cheng
Keyword(s):  
Deep Learning ◽  
Building Blocks ◽  
Learning Networks ◽  
Restricted Boltzmann Machines ◽  
Boltzmann Machines

Restricted Boltzmann machines (RBMs) are the building blocks of some deep learning networks. However, despite their importance, it is our perception that some very important derivations about the RBM are missing in the literature, and a beginner may feel RBM very hard to understand. We provide here these missing derivations. We cover the classic Bernoulli-Bernoulli RBM and the Gaussian-Bernoulli RBM, but leave out the ``continuous'' RBM as it is believed not as mature as the former two. This tutorial can be used as a companion or complement to the famous RBM paper ``Training restricted Boltzmann machines: An introduction'' by Fisher and Igel.

scholarly journals Deep Restricted Kernel Machines Using Conjugate Feature Duality

2017 ◽  
Vol 29 (8) ◽  
pp. 2123-2163 ◽  
Author(s):  
Johan A. K. Suykens
Keyword(s):  
Deep Learning ◽  
Least Squares ◽  
Principal Component ◽  
Feedforward Neural Networks ◽  
Kernel Principal Component Analysis ◽  
Support Vector ◽  
Kernel Machines ◽  
Continuous Variables ◽  
Restricted Boltzmann Machines ◽  
Boltzmann Machines

The aim of this letter is to propose a theory of deep restricted kernel machines offering new foundations for deep learning with kernel machines. From the viewpoint of deep learning, it is partially related to restricted Boltzmann machines, which are characterized by visible and hidden units in a bipartite graph without hidden-to-hidden connections and deep learning extensions as deep belief networks and deep Boltzmann machines. From the viewpoint of kernel machines, it includes least squares support vector machines for classification and regression, kernel principal component analysis (PCA), matrix singular value decomposition, and Parzen-type models. A key element is to first characterize these kernel machines in terms of so-called conjugate feature duality, yielding a representation with visible and hidden units. It is shown how this is related to the energy form in restricted Boltzmann machines, with continuous variables in a nonprobabilistic setting. In this new framework of so-called restricted kernel machine (RKM) representations, the dual variables correspond to hidden features. Deep RKM are obtained by coupling the RKMs. The method is illustrated for deep RKM, consisting of three levels with a least squares support vector machine regression level and two kernel PCA levels. In its primal form also deep feedforward neural networks can be trained within this framework.

scholarly journals Protein Function Prediction Using Deep Restricted Boltzmann Machines

2017 ◽  
Vol 2017 ◽  
pp. 1-9 ◽  
Author(s):  
Xianchun Zou ◽  
Guijun Wang ◽  
Guoxian Yu
Keyword(s):  
Deep Learning ◽  
Language Processing ◽  
Protein Function ◽  
Homo Sapiens ◽  
Protein Function Prediction ◽  
Restricted Boltzmann Machines ◽  
Boltzmann Machines ◽  
Functional Annotations ◽  
Learning Techniques ◽  
Wide Range

Accurately annotating biological functions of proteins is one of the key tasks in the postgenome era. Many machine learning based methods have been applied to predict functional annotations of proteins, but this task is rarely solved by deep learning techniques. Deep learning techniques recently have been successfully applied to a wide range of problems, such as video, images, and nature language processing. Inspired by these successful applications, we investigate deep restricted Boltzmann machines (DRBM), a representative deep learning technique, to predict the missing functional annotations of partially annotated proteins. Experimental results onHomo sapiens,Saccharomyces cerevisiae,Mus musculus,andDrosophilashow that DRBM achieves better performance than other related methods across different evaluation metrics, and it also runs faster than these comparing methods.

scholarly journals Anomaly Detection of Wind Turbines Based on Deep Small-World Neural Network

2020 ◽  
Vol 10 (4) ◽  
pp. 1243 ◽  
Author(s):  
Meng Li ◽  
Shuangxin Wang ◽  
Shanxiang Fang ◽  
Juchao Zhao
Keyword(s):  
Neural Network ◽  
Anomaly Detection ◽  
Wind Turbines ◽  
Small World ◽  
Value Theory ◽  
False Alarms ◽  
Learning Networks ◽  
Restricted Boltzmann Machines ◽  
Boltzmann Machines ◽  
Early Failures

Accurate and efficient condition monitoring is the key to enhance the reliability and security of wind turbines. In recent years, an intelligent anomaly detection method based on deep learning networks has been receiving increasing attention. Since accurately labeled data are usually difficult to obtain in real industries, this paper proposes a novel Deep Small-World Neural Network (DSWNN) on the basis of unsupervised learning to detect the early failures of wind turbines. During network construction, a regular auto-encoder network with multiple restricted Boltzmann machines is first constructed and pre-trained by using unlabeled data of wind turbines. After that, the trained network is transformed into a DSWNN model by randomly add-edges method, where the network parameters are fine-tuned by using minimal amounts of labeled data. In order to guard against the changes and disturbances of wind speed and reduce false alarms, an adaptive threshold based on extreme value theory is presented as the criterion of anomaly judgment. The DSWNN model is excellent in depth mining data characteristics and accurate measurement error. Last, two failure cases of wind turbine anomaly detection are given to demonstrate its validity and accuracy of the proposed methodology contrasted with the deep belief network and deep neural network.

Representational Power of Restricted Boltzmann Machines and Deep Belief Networks

2008 ◽  
Vol 20 (6) ◽  
pp. 1631-1649 ◽  
Author(s):  
Nicolas Le Roux ◽  
Yoshua Bengio
Keyword(s):  
Learning Algorithm ◽  
Network Models ◽  
Building Blocks ◽  
Discrete Distributions ◽  
Belief Networks ◽  
Deep Belief Networks ◽  
Neural Network Models ◽  
Restricted Boltzmann Machines ◽  
Boltzmann Machines ◽  
Representational Power

Deep belief networks (DBN) are generative neural network models with many layers of hidden explanatory factors, recently introduced by Hinton, Osindero, and Teh (2006) along with a greedy layer-wise unsupervised learning algorithm. The building block of a DBN is a probabilistic model called a restricted Boltzmann machine (RBM), used to represent one layer of the model. Restricted Boltzmann machines are interesting because inference is easy in them and because they have been successfully used as building blocks for training deeper models. We first prove that adding hidden units yields strictly improved modeling power, while a second theorem shows that RBMs are universal approximators of discrete distributions. We then study the question of whether DBNs with more layers are strictly more powerful in terms of representational power. This suggests a new and less greedy criterion for training RBMs within DBNs.

Enhanced Gradient for Training Restricted Boltzmann Machines

2013 ◽  
Vol 25 (3) ◽  
pp. 805-831 ◽  
Author(s):  
KyungHyun Cho ◽  
Tapani Raiko ◽  
Alexander Ilin
Keyword(s):  
Building Blocks ◽  
Data Representation ◽  
Learning Rate ◽  
Traditional Learning ◽  
Restricted Boltzmann Machines ◽  
Specific Data ◽  
Boltzmann Machines ◽  
Learning Rules ◽  
Deep Networks ◽  
The Right

Restricted Boltzmann machines (RBMs) are often used as building blocks in greedy learning of deep networks. However, training this simple model can be laborious. Traditional learning algorithms often converge only with the right choice of metaparameters that specify, for example, learning rate scheduling and the scale of the initial weights. They are also sensitive to specific data representation. An equivalent RBM can be obtained by flipping some bits and changing the weights and biases accordingly, but traditional learning rules are not invariant to such transformations. Without careful tuning of these training settings, traditional algorithms can easily get stuck or even diverge. In this letter, we present an enhanced gradient that is derived to be invariant to bit-flipping transformations. We experimentally show that the enhanced gradient yields more stable training of RBMs both when used with a fixed learning rate and an adaptive one.

Sign in / Sign up

Export Citation Format

Share Document