Expansion of Information in the Binary Autoencoder with Random Binary Weights

This letter studies the expansion and preservation of information in a binary autoencoder where the hidden layer is larger than the input. Such expansion is widespread in biological neural networks, as in the olfactory system of a fruit fly or the projection of thalamic inputs to the neocortex. We analyze the threshold model, the kWTA model, and the binary matching pursuit model to find how the sparsity and the dimension of the encoding influence the input reconstruction, similarity preservation, and mutual information across layers. It is shown that the sparser activation of the hidden layer is preferable for preserving information between the input and the output layers. All three models show optimal similarity preservation at dense, not sparse, hidden layer activation. Furthermore, with a large enough hidden layer, it is possible to get zero reconstruction error for any input just by varying the thresholds of neurons. However, we show that the preference for sparsity is due to the noise in the weight matrix between layers. A fixed number of nonzero connections to every neuron achieves better information preservation and input reconstruction for the dense hidden layer activation. The theoretical results give useful insight into models of neural computation based on sparse binary representation and association memory.

Observer Based Multi-Level Fault Reconstruction for Interconnected Systems

The problem of local fault (unknown input) reconstruction for interconnected systems is addressed in this paper. This contribution consists of a geometric method which solves the fault reconstruction (FR) problem via observer based and a differential algebraic concept. The fault diagnosis (FD) problem is tackled using the concept of the differential transcendence degree of a differential field extension and the algebraic observability. The goal is to examine whether the fault occurring in the low-level subsystem can be reconstructed correctly by the output at the high-level subsystem under given initial states. By introducing the fault as an additional state of the low subsystem, an observer based approached is proposed to estimate this new state. Particularly, the output of the lower subsystem is assumed unknown, and is considered as auxiliary outputs. Then, the auxiliary outputs are estimated by a sliding mode observer which is generated by using global outputs and inverse techniques. After this, the estimated auxiliary outputs are employed as virtual sensors of the system to generate a reduced-order observer, which is caplable of estimating the fault variable asymptotically. Thus, the purpose of multi-level fault reconstruction is achieved. Numerical simulations on an intensified heat exchanger are presented to illustrate the effectiveness of the proposed approach.

On dynamical input reconstruction in a distributed second order equation

A second order nonlinear differential equation is considered. An algorithm for reconstructing an input from inaccurate measurements of the solution at discrete times is designed. The algorithm based on the constructions of feedback control theory and theory of ill-posed problems is stable with respect to informational noises and computational errors.

Comparison of Novelty Detection Methods for Detection of Various Rotary Machinery Faults

Condition monitoring is an indispensable element related to the operation of rotating machinery. In this article, the monitoring system for the parallel gearbox was proposed. The novelty detection approach is used to develop the condition assessment support system, which requires data collection for a healthy structure. The measured signals were processed to extract quantitative indicators sensitive to the type of damage occurring in this type of structure. The indicator’s values were used for the development of four different novelty detection algorithms. Presented novelty detection models operate on three principles: feature space distance, probability distribution, and input reconstruction. One of the distance-based models is adaptive, adjusting to new data flowing in the form of a stream. The authors test the developed algorithms on experimental and simulation data with a similar distribution, using the training set consisting mainly of samples generated by the simulator. Presented in the article results demonstrate the effectiveness of the trained models on both data sets.

SurvNet: A Novel Deep Neural Network for Lung Cancer Survival Analysis With Missing Values

Survival analysis is important for guiding further treatment and improving lung cancer prognosis. It is a challenging task because of the poor distinguishability of features and the missing values in practice. A novel multi-task based neural network, SurvNet, is proposed in this paper. The proposed SurvNet model is trained in a multi-task learning framework to jointly learn across three related tasks: input reconstruction, survival classification, and Cox regression. It uses an input reconstruction mechanism cooperating with incomplete-aware reconstruction loss for latent feature learning of incomplete data with missing values. Besides, the SurvNet model introduces a context gating mechanism to bridge the gap between survival classification and Cox regression. A new real-world dataset of 1,137 patients with IB-IIA stage non-small cell lung cancer is collected to evaluate the performance of the SurvNet model. The proposed SurvNet achieves a higher concordance index than the traditional Cox model and Cox-Net. The difference between high-risk and low-risk groups obtained by SurvNet is more significant than that of high-risk and low-risk groups obtained by the other models. Moreover, the SurvNet outperforms the other models even though the input data is randomly cropped and it achieves better generalization performance on the Surveillance, Epidemiology, and End Results Program (SEER) dataset.

The methods of dynamical reconstruction of an input in a system of ordinary differential equations

In the paper, for systems described by ordinary differential equations a review of algorithms of dynamical input reconstruction by results of inaccurate observations of its solutions is given. The problem under discussion is referred to the class of dynamical inverse problems. The proposed algorithms are stable with respect to informational noises and computational errors. They are based on the combination of methods of the theory of ill-posed problems and the theory of feedback control. The essence of the methodology underlying the algorithms suggested in the paper consists in the representation of a reconstruction algorithm in the form of a feedback control algorithm for a certain artificial dynamical system, a model; such an algorithm, whose output is the realization of the control in the model, is dynamical by its definition.

Unsupervised Word Translation with Adversarial Autoencoder

Crosslingual word embeddings learned from monolingual embeddings have a crucial role in many downstream tasks, ranging from machine translation to transfer learning. Adversarial training has shown impressive success in learning crosslingual embeddings and the associated word translation task without any parallel data by mapping monolingual embeddings to a shared space. However, recent work has shown superior performance for non-adversarial methods in more challenging language pairs. In this article, we investigate adversarial autoencoder for unsupervised word translation and propose two novel extensions to it that yield more stable training and improved results. Our method includes regularization terms to enforce cycle consistency and input reconstruction, and puts the target encoders as an adversary against the corresponding discriminator. We use two types of refinement procedures sequentially after obtaining the trained encoders and mappings from the adversarial training, namely, refinement with Procrustes solution and refinement with symmetric re-weighting. Extensive experimentations with high- and low-resource languages from two different data sets show that our method achieves better performance than existing adversarial and non-adversarial approaches and is also competitive with the supervised system. Along with performing comprehensive ablation studies to understand the contribution of different components of our adversarial model, we also conduct a thorough analysis of the refinement procedures to understand their effects.