Task-Agnostic Continual Learning Using Online Variational Bayes with Fixed-Point Updates

Catastrophic forgetting is the notorious vulnerability of neural networks to the changes in the data distribution during learning. This phenomenon has long been considered a major obstacle for using learning agents in realistic continual learning settings. A large body of continual learning research assumes that task boundaries are known during training. However, only a few works consider scenarios in which task boundaries are unknown or not well defined: task-agnostic scenarios. The optimal Bayesian solution for this requires an intractable online Bayes update to the weights posterior. We aim to approximate the online Bayes update as accurately as possible. To do so, we derive novel fixed-point equations for the online variational Bayes optimization problem for multivariate gaussian parametric distributions. By iterating the posterior through these fixed-point equations, we obtain an algorithm (FOO-VB) for continual learning that can handle nonstationary data distribution using a fixed architecture and without using external memory (i.e., without access to previous data). We demonstrate that our method (FOO-VB) outperforms existing methods in task-agnostic scenarios. FOO-VB Pytorch implementation is available at https://github.com/chenzeno/FOO-VB.

Kernel Online System for Fast Principal Component Analysis and its Adaptive Learning

An artificial neural system for data compression that sequentially processes linearly nonseparable classes is proposed. The main elements of this system include adjustable radial-basis functions (Epanechnikov’s kernels), an adaptive linear associator learned by a multistep optimal algorithm, and Hebb-Sanger neural network whose nodes are formed by Oja’s neurons. For tuning the modified Oja’s algorithm, additional filtering (in case of noisy data) and tracking (in case of nonstationary data) properties were introduced. The main feature of the proposed system is the ability to work in conditions of significant nonlinearity of the initial data that are sequentially fed to the system and have a non-stationary nature. The effectiveness of the developed approach was confirmed by the experimental results. The proposed kernel online neural system is designed to solve compression and visualization tasks when initial data form linearly nonseparable classes in general problem of Data Stream Mining and Dynamic Data Mining. The main benefit of the proposed approach is high speed and ability to process data whose characteristics are changed in time.

Optimum wavelet selection for nonparametric analysis toward structural health monitoring for processing big data from sensor network: A comparative study

A critical problem encountered in structural health monitoring of civil engineering structures, and other structures such as mechanical or aircraft structures, is how to convincingly analyze the nonstationary data that is coming online, how to reduce the high-dimensional features, and how to extract informative features associated with damage to infer structural conditions. Wavelet transform among other techniques has proven to be an effective technique for processing and analyzing nonstationary data due to its unique characteristics. However, the biggest challenge frequently encountered in assuring the effectiveness of wavelet transform in analyzing massive nonstationary data from civil engineering structures, and in structural health diagnosis, is how to select the right wavelet. The question of which wavelet function is appropriate for processing and analyzing the nonstationary data in civil engineering structures has not been clearly addressed, and no clear guidelines or rules have been reported in the literature to show how the right wavelet is chosen. Therefore, this study aims to address an important question in this regard by proposing a new framework for choosing a proper wavelet that can be customized for massive nonstationary data analysis, disturbances separation, and extraction of informative features associated with damage. The proposed method takes into account data type, data and wavelet characteristics, similarity, sharing information, and data recovery accuracy. The novelty of this study lies in integrating multi-criteria which are associated directly with features that correlated well with change in structures due to damage, including common criteria such as energy, entropy, linear correlation index, and variance. Also, it introduces and considers new proposed measures, such as wavelet-based nonlinear correlation such as cosh spectral distance and mutual information, wavelet-based energy fluctuation, measures-based recovery accuracy, such as sensitive feature extraction, noise reduction, and others to evaluate various base wavelets’ function capabilities for appropriate decomposition and reconstruction of structural dynamic responses. The proposed method is verified by experimental and simulated data. The results revealed that the proposed method has a satisfactory performance for base wavelet selection and the small order of Daubechies and Symlet provide the best results, especially order 3. The idea behind our proposed framework can be applied to other structural applications.

Spurious Factor Analysis

This paper draws parallels between the principal components analysis of factorless high‐dimensional nonstationary data and the classical spurious regression. We show that a few of the principal components of such data absorb nearly all the data variation. The corresponding scree plot suggests that the data contain a few factors, which is corroborated by the standard panel information criteria. Furthermore, the Dickey–Fuller tests of the unit root hypothesis applied to the estimated “idiosyncratic terms” often reject, creating an impression that a few factors are responsible for most of the nonstationarity in the data. We warn empirical researchers of these peculiar effects and suggest to always compare the analysis in levels with that in differences.

A Novel Approach to Maximize G-mean in Nonstationary Data with Recurrent Imbalance Shifts

One of the noteworthy difficulties in the classification of nonstationary data is handling data with class imbalance. Imbalanced data possess the characteristics of having a lot of samples of one class than the other. It, thusly, results in the biased accuracy of a classifier in favour of a majority class. Streaming data may have inherent imbalance resulting from the nature of dataspace or extrinsic imbalance due to its nonstationary environment. In streaming data, timely varying class priors may lead to a shift in imbalance ratio. The researchers have contemplated ensemble learning, online learning, issue of class imbalance and cost-sensitive algorithms autonomously. They have scarcely ever tended to every one of these issues mutually to deal with imbalance shift in nonstationary data. This correspondence shows a novel methodology joining these perspectives to augment G-mean in no stationary data with Recurrent Imbalance Shifts (RIS). This research modifies the state-of-the-art boosting algorithms,1) AdaC2 to get G-mean based Online AdaC2 for Recurrent Imbalance Shifts (GOA-RIS) and AGOA-RIS (Ageing and G-mean based Online AdaC2 for Recurrent Imbalance Shifts), and 2) CSB2 to get G-mean based Online CSB2 for Recurrent Imbalance Shifts (GOC-RIS) and Ageing and G-mean based Online CSB2 for Recurrent Imbalance Shifts (AGOC-RIS). The study has empirically and statistically analysed the performances of the proposed algorithms and Online AdaC2 (OA) and Online CSB2 (OC) algorithms using benchmark datasets. The test outcomes demonstrate that the proposed algorithms globally beat the performances of OA and OC

Microcluster-Based Incremental Ensemble Learning for Noisy, Nonstationary Data Streams

Data stream classification becomes a promising prediction work with relevance to many practical environments. However, under the environment of concept drift and noise, the research of data stream classification faces lots of challenges. Hence, a new incremental ensemble model is presented for classifying nonstationary data streams with noise. Our approach integrates three strategies: incremental learning to monitor and adapt to concept drift; ensemble learning to improve model stability; and a microclustering procedure that distinguishes drift from noise and predicts the labels of incoming instances via majority vote. Experiments with two synthetic datasets designed to test for both gradual and abrupt drift show that our method provides more accurate classification in nonstationary data streams with noise than the two popular baselines.