Bayesian Optimization of Spiking Neural Network Parameters to Solving the Time Series Classification Task

Time series data from real problems have nonlinear, non-smooth, and multi-scale composite characteristics. This paper first proposes a gated recurrent unit-correction (GRU-corr) network model, which adds a correction layer to the GRU neural network. Then, a adaptive staged variation PSO (ASPSO) is proposed. Finally, to overcome the drawbacks of the imprecise selection of the GRU-corr network parameters and obtain the high-precision global optimization of network parameters, weight parameters and the hidden nodes number of GRU-corr is optimized by ASPSO, and a time series prediction model (ASPSO-GRU-corr) is proposed based on the GRU-corr optimized by ASPSO. In the experiment, a comparative analysis of the optimization performance of ASPSO on a benchmark function was performed to verify its validity, and then the ASPSO-GRU-corr model is used to predict the ship motion cross-sway angle data. The results show that, ASPSO has better optimization performance and convergence speed compared with other algorithms, while the ASPSO-GRU-corr has higher generalization performance and lower architecture complexity. The ASPSO-GRU-corr can reveal the intrinsic multi-scale composite features of the time series, which is a reliable nonlinear and non-steady time series prediction method.

Download Full-text

Bayesian recurrent neural network as a tool for reconstructing dynamical systems from multidimensional data

10.5194/egusphere-egu2020-8787 ◽

2020 ◽

Author(s):

Alexander Feigin ◽

Aleksei Seleznev ◽

Dmitry Mukhin ◽

Andrey Gavrilov ◽

Evgeny Loskutov

Keyword(s):

Neural Network ◽

Time Series ◽

Cost Function ◽

Recurrent Neural Network ◽

Bayesian Optimization ◽

Data Driven ◽

High Dimensional ◽

Model Learning ◽

Low Dimensional ◽

The Cost

We suggest a new method for construction of data-driven dynamical models from observed multidimensional time series. The method is based on a recurrent neural network (RNN) with specific structure, which allows for the joint reconstruction of both a low-dimensional embedding for dynamical components in the data and an operator describing the low-dimensional evolution of the system. The key link of the method is a Bayesian optimization of both model structure and the hypothesis about the data generating law, which is needed for constructing the cost function for model learning. &#160;The form of the model we propose allows us to construct a stochastic dynamical system of moderate dimension that copies dynamical properties of the original high-dimensional system. An advantage of the proposed method is the data-adaptive properties of the RNN model: it is based on the adjustable nonlinear elements and has easily scalable structure. The combination of the RNN with the Bayesian optimization procedure efficiently provides the model with statistically significant nonlinearity and dimension. The method developed for the model optimization aims to detect the long-term connections between system&#8217;s states &#8211; the memory of the system: the cost-function used for model learning is constructed taking into account this factor. In particular, in the case of absence of interaction between the dynamical component and noise, the method provides unbiased reconstruction of the hidden deterministic system. In the opposite case when the noise has strong impact on the dynamics, the method yield a model in the form of a nonlinear stochastic map determining the Markovian process with memory. Bayesian approach used for selecting both the optimal model&#8217;s structure and the appropriate cost function allows to obtain the statistically significant inferences about the dynamical signal in data as well as its interaction with the noise components. Data driven model derived from the relatively short time series of the QG3 model &#8211; the high dimensional nonlinear system producing chaotic behavior &#8211; is shown be able to serve as a good simulator for the QG3 LFV components. The statistically significant recurrent states of the QG3 model, i.e. the well-known teleconnections in NH, are all reproduced by the model obtained. Moreover, statistics of the residence times of the model near these states is very close to the corresponding statistics of the original QG3 model. These results demonstrate that the method can be useful in modeling the variability of the real atmosphere.The work was supported by the Russian Science Foundation (Grant No. 19-42-04121).

Download Full-text

Comparing End-to-End Machine Learning Methods for Spectra Classification

Applied Sciences ◽

10.3390/app112311520 ◽

2021 ◽

Vol 11 (23) ◽

pp. 11520

Author(s):

Yue Sun ◽

Sandor Brockhauser ◽

Péter Hegedűs

Keyword(s):

Neural Network ◽

Time Series ◽

Short Term Memory ◽

Classification Problem ◽

High Energy ◽

High Energy Density ◽

Time Series Classification ◽

Classification Models ◽

Attention Model ◽

End To End

In scientific research, spectroscopy and diffraction experimental techniques are widely used and produce huge amounts of spectral data. Learning patterns from spectra is critical during these experiments. This provides immediate feedback on the actual status of the experiment (e.g., time-resolved status of the sample), which helps guide the experiment. The two major spectral changes what we aim to capture are either the change in intensity distribution (e.g., drop or appearance) of peaks at certain locations, or the shift of those on the spectrum. This study aims to develop deep learning (DL) classification frameworks for one-dimensional (1D) spectral time series. In this work, we deal with the spectra classification problem from two different perspectives, one is a general two-dimensional (2D) space segmentation problem, and the other is a common 1D time series classification problem. We focused on the two proposed classification models under these two settings, the namely the end-to-end binned Fully Connected Neural Network (FCNN) with the automatically capturing weighting factors model and the convolutional SCT attention model. Under the setting of 1D time series classification, several other end-to-end structures based on FCNN, Convolutional Neural Network (CNN), ResNets, Long Short-Term Memory (LSTM), and Transformer were explored. Finally, we evaluated and compared the performance of these classification models based on the High Energy Density (HED) spectra dataset from multiple perspectives, and further performed the feature importance analysis to explore their interpretability. The results show that all the applied models can achieve 100% classification confidence, but the models applied under the 1D time series classification setting are superior. Among them, Transformer-based methods consume the least training time (0.449 s). Our proposed convolutional Spatial-Channel-Temporal (SCT) attention model uses 1.269 s, but its self-attention mechanism performed across spatial, channel, and temporal dimensions can suppress indistinguishable features better than others, and selectively focus on obvious features with high separability.

Download Full-text