Parameter optimisation in fuzzy flip-flop-based neural networks

Recurrent neural networks (RNNs) are useful tools for learning nonlinear relationships between time-varying inputs and outputs with complex temporal dependencies. Recently developed algorithms have been successful at training RNNs to perform a wide variety of tasks, but the resulting networks have been treated as black boxes: their mechanism of operation remains unknown. Here we explore the hypothesis that fixed points, both stable and unstable, and the linearized dynamics around them, can reveal crucial aspects of how RNNs implement their computations. Further, we explore the utility of linearization in areas of phase space that are not true fixed points but merely points of very slow movement. We present a simple optimization technique that is applied to trained RNNs to find the fixed and slow points of their dynamics. Linearization around these slow regions can be used to explore, or reverse-engineer, the behavior of the RNN. We describe the technique, illustrate it using simple examples, and finally showcase it on three high-dimensional RNN examples: a 3-bit flip-flop device, an input-dependent sine wave generator, and a two-point moving average. In all cases, the mechanisms of trained networks could be inferred from the sets of fixed and slow points and the linearized dynamics around them.

Download Full-text

The Flip-flop neuron – A memory efficient alternative for solving challenging sequence processing and decision making problems

10.1101/2021.11.16.468605 ◽

2021 ◽

Author(s):

Sweta Kumari ◽

Vigneswaran C ◽

V. Srinivasa Chakravarthy

Keyword(s):

Neural Networks ◽

Decision Making ◽

Information Integration ◽

Short Term Memory ◽

Previous History ◽

Video Frame ◽

Sequential Decision ◽

Flip Flop ◽

Sequence Processing ◽

Difficult Sequence

Sequential decision making tasks that require information integration over extended durations of time are challenging for several reasons including the problem of vanishing gradients, long training times and significant memory requirements. To this end we propose a neuron model fashioned after the JK flip-flops in digital systems. A flip-flop is a sequential device that can store state information of the previous history. We incorporate the JK flip-flop neuron into several deep network architectures and apply the networks to difficult sequence processing problems. The proposed architectures include flip-flop neural networks (FFNNs), bidirectional flip-flop neural networks (BiFFNNs), convolutional flip-flop neural networks (ConvFFNNs), and bidirectional convolutional flip-flop neural networks (BiConvFFNNs). Learning rules of proposed architectures have also been derived. We have considered the most popular benchmark sequential tasks like signal generation, sentiment analysis, handwriting generation, text generation, video frame prediction, lung volume prediction, and action recognition to evaluate the proposed networks. Finally, we compare the results of our networks with the results from analogous networks with Long Short-Term Memory (LSTM) neurons on the same sequential tasks. Our results show that the JK flip-flop networks outperform the LSTM networks significantly or marginally on all the tasks, with only half of the trainable parameters.

Download Full-text

Robustness of Fuzzy Flip-Flop based Neural Networks

2010 11th International Symposium on Computational Intelligence and Informatics (CINTI) ◽

10.1109/cinti.2010.5672248 ◽

2010 ◽

Cited By ~ 2

Author(s):

Rita Lovassy ◽

Laszlo T. Koczy ◽

Laszlo Gal

Keyword(s):

Neural Networks ◽

Flip Flop

Download Full-text

Optimization in Fuzzy Flip-Flop Neural Networks

Computational Intelligence in Engineering - Studies in Computational Intelligence ◽

10.1007/978-3-642-15220-7_27 ◽

2010 ◽

pp. 337-348

Author(s):

Rita Lovassy ◽

László T. Kóczy ◽

László Gál

Keyword(s):

Neural Networks ◽

Flip Flop

Download Full-text

The Flip-Flop Neuron – A Memory Efficient Alternative for Solving Challenging Sequence Processing and Decision Making Problems

10.21203/rs.3.rs-1040430/v1 ◽

2021 ◽

Author(s):

Sweta Kumari ◽

C Vigneswaran ◽

V. Srinivasa Chakrava

Keyword(s):

Neural Networks ◽

Decision Making ◽

Information Integration ◽

Short Term Memory ◽

Previous History ◽

Video Frame ◽

Sequential Decision ◽

Flip Flop ◽

Sequence Processing ◽

Difficult Sequence

Abstract Sequential decision making tasks that require information integration over extended durations of time are challenging for several reasons including the problem of vanishing gradients, long training times and significant memory requirements. To this end we propose a neuron model fashioned after the JK flip-flops in digital systems. A flip-flop is a sequential device that can store state information of the previous history. We incorporate the JK flip-flop neuron into several deep network architectures and apply the networks to difficult sequence processing problems. The proposed architectures include flip-flop neural networks (FFNNs), bidirectional flip-flop neural networks (BiFFNNs), convolutional flip-flop neural networks (ConvFFNNs), and bidirectional convolutional flip-flop neural networks (BiConvFFNNs). Learning rules of proposed architectures have also been derived. We have considered the most popular benchmark sequential tasks like signal generation, sentiment analysis, handwriting generation, text generation, video frame prediction, lung volume prediction, and action recognition to evaluate the proposed networks. Finally, we compare the results of our networks with the results from analogous networks with Long Short-Term Memory (LSTM) neurons on the same sequential tasks. Our results show that the JK flip-flop networks outperform the LSTM networks significantly or marginally on all the tasks, with only half of the trainable parameters.

Download Full-text