Flip Flop Neural Networks: Modelling Memory for Efficient Forecasting

Recurrent neural networks (RNNs) are useful tools for learning nonlinear relationships between time-varying inputs and outputs with complex temporal dependencies. Recently developed algorithms have been successful at training RNNs to perform a wide variety of tasks, but the resulting networks have been treated as black boxes: their mechanism of operation remains unknown. Here we explore the hypothesis that fixed points, both stable and unstable, and the linearized dynamics around them, can reveal crucial aspects of how RNNs implement their computations. Further, we explore the utility of linearization in areas of phase space that are not true fixed points but merely points of very slow movement. We present a simple optimization technique that is applied to trained RNNs to find the fixed and slow points of their dynamics. Linearization around these slow regions can be used to explore, or reverse-engineer, the behavior of the RNN. We describe the technique, illustrate it using simple examples, and finally showcase it on three high-dimensional RNN examples: a 3-bit flip-flop device, an input-dependent sine wave generator, and a two-point moving average. In all cases, the mechanisms of trained networks could be inferred from the sets of fixed and slow points and the linearized dynamics around them.

Download Full-text

Parameter optimisation in fuzzy flip-flop-based neural networks

International Journal of Reasoning-based Intelligent Systems ◽

10.1504/ijris.2010.036869 ◽

2010 ◽

Vol 2 (3/4) ◽

pp. 237 ◽

Cited By ~ 1

Author(s):

Rita Lovassy ◽

Laszlo T. Koczy ◽

Laszlo Gal

Keyword(s):

Neural Networks ◽

Flip Flop ◽

Parameter Optimisation

Download Full-text

The Flip-flop neuron – A memory efficient alternative for solving challenging sequence processing and decision making problems

10.1101/2021.11.16.468605 ◽

2021 ◽

Author(s):

Sweta Kumari ◽

Vigneswaran C ◽

V. Srinivasa Chakravarthy

Keyword(s):

Neural Networks ◽

Decision Making ◽

Information Integration ◽

Short Term Memory ◽

Previous History ◽

Video Frame ◽

Sequential Decision ◽

Flip Flop ◽

Sequence Processing ◽

Difficult Sequence

Sequential decision making tasks that require information integration over extended durations of time are challenging for several reasons including the problem of vanishing gradients, long training times and significant memory requirements. To this end we propose a neuron model fashioned after the JK flip-flops in digital systems. A flip-flop is a sequential device that can store state information of the previous history. We incorporate the JK flip-flop neuron into several deep network architectures and apply the networks to difficult sequence processing problems. The proposed architectures include flip-flop neural networks (FFNNs), bidirectional flip-flop neural networks (BiFFNNs), convolutional flip-flop neural networks (ConvFFNNs), and bidirectional convolutional flip-flop neural networks (BiConvFFNNs). Learning rules of proposed architectures have also been derived. We have considered the most popular benchmark sequential tasks like signal generation, sentiment analysis, handwriting generation, text generation, video frame prediction, lung volume prediction, and action recognition to evaluate the proposed networks. Finally, we compare the results of our networks with the results from analogous networks with Long Short-Term Memory (LSTM) neurons on the same sequential tasks. Our results show that the JK flip-flop networks outperform the LSTM networks significantly or marginally on all the tasks, with only half of the trainable parameters.

Download Full-text

Robustness of Fuzzy Flip-Flop based Neural Networks

2010 11th International Symposium on Computational Intelligence and Informatics (CINTI) ◽

10.1109/cinti.2010.5672248 ◽

2010 ◽

Cited By ~ 2

Author(s):

Rita Lovassy ◽

Laszlo T. Koczy ◽

Laszlo Gal

Keyword(s):

Neural Networks ◽

Flip Flop

Download Full-text

Optimization in Fuzzy Flip-Flop Neural Networks

Computational Intelligence in Engineering - Studies in Computational Intelligence ◽

10.1007/978-3-642-15220-7_27 ◽

2010 ◽

pp. 337-348

Author(s):

Rita Lovassy ◽

László T. Kóczy ◽

László Gál

Keyword(s):

Neural Networks ◽

Flip Flop

Download Full-text

The Flip-Flop Neuron – A Memory Efficient Alternative for Solving Challenging Sequence Processing and Decision Making Problems

10.21203/rs.3.rs-1040430/v1 ◽

2021 ◽

Author(s):

Sweta Kumari ◽

C Vigneswaran ◽

V. Srinivasa Chakrava

Keyword(s):

Neural Networks ◽

Decision Making ◽

Information Integration ◽

Short Term Memory ◽

Previous History ◽

Video Frame ◽

Sequential Decision ◽

Flip Flop ◽

Sequence Processing ◽

Difficult Sequence

Abstract Sequential decision making tasks that require information integration over extended durations of time are challenging for several reasons including the problem of vanishing gradients, long training times and significant memory requirements. To this end we propose a neuron model fashioned after the JK flip-flops in digital systems. A flip-flop is a sequential device that can store state information of the previous history. We incorporate the JK flip-flop neuron into several deep network architectures and apply the networks to difficult sequence processing problems. The proposed architectures include flip-flop neural networks (FFNNs), bidirectional flip-flop neural networks (BiFFNNs), convolutional flip-flop neural networks (ConvFFNNs), and bidirectional convolutional flip-flop neural networks (BiConvFFNNs). Learning rules of proposed architectures have also been derived. We have considered the most popular benchmark sequential tasks like signal generation, sentiment analysis, handwriting generation, text generation, video frame prediction, lung volume prediction, and action recognition to evaluate the proposed networks. Finally, we compare the results of our networks with the results from analogous networks with Long Short-Term Memory (LSTM) neurons on the same sequential tasks. Our results show that the JK flip-flop networks outperform the LSTM networks significantly or marginally on all the tasks, with only half of the trainable parameters.

Download Full-text

Biophysics (and sociology) of ceramides.

Biochemical Society Symposium ◽

10.1042/bss0720177 ◽

2005 ◽

Vol 72 ◽

pp. 177-188 ◽

Cited By ~ 28

Author(s):

Félix M. Goñi ◽

F-Xabier Contreras ◽

L-Ruth Montes ◽

Jesús Sot ◽

Alicia Alonso

Keyword(s):

Cell Biology ◽

Positive Curvature ◽

Acyl Chain ◽

Cell Signalling ◽

Hexagonal Structure ◽

Lipid Monolayer ◽

Flip Flop ◽

Long Chain ◽

Bilayer Permeability

In the past decade, the long-neglected ceramides (N-acylsphingosines) have become one of the most attractive lipid molecules in molecular cell biology, because of their involvement in essential structures (stratum corneum) and processes (cell signalling). Most natural ceramides have a long (16-24 C atoms) N-acyl chain, but short N-acyl chain ceramides (two to six C atoms) also exist in Nature, apart from being extensively used in experimentation, because they can be dispersed easily in water. Long-chain ceramides are among the most hydrophobic molecules in Nature, they are totally insoluble in water and they hardly mix with phospholipids in membranes, giving rise to ceramide-enriched domains. In situ enzymic generation, or external addition, of long-chain ceramides in membranes has at least three important effects: (i) the lipid monolayer tendency to adopt a negative curvature, e.g. through a transition to an inverted hexagonal structure, is increased, (ii) bilayer permeability to aqueous solutes is notoriously enhanced, and (iii) transbilayer (flip-flop) lipid motion is promoted. Short-chain ceramides mix much better with phospholipids, promote a positive curvature in lipid monolayers, and their capacities to increase bilayer permeability or transbilayer motion are very low or non-existent.

Download Full-text