scholarly journals Optimized Weight Programming for Analogue Memory-based Deep Neural Networks

2021 ◽  
Author(s):  
Charles Mackin ◽  
Malte Rasch ◽  
An Chen ◽  
Jonathan Timcheck ◽  
Robert Bruce ◽  
...  
Keyword(s):  
Neural Networks ◽  
Deep Neural Networks ◽  
Short Term Memory ◽  
Memory Device ◽  
Computational Technique ◽  
Full Potential ◽  
Graphic Processing Units ◽  
Short Term ◽  
Computational Framework ◽  
Long Short Term Memory

Abstract Analogue memory-based Deep Neural Networks (DNNs) provide energy-efficiency and per-area throughput gains relative to state-of-the-art digital counterparts such as graphic processing units (GPUs). Recent advances focus largely on hardware-aware algorithmic training and improvements in circuits, architectures, and memory device characteristics. Optimal translation of software-trained weights into analogue hardware weights---given the plethora of complex memory non-idealities---represents an equally important goal in realizing the full potential of this technology. We report a generalized computational framework that automates the process of crafting complex weight programming strategies for analogue memory-based DNNs, in order to minimize accuracy degradations during inference, particularly over time. This framework is agnostic to DNN structure and is shown to generalize well across Long Short-Term Memory (LSTM), Convolution Neural Networks (CNNs), and Transformer networks. Being a highly-flexible numerical heuristic, our approach can accommodate arbitrary device-level complexity, and is thus broadly applicable to a variety of analogue memories and their continually evolving device characteristics. Interestingly, this computational technique is capable of optimizing inference accuracy without the need to run inference simulations or evaluate large training, validation, or test datasets. Lastly, by quantifying the limit of achievable inference accuracy given imperfections in analogue memory, weight programming optimization represents a unique and foundational tool for enabling analogue memory-based DNN accelerators to reach their full inference potential.

scholarly journals Using Machine Learning Methods in the Financial Market for Technical Analysis Based on Hybrid Models

2020 ◽  
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Hybrid Model ◽  
Deep Neural Network ◽  
Deep Neural Networks ◽  
Short Term Memory ◽  
Short Term ◽  
Term Memory ◽  
Input Structure ◽  
Long Short Term Memory

In this study, it is presented a new hybrid model based on deep neural networks to predict the direction and magnitude of the Forex market movement in the short term. The overall model presented is based on the scalping strategy and is provided for high frequency transactions. The proposed hybrid model is based on a combination of three models based on deep neural networks. The first model is a deep neural network with a multi-input structure consisting of a combination of Long Short Term Memory layers. The second model is a deep neural network with a multi-input structure made of a combination of one-dimensional Convolutional Neural network layers. The third model has a simpler structure and is a multi-input model of the Multi-Layer Perceptron layers. The overall model was also a model based on the majority vote of three top models. This study showed that models based on Long Short-Term Memory layers provided better results than the other models and even hybrid models with more than 70% accurate.

Sentiment Analysis in the Light of LSTM Recurrent Neural Networks

2018 ◽  
Vol 9 (1) ◽  
pp. 33-39 ◽  
Author(s):  
Subarno Pal ◽  
Soumadip Ghosh ◽  
Amitava Nag
Keyword(s):  
Neural Networks ◽  
Sentiment Analysis ◽  
Recurrent Neural Networks ◽  
Deep Neural Networks ◽  
Short Term Memory ◽  
Short Term ◽  
Different Types ◽  
Long Short Term Memory ◽  
Increase In Accuracy ◽  
Temporal Sequences

Long short-term memory (LSTM) is a special type of recurrent neural network (RNN) architecture that was designed over simple RNNs for modeling temporal sequences and their long-range dependencies more accurately. In this article, the authors work with different types of LSTM architectures for sentiment analysis of movie reviews. It has been showed that LSTM RNNs are more effective than deep neural networks and conventional RNNs for sentiment analysis. Here, the authors explore different architectures associated with LSTM models to study their relative performance on sentiment analysis. A simple LSTM is first constructed and its performance is studied. On subsequent stages, the LSTM layer is stacked one upon another which shows an increase in accuracy. Later the LSTM layers were made bidirectional to convey data both forward and backward in the network. The authors hereby show that a layered deep LSTM with bidirectional connections has better performance in terms of accuracy compared to the simpler versions of LSTM used here.

scholarly journals Comparative Performance of Machine Learning Algorithms in the Prediction of Indoor Daylight Illuminances

2020 ◽  
Vol 12 (11) ◽  
pp. 4471 ◽  
Author(s):  
Jack Ngarambe ◽  
Amina Irakoze ◽  
Geun Young Yun ◽  
Gon Kim
Keyword(s):  
Machine Learning ◽  
Neural Networks ◽  
Deep Neural Networks ◽  
Short Term Memory ◽  
Machine Learning Algorithms ◽  
Gradient Boosting ◽  
Short Term ◽  
Term Memory ◽  
Long Short Term Memory ◽  
Indoor Spaces

The performance of machine learning (ML) algorithms depends on the nature of the problem at hand. ML-based modeling, therefore, should employ suitable algorithms where optimum results are desired. The purpose of the current study was to explore the potential applications of ML algorithms in modeling daylight in indoor spaces and ultimately identify the optimum algorithm. We thus developed and compared the performance of four common ML algorithms: generalized linear models, deep neural networks, random forest, and gradient boosting models in predicting the distribution of indoor daylight illuminances. We found that deep neural networks, which showed a determination of coefficient (R2) of 0.99, outperformed the other algorithms. Additionally, we explored the use of long short-term memory to forecast the distribution of daylight at a particular future time. Our results show that long short-term memory is accurate and reliable (R2 = 0.92). Our findings provide a basis for discussions on ML algorithms’ use in modeling daylight in indoor spaces, which may ultimately result in efficient tools for estimating daylight performance in the primary stages of building design and daylight control schemes for energy efficiency.

scholarly journals Transformer-Based Deep Neural Language Modeling for Construct-Specific Automatic Item Generation

2021 ◽  
Author(s):  
Björn Hommel ◽  
Franz-Josef Wollang ◽  
Veronika Kotova ◽  
Hannes Zacher ◽  
Stefan C. Schmukle
Keyword(s):  
Neural Networks ◽  
Language Processing ◽  
Deep Neural Networks ◽  
Short Term Memory ◽  
Automatic Generation ◽  
Short Term ◽  
Item Generation ◽  
Aptitude Testing ◽  
Long Short Term Memory ◽  
Automatic Item Generation

Algorithmic automatic item generation can be used to obtain large quantities of cognitive items in the domains of knowledge and aptitude testing. However, conventional item models used by template-based automatic item generation techniques are not ideal for the creation of items for non-cognitive constructs. Progress in this area has been made recently by employing long short-term memory recurrent neural networks to produce word sequences that syntactically resemble items typically found in personality questionnaires. To date, such items have been produced unconditionally, without the possibility of selectively targeting personality domains. In this article, we offer a brief synopsis on past developments in natural language processing and explain why the automatic generation of construct-specific items has become attainable only due to recent technological progress. We propose that pre-trained causal transformer models can be fine-tuned to achieve this task using implicit parameterization in conjunction with conditional generation. We demonstrate this method in a tutorial-like fashion and finally compare aspects of validity in human- and machine-authored items using empirical data. Our study finds that approximately two-thirds of the automatically generated items show good psychometric properties (factor loadings above .40) and that one-third even have properties equivalent to established and highly curated human-authored items. Our work thus demonstrates the practical use of deep neural networks for non-cognitive automatic item generation.

scholarly journals Application of Deep Learning Methods to Forecasting Changes in Short-Term Currency Trends

2020 ◽  
Vol 7 (2) ◽  
pp. 75-86
Author(s):  
Vasily D. Derbentsev ◽  
Vitalii S. Bezkorovainyi ◽  
Iryna V. Luniak
Keyword(s):  
Neural Networks ◽  
Time Series ◽  
Deep Learning ◽  
Deep Neural Networks ◽  
Short Term Memory ◽  
Learning Models ◽  
Short Term ◽  
Term Memory ◽  
Long Short Term Memory ◽  
Better Than

This study investigates the issues of forecasting changes in short-term currency trends using deep learning models, which is relevant for both the scientific community and for traders and investors. The purpose of this study is to build a model for forecasting the direction of change in the prices of currency quotes based on deep neural networks. The developed architecture was based on the model of valve recurrent node, which is a modification of the model of “Long Short-Term Memory”, but is simpler in terms of the number of parameters and learning time. The forecast calculations of the dynamics of quotations of the currency pair euro/dollar and the most capitalised cryptocurrency Bitcoin/dollar were performed using daily, four-hour and hourly datasets. The obtained results of binary classification (forecast of the direction of trend change) when applying daily and hourly quotations turned out to be generally better than those of time series models or models of neural networks of other architecture (in particular, multilayer perceptron or “Long Short-Term Memory” models). According to the study results, the highest accuracy of classification was for the model of daily quotations for both euro/dollar – about 72%, and for Bitcoin/ dollar – about 69%. For four-hour and hourly time series, the accuracy of classification decreased, which can be explained both by the increase in the impact of “market noise” and the probable overfitting. Computer simulation has demonstrated that models predict a rising trend better than a declining one. The study confirmed the prospects for the application of deep learning models for short-term forecasting of time series of currency quotes. The use of the developed models proved to be effective for both fiat and cryptocurrencies. The proposed system of models based on deep neural networks can be used as a basis for developing an automated trading system in the foreign exchange market

Sign in / Sign up

Export Citation Format

Share Document