Improved ACD-Based Financial Trade Durations Prediction Leveraging LSTM Networks and Attention Mechanism

The liquidity risk factor of security market plays an important role in the formulation of trading strategies. A more liquid stock market means that the securities can be bought or sold more easily. As a sound indicator of market liquidity, the transaction duration is the focus of this study. We concentrate on estimating the probability density function p Δ t i + 1 | G i , where Δ t i + 1 represents the duration of the (i + 1)-th transaction and G i represents the historical information at the time when the (i + 1)-th transaction occurs. In this paper, we propose a new ultrahigh-frequency (UHF) duration modelling framework by utilizing long short-term memory (LSTM) networks to extend the conditional mean equation of classic autoregressive conditional duration (ACD) model while retaining the probabilistic inference ability. And then, the attention mechanism is leveraged to unveil the internal mechanism of the constructed model. In order to minimize the impact of manual parameter tuning, we adopt fixed hyperparameters during the training process. The experiments applied to a large-scale dataset prove the superiority of the proposed hybrid models. In the input sequence, the temporal positions which are more important for predicting the next duration can be efficiently highlighted via the added attention mechanism layer.

Download Full-text

Building Damage Detection Using U-Net with Attention Mechanism from Pre- and Post-Disaster Remote Sensing Datasets

Remote Sensing ◽

10.3390/rs13050905 ◽

2021 ◽

Vol 13 (5) ◽

pp. 905

Author(s):

Chuyi Wu ◽

Feng Zhang ◽

Junshi Xia ◽

Yichen Xu ◽

Guoqing Li ◽

...

Keyword(s):

Damage Assessment ◽

Large Scale ◽

Binary Classification ◽

Open Data ◽

Building Damage ◽

Attention Mechanism ◽

Large Scale Dataset ◽

Data Program ◽

The Impact ◽

Post Disaster

The building damage status is vital to plan rescue and reconstruction after a disaster and is also hard to detect and judge its level. Most existing studies focus on binary classification, and the attention of the model is distracted. In this study, we proposed a Siamese neural network that can localize and classify damaged buildings at one time. The main parts of this network are a variety of attention U-Nets using different backbones. The attention mechanism enables the network to pay more attention to the effective features and channels, so as to reduce the impact of useless features. We train them using the xBD dataset, which is a large-scale dataset for the advancement of building damage assessment, and compare their result balanced F (F1) scores. The score demonstrates that the performance of SEresNeXt with an attention mechanism gives the best performance, with the F1 score reaching 0.787. To improve the accuracy, we fused the results and got the best overall F1 score of 0.792. To verify the transferability and robustness of the model, we selected the dataset on the Maxar Open Data Program of two recent disasters to investigate the performance. By visual comparison, the results show that our model is robust and transferable.

Download Full-text

Estimates of future flood risk in Western Europe and its potential impact on insured losses.

10.5194/egusphere-egu21-12689 ◽

2021 ◽

Author(s):

Remi Meynadier ◽

Hugo Rakotoarimanga ◽

Madeleine-Sophie Deroche ◽

Sylvain Buisine

Keyword(s):

Climate Change ◽

Flood Risk ◽

Large Scale ◽

Western Europe ◽

Complex Nature ◽

Flood Risks ◽

Hazard Exposure ◽

Modelling Framework ◽

Loss Modelling ◽

The Impact

The large-scale and complex nature of climate change makes it difficult to assess and quantify the impact on insurance activities. Climate change is likely affecting the probability of natural hazard occurrence in terms of severity and/or frequency.Natural catastrophe risk is a function of hazard, exposure and vulnerability. As a (re)-insurer it is seen that changes in year-on-year losses are a function of all these components and not just the hazard.The present study focuses, in a first step, on assessing impacts of climate change on fluvial flood risks in Europe solely due to changes in hazard itself. A stochastic catalogue of future flood risk events is derived from Pan-European data sets of river flood probability of occurrence produced within EU FP7 RAIN project. The loss modelling framework internally developed at AXA is then used to provide a geographical view of changes in future flood risks.&#160;

Download Full-text

Emotion-Semantic-Enhanced Bidirectional LSTM with Multi-Head Attention Mechanism for Microblog Sentiment Analysis

Information ◽

10.3390/info11050280 ◽

2020 ◽

Vol 11 (5) ◽

pp. 280

Author(s):

Shaoxiu Wang ◽

Yonghua Zhu ◽

Wenjing Gao ◽

Meng Cao ◽

Mengyao Li

Keyword(s):

Sentiment Analysis ◽

Short Term Memory ◽

Syntactic Structure ◽

Contextual Information ◽

Research Field ◽

Attention Mechanism ◽

Feature Representation ◽

Mechanism Model ◽

Hidden Layer ◽

The Impact

The sentiment analysis of microblog text has always been a challenging research field due to the limited and complex contextual information. However, most of the existing sentiment analysis methods for microblogs focus on classifying the polarity of emotional keywords while ignoring the transition or progressive impact of words in different positions in the Chinese syntactic structure on global sentiment, as well as the utilization of emojis. To this end, we propose the emotion-semantic-enhanced bidirectional long short-term memory (BiLSTM) network with the multi-head attention mechanism model (EBILSTM-MH) for sentiment analysis. This model uses BiLSTM to learn feature representation of input texts, given the word embedding. Subsequently, the attention mechanism is used to assign the attentive weights of each words to the sentiment analysis based on the impact of emojis. The attentive weights can be combined with the output of the hidden layer to obtain the feature representation of posts. Finally, the sentiment polarity of microblog can be obtained through the dense connection layer. The experimental results show the feasibility of our proposed model on microblog sentiment analysis when compared with other baseline models.

Download Full-text

Research on Short-Term Load Prediction Based on Seq2seq Model

Energies ◽

10.3390/en12163199 ◽

2019 ◽

Vol 12 (16) ◽

pp. 3199 ◽

Cited By ~ 1

Author(s):

Gangjun Gong ◽

Xiaonan An ◽

Nawaraj Kumar Mahato ◽

Shuyan Sun ◽

Si Chen ◽

...

Keyword(s):

Power System ◽

Prediction Model ◽

Short Term Memory ◽

Predictive Performance ◽

Weather Conditions ◽

Attention Mechanism ◽

Load Prediction ◽

Short Term ◽

System Load ◽

The Impact

Electricity load prediction is the primary basis on which power-related departments to make logical and effective generation plans and scientific scheduling plans for the most effective power utilization. The perpetual evolution of deep learning has recommended advanced and innovative concepts for short-term load prediction. Taking into consideration the time and nonlinear characteristics of power system load data and further considering the impact of historical and future information on the current state, this paper proposes a Seq2seq short-term load prediction model based on a long short-term memory network (LSTM). Firstly, the periodic fluctuation characteristics of users’ load data are analyzed, establishing a correlation of the load data so as to determine the model’s order in the time series. Secondly, the specifications of the Seq2seq model are given preference and a coalescence of the Residual mechanism (Residual) and the two Attention mechanisms (Attention) is developed. Then, comparing the predictive performance of the model under different types of Attention mechanism, this paper finally adopts the Seq2seq short-term load prediction model of Residual LSTM and the Bahdanau Attention mechanism. Eventually, the prediction model obtains better results when merging the actual power system load data of a certain place. In order to validate the developed model, the Seq2seq was compared with recurrent neural network (RNN), LSTM, and gated recurrent unit (GRU) algorithms. Last but not least, the performance indices were calculated. when training and testing the model with power system load data, it was noted that the root mean square error (RMSE) of Seq2seq was decreased by 6.61%, 16.95%, and 7.80% compared with RNN, LSTM, and GRU, respectively. In addition, a supplementary case study was carried out using data for a small power system considering different weather conditions and user behaviors in order to confirm the applicability and stability of the proposed model. The Seq2seq model for short-term load prediction can be reported to demonstrate superiority in all areas, exhibiting better prediction and stable performance.

Download Full-text

Enhanced credit card fraud detection based on attention mechanism and LSTM deep model

Journal Of Big Data ◽

10.1186/s40537-021-00541-8 ◽

2021 ◽

Vol 8 (1) ◽

Author(s):

Ibtissam Benchaji ◽

Samira Douzi ◽

Bouabid El Ouahidi ◽

Jaafar Jaafari

Keyword(s):

Credit Card ◽

Short Term Memory ◽

Fraud Detection ◽

Input Sequence ◽

Attention Mechanism ◽

Credit Card Fraud ◽

Detection Systems ◽

Deep Model ◽

Efficiency And Effectiveness ◽

Card Payment

AbstractAs credit card becomes the most popular payment mode particularly in the online sector, the fraudulent activities using credit card payment technologies are rapidly increasing as a result. For this end, it is obligatory for financial institutions to continuously improve their fraud detection systems to reduce huge losses. The purpose of this paper is to develop a novel system for credit card fraud detection based on sequential modeling of data, using attention mechanism and LSTM deep recurrent neural networks. The proposed model, compared to previous studies, considers the sequential nature of transactional data and allows the classifier to identify the most important transactions in the input sequence that predict at higher accuracy fraudulent transactions. Precisely, the robustness of our model is built by combining the strength of three sub-methods; the uniform manifold approximation and projection (UMAP) for selecting the most useful predictive features, the Long Short Term Memory (LSTM) networks for incorporating transaction sequences and the attention mechanism to enhance LSTM performances. The experimentations of our model give strong results in terms of efficiency and effectiveness.

Download Full-text

Optimal Power Flow Calculation Considering Large-Scale Photovoltaic Generation Correlation

Frontiers in Energy Research ◽

10.3389/fenrg.2020.590418 ◽

2020 ◽

Vol 8 ◽

Author(s):

He Li ◽

Huijun Li ◽

Weihua Lu ◽

Zhenhao Wang ◽

Jing Bian

Keyword(s):

Power Flow ◽

Power Plants ◽

Optimal Power Flow ◽

Large Scale ◽

Short Term Memory ◽

Photovoltaic System ◽

Power Flow Calculation ◽

Flow Calculation ◽

Optimal Power ◽

The Impact

In order to analyze the impact of large-scale photovoltaic system on the power system, a photovoltaic output prediction method considering the correlation is proposed and the optimal power flow is calculated. Firstly, establish a photovoltaic output model to obtain the attenuation coefficient and fluctuation amount, and analyze the correlation among the multiple photovoltaic power plants through the k-means method. Secondly, the long short-term memory (LSTM) neural network is used as the photovoltaic output prediction model, and the clustered photovoltaic output data is brought into the LSTM model to generate large-scale photovoltaic prediction results with the consideration of the spatial correlation. And an optimal power flow model that takes grid loss and voltage offset as targets is established. Finally, MATLAB is used to verify that the proposed large-scale photovoltaic forecasting method has higher accuracy. The multi-objective optimal power flow calculation is performed based on the NSGA-II algorithm and the modified IEEE systems, and the optimal power flow with photovoltaic output at different times is compared and analyzed.

Download Full-text

Evaluation and Comparison of Random Forest and A-LSTM Networks for Large-scale Winter Wheat Identification

Remote Sensing ◽

10.3390/rs11141665 ◽

2019 ◽

Vol 11 (14) ◽

pp. 1665 ◽

Cited By ~ 11

Author(s):

Tianle He ◽

Chuanjie Xie ◽

Qingsheng Liu ◽

Shiying Guan ◽

Gaohuan Liu

Keyword(s):

Time Series ◽

Random Forest ◽

Winter Wheat ◽

Large Scale ◽

Short Term Memory ◽

State Of The Art ◽

Training Sample ◽

Central China ◽

Surface Reflectance ◽

The Impact

Machine learning comprises a group of powerful state-of-the-art techniques for land cover classification and cropland identification. In this paper, we proposed and evaluated two models based on random forest (RF) and attention-based long short-term memory (A-LSTM) networks that can learn directly from the raw surface reflectance of remote sensing (RS) images for large-scale winter wheat identification in Huanghuaihai Region (North-Central China). We used a time series of Moderate Resolution Imaging Spectroradiometer (MODIS) images over one growing season and the corresponding winter wheat distribution map for the experiments. Each training sample was derived from the raw surface reflectance of MODIS time-series images. Both models achieved state-of-the-art performance in identifying winter wheat, and the F1 scores of RF and A-LSTM were 0.72 and 0.71, respectively. We also analyzed the impact of the pixel-mixing effect. Training with pure-mixed-pixel samples (the training set consists of pure and mixed cells and thus retains the original distribution of data) was more precise than training with only pure-pixel samples (the entire pixel area belongs to one class). We also analyzed the variable importance along the temporal series, and the data acquired in March or April contributed more than the data acquired at other times. Both models could predict winter wheat coverage in past years or in other regions with similar winter wheat growing seasons. The experiments in this paper showed the effectiveness and significance of our methods.

Download Full-text

Transformers in Vision: A Survey

ACM Computing Surveys ◽

10.1145/3505244 ◽

2022 ◽

Author(s):

Salman Khan ◽

Muzammal Naseer ◽

Munawar Hayat ◽

Syed Waqas Zamir ◽

Fahad Shahbaz Khan ◽

...

Keyword(s):

Computer Vision ◽

Video Processing ◽

Architectural Design ◽

Large Scale ◽

Question Answering ◽

Short Term Memory ◽

Super Resolution ◽

Input Sequence ◽

3D Analysis ◽

Comprehensive Overview

Astounding results from Transformer models on natural language tasks have intrigued the vision community to study their application to computer vision problems. Among their salient benefits, Transformers enable modeling long dependencies between input sequence elements and support parallel processing of sequence as compared to recurrent networks e.g. , Long short-term memory (LSTM). Different from convolutional networks, Transformers require minimal inductive biases for their design and are naturally suited as set-functions. Furthermore, the straightforward design of Transformers allows processing multiple modalities ( e.g. , images, videos, text and speech) using similar processing blocks and demonstrates excellent scalability to very large capacity networks and huge datasets. These strengths have led to exciting progress on a number of vision tasks using Transformer networks. This survey aims to provide a comprehensive overview of the Transformer models in the computer vision discipline. We start with an introduction to fundamental concepts behind the success of Transformers i.e., self-attention, large-scale pre-training, and bidirectional feature encoding. We then cover extensive applications of transformers in vision including popular recognition tasks ( e.g. , image classification, object detection, action recognition, and segmentation), generative modeling, multi-modal tasks ( e.g. , visual-question answering, visual reasoning, and visual grounding), video processing ( e.g. , activity recognition, video forecasting), low-level vision ( e.g. , image super-resolution, image enhancement, and colorization) and 3D analysis ( e.g. , point cloud classification and segmentation). We compare the respective advantages and limitations of popular techniques both in terms of architectural design and their experimental value. Finally, we provide an analysis on open research directions and possible future works. We hope this effort will ignite further interest in the community to solve current challenges towards the application of transformer models in computer vision.

Download Full-text

Hybrid LSTM Self-Attention Mechanism Model for Forecasting the Reform of Scientific Research in Morocco

Computational Intelligence and Neuroscience ◽

10.1155/2021/6689204 ◽

2021 ◽

Vol 2021 ◽

pp. 1-14

Author(s):

Asmaa Fahim ◽

Qingmei Tan ◽

Mouna Mazzi ◽

Md Sahabuddin ◽

Bushra Naz ◽

...

Keyword(s):

Time Series ◽

Short Term Memory ◽

Research Area ◽

Vital Role ◽

Attention Mechanism ◽

Percentage Increase ◽

Mechanism Model ◽

Research Outcomes ◽

Future Outcomes ◽

The Impact

Education is the cultivation of people to promote and guarantee the development of society. Education reforms can play a vital role in the development of a country. However, it is crucial to continually monitor the educational model’s performance by forecasting the outcome’s progress. Machine learning-based models are currently a hot topic in improving the forecasting research area. Forecasting models can help to analyse the impact of future outcomes by showing yearly trends. For this study, we developed a hybrid, forecasting time-series model by long short-term memory (LSTM) network and self-attention mechanism (SAM) to monitor Morocco’s educational reform. We analysed six universities’ performance and provided a prediction model to evaluate the best-performing university’s performance after implementing the latest reform, i.e., from 2015–2030. We forecasted the six universities’ research outcomes and tested our proposed methodology’s accuracy against other time-series models. Results show that our model performs better for predicting research outcomes. The percentage increase in university performance after nine years is discussed to help predict the best-performing university. Our proposed algorithm accuracy and performance are better than other algorithms like LSTM and RNN.

Download Full-text

End-to-End Chinese Dialect Discrimination with Self-Attention

10.5121/csit.2021.111425 ◽

2021 ◽

Author(s):

Yangjie Dan ◽

Fan Xu ◽

Mingwen Wang

Keyword(s):

Large Scale ◽

Short Term Memory ◽

Attention Mechanism ◽

Practical Significance ◽

Sequence Information ◽

Discrimination Model ◽

Chinese Dialect ◽

End To End ◽

Low Performance ◽

Phonetic Features

Dialect discrimination has an important practical significance for protecting inheritance of dialects. The traditional dialect discrimination methods pay much attention to the underlying acoustic features, and ignore the meaning of the pronunciation itself, resulting in low performance. This paper systematically explores the validity of the pronunciation features of dialect speech composed of phoneme sequence information for dialect discrimination, and designs an end-to-end dialect discrimination model based on the multi-head self-attention mechanism. Specifically, we first adopt the residual convolution neural network and the multihead self-attention mechanism to effectively extract the phoneme sequence features unique to different dialects to compose the novel phonetic features. Then, we perform dialect discrimination based on the extracted phonetic features using the self-attention mechanism and bi-directional long short-term memory networks. The experimental results on the large-scale benchmark 10-way Chinese dialect corpus released by IFLYTEK 1 show that our model outperforms the state-of-the-art alternatives by large margin.

Download Full-text