Cubic LSTMs for Video Prediction

Predicting future frames in videos has become a promising direction of research for both computer vision and robot learning communities. The core of this problem involves moving object capture and future motion prediction. While object capture specifies which objects are moving in videos, motion prediction describes their future dynamics. Motivated by this analysis, we propose a Cubic Long Short-Term Memory (CubicLSTM) unit for video prediction. CubicLSTM consists of three branches, i.e., a spatial branch for capturing moving objects, a temporal branch for processing motions, and an output branch for combining the first two branches to generate predicted frames. Stacking multiple CubicLSTM units along the spatial branch and output branch, and then evolving along the temporal branch can form a cubic recurrent neural network (CubicRNN). Experiment shows that CubicRNN produces more accurate video predictions than prior methods on both synthetic and real-world datasets.

Download Full-text

Human-Like Hand Reaching by Motion Prediction Using Long Short-Term Memory

Social Robotics - Lecture Notes in Computer Science ◽

10.1007/978-3-319-70022-9_16 ◽

2017 ◽

pp. 156-166 ◽

Cited By ~ 3

Author(s):

Phongtharin Vinayavekhin ◽

Michiaki Tatsubori ◽

Daiki Kimura ◽

Yifan Huang ◽

Giovanni De Magistris ◽

...

Keyword(s):

Short Term Memory ◽

Motion Prediction ◽

Short Term ◽

Term Memory ◽

Long Short Term Memory

Download Full-text

LSTM-PSO: Long Short-Term Memory Ship Motion Prediction Based on Particle Swarm Optimization

2018 IEEE CSAA Guidance, Navigation and Control Conference (CGNCC) ◽

10.1109/gncc42960.2018.9018688 ◽

2018 ◽

Cited By ~ 1

Author(s):

Yuxin Yao ◽

Liang Han ◽

Jiangyun Wang

Keyword(s):

Particle Swarm Optimization ◽

Short Term Memory ◽

Particle Swarm ◽

Ship Motion ◽

Motion Prediction ◽

Short Term ◽

Swarm Optimization ◽

Term Memory ◽

Long Short Term Memory

Download Full-text

A Novel LSTM for Multivariate Time Series with Massive Missingness

Sensors ◽

10.3390/s20102832 ◽

2020 ◽

Vol 20 (10) ◽

pp. 2832

Author(s):

Nazanin Fouladgar ◽

Kary Främling

Keyword(s):

Time Series ◽

Short Term Memory ◽

Time Series Prediction ◽

Multivariate Time Series ◽

Streaming Data ◽

Short Term ◽

Time Intervals ◽

Long Short Term Memory ◽

Real World Datasets ◽

Novel Model

Multivariate time series with missing data is ubiquitous when the streaming data is collected by sensors or any other recording instruments. For instance, the outdoor sensors gathering different meteorological variables may encounter low material sensitivity to specific situations, leading to incomplete information gathering. This is problematic in time series prediction with massive missingness and different missing rate of variables. Contribution addressing this problem on the regression task of meteorological datasets by employing Long Short-Term Memory (LSTM), capable of controlling the information flow with its memory unit, is still missing. In this paper, we propose a novel model called forward and backward variable-sensitive LSTM (FBVS-LSTM) consisting of two decay mechanisms and some informative data. The model inputs are mainly the missing indicator, time intervals of missingness in both forward and backward direction and missing rate of each variable. We employ this information to address the so-called missing not at random (MNAR) mechanism. Separately learning the features of each parameter, the model becomes adapted to deal with massive missingness. We conduct our experiment on three real-world datasets for the air pollution forecasting. The results demonstrate that our model performed well along with other LSTM-derivation models in terms of prediction accuracy.

Download Full-text

Trajectory prediction based on long short-term memory network and Kalman filter using hurricanes as an example

Computational Geosciences ◽

10.1007/s10596-021-10037-2 ◽

2021 ◽

Vol 25 (3) ◽

pp. 1005-1023

Author(s):

Wanting Qin ◽

Jun Tang ◽

Cong Lu ◽

Songyang Lao

Keyword(s):

Kalman Filter ◽

Moving Objects ◽

Short Term Memory ◽

Effective Means ◽

Prediction Algorithm ◽

Trajectory Data ◽

Trajectory Prediction ◽

Short Term ◽

Term Memory ◽

Long Short Term Memory

AbstractTrajectory data can objectively reflect the moving law of moving objects. Therefore, trajectory prediction has high application value. Hurricanes often cause incalculable losses of life and property, trajectory prediction can be an effective means to mitigate damage caused by hurricanes. With the popularization and wide application of artificial intelligence technology, from the perspective of machine learning, this paper trains a trajectory prediction model through historical trajectory data based on a long short-term memory (LSTM) network. An improved LSTM (ILSTM) trajectory prediction algorithm that improves the prediction of the simple LSTM is proposed, and the Kalman filter is used to filter the prediction results of the improved LSTM algorithm, which is called LSTM-KF. Through simulation experiments of Atlantic hurricane data from 1851 to 2016, compared to other LSTM and ILSTM algorithms, it is found that the LSTM-KF trajectory prediction algorithm has the lowest prediction error and the best prediction effect.

Download Full-text

MR-GNN: Multi-Resolution and Dual Graph Neural Network for Predicting Structured Entity Interactions

Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence ◽

10.24963/ijcai.2019/551 ◽

2019 ◽

Author(s):

Nuo Xu ◽

Pinghui Wang ◽

Long Chen ◽

Jing Tao ◽

Junzhou Zhao

Keyword(s):

Neural Network ◽

Short Term Memory ◽

Dual Graph ◽

Material Design ◽

Drug Regimen ◽

Short Term ◽

New Material ◽

Long Short Term Memory ◽

Real World Datasets ◽

Graph Neural Networks

Predicting interactions between structured entities lies at the core of numerous tasks such as drug regimen and new material design. In recent years, graph neural networks have become attractive. They represent structured entities as graphs, and then extract features from each individual graph using graph convolution operations. However, these methods have some limitations: i) their networks only extract features from a fix-sized subgraph structure (i.e., a fix-sized receptive field) of each node, and ignore features in substructures of different sizes, and ii) features are extracted by considering each entity independently, which may not effectively reflect the interaction between two entities. To resolve these problems, we present {\em MR-GNN}, an end-to-end graph neural network with the following features: i) it uses a multi-resolution based architecture to extract node features from different neighborhoods of each node, and, ii) it uses dual graph-state long short-term memory networks (LSTMs) to summarize local features of each graph and extracts the interaction features between pairwise graphs. Experiments conducted on real-world datasets show that MR-GNN improves the prediction of state-of-the-art methods.

Download Full-text

The Deep 3D Convolutional Multi-Branching Spatial-Temporal-Based Unit Predicting Citywide Traffic Flow

Applied Sciences ◽

10.3390/app10217778 ◽

2020 ◽

Vol 10 (21) ◽

pp. 7778 ◽

Cited By ~ 1

Author(s):

Zain Ul Abideen ◽

Heli Sun ◽

Zhou Yang ◽

Amir Ali

Keyword(s):

Traffic Flow ◽

Traffic Congestion ◽

Traffic Management ◽

Short Term Memory ◽

Time Estimation ◽

External Factors ◽

Short Term ◽

Term Memory ◽

Long Short Term Memory ◽

Real World Datasets

Recently, for public safety and traffic management, traffic flow prediction is a crucial task. The citywide traffic flow problem is still a big challenge in big cities because of many complex factors. However, to handle some complex factors, e.g., spatial-temporal and some external factors in the intelligent traffic flow forecasting problem, spatial-temporal data for urban applications (i.e., travel time estimation, trajectory planning, taxi demand, traffic congestion, and the regional rainfall) is inherently stochastic and unpredictable. In this paper, we proposed a deep learning-based novel model called “multi-branching spatial-temporal attention-based long-short term memory residual unit (MBSTALRU)” for the citywide traffic flow from lower-level layers to high-level layers, simultaneously. In our work, initially, we have modeled the traffic flow with spatial correlations multiple 3D volume layers and propose the novel multi-branching scheme to control the spatial-temporal features. Our approach is useful for exploring temporal dependencies through the 3D convolutional neural network (CNN) multiple branches, which aim to merge the spatial-temporal characteristics of historical data with three-time intervals, namely closeness, daily, and weekly, and we have embedded features by attention-based long-short term memory (LSTM). Then, we capture the correlation between traffic inflow and outflow with residual layers units. In the end, we merge the external factors dynamically to predict citywide traffic flow simultaneously. The simulation results have been performed on two real-world datasets, BJTaxi and NYCBike, which show better performance and effectiveness of the proposed method than previous state-of-the-art models.

Download Full-text

A Method of Ground-Based Cloud Motion Predict: CCLSTM + SR-Net

Remote Sensing ◽

10.3390/rs13193876 ◽

2021 ◽

Vol 13 (19) ◽

pp. 3876

Author(s):

Zhiying Lu ◽

Zehan Wang ◽

Xin Li ◽

Jianfeng Zhang

Keyword(s):

Short Term Memory ◽

Morphological Changes ◽

Motion Vector ◽

Prediction Method ◽

Super Resolution ◽

Motion Prediction ◽

Short Term ◽

Cloud Motion ◽

Atmospheric Radiation Measurement ◽

Long Short Term Memory

Ground-based cloud images can provide information on weather and cloud conditions, which play an important role in cloud cover monitoring and photovoltaic power generation forecasting. However, the cloud motion prediction of ground-based cloud images still lacks advanced and complete methods, and traditional technologies based on image processing and motion vector calculation are difficult to predict cloud morphological changes. In this paper, we propose a cloud motion prediction method based on Cascade Causal Long Short-Term Memory (CCLSTM) and Super-Resolution Network (SR-Net). Firstly, CCLSTM is used to estimate the shape and speed of cloud motion. Secondly, the Super-Resolution Network is built based on perceptual losses to reconstruct the result of CCLSTM and, finally, make it clearer. We tested our method on Atmospheric Radiation Measurement (ARM) Climate Research Facility TSI (total sky imager) images. The experiments showed that the method is able to predict the sky cloud changes in the next few steps.

Download Full-text