Predicting Human Mobility with Reinforcement-Learning-Based Long-Term Periodicity Modeling

2021 ◽  
Vol 12 (6) ◽  
pp. 1-23
Author(s):  
Shuo Tao ◽  
Jingang Jiang ◽  
Defu Lian ◽  
Kai Zheng ◽  
Enhong Chen
Keyword(s):  
Reinforcement Learning ◽  
Human Mobility ◽  
Recurrent Network ◽  
Mobility Prediction ◽  
Learning Framework ◽  
Temporal Features ◽  
Wide Range ◽  
Spatio Temporal ◽  
Historical Trajectory

Mobility prediction plays an important role in a wide range of location-based applications and services. However, there are three problems in the existing literature: (1) explicit high-order interactions of spatio-temporal features are not systemically modeled; (2) most existing algorithms place attention mechanisms on top of recurrent network, so they can not allow for full parallelism and are inferior to self-attention for capturing long-range dependence; (3) most literature does not make good use of long-term historical information and do not effectively model the long-term periodicity of users. To this end, we propose MoveNet and RLMoveNet. MoveNet is a self-attention-based sequential model, predicting each user’s next destination based on her most recent visits and historical trajectory. MoveNet first introduces a cross-based learning framework for modeling feature interactions. With self-attention on both the most recent visits and historical trajectory, MoveNet can use an attention mechanism to capture the user’s long-term regularity in a more efficient way. Based on MoveNet, to model long-term periodicity more effectively, we add the reinforcement learning layer and named RLMoveNet. RLMoveNet regards the human mobility prediction as a reinforcement learning problem, using the reinforcement learning layer as the regularization part to drive the model to pay attention to the behavior with periodic actions, which can help us make the algorithm more effective. We evaluate both of them with three real-world mobility datasets. MoveNet outperforms the state-of-the-art mobility predictor by around 10% in terms of accuracy, and simultaneously achieves faster convergence and over 4x training speedup. Moreover, RLMoveNet achieves higher prediction accuracy than MoveNet, which proves that modeling periodicity explicitly from the perspective of reinforcement learning is more effective.

scholarly journals A Reinforcement Learning Framework for Spiking Networks with Dynamic Synapses

2011 ◽  
Vol 2011 ◽  
pp. 1-12 ◽  
Author(s):  
Karim El-Laithy ◽  
Martin Bogdan
Keyword(s):  
Reinforcement Learning ◽  
Spike Timing ◽  
Neural Representation ◽  
Model Parameters ◽  
Learning Framework ◽  
Reference Target ◽  
Wide Range ◽  
Spiking Network ◽  
Dynamic Synapses ◽  
Exclusive Or

An integration of both the Hebbian-based and reinforcement learning (RL) rules is presented for dynamic synapses. The proposed framework permits the Hebbian rule to update the hidden synaptic model parameters regulating the synaptic response rather than the synaptic weights. This is performed using both the value and the sign of the temporal difference in the reward signal after each trial. Applying this framework, a spiking network with spike-timing-dependent synapses is tested to learn the exclusive-OR computation on a temporally coded basis. Reward values are calculated with the distance between the output spike train of the network and a reference target one. Results show that the network is able to capture the required dynamics and that the proposed framework can reveal indeed an integrated version of Hebbian and RL. The proposed framework is tractable and less computationally expensive. The framework is applicable to a wide class of synaptic models and is not restricted to the used neural representation. This generality, along with the reported results, supports adopting the introduced approach to benefit from the biologically plausible synaptic models in a wide range of intuitive signal processing.

scholarly journals Geocoding and spatio-temporal modeling of long-term PM2.5 and NO2 exposure: the Mexican Teachers´ Cohort

2021 ◽  
Author(s):  
KARLA CERVANTES-MARTINEZ ◽  
HORACIO RIOJAS-RODRÍGUEZ ◽  
CARLOS DÍAZ-AVALOS ◽  
HORTENSIA MORENO-MACÍAS ◽  
RUY LÓPEZ-RIDAURA ◽  
...  
Keyword(s):  
Predictive Accuracy ◽  
Human Mobility ◽  
Epidemiological Studies ◽  
Additive Models ◽  
Secondary Source ◽  
Source Information ◽  
Promising Alternative ◽  
Temporal Models ◽  
Spatio Temporal

Epidemiological studies on air pollution in Mexico often use the environmental concentrations of pollutants as measured by monitors closest to the home of participants as exposure proxies, yet this approach does not account for the space gradients of pollutants and ignores intra-city human mobility. This study aimed to develop high-resolution spatial and temporal models for predicting long-term exposure to PM2.5 and NO2 in ~16,500 participants from the Mexican Teachers’ Cohort study. We geocoded the home and work addresses of participants, and used secondary source information on geographical and meteorological variables as well as other pollutants to fit two generalized additive models capable of predicting monthly PM2.5 and NO2 concentrations during the 2004-2019 period. Both models were evaluated through 10-fold cross-validation, and showed high predictive accuracy with out-of-sample data and no overfitting (CV-RMSE=0.102 for PM2.5 and CV-RMSE=4.497 for NO2). Participants were exposed to a monthly average of 24.38 (6.78) mg/m3 of PM2.5 and 28.21 (8.00) ppb of NO2 during the study period. These models offer a promising alternative for estimating PM2.5 and NO2 exposure with high spatio-temporal resolution for epidemiological studies in the Mexico City Metropolitan Area.

HERMAS

2021 ◽  
Vol 5 (3) ◽  
pp. 1-21
Author(s):  
Yiwei Song ◽  
Dongzhe Jiang ◽  
Yunhuai Liu ◽  
Zhou Qin ◽  
Chang Tan ◽  
...  
Keyword(s):  
Large Scale ◽  
Human Mobility ◽  
Representation Learning ◽  
User Profiling ◽  
Similarity Measurement ◽  
Learning Framework ◽  
Computation Efficiency ◽  
Trajectory Similarity ◽  
Spatio Temporal ◽  
Embedding Performance

Efficient representations for spatio-temporal cellular Signaling Data (SD) are essential for many human mobility applications. Traditional representation methods are mainly designed for GPS data with high spatio-temporal continuity, and thus will suffer from poor embedding performance due to the unique Ping Pong Effect in SD. To address this issue, we explore the opportunity offered by a large number of human mobility traces and mine the inherent neighboring tower connection patterns. More specifically, we design HERMAS, a novel representation learning framework for large-scale cellular SD with three steps: (1) extract rich context information in each trajectory, adding neighboring tower information as extra knowledge in each mobility observation; (2) design a sequence encoding model to aggregate the embedding of each observation; (3) obtain the embedding for a trajectory. We evaluate the performance of HERMAS based on two human mobility applications, i.e. trajectory similarity measurement and user profiling. We conduct evaluations based on a 30-day SD dataset with 130,612 users and 2,369,267 moving trajectories. Experimental results show that (1) for the trajectory similarity measurement application, HERMAS improves the Hitting Rate (HR@10) from 15.2% to 39.2%; (2) for the user profiling application, HERMAS improves the F1-score for around 9%. More importantly, HERMAS significantly improves the computation efficiency by over 30x.

scholarly journals Learning Compact Spatio-Temporal Features for Fast Content based Video Retrieval

2019 ◽  
Vol 9 (2) ◽  
pp. 2404-2409
Keyword(s):  
Video Retrieval ◽  
Digital Cameras ◽  
Video Browsing ◽  
Content Filtering ◽  
Effective Learning ◽  
Temporal Features ◽  
Wide Range ◽  
Spatio Temporal ◽  
Two Stages ◽  
Content Based Video Retrieval

Videos are recorded and uploaded daily to the sites like YouTube, Facebook etc. from devices such as mobile phones and digital cameras with less or without metadata (semantic tags) associated with it. This makes extremely difficult to retrieve similar videos based on this metadata without using content based semantic search. Content based video retrieval is problem of retrieving most similar videos to a given query video and has wide range of applications such as video browsing, content filtering, video indexing, etc. Traditional video level features based on key frame level hand engineered features which does not exploit rich dynamics present in the video. In this paper we propose a fast content based video retrieval framework using compact spatio-temporal features learned by deep learning. Specifically, deep CNN along with LSTM is deploy to learn spatio-temporal representations of video. For fast retrieval, binary code is generated by hashing learning component in the framework. For fast and effective learning of hash code proposed framework is trained in two stages. First stage learns the video dynamics and in second stage compact code is learn using learned video’s temporal variation from the first stage. UCF101 dataset is used to test the proposed method and results compared by other hashing methods. Results show that our approach is able to improve the performance over existing methods.

scholarly journals The impact of human mobility data scales and processing on movement predictability

2021 ◽  
Vol 11 (1) ◽  
Author(s):  
Kamil Smolak ◽  
Katarzyna Siła-Nowicka ◽  
Jean-Charles Delvenne ◽  
Michał Wierzbiński ◽  
Witold Rohm
Keyword(s):  
Prediction Models ◽  
Human Mobility ◽  
Reference Value ◽  
Human Movement ◽  
Temporal Data ◽  
Movement Trajectories ◽  
Mobility Data ◽  
Wide Range ◽  
Spatio Temporal ◽  
The Impact

AbstractPredictability of human movement is a theoretical upper bound for the accuracy of movement prediction models, which serves as a reference value showing how regular a dataset is and to what extent mobility can be predicted. Over the years, the predictability of various human mobility datasets was found to vary when estimated for differently processed datasets. Although attempts at the explanation of this variability have been made, the extent of these experiments was limited. In this study, we use high-precision movement trajectories of individuals to analyse how the way we represent the movement impacts its predictability and thus, the outcomes of analyses made on these data. We adopt a number of methods used in the last 11 years of research on human mobility and apply them to a wide range of spatio-temporal data scales, thoroughly analysing changes in predictability and produced data. We find that spatio-temporal resolution and data processing methods have a large impact on the predictability as well as geometrical and numerical properties of human mobility data, and we present their nonlinear dependencies.

scholarly journals A Multi-Agent Reinforcement Learning Framework for Lithium-ion Battery Scheduling Problems

Energies ◽  
2020 ◽  
Vol 13 (8) ◽  
pp. 1982
Author(s):  
Yu Sui ◽  
Shiming Song
Keyword(s):  
Reinforcement Learning ◽  
Scheduling Algorithm ◽  
Lithium Ion ◽  
Thermal Control ◽  
Use Cases ◽  
Limiting Factor ◽  
Scheduling Problems ◽  
Battery Lifetime ◽  
Learning Framework ◽  
Wide Range

This paper presents a reinforcement learning framework for solving battery scheduling problems in order to extend the lifetime of batteries used in electrical vehicles (EVs), cellular phones, and embedded systems. Battery pack lifetime has often been the limiting factor in many of today’s smart systems, from mobile devices and wireless sensor networks to EVs. Smart charge-discharge scheduling of battery packs is essential to obtain super linear gain of overall system lifetime, due to the recovery effect and nonlinearity in the battery characteristics. Additionally, smart scheduling has also been shown to be beneficial for optimizing the system’s thermal profile and minimizing chances of irreversible battery damage. The recent rapidly-growing community and development infrastructure have added deep reinforcement learning (DRL) to the available tools for designing battery management systems. Through leveraging the representation powers of deep neural networks and the flexibility and versatility of reinforcement learning, DRL offers a powerful solution to both roofline analysis and real-world deployment on complicated use cases. This work presents a DRL-based battery scheduling framework to solve battery scheduling problems, with high flexibility to fit various battery models and application scenarios. Through the discussion of this framework, comparisons have also been made between conventional heuristics-based methods and DRL. The experiments demonstrate that DRL-based scheduling framework achieves battery lifetime comparable to the best weighted-k round-robin (kRR) heuristic scheduling algorithm. In the meantime, the framework offers much greater flexibility in accommodating a wide range of battery models and use cases, including thermal control and imbalanced battery.

An adaptive deep reinforcement learning framework enables curling robots with human-like performance in real-world conditions

2020 ◽  
Vol 5 (46) ◽  
pp. eabb9764
Author(s):  
Dong-Ok Won ◽  
Klaus-Robert Müller ◽  
Seong-Whan Lee
Keyword(s):  
Reinforcement Learning ◽  
Real World ◽  
Test Bed ◽  
Good Test ◽  
The Real ◽  
Learning Framework ◽  
Temporal Features ◽  
Level Performance ◽  
Rules Of The Game ◽  
Timing Rules

The game of curling can be considered a good test bed for studying the interaction between artificial intelligence systems and the real world. In curling, the environmental characteristics change at every moment, and every throw has an impact on the outcome of the match. Furthermore, there is no time for relearning during a curling match due to the timing rules of the game. Here, we report a curling robot that can achieve human-level performance in the game of curling using an adaptive deep reinforcement learning framework. Our proposed adaptation framework extends standard deep reinforcement learning using temporal features, which learn to compensate for the uncertainties and nonstationarities that are an unavoidable part of curling. Our curling robot, Curly, was able to win three of four official matches against expert human teams [top-ranked women’s curling teams and Korea national wheelchair curling team (reserve team)]. These results indicate that the gap between physics-based simulators and the real world can be narrowed.

Sign in / Sign up

Export Citation Format

Share Document