Intelligent Traffic Signal Control Based on Reinforcement Learning with State Reduction for Smart Cities

2021 ◽  
Vol 21 (4) ◽  
pp. 1-24
Author(s):  
Li Kuang ◽  
Jianbo Zheng ◽  
Kemu Li ◽  
Honghao Gao
Keyword(s):  
Reinforcement Learning ◽  
Traffic Flow ◽  
Control Algorithm ◽  
The State ◽  
Signal Control ◽  
Traffic Signal Control ◽  
Signal Timing ◽  
State Reduction ◽  
Traffic Demand ◽  
Matching Degree

Efficient signal control at isolated intersections is vital for relieving congestion, accidents, and environmental pollution caused by increasing numbers of vehicles. However, most of the existing studies not only ignore the constraint of the limited computing resources available at isolated intersections but also the matching degree between the signal timing and the traffic demand, leading to high complexity and reduced learning efficiency. In this article, we propose a traffic signal control method based on reinforcement learning with state reduction. First, a reinforcement learning model is established based on historical traffic flow data, and we propose a dual-objective reward function that can reduce vehicle delay and improve the matching degree between signal time allocation and traffic demand, allowing the agent to learn the optimal signal timing strategy quickly. Second, the state and action spaces of the model are preliminarily reduced by selecting a proper control phase combination; then, the state space is further reduced by eliminating rare or nonexistent states based on the historical traffic flow. Finally, a simplified Q-table is generated and used to optimize the complexity of the control algorithm. The results of simulation experiments show that our proposed control algorithm effectively improves the capacity of isolated intersections while reducing the time and space costs of the signal control algorithm.

Reinforcement learning vs. rule-based adaptive traffic signal control: A Fourier basis linear function approximation for traffic signal control

2021 ◽  
pp. 1-15
Author(s):  
Theresa Ziemke ◽  
Lucas N. Alegre ◽  
Ana L.C. Bazzan
Keyword(s):  
Reinforcement Learning ◽  
State Space ◽  
Function Approximation ◽  
Traffic Signals ◽  
The State ◽  
Signal Control ◽  
Traffic Signal Control ◽  
Rule Based ◽  
Fourier Basis ◽  
Linear Function Approximation

Reinforcement learning is an efficient, widely used machine learning technique that performs well when the state and action spaces have a reasonable size. This is rarely the case regarding control-related problems, as for instance controlling traffic signals. Here, the state space can be very large. In order to deal with the curse of dimensionality, a rough discretization of such space can be employed. However, this is effective just up to a certain point. A way to mitigate this is to use techniques that generalize the state space such as function approximation. In this paper, a linear function approximation is used. Specifically, SARSA ( λ ) with Fourier basis features is implemented to control traffic signals in the agent-based transport simulation MATSim. The results are compared not only to trivial controllers such as fixed-time, but also to state-of-the-art rule-based adaptive methods. It is concluded that SARSA ( λ ) with Fourier basis features is able to outperform such methods, especially in scenarios with varying traffic demands or unexpected events.

scholarly journals A MULTI-INTERSECTION MODEL AND SIGNAL TIMING PLAN ALGORITHM FOR URBAN TRAFFIC SIGNAL CONTROL

Transport ◽  
2014 ◽  
Vol 32 (4) ◽  
pp. 368-378 ◽  
Author(s):  
Wenbin Hu ◽  
Huan Wang ◽  
Bo Du ◽  
Liping Yan
Keyword(s):  
Real Time ◽  
Traffic Flow ◽  
Delay Time ◽  
Urban Traffic ◽  
Signal Control ◽  
Traffic Signal Control ◽  
Signal Timing ◽  
Traffic Network ◽  
Intersection Model ◽  
Volume Algorithm

The urban traffic signal control system is complex, non-linear and non-equilibrium in real conditions. The existing methods could not satisfy the requirement of real-time and dynamic control. In order to solve these difficulties and challenges, this paper proposes a novel Multi-Intersection Model (MIM) based on Cellular Automata (CA) and a Multi-Intersection Signal Timing Plan Algorithm (MISTPA), which can reduce the delay time at each intersection and effectively alleviate the traffic pressure on each intersection in the urban traffic network. Our work is divided into several parts: (1) a multi-intersection model based on CA is defined to build the dynamic urban traffic network; (2) MISTPA is proposed, which truly reflects the real-time demand degree to green time of the traffic flow at each intersection. The MISTPA is composed Single Intersection Volume Algorithm (SIVA), Single-Lane Volume Algorithm (SLVA) and single intersection signal timing plan algorithm (SISTPA). Extensive experiments show that when the saturation is greater than 0.3, the MIM and the MISTPA achieve good performance, and can significantly reduce the vehicle delay time at each intersection. The average delay time of the traffic flow at each intersection can obviously be reduced. Finally, a practical case study demonstrates that the proposed model and the corresponding algorithm are correct and effective.

scholarly journals Cooperative Traffic Signal Control with Traffic Flow Prediction in Multi-Intersection

Sensors ◽  
2019 ◽  
Vol 20 (1) ◽  
pp. 137 ◽  
Author(s):  
Daeho Kim ◽  
Okran Jeong
Keyword(s):  
Reinforcement Learning ◽  
Traffic Flow ◽  
Learning Algorithm ◽  
Traffic Signal ◽  
Traffic Information ◽  
Signal Control ◽  
Traffic Signal Control ◽  
Traffic Flow Prediction ◽  
Traffic Conditions ◽  
Flow Prediction

As traffic congestion in cities becomes serious, intelligent traffic signal control has been actively studied. Deep Q-Network (DQN), a representative deep reinforcement learning algorithm, is applied to various domains from fully-observable game environment to traffic signal control. Due to the effective performance of DQN, deep reinforcement learning has improved speeds and various DQN extensions have been introduced. However, most traffic signal control researches were performed at a single intersection, and because of the use of virtual simulators, there are limitations that do not take into account variables that affect actual traffic conditions. In this paper, we propose a cooperative traffic signal control with traffic flow prediction (TFP-CTSC) for a multi-intersection. A traffic flow prediction model predicts future traffic state and considers the variables that affect actual traffic conditions. In addition, for cooperative traffic signal control in multi-intersection, each intersection is modeled as an agent, and each agent is trained to take best action by receiving traffic states from the road environment. To deal with multi-intersection efficiently, agents share their traffic information with other adjacent intersections. In the experiment, TFP-CTSC is compared with existing traffic signal control algorithms in a 4 × 4 intersection environment. We verify our traffic flow prediction and cooperative method.

Control and Coordination of Self-Adaptive Traffic Signal Using Deep Reinforcement Learning

2021 ◽  
Vol 9 (1) ◽  
pp. 373-379
Author(s):  
Pallavi Mandhare, Dr. Jyoti Yadav, Prof. Vilas Kharat, Prof. C.Y. Patil
Keyword(s):  
Reinforcement Learning ◽  
Traffic Flow ◽  
Traffic Signals ◽  
Traffic Signal ◽  
Traffic Signal Control ◽  
Sustainable Mobility ◽  
Traffic Demand ◽  
Traffic Systems ◽  
Traffic Signal Timing ◽  
Traffic Signal Control Systems

The most observable obstacle to sustainable mobility is traffic congestions. These congestions cannot effectively be fixed by traditional control of traffic signals. Safe and smooth movement of traffic is ensured by a self-controlled traffic signal. As such, to coordinate the traffic flow it is necessary to implement dynamic traffic signal subsequences. Primarily, Traffic Signal Controllers (TSC) provides sophisticated control and coordination of vehicles. The control and coordination of traffic signal control systems can be effectively achieved by implementing the Deep Reinforcement Learning (DRL) approaches. The decision-making capabilities at intersections are improved by having variations of traffic signal timing using an adaptive TSC. Alternatively, the actual traffic demand is nothing but managing the traffic systems. It analyses the incoming number and type of vehicles and gives a real-time response at intersection geometrics and controls the traffic signals accordingly. The proposed DRL algorithm observes traffic data and operates optimum management plans for the regulation of the traffic flow. Furthermore, an existing traffic simulator is used to help provide a realistic environment to support the proposed algorithm.  

scholarly journals Dynamic Lane Traffic Signal Control with Group Attention and Multi-Timescale Reinforcement Learning

2021 ◽  
Author(s):  
Qize Jiang ◽  
Jingze Li ◽  
Weiwei Sun ◽  
Baihua Zheng
Keyword(s):  
Reinforcement Learning ◽  
Traffic Flow ◽  
Learning Algorithms ◽  
Traffic Signal ◽  
Signal Control ◽  
Traffic Signal Control ◽  
Traffic Flow Prediction ◽  
Flow Prediction ◽  
Model Training ◽  
Proper Strategy

Traffic signal control has achieved significant success with the development of reinforcement learning. However, existing works mainly focus on intersections with normal lanes with fixed outgoing directions. It is noticed that some intersections actually implement dynamic lanes, in addition to normal lanes, to adjust the outgoing directions dynamically. Existing methods fail to coordinate the control of traffic signal and that of dynamic lanes effectively. In addition, they lack proper structures and learning algorithms to make full use of traffic flow prediction, which is essential to set the proper directions for dynamic lanes. Motivated by the ineffectiveness of existing approaches when controlling the traffic signal and dynamic lanes simultaneously, we propose a new method, namely MT-GAD, in this paper. It uses a group attention structure to reduce the number of required parameters and to achieve a better generalizability, and uses multi-timescale model training to learn proper strategy that could best control both the traffic signal and the dynamic lanes. The experiments on real datasets demonstrate that MT-GAD outperforms existing approaches significantly.

scholarly journals Estimating Signal Timing of Actuated Signal Control Using Pattern Recognition under Connected Vehicle Environment

2021 ◽  
Vol 33 (1) ◽  
pp. 153-163
Author(s):  
Ruochen Hao ◽  
Ling Wang ◽  
Wanjing Ma ◽  
Chunhui Yu
Keyword(s):  
Pattern Recognition ◽  
Traffic Flow ◽  
Quantitative Description ◽  
Estimation Method ◽  
Average Error ◽  
Signal Control ◽  
Signal Timing ◽  
Control Logic ◽  
Traffic Demand ◽  
Actuated Signal

The Signal Phase and Timing (SPaT) message is an important input for research and applications of Connected Vehicles (CVs). However, the actuated signal controllers are not able to directly give the SPaT information since the SPaT is influenced by both signal control logic and real-time traffic demand. This study elaborates an estimation method which is proposed according to the idea that an actuated signal controller would provide similar signal timing for similar traffic states. Thus, the quantitative description of traffic states is important. The traffic flow at each approaching lane has been compared to fluids. The state of fluids can be indicated by state parameters, e.g. speed or height, and its energy, which includes kinetic energy and potential energy. Similar to the fluids, this paper has proposed an energy model for traffic flow, and it has also added the queue length as an additional state parameter. Based on that, the traffic state of intersections can be descripted. Then, a pattern recognition algorithm was developed to identify the most similar historical states and also their corresponding SPaTs, whose average is the estimated SPaT of this second. The result shows that the average error is 3.1 seconds.

Recent Advances in Reinforcement Learning for Traffic Signal Control

2021 ◽  
Vol 22 (2) ◽  
pp. 12-18 ◽  
Author(s):  
Hua Wei ◽  
Guanjie Zheng ◽  
Vikash Gayah ◽  
Zhenhui Li
Keyword(s):  
Reinforcement Learning ◽  
Real World ◽  
Intelligent Transportation Systems ◽  
Transportation Systems ◽  
Traffic Signal ◽  
Signal Control ◽  
Traffic Signal Control ◽  
Control Methods ◽  
Advantages And Disadvantages ◽  
Recent Advances

Traffic signal control is an important and challenging real-world problem that has recently received a large amount of interest from both transportation and computer science communities. In this survey, we focus on investigating the recent advances in using reinforcement learning (RL) techniques to solve the traffic signal control problem. We classify the known approaches based on the RL techniques they use and provide a review of existing models with analysis on their advantages and disadvantages. Moreover, we give an overview of the simulation environments and experimental settings that have been developed to evaluate the traffic signal control methods. Finally, we explore future directions in the area of RLbased traffic signal control methods. We hope this survey could provide insights to researchers dealing with real-world applications in intelligent transportation systems

Sign in / Sign up

Export Citation Format

Share Document