An Effective Spectrum Handoff Based on Reinforcement Learning for Target Channel Selection in the Industrial Internet of Things

The overcrowding of the wireless space has triggered a strict competition for scare network resources. Therefore, there is a need for a dynamic spectrum access (DSA) technique that will ensure fair allocation of the available network resources for diverse network elements competing for the network resources. Spectrum handoff (SH) is a DSA technique through which cognitive radio (CR) promises to provide effective channel utilization, fair resource allocation, as well as reliable and uninterrupted real-time connection. However, SH may consume extra network resources, increase latency, and degrade network performance if the spectrum sensing technique used is ineffective and the channel selection strategy (CSS) is poorly implemented. Therefore, it is necessary to develop an SH policy that holistically considers the implementation of effective CSS, and spectrum sensing technique, as well as minimizes communication delays. In this work, two reinforcement learning (RL) algorithms are integrated into the CSS to perform channel selection. The first algorithm is used to evaluate the channel future occupancy, whereas the second algorithm is used to determine the channel quality in order to sort and rank the channels in candidate channel list (CCL). A method of masking linearly dependent and useless state elements is implemented to improve the convergence of the learning. Our approach showed a significant reduction in terms of latency and a remarkable improvement in throughput performance in comparison to conventional approaches.

Download Full-text

Dynamic Cooperative Spectrum Sensing Based on Deep Multi-User Reinforcement Learning

Applied Sciences ◽

10.3390/app11041884 ◽

2021 ◽

Vol 11 (4) ◽

pp. 1884

Author(s):

Shuai Liu ◽

Jing He ◽

Jiayun Wu

Keyword(s):

Reinforcement Learning ◽

Spectrum Sensing ◽

Dynamic Spectrum Access ◽

Prior Information ◽

Cooperative Spectrum Sensing ◽

Spectrum Access ◽

Spectrum Utilization ◽

Channel Strategy ◽

Large State Space ◽

Network Utility

Dynamic spectrum access (DSA) has been considered as a promising technology to address spectrum scarcity and improve spectrum utilization. Normally, the channels are related to each other. Meanwhile, collisions will be inevitably caused by communicating between multiple PUs or multiple SUs in a real DSA environment. Considering these factors, the deep multi-user reinforcement learning (DMRL) is proposed by introducing the cooperative strategy into dueling deep Q network (DDQN). With no demand of prior information about the system dynamics, DDQN can efficiently learn the correlations between channels, and reduce the computational complexity in the large state space of the multi-user environment. To reduce the conflicts and further maximize the network utility, cooperative channel strategy is explored by utilizing the acknowledge (ACK) signals without exchanging spectrum information. In each time slot, each user selects a channel and transmits a packet with a certain probability. After sending, ACK signals are utilized to judge whether the transmission is successful or not. Compared with other popular models, the simulation results show that the proposed DMRL can achieve better performance on effectively enhancing spectrum utilization and reducing conflict rate in the dynamic cooperative spectrum sensing.

Download Full-text

Reinforcement Learning Based Dynamic Spectrum Access for Software-defined Cognitive Industrial Internet of Things

IEEE Transactions on Industrial Informatics ◽

10.1109/tii.2021.3113949 ◽

2021 ◽

pp. 1-1

Author(s):

Xin Liu ◽

Can Sun ◽

Wei Yu ◽

Mu Zhou

Keyword(s):

Reinforcement Learning ◽

Internet Of Things ◽

Dynamic Spectrum Access ◽

Dynamic Spectrum ◽

Spectrum Access ◽

Industrial Internet Of Things ◽

Industrial Internet

Download Full-text

Distributed Spectrum Management in Cognitive Radio Networks by Consensus-Based Reinforcement Learning

Sensors ◽

10.3390/s21092970 ◽

2021 ◽

Vol 21 (9) ◽

pp. 2970

Author(s):

Dejan Dašić ◽

Nemanja Ilić ◽

Miljan Vučetić ◽

Miroslav Perić ◽

Marko Beko ◽

...

Keyword(s):

Reinforcement Learning ◽

Cognitive Radio ◽

Spectrum Sensing ◽

Cognitive Radio Networks ◽

Channel Selection ◽

Spectrum Management ◽

Radio Networks ◽

Selection Strategy ◽

Distributed Spectrum Sensing ◽

The Individual

In this paper, we propose a new algorithm for distributed spectrum sensing and channel selection in cognitive radio networks based on consensus. The algorithm operates within a multi-agent reinforcement learning scheme. The proposed consensus strategy, implemented over a directed, typically sparse, time-varying low-bandwidth communication network, enforces collaboration between the agents in a completely decentralized and distributed way. The motivation for the proposed approach comes directly from typical cognitive radio networks’ practical scenarios, where such a decentralized setting and distributed operation is of essential importance. Specifically, the proposed setting provides all the agents, in unknown environmental and application conditions, with viable network-wide information. Hence, a set of participating agents becomes capable of successful calculation of the optimal joint spectrum sensing and channel selection strategy even if the individual agents are not. The proposed algorithm is, by its nature, scalable and robust to node and link failures. The paper presents a detailed discussion and analysis of the algorithm’s characteristics, including the effects of denoising, the possibility of organizing coordinated actions, and the convergence rate improvement induced by the consensus scheme. The results of extensive simulations demonstrate the high effectiveness of the proposed algorithm, and that its behavior is close to the centralized scheme even in the case of sparse neighbor-based inter-node communication.

Download Full-text