In this article, we propose a physical layer security (PLS) technique, namely security-aware spatial modulation (SA-SM), in a multiple-input multiple-output-based heterogeneous network, wherein both optical wireless communications and radio-frequency (RF) technologies coexist. In SA-SM, the time-domain signal is altered prior to transmission using a key at the physical layer for combating eavesdropping. Unlike conventional PLS techniques, SA-SM does not rely on channel characteristics for securing the information, as its perception is self-imposed, which allows its adoption in radio-optical networks. Additionally, a novel periodical key selection algorithm is proposed. Instead of having multiple keys stored in the nodes, by using off-the-shelf and low-complexity machine learning (ML) methods, including a support vector machine, logistic regression and a single-layer neural network, SA-SM nodes can estimate the used key. Results show that a positive secrecy capacity can be achieved for both the RF and optical links by using 1000 different keys, with a minimal signal-to-noise ratio penalty of less than 5 dB for the legitimate user using SA-SM versus conventional transmission at a bit-error-rate of 10
−4
. The analysis also includes computational time and classification accuracy evaluation of the various proposed ML techniques using different hardware architectures.
Experimental Study of Machine-Learning-Based Detection and Identification of Physical-Layer Attacks in Optical Networks