Self-organization is a key feature as cellular networks densify and become more heterogeneous,
through the additional small cells such as pico and femtocells. Self- organizing networks (SONs)
can perform self-configuration, self-optimization, and self-healing. These operations can cover
basic tasks such as the configuration of a newly installed base station, resource management, and
fault management in the network. In other words, SONs attempt to minimize human intervention where
they use measurements from the network to minimize the cost of installation, configuration, and
maintenance of the network. In fact, SONs aim to bring two main factors in play: intelligence and
autonomous adaptability. One of the main requirements for achieving such goals is to learn from
sensory data and signal measurements in networks. Therefore, machine learning techniques can play
a major role in processing underutilized sensory data to enhance the performance of SONs.
In the first part of this dissertation, we focus on reinforcement learning as a viable approach
for learning from signal measurements. We develop a general framework in heterogeneous cellular
networks agnostic to the learning approach. We design multiple reward functions and study different
effects of the reward function, Markov state model, learning rate, and cooperation methods on the
performance of reinforcement learning in cellular networks. Further, we look into the optimality of
reinforcement learning solutions and provide insights into how to achieve optimal solutions.
In the second part of the dissertation, we propose a novel architecture based on spatial indexing
for system-evaluation of heterogeneous 5G cellular networks. We develop an open-source platform based
on the proposed architecture that can be used to study large scale directional cellular networks. The
proposed platform is used for generating training data sets of accurate signal-to-interference-plus-noise-ratio
(SINR) values in millimeter-wave communications for machine learning purposes. Then, with taking advantage
of the developed platform, we look into dense millimeter-wave networks as one of the key technologies in
5G cellular networks. We focus on topology management of millimeter-wave backhaul networks and study and
provide multiple insights on the evaluation and selection of proper performance metrics in dense
millimeter-wave networks. Finally, we finish this part by proposing a self-organizing solution to
achieve k-connectivity via reinforcement learning in the topology management of wireless networks.
Inter-Operator Base Station Coordination in Spectrum-Shared Millimeter Wave Cellular Networks
