Subarray-based hybrid beamforming (SBHB) systems represent an efficient architectural solution to realize massive multiple-input multiple-output (MIMO) systems. Thereby, the beamforming is no longer realized exclusively in the digital domain, but in coordination with an analog beamforming network by connecting each digital channel to a dedicated number of antenna elements via phase shifters. Most channel estimation and beamforming algorithms for SBHB systems are based on beam steering approaches, demanding a precise calibration of the analog beamforming network. In this letter, the SBHB system is calibrated by over-the-air (OTA) scattering parameter measurements and two criteria to optimize the beam pattern are formulated exploiting the ambiguities of the phase shifters. Both optimization criteria are evaluated using a designed SBHB receiver system operating at 27.8 GHz.