PurposeThis paper aims to provide a mathematical modeling and design of H-infinity controller for an autonomous vertical take-off and landing (VTOL) Quad Tiltrotor hybrid unmanned aerial vehicles (UAVs). The variation in the aerodynamics and model dynamics of these aerial vehicles due to its tilting rotors are the key issues and challenges, which attracts the attention of many researchers. They carry parametric uncertainties (such as non-linear friction force, backlash, etc.), which drives the designed controller based on the nominal model to instability or performance degradation. The controller needs to take these factors into consideration and still give good stability and performance. Hence, a robust H-infinity controller is proposed that can handle these uncertainties.Design/methodology/approachA unique VTOL Quad Tiltrotor hybrid UAV, which operates in three flight modes, is mathematically modeled using Newton–Euler equations of motion. The contribution of the model is its ability to combine high-speed level flight, VTOL and transition between these two phases. The transition involves the tilting of the proprotors from 90° to 0° and vice-versa in 15° intervals. A robust H-infinity control strategy is proposed, evaluated and analyzed through simulation to control the flight dynamics for different modes of operation.FindingsThe main contribution of this research is the mathematical modeling of three flight modes (vertical takeoff–forward, transition–cruise-back, transition-vertical landing) of operation by controlling the revolutions per minute and tilt angles, which are independent of each other. An autonomous flight control system using a robust H-infinity controller to stabilize the mode of transition is designed for the Quad Tiltrotor UAV in the presence of uncertainties, noise and disturbances using MATLAB/SIMULINK. This paper focused on improving the disturbance rejection properties of the proposed UAV by designing a robust H-infinity controller for position and orientation trajectory regulation in the presence of uncertainty. The simulation results show that the Tiltrotor achieves transition successfully with disturbances, noise and uncertainties being present.Originality/valueA novel VTOL Quad Tiltrotor UAV mathematical model is developed with a special tilting rotor mechanism, which combines both aircraft and helicopter flight modes with the transition taking place in between phases using robust H-infinity controller for attitude, altitude and trajectory regulation in the presence of uncertainty.