A quadrotor with tiltable rotors is a kind of omnidirectional multirotor aerial vehicle (MAV) that has demonstrated advantages of decoupling control of position from the control of orientation. However, quadrotors with tiltable rotors usually suffer from Coriolis term, modeling error and external disturbance. To this end, the extended state observer (ESO)-based controller is designed to estimate and compensate for the above adverse effects. Especially, the controller involves position and attitude controller in parallel. The attitude controller is made up of cascade control-loops: an outer quaternion-based attitude control-loop and an inner ESO-based Proportional derivative angular velocity control-loop. Similarly, the position controller consists of an outer proportional position control-loop and an inner ESO-based PD velocity control-loop. Besides, a linear control allocation strategy, which allocates the controller outputs to tilting angles and motor speed directly, is proposed to avoid the nonlinear allocation matrix. Extensive simulations and flight tests are carried out to illustrate the effectiveness and robustness of the proposed ESO-based controller.