A bending-torsional coupled nonlinear dynamic model which contains the modification parameters of herringbone planetary gear train is presented. A formula of modification incentive is analyzed and deduced. The impact of the straight line and parabolic modification parameters on the amplitude of system transmission error is researched. The optimum modification parameters are acquired according to the minimum amplitude of system transmission error. Different amplitudes of the system transmission error, before and after modification, are compared at different rotational speed. The results indicate that the straight line modification parameters on the amplitude of system transmission error are more sensitive. Modification parameters on the amplitude of system transmission error are researched. When the length of the modification is specified, the amplitude of system transmission error is reduced sharply at first, then increased rapidly with the maximum magnitude of the modification increasing; When the maximum magnitude of the modification is specified, the amplitude of system transmission error is increased weakly at first, then decreased sharply, and increased rapidly in the end, with the length of the modification increasing. The modification parameters could form a crescent-shaped zone which can reduce the system transmission error amplitude significantly. The amplitudes of the system transmission error with modification are all reduced at different rotational speed, especially when there is a sympathetic vibration.