Considering the factors such as backlash, time-varying mesh stiffness and excitation errors, a three-degree-of-freedom torsional vibration model of the gear transmission system is established, in which the wheel/rail adhesion torque is considered as uncertain but bounded parameter.

Ishikawa method is used for analysis and calculation of the time-varying mesh stiffness of the gear pair in meshing process, the curve-fitted stiffness equation with respect to dimensionless time is obtained.

It is shown that small parameter uncertainties might be propagated in the vibration system and lead to relatively large uncertainties of the dynamic response of system.

With the increasing of rotating speed and stiffness of pinion, the gear transmission system is prone to involve one-period, multi-periodic and chaotic state. When the system gets into chaos state, ½ and 1 frequency doubling components are increased except the integer frequency multiplications, and the degree of the gear collision becomes fierce, which leads to the instability and failure of gear pair.

In order to reduce the impact and increase stability, the traction speed, gear motor ratio (number of teeth) and stiffness of the gear are theoretically taken into account based on matching and optimization principles for the locomotive gear system.