Spiral bevel gears in the helicopter's main gearbox exhibit pronounced spatial meshing characteristics. Under the combined effects of complex loads as well as manufacturing and assembly errors, the key excitations such as mesh stiffness and transmission error are difficult to obtain online, which limits the high-accuracy prediction of vibration responses. To address this issue, a digital twin method for vibration prediction was proposed by integrating tooth surface contact analysis with mechanism-guided data-driven modeling. The proposed method adopted collaborative modeling along physical and data paths. On the physical path, a tooth surface with geometric errors was constructed based on the three-coordinate measurement data, and the tooth surface contact analysis was carried out to extract time-varying mesh stiffness and composite transmission error. A condition-related vibration intensity index was then obtained through torsional dynamics analysis and used as an interpretable mechanism-informed feature. On the data path, a Bayesian-optimized XGBoost (eXtreme Gradient Boosting) mapping model BO-XGBoost was constructed to fuse the measured operating parameters with the mechanism features for nonlinear prediction of three-dimensional vibration of the output-shaft. The verification results based on the test-rig data of the transmission system of the main gearbox demonstrated that the model had a higher prediction accuracy for the three-dimensional vibration of the output-shaft, with the determination coefficient R² being higher than 0.97. Compared with baseline models such as BO-XGBoost, SVR (support vector regression), LSTM (long short-term memory), and GRU (gated recurrent unit), its prediction accuracy was the highest. The research results provide a physically interpretable modeling approach for vibration monitoring and performance evaluation of the transmission system of the helicopter's main gearbox.