In order to address the problem that dynamic environmental changes affect the evaluation of cutting tools’ service state during milling, a tool wear evaluation method integrating direct and indirect measurements was proposed. Periodic tool end images were fused with real-time sound data, and the high precision of direct measurement and continuity of indirect measurement were combined to realize robust evaluation of the tool wear state during machining. To improve the evaluation effect, a multi-scale network model was constructed. Wear features at different scales were captured, and multi-source information was deeply fused to enhance the model’s feature extraction capability. Results of milling experiments under different machining parameters show that the proposed method can effectively track feature changes in key features and achieves higher accuracy than comparative models (SPPNet, ASPP, U-Net). In four groups of milling experiments, the proposed method exhibited mean values of 0.05124 for mean absolute error, 0.06273 for root mean square error, and 0.94508 for coefficient of determination. The method showed low evaluation error for the tool service condition and strong dynamic tracking capability.