A VMS-YOLOv11n model for railroad safety wear detection was proposed aiming at the problems of more complex railroad operation scenario, smaller safety wear vision targets and limited resource of edge computing devices. The VABlock feature extraction module was designed to replace the C3k2 module in the backbone network as well as the neck network to reduce computational redundancy while retaining key features. Then the MSF multi-scale feature fusion structure was designed to improve the small target detection accuracy in complex railroad scene by fusing the multilevel feature weighting strategy with the bidirectional feature pyramid network. The spatial pyramid feature fusion structure (SE2SPPF) was constructed to enhance the small target feature learning capability while improving the model detection capability. The experimental results showed that the improved model had precision rate of 98%, recall rate of 92.5%, and average precision mean of 97.5%, which were 2.9%, 2.5% and 2.7% higher, respectively compared with the baseline YOLOv11n. The number of parameters is 2.45×10⁶, which is 5% lower than the baseline. The improved model works well in railroad safety wear detection, effectively balancing detection accuracy and speed.