There are problems such as background interference, multiple and irregularly shaped targets, target overlap, and rapid falling speeds during the transportation of tobacco stems in the tobacco production line. A tobacco stem material target detection algorithm based on improved YOLOv5s was proposed. The backbone and head of the YOLOv5s network were optimized, significantly improving the detection accuracy and substantially reducing the model size. Firstly, the network’s backbone was optimized into the RepViT-m1 structure, enhancing the information extraction efficiency. Secondly, reparameterization techniques were used to better capture the target features, thus improving the detection precision. Dynamic Head, a target detection head based on the attention mechanism, was introduced to make the model be focused on the potential target area to further improve the detection accuracy. Experimental results on self-constructed tobacco stem dataset demonstrated the effectiveness of the improved YOLOv5s model. Compared with the original YOLOv5s model, the improved model achieved an mAP@0.50 of 96.1%, which was improved by 5.8 percentage points; and achieved an mAP@0.50:0.95 of 94.7%, which was improved by 5.7 percentage points. Furthermore, the model size was 12.1 MB, which was decreased by 12.3%. The results provide reliable and accurate support for real-time monitoring systems.