The YOLOv5s network was improved aiming at the problem of missed detection, false detection and low efficiency of existing object detection algorithms for insulator defects in complex backgrounds. K-means++ clustering was used to analyze the insulator dataset to determine the anchor box size preset by the network. The SiLU activation function of convolution module in the third, fifth, and seventh layers of the backbone network was replaced by Hard-Swish activation function, and the convolutional block attention mechanism (CBAM) was added to improve the network generalization ability. CBAMs were added to the skip links between backbone network and neck network to enhance the ability of image feature extraction. Moreover, the residual structure of feature fusion module of the neck network was replaced by the cross convolution to reduce the network parameters and improve the detection speed. The experimental results demonstrated that the detection accuracy and speed for the insulator defect by the improved YOLOv5s network were 88.6% and 69.4 frames per second, respectively, which were better than those of the popular networks such as Faster R-CNN, YOLOv3, YOLOv4 and regular YOLOv5s. The improved YOLOv5s network meets the requirements of insulator defect detection.