A detection algorithm based on improved RT-DETR was proposed, to achieve rapid and accurate detection of minor defects on surfaces of denim fabrics and to overcome the decline in detection performance of existing models under complex textured backgrounds. First, partial convolution (PConv) was utilized to enhance the model’s ability to recognize key features through the Efficient Multi-Scale Attention mechanism. Second, on the basis of adding the S2 feature detection layer, two feature fusion operations, multi-feature encoding (MFE) and multi-scale feature fusion (MSFF), were introduced to enhance the integration of feature information at different scales, and the Deformable Attention mechanism was employed to better address defect deformation. In terms of loss function, a new loss function was proposed, which could accelerate the network convergence and improve the accuracy of network detection. Tests conducted on the fabric defect dataset from Tianchi Cloud demonstrated that the improved RT-DETR model achieved an average mAP@0.5 of 60%, marking 5.3% improvement over the RT-DETR-R18 model, with a 40.1% reduction in the total number of model parameters. Compared to YOLOv5 and YOLOv8, improvements of 9.5% and 9.9% in mAP@0.5 were observed, respectively. The improved RT-DETR model shows significant enhancements in the positioning accuracy and detection precision of defect detection, meeting the requirements for industrial-scale detection. This model serves as an important reference for the intelligent transformation and development of the textile and apparel industry.