Multi-scale targets in unmanned aerial vehicle (UAV) aerial images, especially small targets, have low detection accuracy in complex scenarios such as dense scenes, occlusions, and low illumination. Thus, a convolution-wavelet dual-domain downsampling module (RDWTConv) was proposed to preserve fine details of small targets. Additionally, a three-layer cross-scale residual fusion module (RCDFM) was designed to enhance multi-scale feature interactions. Furthermore, a scale-shape loss function (TSSIoU) was introduced to improve bounding box localization accuracy for varying object scales and shapes under aerial perspectives. On this basis, a series of CF-YOLO models, namely CF-YOLOn, CF-YOLOs, and CF-YOLOm, were constructed based on YOLOv8 to meet diverse computational requirements. Experimental results demonstrated that on the VisDrone dataset, CF-YOLOn achieved a 23.7% reduction in parameters and only a 22.5% increase in computational cost, while improving mAP@0.5 and mAP@0.5:0.95 by 5.5 and 4.0 percentage points, respectively, compared with the baseline YOLOv8n, as well as maintaining a frame rate of 169.1 frames per second. The s and m variants also achieved the highest accuracy within the same frame rate range. After retraining on the Drone-Vehicle dataset, CF-YOLOn’s mAP@0.5:0.95 improved by 3.0 percentage points compared to the baseline. Through the above synergistic improvements, the proposed method not only maintains real-time detection under lightweight computational costs but also effectively enhances multi-scale target detection performance in complex scenarios, achieving state-of-the-art results among comparable methods.