A small object detection algorithm FMCM-YOLO based on feature enhancement and fusion attention mechanism was proposed, aiming at the challenges of limited feature extraction, foreground-background confusion, and severe missed and false detections in small object detection in optical remote sensing images. Firstly, a four-head detection model was designed and a small target detection layer was added to detect numerous small objects in optical remote sensing images. Secondly, a feature enhancement module was proposed in the backbone network, which improved feature extraction capability by designing a multi-branch convolutional structure and introducing dilated convolution of different sizes. Thirdly, channel and spatial attention mechanisms were incorporated into the neck network, and a residual structure was introduced to focus on small objects, facilitating the distinction between targets and backgrounds. Finally, MPDIoU was adopted as the model’s loss function to accelerate convergence and enhance detection performance for small objects. Experimental results demonstrated that the mAP50 of the proposed algorithm on the two public datasets, USOD and AI-TOD, reached 89.9% and 60.6% respectively, which were 2.8 and 5.9 percentage points higher than those of the baseline algorithm YOLOv5m. Especially, the mean average precision for extremely tiny, tiny, and small objects increased by 2.1, 6.5, and 5.1 percentage points, respectively. These results proved that the FMCM-YOLO algorithm effectively improved the detection performance of small targets in optical remote sensing images.