A detection method based on an improved RT-DETR (FES-DETR) was proposed to achieve fast and accurate detection of underwater small organisms, and address the poor detection performance of the existing models in underwater color-cast environments. An efficient multi-scale attention feature extraction (Faster-Rep-EMA) module was designed to optimize the original BasicBlock in the backbone network, thereby improving the feature extraction ability and computational efficiency for weak targets under color-cast interference. The entanglement Transformer block (ETB) was integrated with the attention-based intra-scale feature interaction (AIFI) module in the neck encoding network to achieve the joint optimization of frequency-domain and spatial-domain features, which could enhance the feature representation of color-cast images. A lightweight small object enhancement pyramid (SOEP) module was designed to enhance the detection performance of the model for small targets and reduce the computational redundancy. Experimental results showed that the FES-DETR significantly improved the detection performance. Compared with RT-DETR-r18, the precision and recall were improved by 2.6 and 2.3 percentage points, respectively; the mAP@0.5 and the mAP@0.5:0.95 were improved by 3.2 and 2.1 percentage points, respectively; the number of parameters and computational complexity were decreased by 3.0 M and 8.5 G, respectively; and the FPS was increased to 95.7 frames per second. Compared with mainstream target detection models such as YOLO series, this model showed more superior performance, providing an effective technical approach for the detection of underwater small organisms.