An efficient image dehazing algorithm incorporating Agent Attention and domain feature interaction was developed to address Swin Transformer’s limitations in balancing global dependencies with computational complexity and capturing adequate detail information for image dehazing tasks. The multi-head self-attention was replaced with Agent Attention to construct an encoder-decoder network based on Agent Swin Transformer and Efficient Multi-Scale Attention as fundamental units. This architectural modification reduced the model’s computational complexity while simultaneously enhancing information flow between spatial and channel features. A high-frequency spatial enhancement module and a low-frequency channel enhancement module were designed to reduce spatial feature redundancy and improve the effectiveness of frequency-domain information while extracting features, and spatial domain features were compensated via skip connections. A fast Fourier convolution dense residual structure was constructed in the intermediate layers of the encoder to utilize spectral information for enhancing visual restoration effects. Experiments showed that the proposed algorithm could reduce the model’s computational complexity and feature redundancy, significantly enhance inference speed, restore image detail textures while maintaining their integrity, and achieve superior performance across various objective metrics.