An optimized track-before-detect algorithm for weak and small target detection based on improved multi-kernel correlation filtering was proposed aiming at the problem that radar target tracking under complex sea clutter environments was subject to strong clutter interference and increased tracking difficulty. Weak and small targets were transformed from the time domain with indistinct features to the fractional Fourier domain where amplitude variations are more pronounced by introducing fractional Fourier transform, followed by filtering to enhance detection rates. The template extraction approach was optimized by incorporating Kalman filtering for target matching in order to address the non-robust template extraction issue in multi-kernel correlation filtering (MKCF) methods. Different template extraction methods were adopted according to target types, and prediction results were fused by using maximum likelihood method in order to improve tracking accuracy. Comprehensive multi-frame information was utilized to select optimal trajectory estimation combined with multi-frame track-before-detect (TBD) algorithm. The experimental results demonstrate that the proposed algorithm effectively adapts to complex sea clutter environments, achieves efficient multi-target tracking under low signal-to-noise ratio and strong clutter interference, and demonstrates superior accuracy compared with traditional methods.