A robust feature matching algorithm based on anomaly detection (RFM-AD) was proposed to solve the problem of low generality in feature matching methods that rely on pre-defined parameterized models. Firstly, anomaly detection samples were constructed based on putative feature matches, thereby feature matching problems were transformed into anomaly detection problems, and the local outlier factor (LOF) algorithm was introduced as the foundation for anomaly detection. Secondly, the connectivity-based outlier factor (COF) method was introduced and improved to address the deficiency of LOF algorithm in effectively detecting low-density samples, and a guided matching strategy was used to fuse COF and LOF for enhanced performance. Finally, the parameter settings of the proposed algorithm were tested on 30 randomly selected image pairs involving different transformation models and noise levels, and the globally optimal parameters were determined. Experiments conducted on four public datasets demonstrated the robustness and promising performance of the proposed algorithm when dealing with a large number of outliers. Under the premise of maintaining a high matching precision, the proposed algorithm achieved a leading recall compared to advanced algorithms, such as RANSAC, LPM and RFM-SCAN. Specifically, the proposed algorithm achieved a leading F-score on the Retina dataset, which had the lowest inlier rate.