In reinforcement learning, policy gradient methods are often challenged by excessive variance in gradient estimation, which leads to poor convergence. A stochastic variance-reduced recursive momentum policy gradient method (SVRRM-PG) was proposed. This method combines importance sampling technique with a reference-point switching mechanism to achieve large-batch gradient estimation without resampling, effectively reducing variance and improving training stability, while exhibiting provably advanced sample complexity currently. To improve the general applicability, the SVRRM-PG framework was integrated with the proximal policy optimization (PPO) algorithm, which led to the proposed SVR-PPO method for verifying the efficacy of SVRRM-PG estimator in enhancing the performance of the established reinforcement learning algorithms. A lot of benchmark discrete and continuous control tasks were conducted to compare the performance. The results demonstrate that the SVRRM-PG estimator not only accelerates the convergence significantly and improves the training stability, while achieves superior overall performance in multiple benchmark evaluations, confirming its effectiveness and robustness in complex reinforcement learning environments.