An object tracking method based on policy gradient was proposed aiming at the problems of occlusion, deformation and fast motion in the process of object tracking. The policy gradient algorithm was used to train the policy network. The policy network can make action decisions founded on the reliability of current tracking results to avoid the incorrect template update or re-detect the missing targets. During the decision-making process, the robustness and accuracy of the current tracking result were both analyzed by calculating the weighted confidence margin, which helped the policy network to evaluate the tracking results more accurately. During the re-detection process, an efficient re-detection method was proposed to filter a large number of searching areas, which greatly improved the search efficiency. The decision-making module was utilized to examine the re-detected result, which ensured the accuracy of the re-detected results. The proposed algorithm was evaluated on OTB dataset and LaSOT dataset. The experimental results show that the proposed tracking algorithm improves performance by 2.5%-4.0% based on the original algorithm.