A real-time UAV object tracker based on lightweight and attentional aggregation siamese network with anchor free scheme was proposed, aiming at the problems of viewpoint change, deformation and similar objects around in UAV tracking tasks. Considering the small number of object pixels in the view of UAV high-altitude platform, an efficient channel attention scheme was introduced to the two branches of template. Then semantics information and detail information can be effectively extracted. A spatial attention scheme was constructed to effectively aggregate attention and enlarge the visual field range after fusing the response of two layer. An anchor free mechanism was built to directly classify and predict the object box on each pixel, which can simplify the complexity of model and reduce the calculation cost. The proposed method was conducted on three public UAV data sets such as UAV123@10fps、UAV20L and DTB70, and compared with other state-of-the-art tracking algorithms. The experimental results show that the proposed method can track the target effectively in many challenging scenes with an average speed of 155.2 frame per second on three UAV benchmarks.