A point of interest (POI) trajectory prediction framework with collaborative optimization of large language model (LLM) and lightweight model, namely LLM-RFA, was proposed aiming at the challenge of difficult spatiotemporal information representation and fusion of influencing factor caused by trajectory sparsity and behavioral complexity in POI prediction task. Fused representation of internal and external factor affecting POI trajectory behavior was realized. An end-to-end trajectory prediction method was adopted to generate the candidate POI set, and input token of the large model was reduced. Noise data was mixed into historical trajectory. A historical trajectory point reordering task was designed by temporal graph structure representation in order to guide the LLM to complete the first-round prediction. A lightweight correction model was pre-trained. The model was applied to guide the LLM to conduct reflection from the perspective of access point category preference, trajectory semantic consistency and group behavior influence. Prediction accuracy was improved through the collaboration of large and small model. Experiments conducted on three public datasets including NYC, TKY and CA showed that the prediction accuracy of the model was improved after collaborative optimization. The TOP1 accuracy of the DeepSeek-R1-based model outperformed existing baseline methods by 0.3% to 11%. Ablation experiments showed that both internal and external factor and each component of the model exerted different degree of influence on the prediction result.