Pre-trained long-short spatiotemporal interleaved Transformer for traffic flow prediction applications

doi:10.3785/j.issn.1008-973X.2025.04.002

Journal of ZheJiang University (Engineering Science)

2025, Vol. 59

Issue (4): 669-678 DOI: 10.3785/j.issn.1008-973X.2025.04.002

Pre-trained long-short spatiotemporal interleaved Transformer for traffic flow prediction applications

Li MA1,2(

),Yongshun WANG1,*(

),Yao HU1,Lei FAN3

1. School of Electronic and Information Engineering, Lanzhou Jiaotong University, Lanzhou 730070, China
2. School of Electronic and Electrical Engineering, Lanzhou Petrochemical University of Vocational Technology, Lanzhou 730060, China
3. School ofTelecommunication Engineering, Lanzhou Bowen College of Science and Technology, Lanzhou 730101, China

Download:

HTML

PDF(1333KB) HTML
Export: BibTeX | EndNote (RIS)

Abstract

To mitigate and eliminate the common spatiotemporal illusions in short-term traffic flow prediction, a novel pre-training long-short spatiotemporal interleaved Transformer model was proposed, based on the Transformer network and a self-supervised pre-training to fully supervised training framework. Long-term spatiotemporal heterogeneity was acquired by the self-supervised pre-training, and a spatiotemporal interleaving module was designed to interact and obtain the long-term spatiotemporal heterogeneous interactivity. A short spatiotemporal recurrent Transformer was designed to compress and extract the short-term spatiotemporal sequences onto a spatial slice, which represented the unique spatiotemporal features of the entire short-term sequence. Guided by the long-term spatiotemporal interleaved heterogeneous interactivity, similar features were matched on the future timeline to reconstruct the future short-term spatiotemporal sequence. Different traffic flow prediction models were compared in terms of accuracy and multi-step predictions in four traffic flow benchmark datasets and two traffic speed datasets. Experimental results show that the proposed model improves the accuracy of traffic data prediction compared to current state-of-the-art models.

Key words： intelligent transportation traffic flow prediction Transformer deep learning self-supervision

Received: 23 July 2024 Published: 25 April 2025

CLC:

TP 399

Fund: 国家自然科学基金资助项目（6136606）；甘肃省教育厅高校教师创新基金资助项目（2023B-294）.

Corresponding Authors: Yongshun WANG E-mail: marylovemali@126.com;wangysh@mail.lzjtu.cn

	Service
	E-mail this article
	Add to my bookshelf
	Add to citation manager
	E-mail Alert
	RSS
	Articles by authors
	Li MA
	Yongshun WANG
	Yao HU
	Lei FAN

Cite this article:

Li MA,Yongshun WANG,Yao HU,Lei FAN. Pre-trained long-short spatiotemporal interleaved Transformer for traffic flow prediction applications. Journal of ZheJiang University (Engineering Science), 2025, 59(4): 669-678.

URL:

https://www.zjujournals.com/eng/10.3785/j.issn.1008-973X.2025.04.002 OR https://www.zjujournals.com/eng/Y2025/V59/I4/669

预训练长短时空交错Transformer在交通流预测中的应用

为了削弱和消除短期交通流预测普遍存在的时空幻影现象，基于Transformer网络和自监督预训练-全监督训练框架，提出新型预训练长短时空交错Transformer模型. 采用自监督预训练的方式获得长期时空异质性，设计时空交错模块进行交互获得长期时空异质交互性. 设计短时空循环Transformer，将短期时空序列循环压缩提取至能够表现整个短期时空序列独特时空特征的空间片上. 在长期时空交错的时空异质交互性指导下，将未来时间与近似特征匹配，重建未来短期时空序列. 比较不同交通流预测模型在4个交通流标准数据集和2个交通速度数据集上的预测精度和多步长. 实验结果表明，相比当前先进模型，所提模型提升了交通数据预测的精确性.

关键词： 智能交通, 交通流预测, Transformer, 深度学习, 自监督

Fig.1 Temporal heterogeneity of traffic flow in PEMS04 dataset

Fig.2 Spatial heterogeneity of traffic flow in PEMS04 dataset

Fig.3 Spatiotemporal mirage phenomenon in PEMS04 dataset on January 8, 2018

Fig.4 Local spatiotemporal heterogeneous interactivity of PEMS04 dataset

Fig.5 Pre-trained long-short spatiotemporal interleaved Transformer model framework

Fig.6 Pre-trained long spatiotemporal Transformer encoder layer framework

Fig.7 Spatiotemporal interleaved module framework

Fig.8 Short spatiotemporal recurrent transformer module

Tab.1 Descriptions of spatiotemporal benchmark datasets

Tab.2 Performance comparison of different traffic flow prediction models in four traffic flow benchmark datasets

Fig.9 Comparison of multi-step prediction results of different models in two traffic flow benchmark datasets

Fig.10 Predictive performance comparison of different models in traffic speed datasets

Tab.3 Modular ablation experiments for proposed model


[1]	TEDJOPURNOMO D A, BAO Z, ZHENG B, et al A survey on modern deep neural network for traffic prediction: trends, methods and challenges[J]. IEEE Transactions on Knowledge and Data Engineering, 2022, 34 (4): 1544- 1561

[2]	KUMAR S V, VANAJAKSHI L Short-term traffic flow prediction using seasonal ARIMA model with limited input data[J]. European Transport Research Review, 2015, 7 (3): 21 doi: 10.1007/s12544-015-0170-8

[3]	CASTRO-NETO M, JEONG Y S, JEONG M K, et al Online-SVR for short-term traffic flow prediction under typical and atypical traffic conditions[J]. Expert Systems with Applications, 2009, 36 (3): 6164- 6173 doi: 10.1016/j.eswa.2008.07.069

[4]	ZHANG J, ZHENG Y, QI D Deep spatio-temporal residual networks for citywide crowd flows prediction[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2017, 31 (1): 1655- 1661

[5]	YAO H, TANG X, WEI H, et al Revisiting spatial-temporal similarity: a deep learning framework for traffic prediction[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2019, 33 (1): 5668- 5675 doi: 10.1609/aaai.v33i01.33015668

[6]	SHI X, CHEN Z, WANG H, et al. Convolutional LSTM network [C]// Proceedings of the 29th International Conference on Neural Information Processing Systems - Volume 1 . Montreal: ACM, 2015: 802–810.

[7]	YAO H, WU F, KE J, et al Deep multi-view spatial-temporal network for taxi demand prediction[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2018, 32 (1): 2588- 2595

[8]	LI Y, YU R, SHAHABI C, et al. Diffusion convolutional recurrent neural network: data-driven traffic forecasting [EB/OL]. (2018–02–22)[2024–07–23]. https://arxiv.org/pdf/1707.01926.

[9]	LIN H, BAI R, JIA W, et al. Preserving dynamic attention for long-term spatial-temporal prediction [C]// Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining . [S. l.]: ACM, 2020: 36–46.

[10]	BAI L, YAO L, LI C, et al. Adaptive graph convolutional recurrent network for traffic forecasting [C]// Proceedings of the 34th International Conference on Neural Information Processing Systems . Vancouver: ACM, 2020: 17804–17815.

[11]	YU B, YIN H, ZHU Z. Spatio-temporal graph convolutional networks: a deep learning framework for traffic forecasting [C]// Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence . Stockholm: IJCAI, 2018: 3634–3640.

[12]	WU Z, PAN S, LONG G, et al. Graph WaveNet for deep spatial-temporal graph modeling [C]// Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence . Macao: IJCAI, 2019: 1907−1913.

[13]	CHOI J, CHOI H, HWANG J, et al Graph neural controlled differential equations for traffic forecasting[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2022, 36 (6): 6367- 6374 doi: 10.1609/aaai.v36i6.20587

[14]	黄靖, 钟书远, 文元桥, 等用于交通流预测的自适应图生成跳跃网络[J]. 浙江大学学报: 工学版, 2021, 55 (10): 1825- 1833 HUANG Jing, ZHONG Shuyuan, WEN Yuanqiao, et al Adaptive graph generation jump network for traffic flow prediction[J]. Journal of Zhejiang University: Engineering Science, 2021, 55 (10): 1825- 1833

[15]	贺文武, 裴博彧, 李雅婷, 等基于双向自适应门控图卷积网络的交通流预测[J]. 交通运输系统工程与信息, 2023, 23 (1): 187- 197 HE Wenwu, PEI Boyu, LI Yating, et al Traffic flow forecasting based on bi-directional adaptive gating graph convolutional networks[J]. Journal of Transportation Systems Engineering and Information Technology, 2023, 23 (1): 187- 197

[16]	闫旭, 范晓亮, 郑传潘, 等基于图卷积神经网络的城市交通态势预测算法[J]. 浙江大学学报: 工学版, 2020, 54 (6): 1147- 1155 YAN Xu, FAN Xiaoliang, ZHENG Chuanpan, et al Urban traffic flow prediction algorithm based on graph convolutional neural networks[J]. Journal of Zhejiang University: Engineering Science, 2020, 54 (6): 1147- 1155

[17]	WANG B, LONG Z, SHENG J, et al Spatial–temporal similarity fusion graph adversarial convolutional networks for traffic flow forecasting[J]. Journal of the Franklin Institute, 2024, 361 (17): 107299 doi: 10.1016/j.jfranklin.2024.107299

[18]	VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need [EB/OL]. (2023–08–02)[2024–07–23]. https://arxiv.org/pdf/1706.03762.

[19]	JIANG J, HAN C, ZHAO W X, et al PDFormer: propagation delay-aware dynamic long-range transformer for traffic flow prediction[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2023, 37 (4): 4365- 4373 doi: 10.1609/aaai.v37i4.25556

[20]	LIU H, DONG Z, JIANG R, et al. Spatio-temporal adaptive embedding makes vanilla transformer SOTA for traffic forecasting [C]// Proceedings of the 32nd ACM International Conference on Information and Knowledge Management . Birmingham: ACM, 2023: 4125–4129.

[21]	GAO H, JIANG R, DONG Z, et al. Spatial-temporal-decoupled masked pre-training for spatiotemporal forecasting [EB/OL]. (2024–04–28)[2024–07–23]. https://arxiv.org/pdf/2312.00516.

[22]	DEVLIN J, CHANG M W, LEE K, et al. BERT: pre-training of deep bidirectional transformers for language understanding [C]// Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies . Minneapolis: ACL, 2019: 4171–4186.

[23]	BAO H, DONG L, PIAO S, et al. BEiT: BERT pre-training of image transformers [EB/OL]. (2022–09–03)[2024–07–23]. https://arxiv.org/pdf/2106.08254.

[24]	NIE Y, NGUYEN N H, SINTHONG P, et al. A time series is worth 64 words: long-term forecasting with transformers [EB/OL]. (2023–03–05)[2024–07–23]. https://arxiv.org/pdf/2211.14730.

[25]	WANG Z, LIU J C Translating math formula images to LaTeX sequences using deep neural networks with sequence-level training[J]. International Journal on Document Analysis and Recognition, 2021, 24 (1/2): 63- 75

[26]	LI M, ZHU Z Spatial-temporal fusion graph neural networks for traffic flow forecasting[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2021, 35 (5): 4189- 4196 doi: 10.1609/aaai.v35i5.16542

[27]	GUO S, LIN Y, FENG N, et al Attention based spatial-temporal graph convolutional networks for traffic flow forecasting[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2019, 33 (1): 922- 929 doi: 10.1609/aaai.v33i01.3301922

[28]	LAN S, MA Y, HUANG W, et al. DSTAGNN: dynamic spatial-temporal aware graph neural network for traffic flow forecasting [C]// Proceeding of the 39th International Conference on Machine Learning . [S. l.]: PMLR, 2022, 162: 11906–11917.

[29]	GUO S, LIN Y, WAN H, et al Learning dynamics and heterogeneity of spatial-temporal graph data for traffic forecasting[J]. IEEE Transactions on Knowledge and Data Engineering, 2022, 34 (11): 5415- 5428 doi: 10.1109/TKDE.2021.3056502

[1]	Minghui YAO,Yueyan WANG,Qiliang WU,Yan NIU,Cong WANG. Siamese networks algorithm based on small human motion behavior recognition[J]. Journal of ZheJiang University (Engineering Science), 2025, 59(3): 504-511.

[2]	Liming LIANG,Pengwei LONG,Jiaxin JIN,Renjie LI,Lu ZENG. Steel surface defect detection algorithm based on improved YOLOv8s[J]. Journal of ZheJiang University (Engineering Science), 2025, 59(3): 512-522.

[3]	Kaibo YANG,Mingen ZHONG,Jiawei TAN,Zhiying DENG,Mengli ZHOU,Ziji XIAO. Small-scale sparse smoke detection in multiple fire scenarios based on semi-supervised learning[J]. Journal of ZheJiang University (Engineering Science), 2025, 59(3): 546-556.

[4]	Yongfu HE,Shiwei XIE,Jialu YU,Siyu CHEN. Detection method for spillage risk vehicle considering cross-level feature fusion[J]. Journal of ZheJiang University (Engineering Science), 2025, 59(2): 300-309.

[5]	Zhichao CHEN,Jie YANG,Fan LI,Zhicheng FENG. Review on deep learning-based key algorithm for train running environment perception[J]. Journal of ZheJiang University (Engineering Science), 2025, 59(1): 1-17.

[6]	Bing YANG,Chuyang XU,Jinliang YAO,Xueqin XIANG. 3D hand pose estimation method based on monocular RGB images[J]. Journal of ZheJiang University (Engineering Science), 2025, 59(1): 18-26.

[7]	Dengfeng LIU,Shihai CHEN,Wenjing GUO,Zhilei CHAI. Efficient halftone algorithm based on lightweight residual networks[J]. Journal of ZheJiang University (Engineering Science), 2025, 59(1): 62-69.

[8]	Yi ZHAO,Chun AN,Minghao LI,Jianxiao MA,Shuo HUAI. Selection of lane-changing distance for vehicles in urban expressway interchange weaving section[J]. Journal of ZheJiang University (Engineering Science), 2025, 59(1): 205-212.

[9]	Fan LI,Jie YANG,Zhicheng FENG,Zhichao CHEN,Yunxiao FU. Pantograph-catenary contact point detection method based on image recognition[J]. Journal of ZheJiang University (Engineering Science), 2024, 58(9): 1801-1810.

[10]	Li XIAO,Zhigang CAO,Haoran LU,Zhijian HUANG,Yuanqiang CAI. Elastic metamaterial design based on deep learning and gradient optimization[J]. Journal of ZheJiang University (Engineering Science), 2024, 58(9): 1892-1901.

[11]	Shuhan WU,Dan WANG,Yuanfang CHEN,Ziyu JIA,Yueqi ZHANG,Meng XU. Attention-fused filter bank dual-view graph convolution motor imagery EEG classification[J]. Journal of ZheJiang University (Engineering Science), 2024, 58(7): 1326-1335.

[12]	Linrui LI,Dongsheng WANG,Hongjie FAN. Fact-based similar case retrieval methods based on statutory knowledge[J]. Journal of ZheJiang University (Engineering Science), 2024, 58(7): 1357-1365.

[13]	Jinye LI,Yongqiang LI. Spatial-temporal multi-graph convolution for traffic flow prediction by integrating knowledge graphs[J]. Journal of ZheJiang University (Engineering Science), 2024, 58(7): 1366-1376.

[14]	Xianwei MA,Chaohui FAN,Weizhi NIE,Dong LI,Yiqun ZHU. Robust fault diagnosis method for failure sensors[J]. Journal of ZheJiang University (Engineering Science), 2024, 58(7): 1488-1497.

[15]	Juan SONG,Longxi HE,Huiping LONG. Deep learning-based algorithm for multi defect detection in tunnel lining[J]. Journal of ZheJiang University (Engineering Science), 2024, 58(6): 1161-1173.

Viewed

Full text

Abstract

Cited

Shared

Discussed