Please wait a minute...
浙江大学学报(工学版)  2025, Vol. 59 Issue (4): 669-678    DOI: 10.3785/j.issn.1008-973X.2025.04.002
交通工程     
预训练长短时空交错Transformer在交通流预测中的应用
马莉1,2(),王永顺1,*(),胡瑶1,范磊3
1. 兰州交通大学 电子与信息工程学院,甘肃 兰州 730070
2. 兰州石化职业技术大学 电子电气工程学院,甘肃 兰州 730060
3. 兰州博文科技学院 电信工程学院,甘肃 兰州 730101
Pre-trained long-short spatiotemporal interleaved Transformer for traffic flow prediction applications
Li MA1,2(),Yongshun WANG1,*(),Yao HU1,Lei FAN3
1. School of Electronic and Information Engineering, Lanzhou Jiaotong University, Lanzhou 730070, China
2. School of Electronic and Electrical Engineering, Lanzhou Petrochemical University of Vocational Technology, Lanzhou 730060, China
3. School ofTelecommunication Engineering, Lanzhou Bowen College of Science and Technology, Lanzhou 730101, China
 全文: PDF(1333 KB)   HTML
摘要:

为了削弱和消除短期交通流预测普遍存在的时空幻影现象,基于Transformer网络和自监督预训练-全监督训练框架,提出新型预训练长短时空交错Transformer模型. 采用自监督预训练的方式获得长期时空异质性,设计时空交错模块进行交互获得长期时空异质交互性. 设计短时空循环Transformer,将短期时空序列循环压缩提取至能够表现整个短期时空序列独特时空特征的空间片上. 在长期时空交错的时空异质交互性指导下,将未来时间与近似特征匹配,重建未来短期时空序列. 比较不同交通流预测模型在4个交通流标准数据集和2个交通速度数据集上的预测精度和多步长. 实验结果表明,相比当前先进模型,所提模型提升了交通数据预测的精确性.

关键词: 智能交通交通流预测Transformer深度学习自监督    
Abstract:

To mitigate and eliminate the common spatiotemporal illusions in short-term traffic flow prediction, a novel pre-training long-short spatiotemporal interleaved Transformer model was proposed, based on the Transformer network and a self-supervised pre-training to fully supervised training framework. Long-term spatiotemporal heterogeneity was acquired by the self-supervised pre-training, and a spatiotemporal interleaving module was designed to interact and obtain the long-term spatiotemporal heterogeneous interactivity. A short spatiotemporal recurrent Transformer was designed to compress and extract the short-term spatiotemporal sequences onto a spatial slice, which represented the unique spatiotemporal features of the entire short-term sequence. Guided by the long-term spatiotemporal interleaved heterogeneous interactivity, similar features were matched on the future timeline to reconstruct the future short-term spatiotemporal sequence. Different traffic flow prediction models were compared in terms of accuracy and multi-step predictions in four traffic flow benchmark datasets and two traffic speed datasets. Experimental results show that the proposed model improves the accuracy of traffic data prediction compared to current state-of-the-art models.

Key words: intelligent transportation    traffic flow prediction    Transformer    deep learning    self-supervision
收稿日期: 2024-07-23 出版日期: 2025-04-25
CLC:  TP 399  
基金资助: 国家自然科学基金资助项目(6136606);甘肃省教育厅高校教师创新基金资助项目(2023B-294).
通讯作者: 王永顺     E-mail: marylovemali@126.com;wangysh@mail.lzjtu.cn
作者简介: 马莉(1982—),女,副教授,博士生,从事智能交通研究. orcid.org/0009-0002-1552-096X. E-mail:marylovemali@126.com
服务  
把本文推荐给朋友
加入引用管理器
E-mail Alert
作者相关文章  
马莉
王永顺
胡瑶
范磊

引用本文:

马莉,王永顺,胡瑶,范磊. 预训练长短时空交错Transformer在交通流预测中的应用[J]. 浙江大学学报(工学版), 2025, 59(4): 669-678.

Li MA,Yongshun WANG,Yao HU,Lei FAN. Pre-trained long-short spatiotemporal interleaved Transformer for traffic flow prediction applications. Journal of ZheJiang University (Engineering Science), 2025, 59(4): 669-678.

链接本文:

https://www.zjujournals.com/eng/CN/10.3785/j.issn.1008-973X.2025.04.002        https://www.zjujournals.com/eng/CN/Y2025/V59/I4/669

图 1  PEMS04数据集上的交通流时间异质性
图 2  PEMS04数据集上的交通流空间异质性
图 3  PEMS04数据集上2018年1月8日的时空幻影现象
图 4  PEMS04数据集的局部时空异质交互性
图 5  预训练长短时空交错Transformer模型框架
图 6  预训练长时空Transformer编码层框架
图 7  时空交错模块框架
图 8  短时空循环Transformer模块
数据集NTDS/minNDS采样日期tds/d
PEMS033585262082018年9月—11月91
PEMS043075169922018年1月—2月59
PEMS078835282242017年5月—8月123
PEMS081705178562016年7月—8月62
METR-LA2075342722012年3月—6月123
PEMS-BAY3255521162017年1月—5月62
表 1  时空基准数据集的描述
模型PEMS03数据集PEMS04数据集PEMS07数据集PEMS08数据集
MAERMSEMAPE/%MAERMSEMAPE/%MAERMSEMAPE/%MAERMSEMAPE/%
ARIMA [2]35.3147.5933.7833.7348.8024.1838.1759.2719.4631.0944.3222.73
Transformer[18]17.5030.2416.8023.8337.1915.5726.8042.9512.1118.5228.6813.66
DCRNN[8]18.1830.3118.9124.7038.1217.1225.3038.5811.6617.8627.8311.45
STGCN[11]17.4930.1217.1522.7035.5514.5925.3838.7811.0818.0227.8311.40
GWNet[12]19.8532.9419.3125.4539.7017.2926.8542.7812.1219.1331.0512.68
SVR[3]21.9735.2921.5128.7044.5619.2032.4950.2214.2623.2536.1614.64
LSTM[6]21.3335.1123.3327.1441.5918.2029.9845.8413.2022.2034.0614.20
AGCRN[10]16.0628.4915.8519.8332.2612.9721.2935.128.9715.9525.2210.09
ASTGNN[29]15.0726.8815.8019.2631.1612.6522.2335.959.2515.9825.679.97
DSTAGNN[28]15.5727.2114.6819.3031.4612.7021.4234.519.0115.6724.779.94
STSFGACN[17]14.9826.2414.0719.1431.6412.5620.6133.848.7315.1424.6110.63
ADMSTNODE[30]15.4726.7615.5919.2831.2512.6821.4034.449.0215.5825.099.92
PLSSIFormer14.6726.3614.9218.1129.5112.2220.2933.398.6214.3523.379.48
表 2  不同交通流预测模型在4个交通流标准数据集上的性能对比
图 9  不同模型在2个交通流标准数据集上的多步预测结果对比
图 10  不同模型在交通速度数据集上的预测性能对比
模块PEMS04数据集PEMS08数据集
MAERMSEMAPE/%MAERMSEMAPE/%
基准组件28.646244.03820.208322.792435.04910.1487
STSLT20.294032.95380.136117.622727.87390.1072
PLST+TSI19.769129.80480.125419.769137.50020.1399
PLSSIFormer18.115629.51410.122214.359423.37370.0940
表 3  所提模型的模块消融实验
1 TEDJOPURNOMO D A, BAO Z, ZHENG B, et al A survey on modern deep neural network for traffic prediction: trends, methods and challenges[J]. IEEE Transactions on Knowledge and Data Engineering, 2022, 34 (4): 1544- 1561
2 KUMAR S V, VANAJAKSHI L Short-term traffic flow prediction using seasonal ARIMA model with limited input data[J]. European Transport Research Review, 2015, 7 (3): 21
doi: 10.1007/s12544-015-0170-8
3 CASTRO-NETO M, JEONG Y S, JEONG M K, et al Online-SVR for short-term traffic flow prediction under typical and atypical traffic conditions[J]. Expert Systems with Applications, 2009, 36 (3): 6164- 6173
doi: 10.1016/j.eswa.2008.07.069
4 ZHANG J, ZHENG Y, QI D Deep spatio-temporal residual networks for citywide crowd flows prediction[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2017, 31 (1): 1655- 1661
5 YAO H, TANG X, WEI H, et al Revisiting spatial-temporal similarity: a deep learning framework for traffic prediction[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2019, 33 (1): 5668- 5675
doi: 10.1609/aaai.v33i01.33015668
6 SHI X, CHEN Z, WANG H, et al. Convolutional LSTM network [C]// Proceedings of the 29th International Conference on Neural Information Processing Systems - Volume 1 . Montreal: ACM, 2015: 802–810.
7 YAO H, WU F, KE J, et al Deep multi-view spatial-temporal network for taxi demand prediction[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2018, 32 (1): 2588- 2595
8 LI Y, YU R, SHAHABI C, et al. Diffusion convolutional recurrent neural network: data-driven traffic forecasting [EB/OL]. (2018–02–22)[2024–07–23]. https://arxiv.org/pdf/1707.01926.
9 LIN H, BAI R, JIA W, et al. Preserving dynamic attention for long-term spatial-temporal prediction [C]// Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining . [S. l.]: ACM, 2020: 36–46.
10 BAI L, YAO L, LI C, et al. Adaptive graph convolutional recurrent network for traffic forecasting [C]// Proceedings of the 34th International Conference on Neural Information Processing Systems . Vancouver: ACM, 2020: 17804–17815.
11 YU B, YIN H, ZHU Z. Spatio-temporal graph convolutional networks: a deep learning framework for traffic forecasting [C]// Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence . Stockholm: IJCAI, 2018: 3634–3640.
12 WU Z, PAN S, LONG G, et al. Graph WaveNet for deep spatial-temporal graph modeling [C]// Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence . Macao: IJCAI, 2019: 1907−1913.
13 CHOI J, CHOI H, HWANG J, et al Graph neural controlled differential equations for traffic forecasting[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2022, 36 (6): 6367- 6374
doi: 10.1609/aaai.v36i6.20587
14 黄靖, 钟书远, 文元桥, 等 用于交通流预测的自适应图生成跳跃网络[J]. 浙江大学学报: 工学版, 2021, 55 (10): 1825- 1833
HUANG Jing, ZHONG Shuyuan, WEN Yuanqiao, et al Adaptive graph generation jump network for traffic flow prediction[J]. Journal of Zhejiang University: Engineering Science, 2021, 55 (10): 1825- 1833
15 贺文武, 裴博彧, 李雅婷, 等 基于双向自适应门控图卷积网络的交通流预测[J]. 交通运输系统工程与信息, 2023, 23 (1): 187- 197
HE Wenwu, PEI Boyu, LI Yating, et al Traffic flow forecasting based on bi-directional adaptive gating graph convolutional networks[J]. Journal of Transportation Systems Engineering and Information Technology, 2023, 23 (1): 187- 197
16 闫旭, 范晓亮, 郑传潘, 等 基于图卷积神经网络的城市交通态势预测算法[J]. 浙江大学学报: 工学版, 2020, 54 (6): 1147- 1155
YAN Xu, FAN Xiaoliang, ZHENG Chuanpan, et al Urban traffic flow prediction algorithm based on graph convolutional neural networks[J]. Journal of Zhejiang University: Engineering Science, 2020, 54 (6): 1147- 1155
17 WANG B, LONG Z, SHENG J, et al Spatial–temporal similarity fusion graph adversarial convolutional networks for traffic flow forecasting[J]. Journal of the Franklin Institute, 2024, 361 (17): 107299
doi: 10.1016/j.jfranklin.2024.107299
18 VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need [EB/OL]. (2023–08–02)[2024–07–23]. https://arxiv.org/pdf/1706.03762.
19 JIANG J, HAN C, ZHAO W X, et al PDFormer: propagation delay-aware dynamic long-range transformer for traffic flow prediction[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2023, 37 (4): 4365- 4373
doi: 10.1609/aaai.v37i4.25556
20 LIU H, DONG Z, JIANG R, et al. Spatio-temporal adaptive embedding makes vanilla transformer SOTA for traffic forecasting [C]// Proceedings of the 32nd ACM International Conference on Information and Knowledge Management . Birmingham: ACM, 2023: 4125–4129.
21 GAO H, JIANG R, DONG Z, et al. Spatial-temporal-decoupled masked pre-training for spatiotemporal forecasting [EB/OL]. (2024–04–28)[2024–07–23]. https://arxiv.org/pdf/2312.00516.
22 DEVLIN J, CHANG M W, LEE K, et al. BERT: pre-training of deep bidirectional transformers for language understanding [C]// Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies . Minneapolis: ACL, 2019: 4171–4186.
23 BAO H, DONG L, PIAO S, et al. BEiT: BERT pre-training of image transformers [EB/OL]. (2022–09–03)[2024–07–23]. https://arxiv.org/pdf/2106.08254.
24 NIE Y, NGUYEN N H, SINTHONG P, et al. A time series is worth 64 words: long-term forecasting with transformers [EB/OL]. (2023–03–05)[2024–07–23]. https://arxiv.org/pdf/2211.14730.
25 WANG Z, LIU J C Translating math formula images to LaTeX sequences using deep neural networks with sequence-level training[J]. International Journal on Document Analysis and Recognition, 2021, 24 (1/2): 63- 75
26 LI M, ZHU Z Spatial-temporal fusion graph neural networks for traffic flow forecasting[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2021, 35 (5): 4189- 4196
doi: 10.1609/aaai.v35i5.16542
27 GUO S, LIN Y, FENG N, et al Attention based spatial-temporal graph convolutional networks for traffic flow forecasting[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2019, 33 (1): 922- 929
doi: 10.1609/aaai.v33i01.3301922
28 LAN S, MA Y, HUANG W, et al. DSTAGNN: dynamic spatial-temporal aware graph neural network for traffic flow forecasting [C]// Proceeding of the 39th International Conference on Machine Learning . [S. l.]: PMLR, 2022, 162: 11906–11917.
29 GUO S, LIN Y, WAN H, et al Learning dynamics and heterogeneity of spatial-temporal graph data for traffic forecasting[J]. IEEE Transactions on Knowledge and Data Engineering, 2022, 34 (11): 5415- 5428
doi: 10.1109/TKDE.2021.3056502
[1] 姚明辉,王悦燕,吴启亮,牛燕,王聪. 基于小样本人体运动行为识别的孪生网络算法[J]. 浙江大学学报(工学版), 2025, 59(3): 504-511.
[2] 梁礼明,龙鹏威,金家新,李仁杰,曾璐. 基于改进YOLOv8s的钢材表面缺陷检测算法[J]. 浙江大学学报(工学版), 2025, 59(3): 512-522.
[3] 杨凯博,钟铭恩,谭佳威,邓智颖,周梦丽,肖子佶. 基于半监督学习的多场景火灾小规模稀薄烟雾检测[J]. 浙江大学学报(工学版), 2025, 59(3): 546-556.
[4] 何永福,谢世维,于佳禄,陈思宇. 考虑跨层特征融合的抛洒风险车辆检测方法[J]. 浙江大学学报(工学版), 2025, 59(2): 300-309.
[5] 陈智超,杨杰,李凡,冯志成. 基于深度学习的列车运行环境感知关键算法研究综述[J]. 浙江大学学报(工学版), 2025, 59(1): 1-17.
[6] 杨冰,徐楚阳,姚金良,向学勤. 基于单目RGB图像的三维手部姿态估计方法[J]. 浙江大学学报(工学版), 2025, 59(1): 18-26.
[7] 刘登峰,陈世海,郭文静,柴志雷. 基于轻量残差网络的高效半色调算法[J]. 浙江大学学报(工学版), 2025, 59(1): 62-69.
[8] 赵顗,安醇,李铭浩,马健霄,怀硕. 城市快速路互通交织区车辆的换道持续距离选择[J]. 浙江大学学报(工学版), 2025, 59(1): 205-212.
[9] 李凡,杨杰,冯志成,陈智超,付云骁. 基于图像识别的弓网接触点检测方法[J]. 浙江大学学报(工学版), 2024, 58(9): 1801-1810.
[10] 肖力,曹志刚,卢浩冉,黄志坚,蔡袁强. 基于深度学习和梯度优化的弹性超材料设计[J]. 浙江大学学报(工学版), 2024, 58(9): 1892-1901.
[11] 吴书晗,王丹,陈远方,贾子钰,张越棋,许萌. 融合注意力的滤波器组双视图图卷积运动想象脑电分类[J]. 浙江大学学报(工学版), 2024, 58(7): 1326-1335.
[12] 李林睿,王东升,范红杰. 基于法条知识的事理型类案检索方法[J]. 浙江大学学报(工学版), 2024, 58(7): 1357-1365.
[13] 李劲业,李永强. 融合知识图谱的时空多图卷积交通流量预测[J]. 浙江大学学报(工学版), 2024, 58(7): 1366-1376.
[14] 马现伟,范朝辉,聂为之,李东,朱逸群. 对失效传感器具备鲁棒性的故障诊断方法[J]. 浙江大学学报(工学版), 2024, 58(7): 1488-1497.
[15] 宋娟,贺龙喜,龙会平. 基于深度学习的隧道衬砌多病害检测算法[J]. 浙江大学学报(工学版), 2024, 58(6): 1161-1173.