Please wait a minute...
浙江大学学报(工学版)
浙江大学学报(工学版)  2026, Vol. 60 Issue (11): 2395-2405    DOI: 10.3785/j.issn.1008-973X.2026.11.009
计算机技术     
加速大模型训练的双向交错流水线并行方法
吴厚明1,2(),陈岭1,2,*()
1. 浙江大学 区块链与数据安全全国重点实验室，浙江 杭州 310027
2. 浙江大学 计算机科学与技术学院，浙江 杭州 310027
Bidirectional interleaved pipeline parallelism for accelerating large model training
Houming WU1,2(),Ling CHEN1,2,*()
1. State Key Laboratory of Blockchain and Data Security, Zhejiang University, Hangzhou 310027, China
2. College of Computer Science and Technology, Zhejiang University, Hangzhou 310027, China
 全文: PDF(1112 KB)   HTML
摘要：

现有同步流水线并行方法因阶段间依赖和周期性参数同步引入大量流水线空泡，降低了设备利用率和训练效率，为此提出加速大模型训练的双向交错流水线并行方法BitPipe. 在统一框架下融合双向流水线执行与交错调度，设计双向交错流水线调度机制，以提升设备空闲时隙的利用率并支持更细粒度的阶段调度. 为了缓解并行度提升带来的通信开销，引入V形交错式模型划分策略，将部分跨设备通信转化为本地显存访问，结合设备映射与提前梯度同步优化，实现计算与通信的有效重叠. 基于Megatron-LM框架，所提方法在最多32张GPU上对GPT类与BERT类模型进行实验评估. 结果表明，相较于现有同步流水线并行方法，BitPipe实现了1.05×~1.28×的训练吞吐率提升，有效降低了流水线空泡并提高了训练效率.
关键词： 深度学习;  分布式训练;  流水线并行;  双向流水线;  交错流水线    
Abstract:

Existing synchronous pipeline parallelism methods introduce substantial pipeline bubbles due to inter-stage dependencies and periodic parameter synchronization, which reduce device utilization and training efficiency. To address this limitation, a bidirectional interleaved pipeline parallelism method named BitPipe for accelerating large-scale model training was proposed. Within a unified framework, bidirectional pipeline execution and interleaved scheduling were integrated. A bidirectional interleaved pipeline scheduling mechanism was designed to improve the utilization of idle device time slots and support finer-grained stage scheduling. To alleviate the communication overhead incurred by increased parallelism, a V-shaped interleaved model partitioning strategy was introduced to convert part of cross-device communication into local memory access. In addition, device mapping and eager gradient synchronization were employed to enable effective overlap between computation and communication. The proposed method was implemented based on the Megatron-LM framework and evaluated on GPT-like and BERT-like models using up to 32 GPUs. Experimental results show that BitPipe achieves a training throughput improvement of 1.05×–1.28× compared with existing synchronous pipeline parallelism methods. Moreover, BitPipe effectively reduces pipeline bubbles and improves training efficiency in large-scale model training.
Key words: deep learning    distributed training    pipeline parallelism    bidirectional pipeline    interleaved pipeline
收稿日期: 2025-12-09 出版日期: 2026-10-16
CLC:  TP 18  
通讯作者: 陈岭     E-mail: houmingwu@zju.edu.cn;lingchen@cs.zju.edu.cn
作者简介: 吴厚明（1984—），男，博士生，从事分布式机器学习研究. orcid.org/0009-0004-6793-6812. E-mail：houmingwu@zju.edu.cn
服务　　
把本文推荐给朋友
加入引用管理器
E-mail Alert
作者相关文章　　
吴厚明
陈岭

引用本文:

吴厚明,陈岭. 加速大模型训练的双向交错流水线并行方法[J]. 浙江大学学报(工学版), 2026, 60(11): 2395-2405.

Houming WU,Ling CHEN. Bidirectional interleaved pipeline parallelism for accelerating large model training. Journal of ZheJiang University (Engineering Science), 2026, 60(11): 2395-2405.

链接本文:

https://www.zjujournals.com/eng/CN/10.3785/j.issn.1008-973X.2026.11.009    或    https://www.zjujournals.com/eng/CN/Y2026/V60/I11/2395

符号含义符号含义
$ D $流水线深度$ N $小批次中的微批次数量
$ W $数据并行维度$ \hat{B} $小批次大小（$ \hat{B}=BNW $）
$ P $设备总数（$ P=WD $）$ {M}_{\text{θ}} $单个流水线阶段的权重显存占用
$ B $微批次大小$ {M}_{\text{a}} $单个微批次的激活值显存占用
表 1  关键符号及含义
图 1  循环交错划分和V形交错划分对比
图 2  BitPipe的模型划分与双向交错调度（流水线深度和微批次数均为4）
流水线方法模型划分调度范式设备映射同步时机关键约束
GPipe[7]顺序单向顺序计算后无
DAPPLE[24]顺序单向顺序计算后无
1F1B-Int[6]循环单向交错顺序计算后无
Chimera[29]顺序双向顺序提前同步$ D $为偶数
MixPipe[30]顺序双向跨节点优化提前同步$ D $为偶数
BitPipeV形双向交错跨节点优化提前同步$ D $为偶数
表 2  同步流水线并行方法机制对比（微批次数不小于流水线深度）
图 3  几种同步流水线并行方法在模型划分与微批次调度上的对比（流水线深度和微批次数均为4）
流水线方法空泡率参数显存激活值显存
GPipe[7]$ \dfrac{D-1}{N+D+1} $$ {M}_{\text{θ}} $$ {N\times M}_{\text{a}} $
DAPPLE[24]$ \dfrac{D-1}{N+D+1} $$ {M}_{\text{θ}} $$ [{{M}}_{\text{a}},{D}\times {{M}}_{\text{a}}] $
1F1B-Int[6]$ \dfrac{{D}-1}{2{N}+{D}+1} $$ {{M}}_{\text{θ}} $$ \left[\dfrac{{D}+1}{2}{{M}}_{\text{a}},{D}\times {{M}}_{\text{a}}\right] $
Chimera[29]$ \dfrac{{D}-2}{3{N}/2+{D}-2} $$ 2{{M}}_{\text{θ}} $$ \left[\dfrac{{D}+2}{2}{{M}}_{\text{a}},{D}\times {{M}}_{\text{a}}\right] $
BitPipe$ \dfrac{{D}-2}{3{N}+{D}-2} $$ 2{{M}}_{\text{θ}} $$ \left[\dfrac{{D}+3}{2}{{M}}_{\text{a}},{D}\times {{M}}_{\text{a}}\right] $
表 3  空泡率和显存占用对比
图 4  基于流水线空泡时隙填充的提前梯度同步
图 5  双向流水线的设备映射策略
图 6  微批次数量扩展（流水线深度为4，微批次数为8）
图 7  流水线阶段数量扩展（流水线深度为4）
模型层数注意力头数隐藏维大小序列长度
BERT-64 (5B)64642560512
GPT-96 (11B)963230721024
表 4  用于评估的基准模型
图 8  流水线并行下的显存占用分布 (8 GPU)
图 9  流水线并行和流水线-数据并行方法的吞吐率比较
模型参数备选值8 GPU16 GPU32 GPU
AIMBAIMBAIMB
E$ W ${1, 2, 4, 8}221144444844
$ D ${4, 8, 16}448844448488
$ B ${1, 2, 4, 8}444444442244
G$ W ${1, 2, 4}111122224444
$ D ${8, 16}888888888888
$ B ${1, 2}111111111111
表 5  参数搜索空间及其最终选择
图 10  流水线-数据并行下的显存占用分布(32 GPU)
$ D $$ \hat{B} $T/（sample·s?1）
w/o Vw/o Ew/o MBitPipe
41619.4818.8219.5819.58
46424.1924.1124.2824.28
83238.7838.3239.1739.17
812846.2946.0546.4346.43
166449.2149.0948.5650.67
1625655.9657.0555.2357.44
表 6  模块消融实验结果
图 11  超参数研究结果
1 FARKAS A, KERTÉSZ G, LOVAS R. Parallel and distributed training of deep neural networks: a brief overview [C]// Proceedings of the IEEE 24th International Conference on Intelligent Engineering Systems. Reykjavík: IEEE, 2020: 165–170.
2 VERBRAEKEN J, WOLTING M, KATZY J, et al A survey on distributed machine learning[J]. ACM Computing Surveys, 2021, 53 (2): 1- 33
3 卢凯, 赖志权, 李笙维, 等 并行智能训练技术: 挑战与发展[J]. 中国科学: 信息科学, 2023, 53 (8): 1441- 1468
LU Kai, LAI Zhiquan, LI Shengwei, et al Parallel intelligent computing: development and challenges[J]. Scientia Sinica: Informationis, 2023, 53 (8): 1441- 1468
doi: 10.1360/SSI-2023-0051
4 LI M, ANDERSEN D G, SMOLA A, et al. Communication efficient distributed machine learning with the parameter server [C]// Proceeding of the 28th International Conference on Neural Information Processing Systems: Volume 1. Cambridge: MIT Press, 2014: 19–27.
5 LI S, ZHAO Y, VARMA R, et al. PyTorch distributed: experiences on accelerating data parallel training [EB/OL]. (2022−06−28)[2025−12−09]. https://arxiv.org/pdf/2006.15704.
6 NARAYANAN D, SHOEYBI M, CASPER J, et al. Efficient large-scale language model training on GPU clusters using megatron-LM [C]// Proceedings of the SC21: International Conference for High Performance Computing, Networking, Storage and Analysis. St. Louis: IEEE, 2022: 1–14.
7 HUANG Y, CHENG Y, BAPNA A, et al. GPipe: efficient training of giant neural networks using pipeline parallelism [EB/OL]. (2019−07−25)[2025−12−09]. https://arxiv.org/pdf/1811.06965.
8 NARAYANAN D, HARLAP A, PHANISHAYEE A, et al. PipeDream: generalized pipeline parallelism for DNN training [C]// Proceedings of the 27th ACM Symposium on Operating Systems Principles. Huntsville: ACM, 2019: 1–15.
9 KIM T, KIM H, YU G I, et al. BPIPE: memory-balanced pipeline parallelism for training large language models [C]// Proceedings of the 40th International Conference on Machine Learning. Honolulu: PMLR, 2023: 16639–16653.
10 ALISTARH D, GRUBIC D, LI J, et al. QSGD: communication-efficient SGD via gradient quantization and encoding [C]// Proceedings of the 31th Conference on Neural Information Processing Systems. Long Beach: [s.n.], 2017: 1–12.
11 GUO J, WU J, WANG Z, et al. Compressing large language models by joint sparsification and quantization [C]// Proceedings of the International Conference on Machine Learning. Vienna: PMLR, 2024: 1–13.
12 RENGGLI C, ASHKBOOS S, AGHAGOLZADEH M, et al. SparCML: high-performance sparse communication for machine learning [C]// Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis. Denver: ACM, 2019: 1–15.
13 陈世达, 刘强, 韩亮 降低分布式训练通信的梯度稀疏压缩方法[J]. 浙江大学学报: 工学版, 2021, 55 (2): 386- 394
CHEN Shida, LIU Qiang, HAN Liang Gradient sparsification compression approach to reducing communication in distributed training[J]. Journal of Zhejiang University: Engineering Science, 2021, 55 (2): 386- 394
doi: 10.3785/j.issn.1008-973X.2021.02.019
14 CHEN T, XU B, ZHANG C, et al. Training deep nets with sublinear memory cost [EB/OL]. (2016−04−22)[2025−12−09]. https://arxiv.org/pdf/1604.06174.
15 BEAUMONT O, EYRAUD-DUBOIS L, SHILOVA A. Efficient combination of rematerialization and offloading for training DNNs [C]// Proceeding of the 35th International Conference on Neural Information Processing Systems. [S.l.]: Curran Associates Inc., 2021, 34: 23844–23857.
16 SHOEYBI M, PATWARY M, PURI R, et al. Megatron-LM: training multi-billion parameter language models using model parallelism [EB/OL]. (2020−03−13)[2025−12−09]. https://arxiv.org/pdf/1909.08053.
17 RASLEY J, RAJBHANDARI S, RUWASE O, et al. DeepSpeed: system optimizations enable training deep learning models with over 100 billion parameters [C]// Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. [S.l.]: ACM, 2020: 3505–3506.
18 LANGER M, HE Z, RAHAYU W, et al Distributed training of deep learning models: a taxonomic perspective[J]. IEEE Transactions on Parallel and Distributed Systems, 2020, 31 (12): 2802- 2818
doi: 10.1109/TPDS.2020.3003307
19 GUAN L, LI D S, LIANG J Y, et al Advances of pipeline model parallelism for deep learning training: an overview[J]. Journal of Computer Science and Technology, 2024, 39 (3): 567- 584
doi: 10.1007/s11390-024-3872-3
20 张贵鹏, 孙毓忠 大模型训练的混合并行技术综述[J]. 计算机学报, 2026, 49 (1): 109- 131
ZHANG Guipeng, SUN Yuzhong A survey on hybrid parallelism techniques for large model training[J]. Chinese Journal of Computers, 2026, 49 (1): 109- 131
doi: 10.11897/SP.J.1016.2026.00109
21 YANG B, ZHANG J, LI J, et al. PipeMare: asynchronous pipeline parallel DNN training [C]// Proceedings of the 4th Machine Learning and Systems Conference. San Jose: PMLR, 2021, 3: 269–296.
22 NARAYANAN D, PHANISHAYEE A, SHI K, et al. Memory-efficient pipeline-parallel DNN training [C]// Proceedings of the 38th International Conference on Machine Learning. [S.l.]: PMLR, 2021: 7937–7947.
23 AJANTHAN T, RAMASINGHE S, ZUO Y, et al. Nesterov method for asynchronous pipeline parallel optimization [C]// Proceedings of the 42nd International Conference on Machine Learning. Vancouver: PMLR, 2025: 909–924.
24 FAN S, RONG Y, MENG C, et al. DAPPLE: a pipelined data parallel approach for training large models [C]// Proceedings of the 26th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming. [S.l.]: ACM, 2021: 431–445.
25 LAMY-POIRIER J. Breadth-first pipeline parallelism [C]// Proceedings of the 6th Machine Learning and Systems Conference. Miami: PMLR, 2023, 5: 48–67.
26 LIU Z, CHENG S, ZHOU H, et al. Hanayo: harnessing wave-like pipeline parallelism for enhanced large model training efficiency [C]// Proceedings of the SC23: International Conference for High Performance Computing, Networking, Storage and Analysis. Denver: IEEE, 2024: 1–13.
27 QI P, WAN X, HUANG G, et al. Zero bubble (almost) pipeline parallelism [C]// Proceedings of the International Conference on Learning Representations. Vienna: ICLR, 2024: 1–16.
28 JAIN A, AHMAD AWAN A, ALJUHANI A M, et al. GEMS: GPU-enabled memory-aware model-parallelism system for distributed DNN training [C]// Proceedings of the SC20: International Conference for High Performance Computing, Networking, Storage and Analysis. Atlanta: IEEE, 2021: 1–15.
29 LI S, HOEFLER T. Chimera: efficiently training large-scale neural networks with bidirectional pipelines [C]// Proceedings of the SC21: International Conference for High Performance Computing, Networking, Storage and Analysis. St. Louis: IEEE, 2022: 1–14.
30 ZHANG W, ZHOU B, TANG X, et al. MixPipe: efficient bidirectional pipeline parallelism for training large-scale models [C]// Proceedings of the 60th ACM/IEEE Design Automation Conference. San Francisco: IEEE, 2023: 1–6.
31 LIN J, LIU Z, YOU Y, et al. WeiPipe: weight pipeline parallelism for communication-effective long-context large model training [C]// Proceedings of the 30th ACM SIGPLAN Annual Symposium on Principles and Practice of Parallel Programming. Las Vegas: ACM, 2025: 225–238.
32 WU H, CHEN L. TawPipe: topology-aware weight pipeline parallelism for accelerating long-context large models training [C]// Proceedings of the AAAI Conference on Artificial Intelligence. Singapore: AAAI, 2026, 40(32): 26894–26902.
33 赵海燕, 易庆奥, 汤敬华, 等 分布式模型训练中的通信优化方法: 现状及展望[J]. 小型微型计算机系统, 2024, 45 (12): 2964- 2978
ZHAO Haiyan, YI Qing’ao, TANG Jinghua, et al Communication optimization in distributed model training: current research and prospects[J]. Journal of Chinese Computer Systems, 2024, 45 (12): 2964- 2978
doi: 10.20009/j.cnki.21-1106/TP.2024-0123
34 LIU W, LU K, LAI Z, et al AutoPipe-H: a heterogeneity-aware data-paralleled pipeline approach on commodity GPU servers[J]. IEEE Transactions on Computers, 2025, 74 (4): 1196- 1209
doi: 10.1109/TC.2024.3517748
35 VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need [C]// 31st Conference on Neural Information Processing Systems. Long Beach: Curran Associates Inc., 2017: 1–11.
36 DEVLIN J, CHANG M W, LEE K, et al. BERT: pre-training of deep bidirectional transformers for language understanding [C]// Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Minneapolis: ACL, 2019: 4171–4186.
37 BROWN T B, MANN B, RYDER N, et al. Language models are few-shot learners [C]// Proceedings of the 34th International Conference on Neural Information Processing Systems. [S.l.]: Curran Associates Inc., 2020, 33: 1877–1901.
38 PETERSON J, MEYLAN S, BOURGIN D. Open clone of OpenAI’s unreleased webtext dataset scraper [EB/OL]. (2020−08−15)[2025−12−09]. https://github.com/jcpeterson/openwebtext.
39 SUN Z, CAO H, WANG Y, et al. AdaPipe: optimizing pipeline parallelism with adaptive recomputation and partitioning [C]// Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3. La Jolla: ACM, 2024: 86–100.
[1] 杨恒,王韶涵,董青,赵科渊,杨明亮. 基于注意力增强的跨模态多层融合网络的物体位姿估计[J]. 浙江大学学报(工学版), 2026, 60(9): 1872-1880.
[2] 林洪彬,吕思进,王晨阳,蔡天放,骆鹏伟. 基于残差/梯度高斯自适应采样的径向基网络[J]. 浙江大学学报(工学版), 2026, 60(5): 1119-1127.
[3] 李子祥,陆克成,蔡海兵,解伟帅,张广东. 基于触觉仿生技术的黑暗环境混凝土裂缝检测[J]. 浙江大学学报(工学版), 2026, 60(5): 915-925.
[4] 吴杰,韩贝林,张舣航,邹超,辛莉峰,黄仕平. 微调稳定扩散模型的裂缝图像数据集扩充方法[J]. 浙江大学学报(工学版), 2026, 60(5): 926-934.
[5] 武晓春,郭宁. 基于HMARU-net的隧道渗漏水轻量化检测方法[J]. 浙江大学学报(工学版), 2026, 60(3): 468-477.
[6] 林乐平,李量,欧阳宁. 基于双偏振雷达的高时空分辨率临近降水预报[J]. 浙江大学学报(工学版), 2026, 60(3): 574-584.
[7] 杨明辉,宋牧原,付大喜,郭炎伟,卢贤锥,张文聪,郑伟龙. 基于多头自注意力-Bi-LSTM模型的盾构掘进引发的土体沉降预测[J]. 浙江大学学报(工学版), 2026, 60(2): 415-424.
[8] 曹铭,谌道宽,原鹏飞,周锦鹏,周奎. 基于深度学习的桌面级3D打印参数实时优化方法[J]. 浙江大学学报(工学版), 2026, 60(11): 2355-2363.
[9] 胡欣,阎希玥,常娅姝,程鸿亮,肖剑,罗诗伟,马亮. 基于上下文增强与多目标语义感知的电力缺陷检测算法[J]. 浙江大学学报(工学版), 2026, 60(10): 2176-2185.
[10] 孙月,张兴兰. 基于双重引导的目标对抗攻击方法[J]. 浙江大学学报(工学版), 2026, 60(1): 81-89.
[11] 朱志航,闫云凤,齐冬莲. 基于扩散模型多模态提示的电力人员行为图像生成[J]. 浙江大学学报(工学版), 2026, 60(1): 43-51.
[12] 段继忠,李海源. 基于变分模型和Transformer的多尺度并行磁共振成像重建[J]. 浙江大学学报(工学版), 2025, 59(9): 1826-1837.
[13] 王福建,张泽天,陈喜群,王殿海. 基于多通道图聚合注意力机制的共享单车借还量预测[J]. 浙江大学学报(工学版), 2025, 59(9): 1986-1995.
[14] 张弘,张学成,王国强,顾潘龙,江楠. 基于三维视觉的软体机器人实时定位与控制[J]. 浙江大学学报(工学版), 2025, 59(8): 1574-1582.
[15] 章东平,王大为,何数技,汤斯亮,刘志勇,刘中秋. 基于跨维度特征融合的航空发动机寿命预测[J]. 浙江大学学报(工学版), 2025, 59(7): 1504-1513.