Please wait a minute...
浙江大学学报(工学版)
Journal of ZheJiang University (Engineering Science)  2026, Vol. 60 Issue (11): 2395-2405    DOI: 10.3785/j.issn.1008-973X.2026.11.009
    
Bidirectional interleaved pipeline parallelism for accelerating large model training
Houming WU1,2(),Ling CHEN1,2,*()
1. State Key Laboratory of Blockchain and Data Security, Zhejiang University, Hangzhou 310027, China
2. College of Computer Science and Technology, Zhejiang University, Hangzhou 310027, China
Download: HTML     PDF(1112KB) HTML
Export: BibTeX | EndNote (RIS)      

Abstract  

Existing synchronous pipeline parallelism methods introduce substantial pipeline bubbles due to inter-stage dependencies and periodic parameter synchronization, which reduce device utilization and training efficiency. To address this limitation, a bidirectional interleaved pipeline parallelism method named BitPipe for accelerating large-scale model training was proposed. Within a unified framework, bidirectional pipeline execution and interleaved scheduling were integrated. A bidirectional interleaved pipeline scheduling mechanism was designed to improve the utilization of idle device time slots and support finer-grained stage scheduling. To alleviate the communication overhead incurred by increased parallelism, a V-shaped interleaved model partitioning strategy was introduced to convert part of cross-device communication into local memory access. In addition, device mapping and eager gradient synchronization were employed to enable effective overlap between computation and communication. The proposed method was implemented based on the Megatron-LM framework and evaluated on GPT-like and BERT-like models using up to 32 GPUs. Experimental results show that BitPipe achieves a training throughput improvement of 1.05×–1.28× compared with existing synchronous pipeline parallelism methods. Moreover, BitPipe effectively reduces pipeline bubbles and improves training efficiency in large-scale model training.

Key words： deep learning      distributed training      pipeline parallelism      bidirectional pipeline      interleaved pipeline     
Received: 09 December 2025      Published: 16 October 2026
CLC:  TP 18  
Corresponding Authors: Ling CHEN     E-mail: houmingwu@zju.edu.cn;lingchen@cs.zju.edu.cn
Service
E-mail this article
Add to my bookshelf
Add to citation manager
E-mail Alert
RSS
Articles by authors
Houming WU
Ling CHEN
Cite this article:

Houming WU,Ling CHEN. Bidirectional interleaved pipeline parallelism for accelerating large model training. Journal of ZheJiang University (Engineering Science), 2026, 60(11): 2395-2405.

URL:

https://www.zjujournals.com/eng/10.3785/j.issn.1008-973X.2026.11.009     OR     https://www.zjujournals.com/eng/Y2026/V60/I11/2395


加速大模型训练的双向交错流水线并行方法

现有同步流水线并行方法因阶段间依赖和周期性参数同步引入大量流水线空泡，降低了设备利用率和训练效率，为此提出加速大模型训练的双向交错流水线并行方法BitPipe. 在统一框架下融合双向流水线执行与交错调度，设计双向交错流水线调度机制，以提升设备空闲时隙的利用率并支持更细粒度的阶段调度. 为了缓解并行度提升带来的通信开销，引入V形交错式模型划分策略，将部分跨设备通信转化为本地显存访问，结合设备映射与提前梯度同步优化，实现计算与通信的有效重叠. 基于Megatron-LM框架，所提方法在最多32张GPU上对GPT类与BERT类模型进行实验评估. 结果表明，相较于现有同步流水线并行方法，BitPipe实现了1.05×~1.28×的训练吞吐率提升，有效降低了流水线空泡并提高了训练效率.


关键词： 深度学习,  分布式训练,  流水线并行,  双向流水线,  交错流水线 
符号含义符号含义
$ D $流水线深度$ N $小批次中的微批次数量
$ W $数据并行维度$ \hat{B} $小批次大小（$ \hat{B}=BNW $）
$ P $设备总数（$ P=WD $）$ {M}_{\text{θ}} $单个流水线阶段的权重显存占用
$ B $微批次大小$ {M}_{\text{a}} $单个微批次的激活值显存占用
Tab.1 Key notations and definitions
Fig.1 Comparison between cyclic interleaved partitioning and V-shaped interleaved partitioning
Fig.2 Model partitioning and bidirectional interleaved scheduling of BitPipe (both pipeline depth and number of micro-batches are four)
流水线方法模型划分调度范式设备映射同步时机关键约束
GPipe[7]顺序单向顺序计算后无
DAPPLE[24]顺序单向顺序计算后无
1F1B-Int[6]循环单向交错顺序计算后无
Chimera[29]顺序双向顺序提前同步$ D $为偶数
MixPipe[30]顺序双向跨节点优化提前同步$ D $为偶数
BitPipeV形双向交错跨节点优化提前同步$ D $为偶数
Tab.2 Mechanism comparison of synchronous pipeline parallelism methods (number of micro-batches not less than pipeline depth)
Fig.3 Comparison of different synchronous pipeline parallelism methods in model partitioning and micro-batch scheduling (both pipeline depth and number of micro-batches are four)
流水线方法空泡率参数显存激活值显存
GPipe[7]$ \dfrac{D-1}{N+D+1} $$ {M}_{\text{θ}} $$ {N\times M}_{\text{a}} $
DAPPLE[24]$ \dfrac{D-1}{N+D+1} $$ {M}_{\text{θ}} $$ [{{M}}_{\text{a}},{D}\times {{M}}_{\text{a}}] $
1F1B-Int[6]$ \dfrac{{D}-1}{2{N}+{D}+1} $$ {{M}}_{\text{θ}} $$ \left[\dfrac{{D}+1}{2}{{M}}_{\text{a}},{D}\times {{M}}_{\text{a}}\right] $
Chimera[29]$ \dfrac{{D}-2}{3{N}/2+{D}-2} $$ 2{{M}}_{\text{θ}} $$ \left[\dfrac{{D}+2}{2}{{M}}_{\text{a}},{D}\times {{M}}_{\text{a}}\right] $
BitPipe$ \dfrac{{D}-2}{3{N}+{D}-2} $$ 2{{M}}_{\text{θ}} $$ \left[\dfrac{{D}+3}{2}{{M}}_{\text{a}},{D}\times {{M}}_{\text{a}}\right] $
Tab.3 Comparison on bubble ratio and memory consumption
Fig.4 Eager gradient synchronization by filling pipeline bubble slots
Fig.5 Device mapping strategy for bidirectional pipelines
Fig.6 Scaling to larger micro-batch number (pipeline depth is four, number of micro-batches is eight)
Fig.7 Scaling to larger pipeline stage numbers (pipeline depth is four)
模型层数注意力头数隐藏维大小序列长度
BERT-64 (5B)64642560512
GPT-96 (11B)963230721024
Tab.4 Benchmark models used for evaluation
Fig.8 Memory footprint distribution under pipeline parallelism (8 GPU)
Fig.9 Throughput comparison of pipeline parallelism and pipeline-data parallelism methods
模型参数备选值8 GPU16 GPU32 GPU
AIMBAIMBAIMB
E$ W ${1, 2, 4, 8}221144444844
$ D ${4, 8, 16}448844448488
$ B ${1, 2, 4, 8}444444442244
G$ W ${1, 2, 4}111122224444
$ D ${8, 16}888888888888
$ B ${1, 2}111111111111
Tab.5 Search space of parameters and their final choices
Fig.10 Memory footprint distribution under pipeline-data parallelism (32 GPU)
$ D $$ \hat{B} $T/（sample·s?1）
w/o Vw/o Ew/o MBitPipe
41619.4818.8219.5819.58
46424.1924.1124.2824.28
83238.7838.3239.1739.17
812846.2946.0546.4346.43
166449.2149.0948.5650.67
1625655.9657.0555.2357.44
Tab.6 Results of module ablation study
Fig.11 Hyperparameter study results
[1]   FARKAS A, KERTÉSZ G, LOVAS R. Parallel and distributed training of deep neural networks: a brief overview [C]// Proceedings of the IEEE 24th International Conference on Intelligent Engineering Systems. Reykjavík: IEEE, 2020: 165–170.
[2]   VERBRAEKEN J, WOLTING M, KATZY J, et al A survey on distributed machine learning[J]. ACM Computing Surveys, 2021, 53 (2): 1- 33
[3]   卢凯, 赖志权, 李笙维, 等 并行智能训练技术: 挑战与发展[J]. 中国科学: 信息科学, 2023, 53 (8): 1441- 1468
LU Kai, LAI Zhiquan, LI Shengwei, et al Parallel intelligent computing: development and challenges[J]. Scientia Sinica: Informationis, 2023, 53 (8): 1441- 1468
doi: 10.1360/SSI-2023-0051
[4]   LI M, ANDERSEN D G, SMOLA A, et al. Communication efficient distributed machine learning with the parameter server [C]// Proceeding of the 28th International Conference on Neural Information Processing Systems: Volume 1. Cambridge: MIT Press, 2014: 19–27.
[5]   LI S, ZHAO Y, VARMA R, et al. PyTorch distributed: experiences on accelerating data parallel training [EB/OL]. (2022−06−28)[2025−12−09]. https://arxiv.org/pdf/2006.15704.
[6]   NARAYANAN D, SHOEYBI M, CASPER J, et al. Efficient large-scale language model training on GPU clusters using megatron-LM [C]// Proceedings of the SC21: International Conference for High Performance Computing, Networking, Storage and Analysis. St. Louis: IEEE, 2022: 1–14.
[7]   HUANG Y, CHENG Y, BAPNA A, et al. GPipe: efficient training of giant neural networks using pipeline parallelism [EB/OL]. (2019−07−25)[2025−12−09]. https://arxiv.org/pdf/1811.06965.
[8]   NARAYANAN D, HARLAP A, PHANISHAYEE A, et al. PipeDream: generalized pipeline parallelism for DNN training [C]// Proceedings of the 27th ACM Symposium on Operating Systems Principles. Huntsville: ACM, 2019: 1–15.
[9]   KIM T, KIM H, YU G I, et al. BPIPE: memory-balanced pipeline parallelism for training large language models [C]// Proceedings of the 40th International Conference on Machine Learning. Honolulu: PMLR, 2023: 16639–16653.
[10]   ALISTARH D, GRUBIC D, LI J, et al. QSGD: communication-efficient SGD via gradient quantization and encoding [C]// Proceedings of the 31th Conference on Neural Information Processing Systems. Long Beach: [s.n.], 2017: 1–12.
[11]   GUO J, WU J, WANG Z, et al. Compressing large language models by joint sparsification and quantization [C]// Proceedings of the International Conference on Machine Learning. Vienna: PMLR, 2024: 1–13.
[12]   RENGGLI C, ASHKBOOS S, AGHAGOLZADEH M, et al. SparCML: high-performance sparse communication for machine learning [C]// Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis. Denver: ACM, 2019: 1–15.
[13]   陈世达, 刘强, 韩亮 降低分布式训练通信的梯度稀疏压缩方法[J]. 浙江大学学报: 工学版, 2021, 55 (2): 386- 394
CHEN Shida, LIU Qiang, HAN Liang Gradient sparsification compression approach to reducing communication in distributed training[J]. Journal of Zhejiang University: Engineering Science, 2021, 55 (2): 386- 394
doi: 10.3785/j.issn.1008-973X.2021.02.019
[14]   CHEN T, XU B, ZHANG C, et al. Training deep nets with sublinear memory cost [EB/OL]. (2016−04−22)[2025−12−09]. https://arxiv.org/pdf/1604.06174.
[15]   BEAUMONT O, EYRAUD-DUBOIS L, SHILOVA A. Efficient combination of rematerialization and offloading for training DNNs [C]// Proceeding of the 35th International Conference on Neural Information Processing Systems. [S.l.]: Curran Associates Inc., 2021, 34: 23844–23857.
[16]   SHOEYBI M, PATWARY M, PURI R, et al. Megatron-LM: training multi-billion parameter language models using model parallelism [EB/OL]. (2020−03−13)[2025−12−09]. https://arxiv.org/pdf/1909.08053.
[17]   RASLEY J, RAJBHANDARI S, RUWASE O, et al. DeepSpeed: system optimizations enable training deep learning models with over 100 billion parameters [C]// Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. [S.l.]: ACM, 2020: 3505–3506.
[18]   LANGER M, HE Z, RAHAYU W, et al Distributed training of deep learning models: a taxonomic perspective[J]. IEEE Transactions on Parallel and Distributed Systems, 2020, 31 (12): 2802- 2818
doi: 10.1109/TPDS.2020.3003307
[19]   GUAN L, LI D S, LIANG J Y, et al Advances of pipeline model parallelism for deep learning training: an overview[J]. Journal of Computer Science and Technology, 2024, 39 (3): 567- 584
doi: 10.1007/s11390-024-3872-3
[20]   张贵鹏, 孙毓忠 大模型训练的混合并行技术综述[J]. 计算机学报, 2026, 49 (1): 109- 131
ZHANG Guipeng, SUN Yuzhong A survey on hybrid parallelism techniques for large model training[J]. Chinese Journal of Computers, 2026, 49 (1): 109- 131
doi: 10.11897/SP.J.1016.2026.00109
[21]   YANG B, ZHANG J, LI J, et al. PipeMare: asynchronous pipeline parallel DNN training [C]// Proceedings of the 4th Machine Learning and Systems Conference. San Jose: PMLR, 2021, 3: 269–296.
[22]   NARAYANAN D, PHANISHAYEE A, SHI K, et al. Memory-efficient pipeline-parallel DNN training [C]// Proceedings of the 38th International Conference on Machine Learning. [S.l.]: PMLR, 2021: 7937–7947.
[23]   AJANTHAN T, RAMASINGHE S, ZUO Y, et al. Nesterov method for asynchronous pipeline parallel optimization [C]// Proceedings of the 42nd International Conference on Machine Learning. Vancouver: PMLR, 2025: 909–924.
[24]   FAN S, RONG Y, MENG C, et al. DAPPLE: a pipelined data parallel approach for training large models [C]// Proceedings of the 26th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming. [S.l.]: ACM, 2021: 431–445.
[25]   LAMY-POIRIER J. Breadth-first pipeline parallelism [C]// Proceedings of the 6th Machine Learning and Systems Conference. Miami: PMLR, 2023, 5: 48–67.
[26]   LIU Z, CHENG S, ZHOU H, et al. Hanayo: harnessing wave-like pipeline parallelism for enhanced large model training efficiency [C]// Proceedings of the SC23: International Conference for High Performance Computing, Networking, Storage and Analysis. Denver: IEEE, 2024: 1–13.
[27]   QI P, WAN X, HUANG G, et al. Zero bubble (almost) pipeline parallelism [C]// Proceedings of the International Conference on Learning Representations. Vienna: ICLR, 2024: 1–16.
[28]   JAIN A, AHMAD AWAN A, ALJUHANI A M, et al. GEMS: GPU-enabled memory-aware model-parallelism system for distributed DNN training [C]// Proceedings of the SC20: International Conference for High Performance Computing, Networking, Storage and Analysis. Atlanta: IEEE, 2021: 1–15.
[29]   LI S, HOEFLER T. Chimera: efficiently training large-scale neural networks with bidirectional pipelines [C]// Proceedings of the SC21: International Conference for High Performance Computing, Networking, Storage and Analysis. St. Louis: IEEE, 2022: 1–14.
[30]   ZHANG W, ZHOU B, TANG X, et al. MixPipe: efficient bidirectional pipeline parallelism for training large-scale models [C]// Proceedings of the 60th ACM/IEEE Design Automation Conference. San Francisco: IEEE, 2023: 1–6.
[31]   LIN J, LIU Z, YOU Y, et al. WeiPipe: weight pipeline parallelism for communication-effective long-context large model training [C]// Proceedings of the 30th ACM SIGPLAN Annual Symposium on Principles and Practice of Parallel Programming. Las Vegas: ACM, 2025: 225–238.
[32]   WU H, CHEN L. TawPipe: topology-aware weight pipeline parallelism for accelerating long-context large models training [C]// Proceedings of the AAAI Conference on Artificial Intelligence. Singapore: AAAI, 2026, 40(32): 26894–26902.
[33]   赵海燕, 易庆奥, 汤敬华, 等 分布式模型训练中的通信优化方法: 现状及展望[J]. 小型微型计算机系统, 2024, 45 (12): 2964- 2978
ZHAO Haiyan, YI Qing’ao, TANG Jinghua, et al Communication optimization in distributed model training: current research and prospects[J]. Journal of Chinese Computer Systems, 2024, 45 (12): 2964- 2978
doi: 10.20009/j.cnki.21-1106/TP.2024-0123
[34]   LIU W, LU K, LAI Z, et al AutoPipe-H: a heterogeneity-aware data-paralleled pipeline approach on commodity GPU servers[J]. IEEE Transactions on Computers, 2025, 74 (4): 1196- 1209
doi: 10.1109/TC.2024.3517748
[35]   VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need [C]// 31st Conference on Neural Information Processing Systems. Long Beach: Curran Associates Inc., 2017: 1–11.
[36]   DEVLIN J, CHANG M W, LEE K, et al. BERT: pre-training of deep bidirectional transformers for language understanding [C]// Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Minneapolis: ACL, 2019: 4171–4186.
[37]   BROWN T B, MANN B, RYDER N, et al. Language models are few-shot learners [C]// Proceedings of the 34th International Conference on Neural Information Processing Systems. [S.l.]: Curran Associates Inc., 2020, 33: 1877–1901.
[38]   PETERSON J, MEYLAN S, BOURGIN D. Open clone of OpenAI’s unreleased webtext dataset scraper [EB/OL]. (2020−08−15)[2025−12−09]. https://github.com/jcpeterson/openwebtext.
[39]   SUN Z, CAO H, WANG Y, et al. AdaPipe: optimizing pipeline parallelism with adaptive recomputation and partitioning [C]// Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3. La Jolla: ACM, 2024: 86–100.
[1] Heng YANG,Shaohan WANG,Qing DONG,Keyuan ZHAO,Mingliang YANG. Object pose estimation based on attention enhancement cross-modal multilayer fusion network[J]. Journal of ZheJiang University (Engineering Science), 2026, 60(9): 1872-1880.
[2] Hongbin LIN,Sijin LV,Chenyang WANG,Tianfang CAI,Pengwei LUO. Radial basis network based on residual/gradient Gaussian adaptive sampling[J]. Journal of ZheJiang University (Engineering Science), 2026, 60(5): 1119-1127.
[3] Zixiang LI,Kecheng LU,Haibing CAI,Weishuai XIE,Guangdong ZHANG. Concrete crack detection in dark environments based on biomimetic tactile technology[J]. Journal of ZheJiang University (Engineering Science), 2026, 60(5): 915-925.
[4] Jie WU,Beilin HAN,Yihang ZHANG,Chao ZOU,Lifeng XIN,Shiping HUANG. Method for augmenting crack image datasets via fine-tuning of stable diffusion models[J]. Journal of ZheJiang University (Engineering Science), 2026, 60(5): 926-934.
[5] Xiaochun WU,Ning GUO. Lightweight detection method of water leakage in tunnel based on HMARU-net[J]. Journal of ZheJiang University (Engineering Science), 2026, 60(3): 468-477.
[6] Leping LIN,Liang LI,Ning OUYANG. High spatio-temporal resolution precipitation forecast based on dual-polarization radar[J]. Journal of ZheJiang University (Engineering Science), 2026, 60(3): 574-584.
[7] Minghui YANG,Muyuan SONG,Daxi FU,Yanwei GUO,Xianzhui LU,Wencong ZHANG,Weilong ZHENG. Prediction of shield tunneling-induced soil settlement based on multi-head self-attention-Bi-LSTM model[J]. Journal of ZheJiang University (Engineering Science), 2026, 60(2): 415-424.
[8] Ming CAO,Daokuan CHEN,Pengfei YUAN,Jinpeng ZHOU,Kui ZHOU. Real-time optimization method for desktop level 3D printing parameters based on deep learning[J]. Journal of ZheJiang University (Engineering Science), 2026, 60(11): 2355-2363.
[9] Xin HU,Xiyue YAN,Yashu CHANG,Hongliang CHENG,Jian XIAO,Shiwei LUO,Liang MA. Context-enhanced multi-target semantic perception algorithm for power defect detection[J]. Journal of ZheJiang University (Engineering Science), 2026, 60(10): 2176-2185.
[10] Zhihang ZHU,Yunfeng YAN,Donglian QI. Image generation for power personnel behaviors based on diffusion model with multimodal prompts[J]. Journal of ZheJiang University (Engineering Science), 2026, 60(1): 43-51.
[11] Yue SUN,Xinglan ZHANG. Targeted adversarial attack method based on dual guidance[J]. Journal of ZheJiang University (Engineering Science), 2026, 60(1): 81-89.
[12] Jizhong DUAN,Haiyuan LI. Multi-scale parallel magnetic resonance imaging reconstruction based on variational model and Transformer[J]. Journal of ZheJiang University (Engineering Science), 2025, 59(9): 1826-1837.
[13] Fujian WANG,Zetian ZHANG,Xiqun CHEN,Dianhai WANG. Usage prediction of shared bike based on multi-channel graph aggregation attention mechanism[J]. Journal of ZheJiang University (Engineering Science), 2025, 59(9): 1986-1995.
[14] Hong ZHANG,Xuecheng ZHANG,Guoqiang WANG,Panlong GU,Nan JIANG. Real-time positioning and control of soft robot based on three-dimensional vision[J]. Journal of ZheJiang University (Engineering Science), 2025, 59(8): 1574-1582.
[15] Dongping ZHANG,Dawei WANG,Shuji HE,Siliang TANG,Zhiyong LIU,Zhongqiu LIU. Remaining useful life prediction of aircraft engines based on cross-dimensional feature fusion[J]. Journal of ZheJiang University (Engineering Science), 2025, 59(7): 1504-1513.