加速大模型训练的双向交错流水线并行方法

吴厚明,陈岭

Bidirectional interleaved pipeline parallelism for accelerating large model training

Houming WU,Ling CHEN

表 3 空泡率和显存占用对比

Tab.3 Comparison on bubble ratio and memory consumption

流水线方法 空泡率 参数显存 激活值显存 GPipe[7] $ \dfrac{D-1}{N+D+1} $ $ {M}_{\text{θ}} $ $ {N\times M}_{\text{a}} $ DAPPLE[24] $ \dfrac{D-1}{N+D+1} $ $ {M}_{\text{θ}} $ $ [{{M}}_{\text{a}},{D}\times {{M}}_{\text{a}}] $ 1F1B-Int[6] $ \dfrac{{D}-1}{2{N}+{D}+1} $ $ {{M}}_{\text{θ}} $ $ \left[\dfrac{{D}+1}{2}{{M}}_{\text{a}},{D}\times {{M}}_{\text{a}}\right] $ Chimera[29] $ \dfrac{{D}-2}{3{N}/2+{D}-2} $ $ 2{{M}}_{\text{θ}} $ $ \left[\dfrac{{D}+2}{2}{{M}}_{\text{a}},{D}\times {{M}}_{\text{a}}\right] $ BitPipe $ \dfrac{{D}-2}{3{N}+{D}-2} $ $ 2{{M}}_{\text{θ}} $ $ \left[\dfrac{{D}+3}{2}{{M}}_{\text{a}},{D}\times {{M}}_{\text{a}}\right] $