加速大模型训练的双向交错流水线并行方法

吴厚明,陈岭

Bidirectional interleaved pipeline parallelism for accelerating large model training

Houming WU,Ling CHEN

图 4 基于流水线空泡时隙填充的提前梯度同步

Fig.4 Eager gradient synchronization by filling pipeline bubble slots