Publications

TensorExpress: In-Network Communication Scheduling for Distributed Deep Learning

2020 IEEE 13th International Conference on Cloud Computing (CLOUD), work-in-progress, Acceptance rate 26.3%, Virtual (originally in, Beijing, China), 2020, pp. 25-27, doi: 10.1109/CLOUD49709.2020.00014.

Minkoo Kang, Gyeongsik Yang, Yeonho Yoo, Chuck Yoo

Download PDF

Abstract

TensorExpress provides in-network communication scheduling for distributed deep learning (DDL). In cloud-based DDL, parameter communication over a network is a key bottleneck. Previous studies proposed tensor packet reordering approaches to reduce network blocking time. However, network contention still exists in DDL. TensorExpress mitigates network contention and reduces overall training time. It schedules tensor packets in-network using P4, a switch programming language. TensorExpress improves latency and network blocking time up to 2.5 and 2.44 times, respectively.

Paper summary

Send the tensors needed next first

Network scheduling can prioritize the model layers that block the next training iteration.

Distributed training sends layer tensors through shared switch queues. TensorExpress uses layer information to prioritize communication that the next forward pass needs first.

Map layer dependencies to switch priorities

A ticketer places the layer identifier in the packet’s IP option field. A P4 switch reads it and maps layers to a limited set of priority queues. A sliding priority window updates that mapping as tensors from successive layers arrive.

Worker ticketers identify tensor layers, allowing the switch’s sliding priority window to assign queue priority.View full-size figure
Worker ticketers identify tensor layers, allowing the switch’s sliding priority window to assign queue priority.

Measure the wait for the next iteration

In the bmv2 evaluation, TensorExpress reduces network blocking time by 46% on average and by up to 59% for the evaluated workers. Network blocking time measures the transfer wait for layer-zero tensors.

Reduction in layer-zero tensor transfer waiting time in the bmv2 evaluation.
Network blocking timeReduction
Average across evaluated workers46%
Largest measured reduction59%

Key takeaways

  1. Tensor traffic has dependencies that ordinary packet queues cannot see. Layer metadata lets the switch prioritize communication according to training progress.
  2. A small queue budget can still express changing priorities. The sliding window reuses queues for the layers that matter at the current stage.