Publications

Xonar: Profiling-based Job Orderer for Distributed Deep Learning

2022 IEEE 15th International Conference on Cloud Computing (CLOUD), work-in-progress, Acceptance rate 25.5% (49/192), Barcelona, Spain, 2022, pp. 112-114, doi: 10.1109/CLOUD55607.2022.00030.

Changyong Shin, Gyeongsik Yang, Yeonho Yoo, Jeunghwan Lee, Chuck Yoo

Download PDF

Abstract

Deep learning models have a wide spectrum of GPU execution time and memory size. When running distributed training jobs, however, their GPU execution time and memory size have not been taken into account, which leads to the high variance of job completion time (JCT). Moreover, the jobs often run into the GPU out-of-memory (OoM) problem so that the unlucky job has to restart all over. To address the problems, we propose Xonar to profile the deep learning jobs and order them in the queue. The experiments show that Xonar with TensorFlow v1.6 reduces the tail JCT by 44% with the OoM problem eliminated.

Paper summary

Pair training jobs by their active and idle periods

Two training jobs can share a GPU more effectively when their work fits into each other’s idle periods.

Distributed training alternates GPU computation with idle periods such as communication. Xonar profiles those periods and peak GPU memory, then uses them to choose which queued jobs should run together.

Filter by memory, then match activity

The profiler instruments TensorFlow operations to measure active and idle duration. The orderer first excludes jobs that do not fit the remaining memory, then selects a partner with a complementary active-to-idle ratio.

The profiler measures queued jobs and the orderer chooses a complementary pair for GPU execution.View full-size figure
The profiler measures queued jobs and the orderer chooses a complementary pair for GPU execution.

Compare completion time and memory failures

Using offline profiles of 600 configurations, the evaluation samples queues of ten jobs and repeats the experiment 50 times. Xonar reduces average job completion time by up to 43.6% versus serial training and 13.7% versus ordinary parallel training. No out-of-memory failures occur in the evaluated Xonar runs.

Reported average job-completion-time reductions with Xonar.
ComparisonReduction
Versus serial trainingUp to 43.6%
Versus ordinary parallel trainingUp to 13.7%

Key takeaways

  1. Good job pairs are complementary over time. Profiling exposes opportunities that queue order alone cannot identify.
  2. Complementarity is useful only within memory capacity. Filtering by peak memory keeps the pairing decision grounded in whether the jobs can coexist.