Publications

Prediction-based GPU Sharing for Distributed Training

Future Generation Computer Systems, Volume 181, 108413, JCR 2023 IF 6.2 Top 9.4%, 2026.

Changyong Shin, Younghun Go, Yeonho Yoo, Jinwoo Jeong, Jaehyun Hwang, Gyeongsik Yang, Chuck Yoo

Download PDF

Abstract

GPU sharing aims to enhance the efficiency of GPU utilization by running distributed deep learning training jobs concurrently. However, GPU sharing poses a significant challenge: the increase in job completion time (JCT) caused by interference between jobs is inconsistent, complicating job scheduling. Our experiments reveal that the degree of JCT increase varies by as much as  ∼ 3.7 × . While previous studies have analyzed this JCT inconsistency problem, none of them have been able to minimize the inconsistency. We propose TensorShare, a proactive GPU sharing technique that leverages a deep learning model to predict the extent of JCT increase. This study defines a new metric, called GPU SLA, which represents the upper threshold of JCT increase. TensorShare then introduces a novel scheduler that proactively identifies which jobs meet GPU SLA while minimizing the JCT increase. Our evaluation shows that TensorShare improves GPU SLA satisfaction rates by 26.1 × –47.3 ×  and reduces the JCT increase by 37%–60%. Furthermore, we evaluate TensorShare with large language models that are not included in training TensorShare’s prediction model, achieving  ∼ 7 ×  and  ∼ 10.3 ×  improvements in GPU SLA satisfaction and JCT inconsistency, respectively.

Paper summary

Predict interference before sharing a GPU

GPU sharing is more useful when the scheduler accounts for how much each job can be slowed down.

Running training jobs together can improve GPU utilization, but interference changes completion time unevenly across job combinations. TensorShare introduces a GPU SLA: the maximum acceptable ratio of completion time under sharing to completion time with dedicated use.

Filter job pairs using predicted slowdown

An introspector profiles each job in isolation, and a learned predictor estimates slowdown for candidate pairs. The scheduler first checks matching worker counts, available GPU memory and both jobs’ GPU SLAs.

A feasible pairing must satisfy both GPU capacity and each job’s slowdown limit.

Among feasible pairs, TensorShare selects combinations to reduce completion time and GPU time. Jobs run alone when no pairing satisfies the constraints.

Job profiles feed slowdown prediction; the scheduler filters candidate pairs and selects feasible GPU-sharing combinations.View full-size figure
Job profiles feed slowdown prediction; the scheduler filters candidate pairs and selects feasible GPU-sharing combinations.

Measure SLA violations in a physical testbed

The physical evaluation uses two GPU machines managed through Kubernetes. Experiments form queues by sampling from a pool of 29 training jobs and compare TensorShare with ordinary and reactive GPU sharing.

TensorShare reports a 0.7% GPU-SLA violation rate, reducing violations by 47.3× relative to ordinary sharing and 30.3× relative to reactive sharing. Simulated production traces and unseen-model tests are evaluated separately.

TensorShare’s GPU-SLA violation results in the physical-cluster evaluation.
MetricResult
Violation rate0.7%
Reduction versus ordinary sharing47.3×
Reduction versus reactive sharing30.3×

Key takeaways

  1. GPU sharing is a constrained performance choice. Each job’s slowdown tolerance defines which utilization gains are acceptable.
  2. Interference is a property of the pair. Predicting both jobs’ behavior lets the scheduler reject unsuitable combinations before co-execution.
  3. Capacity and SLA checks precede optimization. Completion time and GPU time guide selection among pairs that satisfy those requirements.