Publications

Making Sense of Job Preemption for Distributed Deep Learning Acceleration

63rd ACM/IEEE Design Automation Conference (DAC '26), Acceptance rate 22.3%, Long Beach, California, USA, 2026. (Accepted)

Younghun Go, Changyong Shin, Minchul Kang, Jaehyun Hwang, Chuck Yoo, Gyeongsik Yang

Younghun Go, Changyong Shin, Minchul Kang, Jaehyun Hwang, Chuck Yoo, and Gyeongsik Yang. 2026. Making Sense of Job Preemption for Distributed Deep Learning Acceleration. 63rd ACM/IEEE Design Automation Conference (DAC ’26), July 26–29, 2026, Long Beach, CA, USA. ACM, 7 pages.

Download PDF

Abstract

Preemptive scheduling is gaining attention in GPU scheduling for distributed training because it reduces job completion times (JCT). However, our analysis of production traces reveals that, surprisingly, it can increase JCT in practice. We identify two key contributors to this inefficiency. First, preemptions become "futile" when a job is preempted right after being loaded, before it begins execution. We find this futile preemption wastes the load time and so inflates JCT ~1.6⨉. Second, existing GPU schedulers run at fixed intervals (e.g., 360 s) rather than upon new job arrivals or when a job completes. So, new jobs must wait until the scheduler kicks in, which, we find, increases JCT ~2.6⨉. To address the problems, we introduce Lazer, a novel job scheduler that predicts when to preempt jobs based on job-specific and cluster conditions. Lazer is designed to efficiently explore the scheduling space and adapt to diverse job and GPU cluster characteristics based on Bayesian optimization. Our extensive evaluation shows that Lazer significantly outperforms state-of-the-art schedulers---reducing JCT by 1.2⨉--233.3⨉, waiting time by 2⨉--11690⨉, and futile preemptions by 2⨉--67⨉.

Paper summary

When switching jobs costs more than it saves

Choosing when to pause GPU training can reduce wasted work and help jobs finish sooner.

Giving a short training job priority requires pausing a running job and loading a new model. If another job takes its place before training starts, that loading work is wasted. Lazer weighs a job’s priority against the cost of switching to it.

Two sources of avoidable delay

Scheduling at fixed intervals can prevent repeated switches, but makes newly arrived jobs wait. The challenge is to reduce both wasted loading, called futile preemption, and queueing delay.

The paper therefore measures job completion time (JCT) from arrival to completion, including waiting, loading, training, and saving work. The target is earlier job completion, including time spent waiting and moving models.

Choose the jobs, then choose when to switch

Lazer first identifies lower-priority jobs that could release enough GPUs for an arriving job. It then uses Bayesian optimization to choose a delay of 0–100 seconds, informed by recent arrivals, estimated remaining training time, and loading and saving costs.

Selected running jobs keep training during the delay. When it ends, the orchestrator chooses preemption candidates again using their updated states. A completion handler separately assigns freed GPUs to waiting jobs. Deferring one preemption does not block scheduling for newly arriving jobs.

Lazer architecture: scheduling orchestrator, deferral predictor, and completion handlerView full-size figure
Lazer separates job selection from the decision about when to preempt.

What the experiments show

To test whether this policy reduces completion time on real hardware, the authors use four servers with 16 heterogeneous NVIDIA GPUs and 40 GbE. They run 100 jobs sampled from the Earth trace while preserving its arrival-time distribution, with a dedicated scheduler CPU and separate model-storage server.

Each ratio divides the comparison scheduler’s JCT by Lazer’s; larger values favor Lazer. P95 is the 95th percentile of job completion time.

In this testbed, futile-preemption time is zero at both P50 and P95. The timing decisions add overhead, however: average scheduler decision latency is 7.1 ms, versus 3.8 ms for SJF and SRTF.

What larger trace simulations add

The authors separately replay the full Earth, Saturn, and Uranus traces in simulation. On Uranus, P95 JCT is 6,999.9 minutes for SJF and 29.9 minutes for Lazer, a comparison the paper reports as 233.3×. This comparison measures simulated tail job-completion time.

Job completion time on the physical testbed
Comparison schedulerMedian JCT / LazerP95 JCT / Lazer
SJF1.3×1.16×
SRTF1.5×1.2×
Pollux2.1×1.4×
Muri5.3×2.9×
Optimus8.8×3.8×

Key takeaways

  1. Preemption can waste work before useful training starts. Avoiding interrupted model loads with fixed scheduling intervals also creates waiting, exposing a tradeoff in when to switch.
  2. Timing is part of the scheduling decision. Lazer predicts a deferral from job and cluster conditions while selected jobs continue running.
  3. Less wasted loading can accompany shorter waits. On the 16-GPU, 100-job testbed, median JCT and waiting time improve by 8.8× and 22.2× over Optimus, with zero futile-preemption time at P50 and P95.