Publications

Training Time Prediction for Mixed Precision-based Distributed Training

arXiv preprint arXiv:2604.16145, 2026.

Minchul Kang, Changyong Shin, Jinwoo Jeong, Hyunho Lee, Younghun Go, Gyeongmin Kim, Gyeongsik Yang, Chuck Yoo

Download PDF

Abstract

Accurate prediction of training time in distributed deep learning is crucial for resource allocation, cost estimation, and job scheduling. We observe that the floating-point precision setting is a key determinant of training time, leading to training time variations of ~2.4x over its minimum. However, existing studies on distributed training time prediction rely on static model computation graphs that do not capture precision variations, including mixed precision. According to our experiments, training time prediction without considering precision results in significant prediction errors - reaching up to 147.85% in mean absolute percentage error (MAPE). To address this issue, we propose a precision-aware distributed training time predictor that achieves robust accuracy across diverse precision settings, including mixed precision, with 9.8% MAPE.

Paper summary

Predict the precision that each operation uses

A training-time estimate needs to account for mixed precision as well as how the model is split across GPUs.

Mixed-precision training changes the work and tensor sizes of individual operations. A static computation graph that assumes one precision can therefore misestimate both compute time and communication. This paper predicts a single training iteration with operator-level precision information.

Partition the graph and inspect precision

The predictor extracts operators and partitions the model into GPU-specific subgraphs using data, tensor and pipeline parallelism settings. For mixed precision, it observes the casts applied by automatic mixed precision and profiles operations with the resulting precision and tensor shapes.

It aggregates forward and backward operator times and adds modeled communication and pipeline overheads for the specified configuration. Precision thus informs both local computation and the amount of data exchanged.

Prediction MAPE across mixed-precision and FP16 settings; each horizontal-axis tuple gives data, tensor and pipeline parallelism degrees. Lower error is better.View full-size figure
Prediction MAPE across mixed-precision and FP16 settings; each horizontal-axis tuple gives data, tensor and pipeline parallelism degrees. Lower error is better.

Compare estimates across parallel strategies

Tests use LLaMA 3.1-8B on C4 with eight NVIDIA H100 GPUs. Across the evaluated parallelism combinations, average mean absolute percentage error is 9.8% for mixed precision and 10.64% for the FP16 setting described as unseen in the paper.

Prediction error is calculated from measured per-iteration times.

Average iteration-time prediction error for LLaMA 3.1-8B on eight H100 GPUs.
Precision settingMAPE
Mixed precision9.8%
FP1610.64%

Key takeaways

  1. Mixed precision changes operator types and adds casts to the execution graph. Operator-level casts belong in a training-time model alongside arithmetic work.
  2. Distributed iteration time includes coordination. Communication and pipeline effects explain why isolated computation estimates are insufficient.