Publications

Accurate Simulation of Distributed Training Jobs with Network Contention Modeling

The 34th IEEE International Symposium on the Modeling, Analysis, and Simulation of Computer and Telecommunication Systems (MASCOTS 2026), Acceptance rate 23%, Genova, Italy, 2026. (Accepted)

Yeonho Yoo, Hyunho Lee, Hyunmok Choi, Chuck Yoo, Gyeongsik Yang

Download PDF

Abstract

Trace-driven simulation is widely used to evaluate distributed training (DT) jobs in GPU clusters, but existing simulators either ignore network contention or approximate it with a fixed penalty. This misses how scheduling decisions determine which jobs share server network interfaces and inter-server links, thereby changing networking time during training. As a result, our motivating experiments demonstrate that they incur large errors, reaching up to 73.64% mean absolute percentage error (MAPE) in average job completion time (JCT). This paper introduces MoSim, a GPU-cluster simulator that models DT job execution under dynamic network contention. MoSim combines GPU-free characterization with network contention model: it obtains each job's compute time, networking time, and networking volume without GPUs, then uses the current worker assignment to estimate how shared network interfaces affect each job's iteration time. Our evaluation shows that, compared with existing simulators, MoSim reduces simulation error for average JCT by up to 3.28×, tail (99th-percentile) JCT by up to 7.79×, and makespan by up to 8.49×, while modeling NIC contention factors with only 8.63% error on average. By avoiding real-GPU profiling, MoSim also reduces input construction overhead by 44.6×.

Paper summary

Let placement change the simulated network cost

A training simulator becomes more faithful when it accounts for which running jobs actually share network capacity.

Distributed training repeatedly synchronizes gradients. A scheduler decides which jobs overlap and whether their workers share server network interfaces or inter-server links. Simulating every job at its isolated speed—or adding the same fixed slowdown—misses how these decisions change completion times.

Update contention as the cluster changes

MOSIM augments public job traces with workload descriptions and uses GPU-free characterization to obtain compute time, networking time and communication volume. A discrete-event simulator then applies scheduling decisions and tracks arrivals, completions and progress.

Network cost changes with the current placement and overlap of jobs.

The contention model derives demand on shared network interfaces from worker assignments and communication volume, then updates affected jobs’ iteration times. The paper’s bandwidth-demand formulation assumes synchronous data-parallel training with ring all-reduce.

Workload characterization supplies a discrete-event simulator whose network model updates job progress under contention.View full-size figure
Workload characterization supplies a discrete-event simulator whose network model updates job progress under contention.

Compare simulated times with a physical cluster

The main validation replays a 13-hour, 60-job workload derived from the Philly trace on four eight-V100 servers. Physical measurements provide ground truth under both bin-packing and load-balancing placement.

Under bin packing, MOSIM’s mean absolute percentage error for average job completion time is 11.10%, versus 36.38% for Tiresias and 33.00% for Pollux. Under load balancing, its P99 completion-time error is 8.38%, versus 65.28% and 63.89%. The errors compare simulated completion times with physical-cluster measurements.

Job-completion-time prediction MAPE against physical-cluster measurements; lower is better.
Metric / placementMOSIMTiresiasPollux
Average JCT / bin packing11.10%36.38%33.00%
P99 JCT / load balancing8.38%65.28%63.89%

Key takeaways

  1. Network contention changes with scheduling decisions. A simulator must update shared capacity as jobs overlap.
  2. GPU-free characterization can support cluster-level prediction. Separating compute and network behavior supplies the inputs used to reproduce completion-time behavior under different placements.