Publications

GeoMesh: Workload-Balanced and Sign-Compressed Geo-Distributed LLM Training

The 2026 Conference on Empirical Methods in Natural Language Processing (EMNLP 2026), Findings, Acceptance rate 14.3%, Budapest, Hungary, 2026.

Changyong Shin, Jaerim Park, Minchul Kang, Younghun Go, Zhixiong Niu, Yongqiang Xiong, Gyeongsik Yang, Chuck Yoo

Download PDF

Abstract

Large language models are increasingly trained on GPUs distributed across multiple regions, but geo-distributed training is challenging in practice. Real clusters often contain GPUs with different speeds and memory capacities, and they communicate over slow wide-area networks. Our analysis shows that this creates serious problems: existing synchronous methods preserve stable updates, but fast GPUs wait up to 20.9% of their runtime for slower ones, and all workers spend, on average, 65.8% of their runtime on synchronization. Recent asynchronous methods reduce waiting time but worsen the model accuracy due to stale updates. To address the problems, we present GeoMesh, a synchronous geo-distributed training framework for heterogeneous GPUs. GeoMesh balances per-worker workloads by assigning each GPU a suitable batch size and number of inner steps, so faster GPUs do more useful work instead of waiting. It also reduces communication volume by nearly 32× by exchanging compressed sign-based pseudo-gradients with lightweight magnitude and token count. Across heterogeneous GPUs and Azure-derived WAN, GeoMesh reduces time-to-target perplexity by up to 70.2% over representative baselines and lowers straggler- and WAN-induced GPU idle by up to 8.0× and 5.6×, respectively, while preserving comparable zero-shot accuracy.

Paper summary

Reduce two different kinds of GPU waiting

Unequal GPUs can train together more efficiently when each receives suitable work and exchanges smaller updates.

Synchronous training across regions makes faster GPUs wait for slower workers, then makes all workers wait for updates to cross the wide-area network. GeoMesh treats these as separate problems. It retains synchronized updates while changing both how work is assigned and how updates are communicated.

Balance local work and compress shared updates

A one-time profiler estimates each worker’s feasible batch size and training speed. GeoMesh assigns batch sizes and inner-step counts to bring workers’ local training times closer together according to each GPU’s measured capability.

Faster GPUs receive more local work to balance the time spent before synchronization.

Workers exchange bit-packed pseudo-gradient signs with magnitude and token-count metadata. Token-weighted aggregation accounts for their unequal workloads, while the magnitude information restores the update scale lost by sign compression.

Workload balancing assigns unequal work to heterogeneous GPUs; compressed synchronization combines their updates.View full-size figure
Workload balancing assigns unequal work to heterogeneous GPUs; compressed synchronization combines their updates.

Training with heterogeneous GPUs and an emulated WAN

The main evaluation uses H100, A100, L40S and RTX 6000 Ada workers, with WAN rates emulated from Azure inter-region measurements. The authors train 150M, 300M and 500M LLaMA-style models on C4.

For the 150M model, GeoMesh reaches perplexity 30 in 1.1 hours: 70.2% less time than DiLoCo and 43.8% less than HALoS. For 300M and 500M, it reaches the respective perplexity targets of 25 and 20 in 2.6 and 7.5 hours, versus HALoS’s 7.9 and 22.8 hours. DiLoCo does not reach those two targets within the stated 30- and 50-hour budgets. These main results are reported from one run per configuration.

Time to the stated perplexity target for GeoMesh and HALoS on C4.
Model / targetGeoMeshHALoS
300M / 252.6 h7.9 h
500M / 207.5 h22.8 h

Key takeaways

  1. Geographic training has two distinct waiting costs. Unequal worker speeds and WAN synchronization require workload balancing and communication reduction together.
  2. Unequal work must remain visible after compression. Token counts and magnitude metadata carry information needed to combine the updates.
  3. The measured benefit reaches training quality sooner. On the 150M model, GeoMesh reaches perplexity 30 ahead of both synchronous and asynchronous baselines.