Heterogeneous Parallelism for Multimodal Large Language Model Training

Akash Mehra; Ali Roshan Ghias; Ashwath Aithal; Eric Harper; Hao Wu; Kamran Jafari; Li Ding; Nima Tajbakhsh; Parth Mannan; Pranav Prashant Thombre

arxiv: 2605.27678 · v1 · pith:JSC4J7ABnew · submitted 2026-05-26 · 💻 cs.LG · cs.DC

Heterogeneous Parallelism for Multimodal Large Language Model Training

Yashaswi Karnati , Kamran Jafari , Akash Mehra , Li Ding , Pranav Prashant Thombre , Ali Roshan Ghias , Shifang Xu , Parth Mannan

show 5 more authors

Yu Yao Hao Wu Eric Harper Ashwath Aithal Nima Tajbakhsh

This is my paper

Pith reviewed 2026-06-29 18:39 UTC · model grok-4.3

classification 💻 cs.LG cs.DC

keywords heterogeneous parallelismmultimodal LLM trainingboundary communicatorscolocated executionnon-colocated executiontensor parallelismpipeline parallelismtraining throughput

0 comments

The pith

Multimodal LLMs can assign independent parallelism layouts to encoders and core models, raising TFLOPS per GPU by up to 49.3 percent on shared GPUs.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that a single shared parallelism layout for an entire multimodal model forces encoders to inherit choices optimized for the language model, creating communication overhead and underused parallelism especially at long contexts. It introduces an abstraction that lets separate modules run on their own tensor, pipeline, data, and expert parallelism layouts while still forming one training graph. Boundary communicators handle the necessary forward activation materialization and backward gradient routing between those layouts. Both colocated execution on the same GPUs and non-colocated execution on disjoint rank sets are supported through scheduling extensions. Measurements across workloads show the resulting efficiency gains while loss curves stay matched to homogeneous baselines.

Core claim

Heterogeneous parallelism lets modules in one end-to-end graph use independent layouts and rank placements. Boundary communicators implement forward and backward layout transforms that preserve tensor semantics across those layouts. The design supports colocated execution on shared GPUs and non-colocated execution on disjoint rank sets, with added scheduling logic for each mode. Evaluation across multimodal workloads and GPU scales shows colocated heterogeneity improves TFLOPS per GPU by up to 49.3 percent while non-colocated heterogeneity improves aggregate token throughput by up to 13.0 percent and TFLOPS per GPU by up to 9.6 percent, with loss convergence parity to homogeneous baselines.

What carries the argument

Boundary communicators that materialize forward activations for the destination layout and route backward gradients back to the source layout.

If this is right

Colocated heterogeneous configurations can raise TFLOPS per GPU by up to 49.3 percent.
Non-colocated heterogeneous configurations can raise aggregate token throughput by up to 13.0 percent.
Non-colocated heterogeneous configurations can raise TFLOPS per GPU by up to 9.6 percent.
Loss convergence remains equivalent to homogeneous baselines across the tested workloads.
The gains appear across different multimodal workloads and GPU cluster scales.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The same boundary handling could be tested on models that contain more than two distinct modules, such as separate vision, audio, and language components.
Dynamic rank allocation that assigns hardware clusters sized to each module's optimal layout becomes feasible once boundary costs are controlled.
The approach may reduce the hardware homogeneity requirement for large training runs when modules have mismatched scaling needs.

Load-bearing premise

Boundary communicators can move activations and gradients between independent layouts without introducing correctness errors or prohibitive overhead.

What would settle it

An experiment that applies heterogeneous layouts to a multimodal workload and measures either higher final loss than the matched homogeneous run or no gain in TFLOPS per GPU would falsify the performance claims.

read the original abstract

Foundation model training is becoming multimodal, from post-training pipelines to large-scale pretraining. As modality coverage broadens, context windows grow, and encoder LLM scales diverge, a single LLM-centric TP/CP/PP/DP/EP layout increasingly limits throughput. This coupling forces encoders to inherit LLM-driven sharding and placement choices that can add communication, limit encoder parallelism, or constrain the LLM schedule; the mismatch is most pronounced at long contexts, where LLM context parallelism is needed for the fused multimodal sequence but encoder inputs remain bounded. We present heterogeneous parallelism for multimodal large language model training, an abstraction that lets modules in one end-to-end graph use independent layouts and rank placements, supporting colocated execution on shared GPUs and non-colocated execution on disjoint rank sets. The key challenge is preserving boundary tensor semantics across independent layouts: forward activations must be materialized for the destination layout, while backward gradients must be routed back to the source layout. We address this with boundary communicators that implement forward and backward layout transforms, plus scheduling extensions for both placement modes. We evaluate optimized homogeneous, colocated heterogeneous, and non-colocated heterogeneous configurations across multimodal workloads and GPU scales to characterize when added layout and placement freedom exposes a better operating point. Across this sweep, colocated heterogeneity improves TFLOPS/GPU by up to 49.3%, while non-colocated heterogeneity improves aggregate token throughput by up to 13.0% and TFLOPS/GPU by up to 9.6%. We validate loss convergence parity against homogeneous baselines and release the system as an open-source Megatron-LM extension.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

The boundary communicator abstraction lets encoders and LLMs keep independent layouts in one graph, and the reported colocated gains look real enough to try, but the overhead evidence is still thin.

read the letter

The paper's core move is to decouple the parallelism layout of different modules inside a single multimodal training graph. Encoders can run their own TP or DP while the LLM uses CP for long contexts, and boundary communicators handle the forward activation reshape and backward gradient routing between those layouts. Both colocated (shared GPU) and non-colocated (disjoint ranks) modes are supported with scheduling tweaks.

They show concrete speedups: up to 49.3% better TFLOPS/GPU in colocated heterogeneous mode and smaller but positive gains in the non-colocated case, plus matching loss curves against homogeneous baselines. Releasing the code as a Megatron-LM extension is the right call; it lets others measure the actual trade-offs.

The weakest part is still the communication cost of those boundary transforms. Convergence parity confirms numerical correctness but does not bound extra all-to-all volume or scheduling bubbles when layouts diverge sharply or context lengths grow. The abstract does not break out communicator time or show scaling curves for that overhead, so the 49% and 9.6% deltas could shrink once the transforms are instrumented.

This is a systems paper aimed at teams already running large multimodal jobs on existing hardware. Practitioners who need to mix short encoder inputs with long LLM sequences will find the implementation details useful. The work is grounded enough and the problem is practical enough that it should go to peer review rather than desk reject; a referee can ask for the missing overhead breakdowns and scaling plots.

Referee Report

2 major / 1 minor

Summary. The paper introduces heterogeneous parallelism for multimodal LLM training, allowing independent TP/CP/PP/DP/EP layouts and rank placements for different modules (e.g., encoders vs. LLM) within one end-to-end graph. It addresses cross-layout tensor semantics via boundary communicators that materialize forward activations and route backward gradients, plus scheduling extensions for colocated (shared-GPU) and non-colocated (disjoint ranks) modes. Empirical evaluation across workloads reports colocated heterogeneity improving TFLOPS/GPU by up to 49.3%, non-colocated improving aggregate token throughput by up to 13.0% and TFLOPS/GPU by up to 9.6%, with loss convergence parity to homogeneous baselines; the system is released as an open-source Megatron-LM extension.

Significance. If the reported gains hold after accounting for boundary communicator costs, the work could meaningfully improve efficiency in multimodal pretraining and post-training by decoupling modality-specific parallelism choices, especially at long contexts. The open-source release is a clear strength supporting reproducibility.

major comments (2)

[Boundary communicators design] Boundary communicators design: The central TFLOPS/GPU claims (49.3% colocated, 9.6% non-colocated) rest on the assumption that layout transforms add only negligible communication and synchronization overhead. No quantitative breakdown of added all-to-all or point-to-point volume, nor scaling with context length or layout divergence (e.g., CP active on LLM but not encoder), is provided to bound this cost.
[Evaluation] Evaluation: The peak improvements are stated without accompanying details on the exact model sizes, GPU counts, workload characteristics, number of runs, variance, or conditions achieving the 49.3%, 13.0%, and 9.6% figures, nor any error analysis, which prevents full assessment of robustness against the homogeneous baselines.

minor comments (1)

[Abstract] The abstract refers to results 'across this sweep' without enumerating the range of configurations or workloads tested.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the detailed and constructive review. The two major comments highlight areas where additional analysis and experimental details would strengthen the presentation. We address each point below and will revise the manuscript accordingly.

read point-by-point responses

Referee: [Boundary communicators design] Boundary communicators design: The central TFLOPS/GPU claims (49.3% colocated, 9.6% non-colocated) rest on the assumption that layout transforms add only negligible communication and synchronization overhead. No quantitative breakdown of added all-to-all or point-to-point volume, nor scaling with context length or layout divergence (e.g., CP active on LLM but not encoder), is provided to bound this cost.

Authors: We agree that an explicit quantitative breakdown of boundary communicator overhead would improve the paper. The current manuscript reports net end-to-end TFLOPS/GPU and throughput gains after all communication (including boundary transforms), which implicitly shows that the added cost is more than offset by the parallelism benefits. However, we did not include a dedicated micro-benchmark isolating all-to-all/point-to-point volume or its scaling with context length and layout divergence. In the revision we will add a new subsection (and associated figure) that measures this overhead across the evaluated workloads, including cases with and without CP on the encoder. revision: yes
Referee: [Evaluation] Evaluation: The peak improvements are stated without accompanying details on the exact model sizes, GPU counts, workload characteristics, number of runs, variance, or conditions achieving the 49.3%, 13.0%, and 9.6% figures, nor any error analysis, which prevents full assessment of robustness against the homogeneous baselines.

Authors: We acknowledge that the main text could more explicitly enumerate the precise configurations and statistical details behind the reported peaks. The evaluation section and appendix already contain the model sizes, GPU counts, and workload descriptions, and all experiments were run with multiple seeds; however, variance and error bars are not shown in the primary figures. We will revise the evaluation section to list the exact conditions for each peak number, state the number of runs (three), and add error bars or standard-deviation shading to the relevant plots. revision: yes

Circularity Check

0 steps flagged

No circularity detected; results are direct empirical measurements

full rationale

The paper is a systems/engineering contribution that introduces boundary communicators and scheduling extensions for heterogeneous parallelism, then reports measured TFLOPS/GPU and throughput deltas from implemented colocated and non-colocated configurations. No mathematical derivation chain, fitted parameters presented as predictions, or load-bearing self-citations exist in the provided text. All performance claims are framed as outcomes of direct execution on multimodal workloads rather than reductions to inputs by construction. The loss-convergence parity check is an external validation step, not a self-referential fit.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 1 invented entities

The central claim rests on the new software abstraction and its empirical validation; no free parameters are mentioned. The key domain assumption is that tensor semantics can be maintained across layouts.

axioms (1)

domain assumption Tensor semantics must be preserved across independent parallelism layouts via explicit forward and backward transforms
Invoked in the description of boundary communicators and scheduling extensions

invented entities (1)

boundary communicators no independent evidence
purpose: Implement forward activation materialization and backward gradient routing between source and destination layouts
New component introduced to address the key challenge of preserving boundary tensor semantics

pith-pipeline@v0.9.1-grok · 5872 in / 1302 out tokens · 32504 ms · 2026-06-29T18:39:32.705429+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

23 extracted references · 15 canonical work pages · 7 internal anchors

[1]

Chiang, Z

W.-L. Chiang, Z. Li, Z. Lin, Y. Sheng, Z. Wu, H. Zhang, L. Zheng, S. Zhuang, Y. Zhuang, J. E. Gonzalez, I. Stoica, and E. P. Xing. Vicuna: An open-source chatbot impressing GPT-4 with 90%* ChatGPT quality. https://lmsys.org/blog/2023-03-30-vicuna/, March 2023

2023
[2]

Using Cornstarch 5d parallelism.https://cornstarch-org.github.io/ parallelization/cornstarch_parallel/, 2025

Cornstarch Project. Using Cornstarch 5d parallelism.https://cornstarch-org.github.io/ parallelization/cornstarch_parallel/, 2025. Accessed: 2026-05-06

2025
[3]

W. Feng, Y. Chen, S. Wang, Y. Peng, H. Lin, and M. Yu. Optimus: Accelerating large-scale multi-modal LLM training by bubble exploitation. In2025 USENIX Annual Technical Conference (USENIX ATC 25). USENIX Association, 2025. URLhttps://www.usenix.org/conference/atc25/ presentation/feng

2025
[4]

Huang, Z

J. Huang, Z. Zhang, S. Zheng, F. Qin, and Y. Wang. DISTMM: Accelerating distributed multimodal model training. In21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24), pages 1157–1171. USENIX Association,
[5]

URLhttps://www.usenix.org/conference/ nsdi24/presentation/huang
[6]

I. Jang, R. Lu, N. Bansal, A. Chen, and M. Chowdhury. Efficient distributed MLLM training with Cornstarch.arXiv preprint arXiv:2503.11367, 2025

work page internal anchor Pith review Pith/arXiv arXiv 2025
[7]

B. Jeon, M. Wu, S. Cao, S. Kim, S. Park, N. Aggarwal, C. Unger, D. Arfeen, P. Liao, X. Miao, M. Alizadeh, G. R. Ganger, T. Chen, and Z. Jia. GraphPipe: Improving performance and scalability of DNN training with graph pipeline parallelism. arXiv preprint arXiv:2406.17145, 2024

work page arXiv 2024
[8]

Jiang, Z

C. Jiang, Z. Cai, Y. Tian, Z. Jia, Y. Wang, and C. Wu. DCP: Addressing input dynamism in long-context training via dynamic context parallelism. InACM SIGOPS 31st Symposium on Operating Systems Principles (SOSP 25), 2025. doi: 10.1145/3731569.3764849

work page doi:10.1145/3731569.3764849 2025
[9]

H. Li, F. Fu, S. Lin, H. Ge, X. Wang, J. Niu, J. Xue, Y. Tao, D. Wang, J. Jiang, and B. Cui. Hydraulis: Balancing large transformer model training via co-designing parallel strategies and data assignment. arXiv preprint arXiv:2412.07894, 2024

work page arXiv 2024
[10]

Narayanan et al

D. Narayanan et al. Efficient large-scale language model training on GPU clusters using Megatron-LM. arXiv preprint arXiv:2104.04473, 2021

work page arXiv 2021
[11]

Y. Niu, H. Xiao, D. Liu, W. Zhou, and J. Li. DHP: Efficient scaling of MLLM training with dynamic hybrid parallelism.arXiv preprint arXiv:2602.21788, 2026

work page internal anchor Pith review Pith/arXiv arXiv 2026
[12]

Radford, J

A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever. Learning transferable visual models from natural language supervision. InProceedings of the 38th International Conference on Machine Learning, volume 139 ofProceedings of Machine Learning Research, pages 8748–8763. PM...

2021
[13]

ZeRO: Memory Optimizations Toward Training Trillion Parameter Models

S. Rajbhandari, J. Rasley, O. Ruwase, and Y. He. ZeRO: Memory optimizations toward training trillion parameter models.arXiv preprint arXiv:1910.02054, 2020

work page internal anchor Pith review Pith/arXiv arXiv 1910
[14]

Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism

M. Shoeybi et al. Megatron-LM training multi-billion parameter language models using model parallelism.arXiv preprint arXiv:1909.08053, 2019

work page internal anchor Pith review Pith/arXiv arXiv 1909
[15]

Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model

S. Smith et al. Using DeepSpeed and Megatron to train Megatron-Turing NLG 530B, a large-scale generative language model.arXiv preprint arXiv:2201.11990, 2022

work page internal anchor Pith review Pith/arXiv arXiv 2022
[16]

Y. Wang, S. Wang, S. Zhu, F. Fu, X. Liu, X. Xiao, H. Li, J. Li, F. Wu, and B. Cui. FlexSP: Accelerating large language model training via flexible sequence parallelism.arXiv preprint arXiv:2412.01523, 2024

work page arXiv 2024
[17]

Y. Wang, S. Zhu, F. Fu, X. Miao, J. Zhang, J. Zhu, F. Hong, Y. Li, and B. Cui. Efficient multi-task large model training via data heterogeneity-aware model management.Proceedings of the VLDB Endowment, 18(1), 2025

2025
[18]

B. Xiao, Y. Zheng, L. Shi, X. Li, F. Wu, T. Li, X. Xiao, Y. Zhang, Y. Wang, and S. Liu. OrchMLLM: Orchestrate multimodal data with batch post-balancing to accelerate multimodal large language model training.arXiv preprint arXiv:2503.23830, 2025

work page arXiv 2025
[19]

Z. Xue, H. Hu, X. Chen, Y. Jiang, Y. Song, Z. Mi, Y. Zhu, D. Jiang, Y. Xia, and H. Chen. PipeWeaver: Addressing data dynamicity in large multimodal model training with dynamic interleaved pipeline. InProceedings of the 31st ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS), 2026. doi: 10.1145/37...

work page doi:10.1145/3779212.3790154 2026
[20]

Zhang, Y

Z. Zhang, Y. Zhong, Y. Jiang, H. Hu, J. Sun, Z. Ge, Y. Zhu, D. Jiang, and X. Jin. DistTrain: Addressing model and data heterogeneity with disaggregated training for multimodal large language models. arXiv preprint arXiv:2408.04275, 2024

work page arXiv 2024
[21]

J. Zhao, Q. Lu, W. Jia, B. Wan, L. Zuo, J. Feng, J. Jiang, Y. Chen, S. Cao, J. He, K. Jiang, Y. Hu, S. Nong, Y. Peng, H. Lin, and C. Wu. MegaScale-Data: Scaling dataloader for multisource 10 Heterogeneous Parallelism for Multimodal Large Language Model Training large foundation model training.arXiv preprint arXiv:2504.09844, 2025

work page internal anchor Pith review Pith/arXiv arXiv 2025
[22]

PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel

Y. Zhao et al. PyTorch FSDP: Experiences on scaling fully sharded data parallel.arXiv preprint arXiv:2304.11277, 2023. 11 Heterogeneous Parallelism for Multimodal Large Language Model Training A. Appendix This appendix reports the layouts, peak memory, step times, tokens/s gains, and TFLOPS/GPU gains supporting Section 4. All parallelism columns useTP/CP/...

work page internal anchor Pith review Pith/arXiv arXiv 2023
[23]

language

Each colocated bridge group is therefore a 2-rank all-gather, not a global DP32 or DP16 collective. This small-group fan-in explains why colocated forward bridge time is low. NC forward is larger because it includes both cross-island activation transfer and LLM-side activation fanout: the encoder island sends activations to the LLM receiver ranks, and tho...

[1] [1]

Chiang, Z

W.-L. Chiang, Z. Li, Z. Lin, Y. Sheng, Z. Wu, H. Zhang, L. Zheng, S. Zhuang, Y. Zhuang, J. E. Gonzalez, I. Stoica, and E. P. Xing. Vicuna: An open-source chatbot impressing GPT-4 with 90%* ChatGPT quality. https://lmsys.org/blog/2023-03-30-vicuna/, March 2023

2023

[2] [2]

Using Cornstarch 5d parallelism.https://cornstarch-org.github.io/ parallelization/cornstarch_parallel/, 2025

Cornstarch Project. Using Cornstarch 5d parallelism.https://cornstarch-org.github.io/ parallelization/cornstarch_parallel/, 2025. Accessed: 2026-05-06

2025

[3] [3]

W. Feng, Y. Chen, S. Wang, Y. Peng, H. Lin, and M. Yu. Optimus: Accelerating large-scale multi-modal LLM training by bubble exploitation. In2025 USENIX Annual Technical Conference (USENIX ATC 25). USENIX Association, 2025. URLhttps://www.usenix.org/conference/atc25/ presentation/feng

2025

[4] [4]

Huang, Z

J. Huang, Z. Zhang, S. Zheng, F. Qin, and Y. Wang. DISTMM: Accelerating distributed multimodal model training. In21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24), pages 1157–1171. USENIX Association,

[5] [5]

URLhttps://www.usenix.org/conference/ nsdi24/presentation/huang

[6] [6]

I. Jang, R. Lu, N. Bansal, A. Chen, and M. Chowdhury. Efficient distributed MLLM training with Cornstarch.arXiv preprint arXiv:2503.11367, 2025

work page internal anchor Pith review Pith/arXiv arXiv 2025

[7] [7]

B. Jeon, M. Wu, S. Cao, S. Kim, S. Park, N. Aggarwal, C. Unger, D. Arfeen, P. Liao, X. Miao, M. Alizadeh, G. R. Ganger, T. Chen, and Z. Jia. GraphPipe: Improving performance and scalability of DNN training with graph pipeline parallelism. arXiv preprint arXiv:2406.17145, 2024

work page arXiv 2024

[8] [8]

Jiang, Z

C. Jiang, Z. Cai, Y. Tian, Z. Jia, Y. Wang, and C. Wu. DCP: Addressing input dynamism in long-context training via dynamic context parallelism. InACM SIGOPS 31st Symposium on Operating Systems Principles (SOSP 25), 2025. doi: 10.1145/3731569.3764849

work page doi:10.1145/3731569.3764849 2025

[9] [9]

H. Li, F. Fu, S. Lin, H. Ge, X. Wang, J. Niu, J. Xue, Y. Tao, D. Wang, J. Jiang, and B. Cui. Hydraulis: Balancing large transformer model training via co-designing parallel strategies and data assignment. arXiv preprint arXiv:2412.07894, 2024

work page arXiv 2024

[10] [10]

Narayanan et al

D. Narayanan et al. Efficient large-scale language model training on GPU clusters using Megatron-LM. arXiv preprint arXiv:2104.04473, 2021

work page arXiv 2021

[11] [11]

Y. Niu, H. Xiao, D. Liu, W. Zhou, and J. Li. DHP: Efficient scaling of MLLM training with dynamic hybrid parallelism.arXiv preprint arXiv:2602.21788, 2026

work page internal anchor Pith review Pith/arXiv arXiv 2026

[12] [12]

Radford, J

A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever. Learning transferable visual models from natural language supervision. InProceedings of the 38th International Conference on Machine Learning, volume 139 ofProceedings of Machine Learning Research, pages 8748–8763. PM...

2021

[13] [13]

ZeRO: Memory Optimizations Toward Training Trillion Parameter Models

S. Rajbhandari, J. Rasley, O. Ruwase, and Y. He. ZeRO: Memory optimizations toward training trillion parameter models.arXiv preprint arXiv:1910.02054, 2020

work page internal anchor Pith review Pith/arXiv arXiv 1910

[14] [14]

Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism

M. Shoeybi et al. Megatron-LM training multi-billion parameter language models using model parallelism.arXiv preprint arXiv:1909.08053, 2019

work page internal anchor Pith review Pith/arXiv arXiv 1909

[15] [15]

Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model

S. Smith et al. Using DeepSpeed and Megatron to train Megatron-Turing NLG 530B, a large-scale generative language model.arXiv preprint arXiv:2201.11990, 2022

work page internal anchor Pith review Pith/arXiv arXiv 2022

[16] [16]

Y. Wang, S. Wang, S. Zhu, F. Fu, X. Liu, X. Xiao, H. Li, J. Li, F. Wu, and B. Cui. FlexSP: Accelerating large language model training via flexible sequence parallelism.arXiv preprint arXiv:2412.01523, 2024

work page arXiv 2024

[17] [17]

Y. Wang, S. Zhu, F. Fu, X. Miao, J. Zhang, J. Zhu, F. Hong, Y. Li, and B. Cui. Efficient multi-task large model training via data heterogeneity-aware model management.Proceedings of the VLDB Endowment, 18(1), 2025

2025

[18] [18]

B. Xiao, Y. Zheng, L. Shi, X. Li, F. Wu, T. Li, X. Xiao, Y. Zhang, Y. Wang, and S. Liu. OrchMLLM: Orchestrate multimodal data with batch post-balancing to accelerate multimodal large language model training.arXiv preprint arXiv:2503.23830, 2025

work page arXiv 2025

[19] [19]

Z. Xue, H. Hu, X. Chen, Y. Jiang, Y. Song, Z. Mi, Y. Zhu, D. Jiang, Y. Xia, and H. Chen. PipeWeaver: Addressing data dynamicity in large multimodal model training with dynamic interleaved pipeline. InProceedings of the 31st ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS), 2026. doi: 10.1145/37...

work page doi:10.1145/3779212.3790154 2026

[20] [20]

Zhang, Y

Z. Zhang, Y. Zhong, Y. Jiang, H. Hu, J. Sun, Z. Ge, Y. Zhu, D. Jiang, and X. Jin. DistTrain: Addressing model and data heterogeneity with disaggregated training for multimodal large language models. arXiv preprint arXiv:2408.04275, 2024

work page arXiv 2024

[21] [21]

J. Zhao, Q. Lu, W. Jia, B. Wan, L. Zuo, J. Feng, J. Jiang, Y. Chen, S. Cao, J. He, K. Jiang, Y. Hu, S. Nong, Y. Peng, H. Lin, and C. Wu. MegaScale-Data: Scaling dataloader for multisource 10 Heterogeneous Parallelism for Multimodal Large Language Model Training large foundation model training.arXiv preprint arXiv:2504.09844, 2025

work page internal anchor Pith review Pith/arXiv arXiv 2025

[22] [22]

PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel

Y. Zhao et al. PyTorch FSDP: Experiences on scaling fully sharded data parallel.arXiv preprint arXiv:2304.11277, 2023. 11 Heterogeneous Parallelism for Multimodal Large Language Model Training A. Appendix This appendix reports the layouts, peak memory, step times, tokens/s gains, and TFLOPS/GPU gains supporting Section 4. All parallelism columns useTP/CP/...

work page internal anchor Pith review Pith/arXiv arXiv 2023

[23] [23]

language

Each colocated bridge group is therefore a 2-rank all-gather, not a global DP32 or DP16 collective. This small-group fan-in explains why colocated forward bridge time is low. NC forward is larger because it includes both cross-island activation transfer and LLM-side activation fanout: the encoder island sends activations to the LLM receiver ranks, and tho...