Learning to Shard: RL for Co-optimizing the Parallelism Degrees and Per-operator Sharding Dimensions in Distributed LLM Inference

Apala Guha; Hokchhay Tann; Preyas Shah; Ruokai Yin; Sattwik Deb Mishra; Xuan Zuo

REVIEW 3 major objections 5 minor 21 references

Learning to Shard: RL for Co-optimizing the Parallelism Degrees and Per-operator Sharding Dimensions in Distributed LLM Inference

T0 review · 3 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Learn to Shard uses a PPO agent with an attention-based elite-history policy to jointly choose parallelism degrees and per-operator sharding dimensions, reporting up to 1.06x throughput over Megatron heuristics and up to 3.5x over simulated

desk verdict Genuinely new co-optimization formulation, but the headline numbers are entirely simulator-produced and the simulator's accuracy for the novel all-gather strategies is unverified—treat the results as promising, not proven. read the letter →

arxiv 2509.00217 v1 pith:UHAXZSIE submitted 2025-08-29 cs.LG cs.DC

Ruokai Yin , Sattwik Deb Mishra , Xuan Zuo , Hokchhay Tann , Preyas Shah , Apala Guha This is my paper

classification cs.LGcs.DC

keywords reinforcementlearningparallelismstrategysearchper-operatorshardingLLMinferencemixture-of-expertsPPOdistributedthroughputoptimization

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes Learn to Shard, an RL-based search that jointly chooses coarse-grained parallelism degrees (tensor, expert, pipeline, batch) and the per-operator sharding dimension of each fused operator in a distributed LLM inference graph. The central claim is that this co-optimization finds strategies that outperform both metaheuristic search (simulated annealing, random walk) and the fixed heuristics used by Megatron-LM. On synthetic MoE workloads up to 1.6T parameters, with a budget of only 4,000 simulated evaluations out of roughly 10^9 possible strategies, the method reports up to 2.76x throughput over random walk, up to 3.5x over simulated annealing, and up to 1.06x over Megatron heuristic sharding. The gains come from discovering non-standard patterns, such as sharding an MLP's second feedforward operator along the hidden dimension and trading all-reduce for all-gather collectives. These numbers come from an in-house roofline simulator validated only against the Megatron heuristic, so real H100 hardware gains are not directly measured.

What carries the argument

The elite-history attention policy: a fixed-size deque of the top T strategies, each embedded, passed through a transformer encoder, mean-pooled across the history, and projected to sub-strategy logits for every action component. This gives the agent a compact, learned notion of what good strategies look like, guiding exploration in a sparse reward space. The simulator acts as the environment; its fidelity is what carries the reported improvements to real hardware.

What would settle it

Run the agent-found sharding strategy for a GPT-MoE 1.6T model on an actual H100 cluster with, say, 64 GPUs and a 16k context, and compare measured decoding throughput against Megatron-LM's heuristic strategy under identical SLOs. If the measured ratio does not show at least the simulated 1.06x edge, the central claim fails.

Watch

Extended reading notes

Core claim

Learn to Shard models a parallelization strategy as a joint action a = (a_TP, a_EP, a_PP, a_B, {a_dim^l}), where the first terms set parallelism degrees and batch size, and each a_dim^l selects a sharding dimension (0, 1, or none) for each fused operator. A PPO-trained policy, conditioned on an attention-based encoding of the top-performing 'elite' strategies seen so far, proposes actions; an in-house throughput simulator scores them. The reward adds an improvement bonus over the best strategy found so far, and a confidence-based early exit restarts the agent with inherited history to avoid local optima. The paper claims this is the first RL-based approach to co-optimize coarse and fine-grai

Load-bearing premise

The reported speedups assume the in-house roofline simulator predicts real H100 throughput for non-standard strategies like the all-gather pattern, but the simulator was validated only against the Megatron heuristic, so the 1.06x and 3.5x numbers may not reproduce on physical hardware.

Editorial extensions

If this is right

If the simulator fidelity holds, inference systems can automate discovery of non-standard sharding strategies like the all-gather-based MLP pattern that current heuristics miss.
Search cost is low enough (thousands of simulated calls, under 10 minutes) to re-optimize per workload and topology, making per-deployment strategy search practical.
The 1.06x edge over Megatron suggests per-operator sharding dimensions are a meaningful, under-exploited lever even when parallelism degrees are already near-optimal.
The same RL loop can absorb additional axes such as sequence sharding, multi-dimensional sharding, and collective-algorithm choices as future work.
Automated co-optimization can replace hand-tuned rules as model scale and interconnect topologies diversify, reducing reliance on expert heuristics.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The agent's all-gather pattern suggests that communication-collective selection, not just tensor sharding dimension, is the real driver; the paper folds that into the strategy but never isolates it as a separate knob, so a fair comparison would hold parallelism degrees fixed and vary only the collective.
Because the simulator is validated only against Megatron's strategy, the 1.06x claim over Megatron is best read as a simulator prediction; the actual hardware margin could be smaller or larger.
The elite-history attention mechanism is a general trick for combinatorial optimization with expensive evaluators and could transfer to other systems search spaces such as query planning or scheduling.
The same co-optimization approach could extend to training pipelines, where heterogeneous topologies and per-layer shapes create similar joint search spaces.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague—

This paper is worth knowing about: it's the first RL formulation I've seen that co-optimizes coarse parallelism degrees (TP/EP/PP/batch) with per-operator sharding dimensions. That's a real gap. The policy network—attention over an elite history of good strategies—is a sensible way to bias exploration in a huge discrete space, and the authors are honest about the budget (4000 simulator calls out of ~1e9). The design is clean and the writing is straightforward.

But the empirical claims rest entirely on an in-house roofline simulator. The abstract says "Evaluated on H100 clusters," which is misleading—no real H100 runs appear anywhere. The simulator is validated only against Megatron's heuristic (footnote 5), and the agent's discovered strategy replaces all-reduce with all-gather between FFN1 and FFN2. The stress-test concern is on target: if the simulator undercounts all-gather cost or launch overhead, the 1.06x over Megatron could shrink or flip. The 3.5x number is over SA and RW, which are deliberately weak baselines; the honest comparison is the 1.06x, and that's modest. So the headline is a promise, not a result.

What's good: the problem is well-motivated, related work is covered (ALPA, FlexFlow, COSMIC, TAPAS, Helix, Chimera), and the claim of firstness seems plausible. The formalization of strategies as integer tuples is clean.

Where it's soft: no simulator release, no real-hardware validation, no sensitivity analysis on α, β, T, τ. The reward includes an improvement bonus which could bias policy exploration, but that's fine. The "up to 1.06x" is thin—it's one configuration, best of 10 runs, and we don't see variance.

For peer review: I'd send it out. The idea deserves scrutiny, and a good referee could push the authors to either release the simulator or run a single small real cluster experiment. If the method holds up on hardware, it's a useful contribution. Without that, it's an interesting system paper with an unverified center.

My recommendation: read it, think about the search formulation, but don't change your production stack based on the numbers. I'd cite it for the co-optimization idea, not for the measurements.

Referee Report

3 major / 5 minor

Summary. The paper presents Learn to Shard, a PPO-based RL agent that jointly optimizes coarse-grained parallelism degrees (TP/EP/PP/batch size) and fine-grained per-operator sharding dimensions for distributed LLM inference. Strategies are encoded as multi-discrete actions; the environment is an in-house roofline-based H100 simulator that returns token/s/chip throughput. The policy is an attention-based Transformer over an elite history of high-performing strategies, with a confidence-based early exit. Experiments on synthetic MoE models up to 1.6T parameters report up to 2.76x improvement over random walk, a derived up-to-3.5x improvement over simulated annealing, and up to 1.06x over Megatron-LM heuristic sharding, all under a 4000-call search budget.

Significance. If the simulator faithfully predicts real H100 behavior, the paper makes a useful contribution: it is, to my knowledge, the first RL-based method to co-optimize coarse-grained parallelism degrees and per-operator sharding dimensions, and the attention-based elite-history policy is a plausible mechanism for learning from prior high-performing configurations. The problem is well motivated and the RL formulation is presented consistently. However, the empirical significance is currently contingent on an unverified simulation assumption: all reported gains come from an in-house simulator validated only against the Megatron-LM all-reduce heuristic, while the agent's discovered best strategy relies on all-gather patterns that are never checked on real hardware. The paper also describes the evaluation as 'on H100 clusters' when no physical hardware measurements are reported. These issues make the headline results provisional.

major comments (3)

[Section 4, footnote 5, and Appendix B] The central empirical claims (up to 3.5x over SA, 1.06x over Megatron) rest entirely on the in-house roofline simulator, whose only stated validation is against Megatron-LM's heuristic all-reduce strategy. Appendix B shows that the agent's discovered strategy replaces all-reduce with all-gather between FFN1 and FFN2. No validation or real-hardware measurement covers all-gather-based sharding; if the simulator's all-gather model is optimistic, the reported margins may shrink or invert on physical H100 clusters. Please add real H100 measurements of at least the agent-discovered strategies, or targeted simulator calibration against all-gather/all-reduce collectives (e.g., predicted vs. measured throughput for representative TP=4/8 configurations).
[Abstract and Section 4] The abstract states 'Evaluated on H100 clusters with MoE models up to 1.6T parameters,' but Section 4 says performance is measured using 'an in-house roofline-based simulator' and no real hardware experiment is reported anywhere in the manuscript. This overstates the evidence. The claims should be reworded as simulator-based evaluations unless actual H100 measurements are included.
[Section 4, Table 1, and Section 5] The abstract/conclusion claim of 'up to 3.5x throughput improvement over metaheuristic baselines' is not directly reported. Table 1 normalizes throughput to random walk; the SA column is also normalized to RW. The 3.5x value only appears as a ratio of normalized values (e.g., 2.31/0.66), which is not a transparent comparison. Please report direct pairwise Ours-vs-SA ratios, ideally with variance/confidence intervals across the 10 runs, so the reader can assess the significance of the 1.06x and 1.01x margins in Figure 3.

minor comments (5)

[Footnote 5] The simulator validation is described in a single sentence. Please specify what was compared, on which configurations, and the achieved error/accuracy; this is essential given the load-bearing role of the simulator.
[Abstract and Section 4] The paper alternates between 'NPUs' (abstract) and 'H100 GPUs' (Section 4). Please use consistent terminology.
[Section 4] The device budget is stated as 'up to 24k GPUs,' but the introduction mentions hundreds to thousands of NPUs. Clarify whether 24k is the search-space device limit or an actual cluster size, and how many devices each evaluated strategy uses.
[Section 3 and Appendix A] The reward function in Eq. (1) depends on scaling factors α and β, and early exit depends on τ and chunk count, but no ablation or sensitivity analysis is provided for these hyperparameters. This is not blocking, but reporting robustness would strengthen the method.
[Figure 3] The caption says 'best of 10 runs,' while Table 1 reports means. Please clarify which statistic is shown in each figure and add error bars or dispersion information.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the RL search optimizes simulator reward and is compared against baselines under the same simulator; no fitted parameter is renamed as prediction and no load-bearing self-citation exists.

full rationale

The paper's derivation chain is self-contained. The reward r(a) is defined directly from the simulator's raw throughput raw(a) (Eq. 1), and the final strategy is selected as the highest-reward configuration encountered. Reporting that this strategy scores higher in the same simulator than baselines searched under the same budget is an optimization result, not a circular prediction: the comparison is between different search algorithms on a common objective, and no parameter is fitted to a subset of data and then 'predicted' on a closely related quantity. The only in-scope limitation is footnote 5, which states the in-house simulator is 'Validated against Megatron-LM heuristic parallelization strategy.' That is an external benchmark, and validation against one baseline does not define the agent's discovered all-gather strategy into existence or force the reported improvements; it is a correctness/validity risk about simulator fidelity for non-standard sharding patterns, not a circularity. There are no self-citations that are load-bearing, no imported uniqueness theorem, and no ansatz smuggled in via citation. The evaluation does depend on the unverified accuracy of the simulator for non-Megatron patterns, but that is an empirical assumption, not a definitional reduction.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the accuracy of an unreleased in-house simulator and on a restricted action space. The RL hyperparameters are reported, but the simulator's internal parameters and validation are not disclosed beyond a single check against the Megatron heuristic.

free parameters (4)

Reward scaling factors α and β = unspecified
Defined in Eq. 1 as positive scaling factors, but their numeric values are not reported. They affect the reward magnitude and the balance between raw throughput and improvement bonus.
Elite history buffer size T = 3
Set to 3 in Appendix A. Controls the context length of the policy input.
Confidence threshold τ = 0.95
Set to 0.95 for early exit. Determines when the policy is considered deterministic.
Budget chunk count = 5
The 4000-call budget is split into 5 chunks for restarts. Affects exploration.

assumptions (4)

domain assumption The in-house roofline simulator accurately models the H100 cluster for arbitrary sharding strategies.
All reported throughput values come from this simulator (Section 4, footnote 5). No real hardware measurements are provided.
domain assumption The space of sharding dimensions {0,1,∅} over L=12 fused ops contains near-optimal strategies.
The action space is defined in Section 3; footnote 4 explicitly excludes context-length sharding, so the optimum could lie outside.
ad hoc to paper The simulator's validation against Megatron-LM's heuristic ensures accuracy for non-heuristic strategies.
Footnote 5 states validation only against Megatron-LM; there is no evidence it generalizes to unusual sharding patterns.
domain assumption PPO with an attention policy converges to good strategies within 4000 simulator calls.
The paper uses hyperparameters from Appendix A and reports results over 10 runs, but convergence is not proven.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Learning to Shard: RL for Co-optimizing the Parallelism Degrees and Per-operator Sharding Dimensions in Distributed LLM Inference." pith.science (2026). https://pith.science/paper/UHAXZSIE

@misc{pith2026250900217,
  author       = {Pith},
  title        = {Pith review of: Learning to Shard: RL for Co-optimizing the Parallelism Degrees and Per-operator Sharding Dimensions in Distributed LLM Inference},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UHAXZSIE}},
  note         = {Machine review of arXiv:2509.00217}
}

read the original abstract

Distributed LLM inference requires careful coordination of parallelization strategies across hundreds to thousands of NPUs to meet production SLOs. Current systems like Megatron-LM rely on static heuristics that separately configure parallelism degrees and per-operator sharding dimensions, leaving significant performance on the table as models scale and hardware topologies diversify. We introduce Learn to Shard, to our knowledge, the first RL-based approach to co-optimize both coarse-grained parallelism degrees and fine-grained per-operator sharding dimensions for distributed LLM inference. Our method employs an attention-based policy over an elite history that learns from high-performing strategies to efficiently navigate the vast combinatorial search space. Evaluated on H100 clusters with MoE models up to 1.6T parameters, Learn to Shard achieves up to 3.5x throughput improvement over metaheuristic baselines and 1.06x over Megatron heuristics.

Figures

Figures reproduced from arXiv: 2509.00217 by the authors.

**Figure 1.** An illustrative example of the standard TP parallelism strategy. For agentic search, we need a formal representation of the sharding strategy. We define a parallelization strategy with two parts: the coarse-grained and the fine-grained strategy. The coarse-grained strategy refers to the parallelization degrees, which refer to the number of NPUs that each specific parallelization type requires. In the TP-only examp… view at source ↗

**Figure 2.** Overall search loop and the policy network design. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Normalized throughput of jointly optimized sharding vs. Megatron-LM heuristics (best of 10 runs; decoding, 16k context). We further measure the improvement of the agent-found parallelization strategies over the heuristic ones. We set the heuristic strategy baseline to use the standard per-operator sharding dimensions from Megatron-LM (Shoeybi et al., 2019). We then fix Megatron’s peroperator sharding dimensions a… view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Illustration of the all-gather based sharding strategy found by our [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

21 extracted references · 11 canonical work pages

[1]

Bhatia, N., More, A., Borkar, R., Mitra, T., Matas, R., Zhao, R., Golub, M., Mudigere, D., Pharris, B., and Rouhani, B. D. Helix parallelism: Rethinking sharding strategies for interactive multi-million-token llm decoding. arXiv preprint arXiv:2507.07120, 2025

arXiv 2025
[2]

Beyond data and model parallelism for deep neural networks

Jia, Z., Zaharia, M., and Aiken, A. Beyond data and model parallelism for deep neural networks. In Proceedings of Machine Learning and Systems (MLSys), 2019

work page 2019
[3]

Dfmodel: Design space optimization of large-scale systems exploiting dataflow mappings

Ko, S., Zhang, N., Hsu, O., Pedram, A., and Olukotun, K. Dfmodel: Design space optimization of large-scale systems exploiting dataflow mappings. arXiv preprint arXiv:2412.16432, 2024

arXiv 2024
[4]

Sequence parallelism: Long sequence training from system perspective

Li, S., Xue, F., Baranwal, C., Li, Y., and You, Y. Sequence parallelism: Long sequence training from system perspective. arXiv preprint arXiv:2105.13120, 2021

arXiv 2021
[5]

Uniap: Unifying inter-and intra-layer automatic parallelism by mixed integer quadratic programming

Lin, H., Wu, K., Li, J., Li, J., and Li, W.-J. Uniap: Unifying inter-and intra-layer automatic parallelism by mixed integer quadratic programming. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp.\ 20947--20957, 2025

work page 2025
[6]

Deepseek-v3 technical report

Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024

arXiv 2024
[7]

GTC 2024 Presentation Slides

NVIDIA Corporation . GTC 2024 Presentation Slides . Conference presentation at NVIDIA GTC 2024, March 2024. URL https://www.youtube.com/watch?v=f8DKD78BrQA

work page 2024
[8]

XLA: A Machine Learning Compiler

OpenXLA Project . XLA: A Machine Learning Compiler . https://github.com/openxla/xla, 2025

work page 2025

Show all 21 references

[9]

Pytorch: An imperative style, high-performance deep learning library

Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019

2019
[10]

Chimera: Communication fusion for hybrid parallelism in large language models

Qin, L., Cui, J., Cai, W., and Huang, J. Chimera: Communication fusion for hybrid parallelism in large language models. In Proceedings of the 52nd Annual International Symposium on Computer Architecture, pp.\ 498--513, 2025

2025
[11]

Stable-baselines3: Reliable reinforcement learning implementations

Raffin, A., Hill, A., Gleave, A., Kanervisto, A., Ernestus, M., and Dormann, N. Stable-baselines3: Reliable reinforcement learning implementations. Journal of machine learning research, 22 0 (268): 0 1--8, 2021

2021
[12]

Raju, A., Ni, J., Won, W., Man, C., Krishnan, S., Sridharan, S., Yazdanbakhsh, A., Krishna, T., and Reddi, V. J. Cosmic: Enabling full-stack co-design and optimization of distributed machine learning systems. arXiv preprint arXiv:2505.15020, 2025

2025 arXiv
[13]

TAPAS : Fast and automatic derivation of tensor parallel strategies for large neural networks

Shi, Z., Jiang, L., Wang, A., Zhang, J., Wu, C., Li, Y., Xiao, X., Lin, W., and Li, J. TAPAS : Fast and automatic derivation of tensor parallel strategies for large neural networks. In Proceedings of the 54th International Conference on Parallel Processing, ICPP '25, New York,...

2025 arXiv
[14]

Megatron-lm: Training multi-billion parameter language models using model parallelism

Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., and Catanzaro, B. Megatron-lm: Training multi-billion parameter language models using model parallelism. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analys...

2019
[15]

Seesaw: High-throughput llm inference via model re-sharding

Su, Q., Zhao, W., Li, X., Andoorveedu, M., Jiang, C., Zhu, Z., Song, K., Giannoula, C., and Pekhimenko, G. Seesaw: High-throughput llm inference via model re-sharding. arXiv preprint arXiv:2503.06433, 2025

2025 arXiv
[16]

Llama: Open and efficient foundation language models

Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozi \`e re, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

2023 arXiv
[17]

Gspmd: General and scalable parallelization for ml computation graphs

Xu, S., Chen, X., Xu, Y., Johnson, D., Krikun, M., Chen, K., Liu, Y., Zhao, K., Jouppi, N., Laudon, J., et al. Gspmd: General and scalable parallelization for ml computation graphs. In Proceedings of Machine Learning and Systems (MLSys), 2021

2021
[18]

E., and Stoica, I

Yu, C., Yan, Y., Zheng, L., Zhang, W., Chen, M., Xing, E., Gonzalez, J. E., and Stoica, I. Alpaserve: Statistical multiplexing with model parallelism for deep learning serving. In Proceedings of the ACM Symposium on Cloud Computing (SoCC), 2023

2023
[19]

E., Stoica, I., Jin, X., Xing, E., Chen, Q

Zheng, L., Yu, C., Yan, Y., Xu, Z., Zhang, H., Gonzalez, J. E., Stoica, I., Jin, X., Xing, E., Chen, Q. H., et al. Alpa: Automating inter- and intra-operator parallelism for distributed deep learning. In Proceedings of the 16th USENIX Symposium on Operating Systems Design and ...

2022
[20]

Transferable graph optimizers for ml compilers

Zhou, Y., Roy, S., Abdolrashidi, A., Wong, D., Ma, P., Xu, Q., Liu, H., Phothilimtha, P., Wang, S., Goldie, A., et al. Transferable graph optimizers for ml compilers. Advances in Neural Information Processing Systems, 33: 0 13844--13855, 2020

2020
[21]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.

[1] [1]

Bhatia, N., More, A., Borkar, R., Mitra, T., Matas, R., Zhao, R., Golub, M., Mudigere, D., Pharris, B., and Rouhani, B. D. Helix parallelism: Rethinking sharding strategies for interactive multi-million-token llm decoding. arXiv preprint arXiv:2507.07120, 2025

arXiv 2025

[2] [2]

Beyond data and model parallelism for deep neural networks

Jia, Z., Zaharia, M., and Aiken, A. Beyond data and model parallelism for deep neural networks. In Proceedings of Machine Learning and Systems (MLSys), 2019

work page 2019

[3] [3]

Dfmodel: Design space optimization of large-scale systems exploiting dataflow mappings

Ko, S., Zhang, N., Hsu, O., Pedram, A., and Olukotun, K. Dfmodel: Design space optimization of large-scale systems exploiting dataflow mappings. arXiv preprint arXiv:2412.16432, 2024

arXiv 2024

[4] [4]

Sequence parallelism: Long sequence training from system perspective

Li, S., Xue, F., Baranwal, C., Li, Y., and You, Y. Sequence parallelism: Long sequence training from system perspective. arXiv preprint arXiv:2105.13120, 2021

arXiv 2021

[5] [5]

Uniap: Unifying inter-and intra-layer automatic parallelism by mixed integer quadratic programming

Lin, H., Wu, K., Li, J., Li, J., and Li, W.-J. Uniap: Unifying inter-and intra-layer automatic parallelism by mixed integer quadratic programming. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp.\ 20947--20957, 2025

work page 2025

[6] [6]

Deepseek-v3 technical report

Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024

arXiv 2024

[7] [7]

GTC 2024 Presentation Slides

NVIDIA Corporation . GTC 2024 Presentation Slides . Conference presentation at NVIDIA GTC 2024, March 2024. URL https://www.youtube.com/watch?v=f8DKD78BrQA

work page 2024

[8] [8]

XLA: A Machine Learning Compiler

OpenXLA Project . XLA: A Machine Learning Compiler . https://github.com/openxla/xla, 2025

work page 2025