Pith. sign in

REVIEW 4 major objections 5 minor 34 references

Long-context LLM training can recover near-linear scaling by balancing attention over fixed-size pools.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-07-31 23:58 UTC pith:N2KQMU2D

load-bearing objection Libra presents a genuinely new bounded-pool principle for attention load balancing with large measured gains, but the motivation numbers conflict and the FLOPs-to-time proxy is under-calibrated; still worth a serious referee. the 4 major comments →

arxiv 2607.23250 v1 pith:N2KQMU2D submitted 2026-07-25 cs.DC

Libra: Taming Attention Workload Skew in Long-Context LLM Training with Bounded Sequence Pool

Yan Wang , Xiulong Yuan , Kaiming Yang , Jiaxuan Peng , Pengju Lu , Mingzhen Li , Zhipeng Zhang , Chang Si
show 11 more authors
Zhixiang Ruan Hongqing Chen Linlang Jiang Siyu Wang Langshi Chen Rui Men Man Yuan Guangming Tan Yong Li Weile Jia Jingren Zhou
This is my paper
classification cs.DC
keywords long-context LLM trainingload balancingsequence packingattention FLOPscontext parallelismlaw of large numbersdata-parallel scalingtiled attention
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Sequence packing balances token counts but leaves attention FLOPs wildly uneven, because attention cost grows as the sum of squared sample lengths. This paper claims that the right balancing scope is a fixed-size sequence pool, not a global worker pool: as data-parallel scale grows, add more pools rather than enlarge each one. To make that work on finite long-tailed workloads, it pairs a variance-reduced placement heuristic with tile-level attention migration inside each pool. If correct, long-context training can recover most of the lost data-parallel scaling (from 27.6% to 70.3% efficiency at DP=16 on 1M-token workloads) while keeping communication bounded.

Core claim

The central claim is that the attention-balancing pool size can be decoupled from data-parallel degree, so that attention exchange stays bounded while the cluster scales. The paper's sequence pool holds a fixed number of packed sequences and their context-parallel groups; VRSP assigns sequences with complementary attention FLOPs to pools, and Tiled Attention Pooling splits each sequence into sequence-by-head tiles that are placed to equalize per-GPU FLOPs. The result is a drop-in context-parallel attention operator that restores near-linear scaling, cutting per-microbatch forward-time spread from a 23x range to about 3x on 1M-token data.

What carries the argument

The sequence pool is the central object: a fixed-size group of packed sequences and the context-parallel workers that process them, forming the only domain over which attention work may be exchanged. VRSP, an exact-cardinality longest-processing-time-first placement, flattens inter-pool FLOPs imbalance (from over 100% overshoot to below 0.7% at pool size 8). TAP decomposes attention into SH-Tiles—sequence-block by head-shard units—and a communication-aware greedy placer balances estimated FLOPs across workers while reusing resident KV groups; a pipelined runtime overlaps tile exchange with attention to hide communication.

Load-bearing premise

The load-bearing premise is that estimated attention FLOPs—computed from the sum of squared packed-sample lengths—faithfully predict wall-clock per-worker time; the paper measures runtime spread on identical GPUs and kernel-shape efficiency effects but does not feed either into the balancing objective.

What would settle it

Run one optimizer-step window at pool size 8 with VRSP and TAP, and compare the actual slowest-worker time against the FLOPs-balance prediction; if real time spread stays near the measured 7-15% hardware variance rather than collapsing toward zero, the proxy fails. Alternatively, replace the FLOPs objective with measured runtime and show it changes the placement; that would demonstrate the proxy is not the right cost function.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • At DP=16 on 1M-token workloads, data-parallel scaling efficiency rises from 27.6% to 70.3%, translating to up to 2.54x end-to-end throughput over a head-axis context-parallel baseline.
  • Worst-step straggler attention latency drops up to 3.14x in microbenchmarks, which tightens the effective pipeline bubble that limits long-context training.
  • Scaling out DP adds more fixed-size pools instead of enlarging each pool, so every attention exchange stays within a bounded locality domain.
  • The mechanism preserves the raw-sample multiset of each optimizer step and plugs in without changes to model layers, optimizers, or pipeline schedules.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the FLOPs proxy is replaced by a runtime-calibrated cost model, the same bounded-pool principle should apply to sparse or linear attention variants, where the quadratic proxy no longer holds.
  • The LLN insight suggests a general design heuristic for any distributed workload with heavy-tailed per-item costs: bound the pooling domain and use variance-reduced placement within each step, rather than growing the scheduler's scope.
  • Testable extension: on heterogeneous GPU clusters, the fixed pool size may need to shrink or the placement objective should weight per-device speeds; the paper's measurements of 7-15% runtime spread on identical GPUs suggest this matters as pool size grows.
  • Because VRSP only reorders complete packed sequences, it can be composed with any future packing algorithm, as long as the optimizer-step sample multiset is preserved.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper targets cross-CP-group attention workload skew in long-context LLM training. It introduces sequence pools of fixed size, motivated by the law of large numbers: pool size need not grow with the data-parallel degree. VRSP balances estimated attention FLOPs across pools, TAP tiles attention along sequence and head dimensions and places SH-Tiles within a pool, and a Pipeliner overlaps tile exchange with computation. The evaluation on Qwen3-Turbo training reports up to 2.54× end-to-end throughput over Ulysses, near-linear DP scaling, a 3.14× worst-step straggler-attention speedup in microbenchmarks, and extensive production deployment. The central mechanism is evaluated with measured end-to-end and microbenchmark times, not derived from the analytical proxy alone, and the paper is transparent about several limitations.

Significance. If the results hold, this is a significant contribution: it reframes cross-CP-group load balancing as statistical concentration over bounded pools, offers a concrete two-level design (VRSP plus TAP) that preserves optimizer-step sample multisets and requires no model or pipeline-schedule changes, and provides strong production evidence. The paper is honest about its assumptions: Eq. (3) is explicitly an analytical approximation compared with random-grouping data, the FLOPs proxy is stated as a proxy, and the P-selection mismatch in §6.4 is disclosed. The claimed speedups are measured, not fitted. However, the evaluation has several load-bearing gaps: the motivation figures are internally inconsistent, the warm-up baselines are not run as published, and the relationship between the FLOPs-balancing objective and wall-clock straggler time is not quantified. These issues are fixable but require additional experimental work.

major comments (4)
  1. [§3.1, Figure 1] The flagship motivation is internally inconsistent. The introduction states that with 1M packed sequences and CP=16, scaling DP from 1 to 16 improves throughput by only 4.42×, citing Figure 1. Figure 1, however, reports CP=16 and shows 7.63× at DP=16 (matching the 256K result in §6.2, not the 1M result). §3.1 further states 'when CP=32, increasing DP from 1 to 8 yields only a 3.51× throughput gain,' which contradicts Figure 1's 4.66× at DP=8 with CP=16. Since this scaling gap motivates the entire paper, the caption, configuration, and text must be reconciled.
  2. [Eq. (6), Algorithm 2] Algorithm 2 balances estimated attention FLOPs (Eq. 6) with a 3% target, but the actual straggler metric is wall-clock time. Figure 3 shows 7.01% mean and 15.07% max runtime spread on identical GPUs with identical inputs, and Figure 5 shows FlashAttention throughput varying strongly with per-call block length and head count; neither effect is incorporated into Eq. 6 or Algorithm 2. This does not make the measured speedups circular, but it means the proxy-to-time error is unquantified. Since the worst-step max is the metric most sensitive to residual imbalance, the reported 3.14× figure needs error bars/per-step distributions and a sensitivity check on τ (and M), or a calibration of the FLOPs proxy by per-shape kernel-time coefficients.
  3. [§6.3] The comparisons to WLB-LLM and DistCA are not faithful to the published systems. WLB-LLM is reimplemented with its outlier-deferral mechanism excluded, and DistCA is emulated only as a cluster-wide pool with Libra's own SH-Tile granularity and Pipeliner overlap, omitting token-level dispatch, the dedicated attention-server pool, and ping-pong execution. Consequently Figure 13 and the associated text ('WLB-LLM cannot cut the max', 'DistCA pays the communication cost') are claims about weakened variants, not about the actual baselines. Either run the public implementations or explicitly relabel these as 'reimplemented variants excluding [mechanism]' and soften the comparative conclusions.
  4. [§6.4, Figure 14] The paper admits that Libra(full) at P=4 slightly outperforms P=8 on both datasets (0.33 vs 0.34 on 256K; 0.37 vs 0.39 on 1M), yet fixes P=8 throughout because the offline simulator selects it on an 'estimated balance–communication trade-off' and 'does not claim to solve a fixed optimization objective'. This is an admitted mismatch between the configuration-selection criterion and the primary measured metric. The mismatch does not invalidate the overall speedup claims, but it does invalidate the claim that the simulator provides trustworthy configuration guidance. Please report a measured-latency sweep as the selection procedure, or justify P=8 by a different explicit criterion.
minor comments (5)
  1. [§8] The conclusion says '3.14× worst-step straggler-attention throughput over Ulysses'; this should be 'speedup' or 'latency reduction', not 'throughput'.
  2. [Figures 11, 13, 14] No error bars or run counts are reported for the central end-to-end and microbenchmark plots. Given the runtime variability documented in Figure 3, per-configuration means over multiple runs (or at least shaded min–max ranges) are needed to assess the stability of the 2.54× and 3.14× claims.
  3. [§6.2, PP Imbalance] The pipeline-bubble reduction is only proxied by the compression of per-microbatch forward-time distributions, and the text explicitly notes that no pipeline trace was collected. This is acceptable as indirect evidence, but the wording in the abstract and conclusion ('pipeline bubbles') should make the proxy nature explicit, e.g., 'reduced compute-induced bubble risk'.
  4. [§5, Configuration simulator] The simulator 'does not claim to solve a fixed optimization objective', yet P=8, H, B, τ=0.03, and M=4 are chosen/fixed with no sensitivity study over τ and M. The dependence of the results on these four hidden parameters should be reported or at least discussed.
  5. [§6.1, baselines] The DistCA emulation is described as reusing Libra's SH-Tile granularity and Pipeliner overlap. This gives DistCA credit for Libra's intra-pool communication optimizations, which is favorable to the baseline; please state whether the emulated DistCA includes a similar communication-overlap mechanism.

Circularity Check

0 steps flagged

No significant circularity: the headline gains are wall-clock measurements, the FLOPs proxy is an explicit modeling assumption rather than a fitted input, and self-citations are background only.

full rationale

I walked the paper's derivation chain. The central load-bearing steps are: (1) LLN-guided fixed-size sequence pools, justified by the analytical CLT/Gaussian-maximum approximation in Eq. 3; this is compared against empirical random grouping in Fig. 4 and contains no fitted constants. (2) VRSP, which minimizes estimated attention-FLOPs sums (Eq. 4) with an exact-cardinality LPT heuristic; its imbalance reduction is measured in Table 1 and its wall-clock contribution is measured in Fig. 13/14. (3) TAP, whose Algorithm 2 balances the estimated FLOPs proxy in Eq. 6, but the abstract's 2.54x/3.14x claims are measured end-to-end and microbenchmark wall-clock times, not outputs of that FLOPs objective. The FLOPs-to-time relationship is an explicitly stated assumption, not a fitted parameter renamed as a prediction; if it failed, the quantitative speedups would be smaller, but that is a correctness/robustness risk, not circularity. The paper even discloses evidence that would challenge the proxy: Fig. 3 shows 7.01% mean/15.07% max runtime spread on identical GPUs, Fig. 5 shows FlashAttention efficiency varies with block/head shape, and §6.4 admits P=4 slightly beats the simulator-selected P=8. Such disclosure is inconsistent with constructing results to match the proxy. Self-citations to prior Alibaba group work ([14], [26], [29]) appear only in related-work context and are not load-bearing for the design or evaluation. No derivation step reduces to its own input by definition, and no load-bearing assertion rests on an unverified self-citation. Therefore the paper shows no significant circularity.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 3 invented entities

Central claims rest on the workload proxy Σl^2, the parameter-free-execution property, and the CLT approximation; all are stated. No fitted constants are hidden in the main derivation. The paper's design constants (P, τ, M, tile geometry) are configurable knobs rather than fitted physical parameters, but they do affect the reported gains.

free parameters (4)
  • Sequence pool size P = 8
    Selected by offline simulator over balance-communication trade-off; used for both 256K and 1M configs. Microbenchmarks show P=4 slightly faster (0.33 vs 0.34 on 256K, 0.37 vs 0.39 on 1M), but P=8 is used throughout (§5, §6.4). Central claims depend on this choice.
  • Load-target slack τ = 0.03
    Fixes soft load-target C=(1+τ)·mean load in Algorithm 2; no sensitivity analysis reported (§4.3.2).
  • Pipeliner chunk count M = 4
    Configurable default; overlap bound is (M−1)/M; no sensitivity analysis reported (§4.4).
  • TAP tile geometry (B,H) = B=4096,H=1 (256K); B=8192,H=1 (1M)
    Chosen via grid search over B∈{2048..32768}, H∈{1..8}; performance varies from 0.81x to 1.89x across configurations (§6.6).
axioms (5)
  • domain assumption Dense causal attention cost is dominated by O(l^2) core attention, with workload proxy w(S) ∝ Σ_j l_j^2 (Eq. 2).
    Underlies the whole problem statement; sparse/linear attention variants are explicitly out of scope (§2.2).
  • domain assumption Core attention is parameter-free and can be executed by any worker given Q/K/V, mask, and metadata (§2.2).
    Justifies cross-worker migration of SH-Tiles; true in the framework implementation but relies on exact tensor state availability.
  • standard math CLT/Gaussian-maximum approximation E[R] ≈ 1 + CV/√P · √(2 ln(D/P)) (Eq. 3).
    Used as the LLN scaling principle; authors state it is an analytical approximation and that samples are not strictly IID (§3.2).
  • domain assumption Step-equivalence: the raw-sample multiset of each optimizer step must be preserved.
    Design constraint that rules out WLB-LLM's outlier deferral and motivates VRSP (§2.3, §3.2).
  • domain assumption Fixed-token packing with block-diagonal causal mask and mbs=1.
    Evaluation and design assume this configuration (§2.3, §2.1); results may not transfer to variable-token packing.
invented entities (3)
  • Sequence pool no independent evidence
    purpose: Fixed-size set of packed sequences and their CP groups over which attention work can be redistributed; decouples balancing scope from DP degree.
    Implementation abstraction, fully defined in §3.2/§4.1; no external falsifiable handle beyond the paper's own measurements.
  • SH-Tile no independent evidence
    purpose: Schedulable sequence-block × head-shard attention unit for intra-pool load balancing.
    Decomposition concept; evidence is the grid search in §6.6, inside the paper.
  • KV group no independent evidence
    purpose: Reuse unit for K/V tensors per raw sample and head shard, used to deduplicate tile-fetch communication.
    Bookkeeping abstraction in §4.3, not an empirical entity.

pith-pipeline@v1.3.0-alltime-deepseek · 22198 in / 22112 out tokens · 186751 ms · 2026-07-31T23:58:14.263499+00:00 · methodology

0 comments
read the original abstract

Long-context LLM training suffers from a load-balancing problem that sequence packing does not solve. Packing samples into fixed-token sequences balances memory and linear-cost operators, but the dominant attention cost scales with the sum of squared sequence lengths. Thus, equally sized packed sequences drawn from a long-tailed corpus can carry substantially different attention workloads, creating data-parallel stragglers and pipeline bubbles. Existing approaches either balance at the granularity of sequences or microbatches, where an outlier can dominate an assignment, or disaggregate attention over a global worker pool whose communication domain grows with the data-parallel (DP) degree. We present Libra, which operationalizes the law of large numbers (LLN) as a scaling principle for load balancing: the attention-balancing pool need not grow with the DP degree. Libra groups packed sequences and their CP groups into fixed-size sequence pools. As DP scales out, Libra adds pools rather than enlarging each one, bounding every attention exchange. Variance-Reduced Sequence Placement makes this effective for finite, long-tailed workloads by co-locating sequences with complementary attention workloads to reduce residual inter-pool skew. Within each pool, Tiled Attention Pooling dispatches sequence-head SH-Tiles across GPUs, while a pipelined runtime overlaps tile exchange with attention. Libra exposes a drop-in context-parallel attention operator and a pluggable data sampler, requiring no changes to model layers, optimizers, or pipeline schedules. On Qwen3-Turbo training with 256K- and 1M-token workloads, Libra improves end-to-end throughput by up to 2.54x over Ulysses, with up to 3.14x worst-step straggler-attention speedup in microbenchmarks. Libra has run for hundreds of thousands of GPU-hours in production on jobs spanning 32K to 1M tokens while preserving training semantics.

Figures

Figures reproduced from arXiv: 2607.23250 by Chang Si, Guangming Tan, Hongqing Chen, Jiaxuan Peng, Jingren Zhou, Kaiming Yang, Langshi Chen, Linlang Jiang, Man Yuan, Mingzhen Li, Pengju Lu, Rui Men, Siyu Wang, Weile Jia, Xiulong Yuan, Yan Wang, Yong Li, Zhipeng Zhang, Zhixiang Ruan.

Figure 1
Figure 1. Figure 1: Normalized cluster throughput of Qwen3-Turbo training with 1M-token packed sequences and CP= 16, as DP scales from 1 to 16. 1 Introduction Long contexts are increasingly important to LLM applica￾tions such as coding, reasoning, and data analysis [1, 9, 28]. Training on long contexts, however, exposes an acute load￾balancing problem. Sequence lengths in production corpora are highly skewed: our long-context… view at source ↗
Figure 3
Figure 3. Figure 3: Runtime variation across 256 identical GPUs ex￾ecuting the same 32K causal-attention input for 500 steps. At each step we discard one fastest and one slowest outlier, then report (slowest − fastest)/fastest over the remaining workers. and FlexSP [23]. All such approaches face an indivisible￾outlier lower bound: rearranging other samples cannot make a single outlier cheaper. Methods that defer an outlier to… view at source ↗
Figure 4
Figure 4. Figure 4: Residual inter-pool imbalance versus pool size for the 256K (left) and 1M (right) datasets at GBS= 128. Distribu￾tions are obtained by empirical random grouping; markers show their means, and the analytical curve follows Eq. 3. Under the analytical approximation that packed-sequence workloads are grouped randomly and are independent and identically distributed (IID) with finite variance, the Central Limit … view at source ↗
Figure 6
Figure 6. Figure 6: Design overview of Libra. REORDER GA 0 Worker Group 0 GA 1 GA 0 GA 1 Worker Group 1 Worker Group 2 Worker Group 3 Worker Group 0 Worker Group 1 Worker Group 2 Worker Group 3 Pool GA Worker Group [PITH_FULL_IMAGE:figures/full_fig_p006_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: GA-index/pool layout (GBS=16, DP=8, 𝑃=2). At each GA index, four pools occupy disjoint groups of two DP ranks; the next GA index reuses the same four worker groups. Libra uses three components to address two levels of skew. Variance-Reduced Sequence Placement (VRSP; §4.2) bal￾ances aggregate attention FLOPs across sequence pools. Tiled Attention Pooling (TAP; §4.3) decomposes each packed se￾quence along th… view at source ↗
Figure 8
Figure 8. Figure 8: shows the resulting reduction. At 𝑃=8, VRSP lowers 𝑅−1 below 0.7% on both datasets, compared with 90% and 108% under random grouping. It also consistently outperforms Zigzag and Zigzag-with-swap across the evalu￾ated pool sizes. Thus, VRSP supplies the per-step placement mechanism that bare LLN lacks: LLN motivates keeping each pool bounded, while workload-aware placement balances the actual pool instances… view at source ↗
Figure 9
Figure 9. Figure 9: SH-Tile construction and communication-aware placement (𝐵=2048, 𝐻=2, 𝑃=2, CP=2). The placer balances its FLOPs while favoring destinations that avoid Q/output migration or reuse resident KV groups. ties by communication and worker ID. This fallback guar￾antees a complete assignment for every input. 𝐶 guides the balance–communication trade-off rather than providing a worst-case load guarantee; in particular… view at source ↗
Figure 10
Figure 10. Figure 10: TAP Pipeliner with 𝑀=4. Nonblocking dispatch and return operations overlap the computation of adjacent equal-head chunks; the first dispatch and final return remain exposed. correspondingly transferred in 𝑀 head pieces. We use the configurable default 𝑀 = 4 in all experiments. Asynchronous overlap. The runtime issues nonblock￾ing communication and delays synchronization until the transferred tensors are c… view at source ↗
Figure 11
Figure 11. Figure 11: Throughput scaling vs. DP size (relative to DP=1; ideal is linear) on the 256K and 1M datasets. Metrics. For end-to-end training, we report throughput in tokens per GPU per second as the primary efficiency met￾ric. For standalone microbenchmarks, we report the core￾attention latency, including both computation and commu￾nication, measured on the slowest worker in the step, where a step is the execution of… view at source ↗
Figure 13
Figure 13. Figure 13: Per-step core-attention latency of Libra and all baselines on the 256K and 1M datasets. We compare Libra against all baselines (Ulysses, WLB￾LLM, DistCA) on the 256K and 1M datasets, isolating the core-attention layer. We take each step’s straggler latency (the slowest worker) as its time, and report the mean and max across steps in a GBS window: the mean reflects throughput without PP, while the max prox… view at source ↗
Figure 14
Figure 14. Figure 14: Per-step core-attention latency vs. pool size (256K, 1M), normalized to the Ulysses mean. We sweep the pool size 𝑃 under the same parallelism layout (CP, DP, GBS) as §6.3, comparing all three variants. We report the executor time of the slowest pool, averaged across steps and normalized to the Ulysses mean executor time (530.6 ms on 256K, 3955.0 ms on 1M). For Libra (TAP) and Libra (TAP+VRSP) we split eac… view at source ↗
Figure 15
Figure 15. Figure 15: Core-attention speedup over Ulysses across num_head_split (𝐻) and block size (𝐵) on 256K and 1M. 7 Related Work Attention within a CP group. Most systems divide one packed sequence within a context-parallel (CP) group. Ring schemes include Ring Attention [13], Zigzag-Ring [10], and Striped Attention [3]; they circulate KV blocks and arrange sequence shards to reduce causal imbalance. Burst￾Engine [20] ove… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

34 extracted references · 1 canonical work pages

  1. [1]

    Shengnan An, Zexiong Ma, Zeqi Lin, Nanning Zheng, Jian-Guang Lou, and Weizhu Chen. 2024. Make your llm fully utilize the context. Advances in Neural Information Processing Systems37 (2024), 62160– 62188

    2024

  2. [2]

    Yushi Bai, Xin Lv, Jiajie Zhang, Yuze He, Ji Qi, Lei Hou, Jie Tang, Yuxiao Dong, and Juanzi Li. 2024. LongAlign: A Recipe for Long Context Alignment of Large Language Models. InFindings of the Association for Computational Linguistics: EMNLP 2024, Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (Eds.). Association for Computational Linguistics, Miami, F...

    doi:10.18653/v1/2024 2024

  3. [3]

    William Brandon, Aniruddha Nrusimha, Kevin Qian, Zachary Ankner, Tian Jin, Zhiye Song, and Jonathan Ragan-Kelley. 2023. Striped Attention: Faster Ring Attention for Causal Transformers. arXiv:2311.09431 [cs.LG]https://arxiv.org/abs/2311.09431

    Pith/arXiv arXiv 2023

  4. [4]

    Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. 2024. LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models. InInternational Con- ference on Learning Representations, B. Kim, Y. Yue, S. Chaud- huri, K. Fragkiadaki, M. Khan, and Y. Sun (Eds.), Vol. 2024. 8220–8238.https://proceedings.iclr.cc/paper_fil...

    2024

  5. [5]

    Diandian Gu, Peng Sun, Qinghao Hu, Ting Huang, Xun Chen, Ying- tong Xiong, Guoteng Wang, Qiaoling Chen, Shangchun Feng, Jie Fang, Yuhui Wang, Yong Li, Cong Wang, Wei Wei, Zhangsong Yang, Yunhai Tong, Yonggang Wen, Tianwei Zhang, and Yang You. 2024. Loong- Train: Efficient Training of Long-Sequence LLMs with Head-Context Parallelism. arXiv:2406.18485 [cs.D...

    Pith/arXiv arXiv 2024

  6. [6]

    Le, Yonghui Wu, and Zhifeng Chen

    Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V. Le, Yonghui Wu, and Zhifeng Chen. 2019. GPipe: Efficient training of giant neural networks using pipeline parallelism. InAdvances in Neural Information Processing Systems (NeurIPS)

    2019

  7. [7]

    Sam Ade Jacobs, Masahiro Tanaka, Chengming Zhang, Minjia Zhang, Shuaiwen Leon Song, Samyam Rajbhandari, and Yuxiong He. 2023. DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models. arXiv:2309.14509 [cs.LG] https://arxiv.org/abs/2309.14509

    Pith/arXiv arXiv 2023

  8. [8]

    Chenyu Jiang, Zhenkun Cai, Ye Tian, Zhen Jia, Yida Wang, and Chuan Wu. 2025. DCP: Addressing Input Dynamism In Long-Context Train- ing via Dynamic Context Parallelism. InProceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles(Lotte Hotel World, Seoul, Republic of Korea)(SOSP ’25). Association for Computing Ma- chinery, New York, NY, U...

    arXiv 2025

  9. [9]

    Hongye Jin, Xiaotian Han, Jingfeng Yang, Zhimeng Jiang, Zirui Liu, Chia-Yuan Chang, Huiyuan Chen, and Xia Hu. 2024. Llm maybe longlm: Self-extend llm context window without tuning.arXiv preprint arXiv:2401.01325(2024)

    Pith/arXiv arXiv 2024

  10. [10]

    Vijay Anand Korthikanti, Jared Casper, Sangkug Lym, Lawrence McAfee, Michael Andersch, Mohammad Shoeybi, and Bryan Catan- zaro. 2023. Reducing Activation Recomputation in Large Transformer Models. InProceedings of Machine Learning and Systems (MLSys)

    2023

  11. [11]

    Perez, and Andrew Fitzgib- bon

    Mario Michael Krell, Matej Kosec, Sergio P. Perez, and Andrew Fitzgib- bon. 2021. Efficient Sequence Packing without Cross-contamination: Accelerating Large Language Models without Impacting Performance. arXiv:2107.02027 [cs.LG]https://arxiv.org/abs/2107.02027

    Pith/arXiv arXiv 2021

  12. [12]

    Zhouyang Li, Yuliang Liu, Wei Zhang, Tailing Yuan, Bin Chen, and Chengru Song. 2025. SlimPipe: Memory-Thrifty and Efficient Pipeline Parallelism for Long-Context LLM Training. InProceedings of the In- ternational Conference for High Performance Computing, Networking, Storage and Analysis. 1409–1428

    2025

  13. [13]

    Hao Liu, Matei Zaharia, and Pieter Abbeel. 2024. RingAttention with Blockwise Transformers for Near-Infinite Context. InThe Twelfth 14 Libra Conference’17, July 2017, Washington, DC, USA International Conference on Learning Representations (ICLR).https: //openreview.net/forum?id=WsRHpHH4s0

    2024

  14. [14]

    Weijian Liu, Mingzhen Li, Guangming Tan, and Weile Jia. 2025. Mario: Near zero-cost activation checkpointing in pipeline parallelism. In Proceedings of the 30th ACM SIGPLAN Annual Symposium on Principles and Practice of Parallel Programming. 197–211

    2025

  15. [15]

    Parth Mannan, Kunlun Li, Tailai Ma, Sophia Yang, Guohao Wu, and Chenyu Wang. 2026. Speeding Up Variable-Length Training with Dynamic Context Parallelism and NVIDIA Megatron Core. NVIDIA Technical Blog.https://developer.nvidia.cn/blog/speeding- up-variable-length-training-with-dynamic-context-parallelism- and-nvidia-megatron-core/Accessed: 2026-06-05

    2026

  16. [16]

    Devanur, Gregory R

    Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil R. Devanur, Gregory R. Ganger, Phillip B. Gibbons, and Matei Zaharia. 2019. PipeDream: Generalized Pipeline Parallelism for DNN Training. InProceedings of the 27th ACM Symposium on Operating Systems Principles (SOSP). 1–15. doi:10.1145/3341301.3359646

    arXiv 2019

  17. [17]

    Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGres- ley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, Amar Phanishayee, and Matei Zaharia. 2021. Efficient large-scale language model training on GPU clusters using Megatron-LM. InProceedings of the International Conference for High...

    2021

  18. [18]

    Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He

  19. [19]

    Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019. Megatron-lm: Training multi- billion parameter language models using model parallelism.arXiv preprint arXiv:1909.08053(2019)

    Pith/arXiv arXiv 2019

  20. [20]

    Ao Sun, Weilin Zhao, Xu Han, Cheng Yang, Zhiyuan Liu, Chuan Shi, and Maosong Sun. 2025. BurstEngine: An efficient distributed frame- work for training transformers On extremely Long sequences of over 1M tokens. InProceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC ’25). As- sociation for Computi...

    arXiv 2025

  21. [21]

    Gomez, Łukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. InProceedings of the 31st International Con- ference on Neural Information Processing Systems(Long Beach, Cali- fornia, USA)(NIPS’17). Curran Associates Inc., Red Hook, NY, USA, 6000–6010

    2017

  22. [22]

    Shuhe Wang, Guoyin Wang, Yizhong Wang, Jiwei Li, Eduard Hovy, and Chen Guo. 2025. Packing Analysis: Packing Is More Appropriate for Large Models or Datasets in Supervised Fine-tuning. InFindings of the Association for Computational Linguistics: ACL 2025, Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (Eds.). Association for Co...

    work page doi:10.18653/v1/2025.findings-acl.256 2025

  23. [23]

    Yujie Wang, Shiju Wang, Shenhan Zhu, Fangcheng Fu, Xinyi Liu, Xuefeng Xiao, Huixia Li, Jiashi Li, Faming Wu, and Bin Cui. 2025. FlexSP: Accelerating Large Language Model Training via Flexible Sequence Parallelism. InProceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2(Rotte...

    arXiv 2025

  24. [24]

    Zheng Wang, Anna Cai, Xinfeng Xie, Zaifeng Pan, Yue Guan, Weiwei Chu, Jie Wang, Shikai Li, Jianyu Huang, Chris Cai, Yuchen Hao, and Yufei Ding. 2025. WLB-LLM: workload-balanced 4D parallelism for large language model training. InProceedings of the 19th USENIX Con- ference on Operating Systems Design and Implementation(Boston, MA, USA)(OSDI ’25). USENIX As...

    2025

  25. [25]

    Maurice Weber, Daniel Fu, Quentin Anthony, Yonatan Oren, Shane Adams, Anton Alexandrov, Xiaozhong Lyu, Huu Nguyen, Xiaozhe Yao, Virginia Adams, Ben Athiwaratkun, Rahul Chalamala, Kezhen Chen, Max Ryabinin, Tri Dao, Percy Liang, Christopher Ré, Irina Rish, and Ce Zhang. 2024. RedPajama: an Open Dataset for Train- ing Large Language Models. InAdvances in Ne...

    Pith/arXiv arXiv 2024

  26. [26]

    Hongtao Xu, Wenting Shen, Yuanxin Wei, Ang Wang, Guo Run- fan, Tianxing Wang, Yong Li, Mingzhen Li, and Weile Jia. 2026. Skrull: Towards Efficient Long Context Fine-tuning through Dynamic Data Scheduling. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems.https://openreview.net/forum?id= WBEknRZBpT

    2026

  27. [27]

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. 2025. Qwen3 technical report.arXiv preprint arXiv:2505.09388(2025)

    Pith/arXiv arXiv 2025

  28. [28]

    Hongli Yu, Tinghong Chen, Jiangtao Feng, Jiangjie Chen, Weinan Dai, Qiying Yu, Ya-Qin Zhang, Wei-Ying Ma, Jingjing Liu, Mingxuan Wang, et al. 2025. Memagent: Reshaping long-context llm with multi-conv rl-based memory agent.arXiv preprint arXiv:2507.02259(2025)

    Pith/arXiv arXiv 2025

  29. [29]

    Xiulong Yuan, Hongtao Xu, Wenting Shen, Ang Wang, Xiafei Qiu, Jie Zhang, Yuqiong Liu, Bowen Yu, Junyang Lin, Mingzhen Li, Weile Jia, Yong Li, and Wei Lin. 2025. Efficient Long Context Fine-tuning with Chunk Flow. InForty-second International Conference on Machine Learning.https://openreview.net/forum?id=rzn2OgflOK

    2025

  30. [30]

    Tao Zewei and Huang Yunpeng. 2025. MagiAttention: A Dis- tributed Attention Towards Linear Scalability for Ultra-Long Con- text, Heterogeneous Mask Training.https://github.com/SandAI- org/MagiAttention/

    2025

  31. [31]

    Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, Alban Desmaison, Can Balioglu, Pritam Damania, Bernard Nguyen, Geeta Chauhan, Yuchen Hao, Ajit Mathews, and Shen Li. 2023. PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel.Proceedings of the VLDB Endowment (PVLDB...

    arXiv 2023

  32. [32]

    Yilong Zhao, Xiaonan Nie, Kan Zhu, Shuang Ma, Zhichao Lai, Hongxi- ang Hao, Yang Zhou, Baris Kasikci, and Ion Stoica. 2026. Unleashing Scalable Context Parallelism for Foundation Models Pre-Training via FCP. InNinth Conference on Machine Learning and Systems.https: //openreview.net/forum?id=MPVycRsIn6

    2026

  33. [33]

    Yonghao Zhuang, Junda Chen, Bo Pang, Yi Gu, Yibo Zhu, Yimin Jiang, Ion Stoica, Hao Zhang, and Eric P. Xing. 2026. Efficient Long-Context Language Model Training by Core Attention Disaggregation. InNinth Conference on Machine Learning and Systems.https://openreview.net/ forum?id=oIonqkc8hM 15

    2026

  34. [2020]

    InProceedings of the International Conference for High Perfor- mance Computing, Networking, Storage and Analysis(Atlanta, Georgia) (SC ’20)

    ZeRO: memory optimizations toward training trillion parameter models. InProceedings of the International Conference for High Perfor- mance Computing, Networking, Storage and Analysis(Atlanta, Georgia) (SC ’20). IEEE Press, Article 20, 16 pages