Pith. sign in

REVIEW 4 major objections 5 minor 33 references

The paper claims that processing attention heads in small reusable chunks instead of all at once breaks the activation-memory barrier for long-context Transformer training, letting an 8B model reach 5M-token contexts on a single 8-GPU node

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 21:06 UTC pith:I4OZD25R

load-bearing objection UPipe's headwise chunking is a real memory win for Ulysses-style context parallelism, but the headline 25% over FPDT needs an explanation and the GQA schedule needs a correctness check. the 4 major comments →

arxiv 2602.21196 v2 pith:I4OZD25R submitted 2026-02-24 cs.LG cs.DC

Untied Ulysses: Memory-Efficient Context Parallelism via Headwise Chunking

Ravi Ghadia , Maksim Abraham , Sergei Vorobyov , Max Ryabinin This is my paper
classification cs.LG cs.DC
keywords UPipecontext parallelismheadwise chunkinglong-context trainingactivation memorygrouped query attentionsequence parallelism
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper aims to show that the main memory wall for training Transformers on very long sequences can be attacked from the head dimension, not just the sequence dimension. It introduces UPipe, a context-parallel schedule that processes U attention heads per stage instead of all H heads, reusing the same QKV and all-to-all buffers across stages. This lowers attention activation memory from depending on total head count to depending only on chunk size U; with U equal to the number of parallel devices, memory no longer scales with the model's head count. The authors demonstrate the payoff by training an 8B model at 5M-token context on a single 8×H100 node, a 25% context-length gain over the previous best method, and by reducing attention intermediate tensors by up to 87.5% for a 32B model. A grouped-query variant reorders query heads so KV tensors are communicated once, so the memory gains do not come with extra communication overhead.

Core claim

The central claim is that serializing attention at the head level—chunking the H heads into H/U groups, running one group per stage, and reusing buffers—makes the peak activation memory proportional to U rather than H. Because the smallest valid chunk size is U equal to the context-parallel degree C, the attention activation footprint can be made independent of model head count. On the authors' testbed, this is enough to fit a 5-million-token context for an 8-billion-parameter model on one eight-GPU node, extending the reachable context length by about 25% over the previous record while keeping throughput close to non-memory-optimized context parallelism. For a 32-billion-parameter model wit

What carries the argument

The central mechanism is headwise chunking with buffer reuse: UPipe projects only U query/key/value heads, runs an all-to-all reshuffle that swaps sequence shards for head shards on those U heads, computes attention, then reuses the same buffers for the next U heads. The memory-scaling law follows directly: intermediate tensor memory drops from 12·(S/C)·H·d_head bytes to 12·(S/C)·U·d_head bytes. The companion grouped-query schedule is an out-of-order query dispatch: all unique KV heads are communicated in the first stage, then the remaining queries of each group are processed over the next G stages without re-sending K or V, beating the naive communication volume of repeated head groups.

Load-bearing premise

The load-bearing premise is that for long sequences and sufficiently large models, a subset of attention heads is enough to keep the GPUs compute-bound—so chunking heads in stages does not cost throughput; if that premise fails for smaller models, shorter sequences, or unusual head counts, the memory savings survive but the speed-parity claim collapses.

What would settle it

A direct test: run UPipe at U=C on a small model (say 1B with 16 heads) at a mid-length sequence (e.g., 128K) and compare per-GPU throughput against running with U=H; if the U=C run is materially slower while memory savings are small, the compute-bound premise fails. A second check: use a model where the number of unique KV heads is not equal to C and measure whether the GQA schedule's communication volume stays below naive processing; if it does not, the claimed communication benefit fails in that configuration.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • On a single 8×H100 node, an 8B Transformer can be trained at context lengths up to 5M tokens—25% beyond the prior best—with throughput essentially unchanged at lengths of 2M and above.
  • On two nodes with hybrid intra-node/inter-node parallelism, the same method reaches 8M tokens, a 33% context-length improvement over the prior hybrid approach.
  • Attention activation memory scales with chunk size U rather than total head count H; setting U=C makes it head-count-independent, which matters as models grow more heads.
  • The grouped-query schedule keeps the memory benefit without paying extra communication: KV heads are sent once and reused across query groups.
  • Headwise chunking is orthogonal to sequence-chunking and offload-based methods, so the memory savings can be stacked with those techniques for even longer contexts.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the memory-scaling claim holds beyond the two tested models, headwise chunking offers a simple practical recipe: on any cluster, choose U to hit a target memory ceiling and let sequence length grow until the compute-bound assumption starts to fail.
  • The same out-of-order head scheduling idea could generalize to any attention variant that shares K/V across heads or positions—cross-attention, sliding-window attention—where grouped reuse of K/V would yield communication savings beyond GQA.
  • The paper only explores U=C plus one ablation on one model; a natural next step is a per-layer or per-phase scheduler that picks U dynamically, using the demonstrated memory–throughput tradeoff instead of fixing U at C.
  • The paper's GQA analysis assumes the number of unique KV heads equals the parallel degree; a testable extension would handle H/G≠C while preserving the communication savings.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces UPipe, a context-parallelism method built on DeepSpeed-Ulysses. Instead of performing attention over all heads at once, UPipe chunks the attention computation headwise into H/U stages, processing U heads per stage and reusing QKV/all-to-all buffers across stages. This reduces the intermediate attention activation memory from O(H) to O(U); with U=C the per-device attention buffer becomes 12·S·d_head, independent of H, giving an 87.5% reduction for a 64-head model at C=8. The paper also proposes a GQA-aware schedule that communicates unique KV heads once per group and then reuses them for the remaining queries. Experiments on Llama3-8B and Qwen3-32B report throughput and peak-memory numbers against Ulysses, Ring, FPDT, and native PyTorch baselines, claiming 5M-token context on a single 8xH100 node for Llama3-8B (25% beyond FPDT) and 8M on two nodes. The central memory accounting is straightforward and appears correct, but the headline context-length claim depends on an unexplained FPDT failure, and the paper does not verify that the reordered execution produces the same loss/gradients as standard Ulysses.

Significance. If the claims hold, UPipe is a simple and practical improvement: it reduces the dominant attention activation memory while preserving Ulysses-like throughput, and the headwise chunking is orthogonal to sequence-chunking and offloading methods, so it may compose with existing long-context training recipes. The memory savings are derived from an explicit algebraic accounting identity (12·S/C·U·d_head vs 12·S/C·H·d_head), not from fitted parameters, and the paper includes a code release. The GQA schedule is also a useful practical detail. However, the central quantitative claim — 25% longer context than the prior state of the art — rests on a single unexplained failure point (FPDT at >4M), and there is no numerical equivalence test showing that UPipe reproduces Ulysses training dynamics. The contribution would still have value as a memory/throughput optimization at equal context lengths, but the paper's strongest headline claim is not yet substantiated.

major comments (4)
  1. [Table 3 / Table 4, §5.2] The 25% improvement over FPDT is load-bearing and unsupported. FPDT's measured peak GPU memory at 4M is 51.42 GiB (Table 4). If its memory scales roughly linearly with sequence length, a 5M run would be expected to fit in about 64 GiB, below the 80 GiB HBM limit. The only evidence that FPDT cannot reach 5M is the appended note 'FPDT execution fails at lengths >4M' with no explanation. The failure could be a CPU-RAM constraint, a hard-coded limit in the patched FA3 code, or an implementation bug — none of which supports a claim that UPipe extends the context frontier over FPDT. Please either provide a direct FPDT measurement at 5M, or identify and demonstrate the exact resource/implementation limit that prevents FPDT from running at 5M. Without this, the 'over 25%' improvement in the abstract is not established.
  2. [§3.3, §5] The paper never verifies that UPipe produces the same loss and gradients as standard Ulysses. UPipe changes the execution order of attention heads, reuses buffers across stages, and (in the GQA schedule) processes queries out of group order. Such a scheduling change can introduce subtle numerical mismatches or even correctness bugs (e.g., in the all-to-all indexing or the backward pass). None of the experiments reports training loss, gradient norms, or a numerical equivalence check against Ulysses; all tables report only throughput and peak memory. Please add a correctness test — ideally a bitwise or close-tolerance comparison of loss and gradients between UPipe and Ulysses for a small model, or at least a training curve demonstrating matching loss.
  3. [§4.1, GQA scheduling] The GQA schedule is only analyzed for the case where the number of unique KV heads H/G equals the context-parallel degree C (both test models satisfy 32/4=8 and 64/8=8). The general case H/G ≠ C is not discussed. The text says UPipe 'communicates as many unique key/value heads as possible' in stage-0, but the subsequent reuse pattern and the claimed communication-volume formula O((3+G−1)·H/(C·G)·(C−1)) depend on the number of unique KV heads being exactly C so that each device receives one unique KV head per stage. For arbitrary H/G, it is unclear how the schedule generalizes and what the communication volume becomes. Please either extend the analysis to general H/G or explicitly state the divisibility requirement as a limitation of the proposed GQA schedule.
  4. [§3.2, Figure 6] The throughput-parity claim relies on the premise that 'for long sequences and large enough models, a subset of heads is enough to reach the compute-bound regime.' The only direct support is the ablation in Figure 6 for one model (Llama3-8B) at one context length (512K) on 4 GPUs. This is a plausible systems assumption, but it is not quantified for other model sizes, head counts, or sequence lengths. The empirical tables do show that UPipe matches Ulysses at 2M–3M for Llama3-8B, but the general claim in §3.2 is broader. Please either provide additional evidence across configurations or soften the claim to be an empirical observation rather than a design principle.
minor comments (5)
  1. [Abstract/§1] The abstract says '87.5% for 32B Transformers' without specifying that this is the attention intermediate tensor memory, not total memory. The reduction is also only valid for U=C and H/C=8; please state these conditions in the abstract or at least in the conclusion.
  2. [§2.2, Table 1] The Table 1 memory analysis omits the GQA factor in the QKV term: it writes 6·S·H·d_head for all phases, while later Table 2 correctly introduces γ = 1 + 2/g. Please reconcile the two notations or clarify that Table 1 assumes MHA.
  3. [§5.1] The paper says it uses 'full activation-checkpointing with CPU offloading, to maintain consistency with FPDT,' but does not specify the checkpointing granularity or whether the offloading applies to all layers. A sentence on the exact activation-checkpoint schedule would help reproducibility.
  4. [§5.3.2, Qwen3-32B] The claim that UPipe 'delivering 8.3% better performance than FPDT' at 4M is only for Qwen3-32B; please make the comparison context explicit in the text to avoid ambiguity.
  5. [Appendix Table 4] The note 'FPDT execution fails at lengths>4M' is repeated in the table caption but never explained. This is directly connected to Major Comment 1 and should be addressed with an actual failure analysis.

Circularity Check

0 steps flagged

No circular derivation: UPipe's memory savings are a direct buffer-accounting identity, and the throughput claim rests on an empirically tested (if fragile) assumption rather than on its own output.

full rationale

The paper's central memory-saving claim is not circular. Section 3.4 derives DS-Ulysses intermediate tensor usage as 12·S/C·H·d_head and UPipe as 12·S/C·U·d_head; these are direct counts of the QKV and all-to-all buffers for processing U heads per stage, with U chosen by design (U=C in the experiments). No parameter is fitted to the reported memory numbers, and the 87.5% reduction for Qwen3-32B (H=64, C=8, U=8) follows arithmetically. The GQA scheduling claim in Section 4.1 is also derived from the communication schedule: unique KV heads are sent once per G stages and only query heads thereafter, so the O((3+G−1)·H/(C·G)·(C−1)) volume follows from the schedule rather than being fit to measured results. Throughput parity rests on the explicit premise in Section 3.2 that 'for long sequences and large enough models, a subset of heads is enough to reach the compute-bound regime'; this is an empirical assumption that is directly ablated in Figure 6. If the premise fails at other model sizes or lengths, the matching-throughput claim would fail, but that is a correctness risk, not circularity. The paper's use of ALST tiling and Liger-Kernel loss is tooling reuse, not load-bearing self-citation; there is no cited uniqueness theorem or prior author result invoked to force the design. One flagged limitation is relevant to correctness rather than circularity: both Table 3 and Table 4 append 'FPDT execution fails at lengths>4M' without explaining why, even though FPDT's 4M memory (51.42 GiB) appears to extrapolate below the 80 GiB HBM limit. If FPDT's failure is an implementation artifact rather than a fundamental memory limit, the headline '25%' improvement over FPDT would be overstated. This does not make the UPipe derivation circular; it is an evidence gap in the external baseline comparison.

Axiom & Free-Parameter Ledger

1 free parameters · 4 axioms · 0 invented entities

The central memory-saving claim is a buffer-accounting identity, not a physical derivation. It relies on an asserted compute-bound regime and an implicit GQA head-count divisibility condition. No new particles, forces, or entities are introduced.

free parameters (1)
  • Head chunk size U = U=C=8 for all reported experiments; 4–32 explored in ablation
    The memory formula scales as 1/U, and U is the key tunable hyperparameter. Setting U=C maximizes memory efficiency; the paper asserts this retains throughput because 'a subset of heads is enough to reach the compute-bound regime' (§3.2). The headline 87.5% and 5M/8M results all use this hand-chosen value.
axioms (4)
  • domain assumption For long sequences and large enough models, a subset of U<H heads is enough to reach the compute-bound regime, so headwise serialization does not cost throughput.
    Stated in §3.2 and used to justify U=C. The only direct evidence is the Figure 6 ablation at 512K on 4 GPUs, which shows a throughput–memory tradeoff; the 5M/32B parity is extrapolation.
  • domain assumption All-to-all communication requires an additional buffer the same size as the tensor being exchanged, and these buffers dominate peak attention memory.
    Used in §3.4 to derive DS-Ulysses memory as 12·S/C·H·d_head and UPipe as 12·S/C·U·d_head. This is the foundation of the 87.5% reduction figure.
  • standard math FlashAttention computes attention without materializing the S×S attention matrix.
    Invoked in Table 1 and §2.2; standard result from Dao et al. 2022, cited in the paper.
  • ad hoc to paper The GQA scheduling in §4.1 implicitly assumes that the number of unique KV heads H/G equals the context-parallel degree C (or a compatible divisibility structure).
    The scheme communicates all unique K/V heads in stage 0 and reuses them for G stages. Both evaluated models satisfy H/G=C: Llama3-8B has 32/4=8 and Qwen3-32B has 64/8=8. The paper does not state or analyze what happens when H/G≠C.

pith-pipeline@v1.3.0-alltime-deepseek · 19666 in / 19481 out tokens · 181835 ms · 2026-08-02T21:06:32.142343+00:00 · methodology

0 comments
read the original abstract

Efficiently processing long sequences with Transformer models usually requires splitting the computations across accelerators via context parallelism. The dominant approaches in this family of methods, such as Ring Attention or DeepSpeed Ulysses, enable scaling over the context dimension but do not focus on memory efficiency, which limits the sequence lengths they can support. More advanced techniques, such as Fully Pipelined Distributed Transformer or activation offloading, can further extend the possible context length at the cost of training throughput. In this paper, we present UPipe, a simple yet effective context parallelism technique that performs fine-grained chunking at the attention head level. This technique significantly reduces the activation memory usage of self-attention, breaking the activation memory barrier and unlocking much longer context lengths. Our approach reduces intermediate tensor memory usage in the attention layer by as much as 87.5$\%$ for 32B Transformers, while matching previous context parallelism techniques in terms of training speed. UPipe can support the context length of 5M tokens when training Llama3-8B on a single 8$\times$H100 node, improving upon prior methods by over 25$\%$.

Figures

Figures reproduced from arXiv: 2602.21196 by Maksim Abraham, Max Ryabinin, Ravi Ghadia, Sergei Vorobyov.

Figure 1
Figure 1. Figure 1: Comparison of context parallelism approaches on long￾sequence training for Llama 3-8B using 8 × H100s. UPipe pro￾vides maximum efficiency, resulting in longer maximum context length (5M tokens) while retaining throughput. even audio processing (Hori et al., 2021). However, training models to effectively process such long sequences is limited by the accelerator hardware: beyond a certain limit, even keeping… view at source ↗
Figure 2
Figure 2. Figure 2: Memory usage breakdown when training Llama3-8B with a sequence length of 3M tokens across 8 H100 GPUs. AC stands for Activation Checkpointing, AO denotes AC with offloading, OOM stands for Out of Memory. algorithm: similarly to DeepSpeed Ulysses, it uses the same kernels to compute attention as non-distributed training. On Llama 3-8B, our method is able to fit context lengths of up to 5 million tokens on a… view at source ↗
Figure 3
Figure 3. Figure 3: Illustration of (a) DeepSpeed-Ulysses and (b) UPipe designs. UPipe processes attention in a headwise untied manner, so that in each stage, attention is performed only on a subset of heads. This allows memory reuse across different stages, significantly reducing the peak memory usage due to attention activations. HBM (High-Bandwidth Memory) usage illustrates memory utilization due to the intermediate buffer… view at source ↗
Figure 4
Figure 4. Figure 4: Illustration of UPipe’s GQA scheduling algorithm. We communicate as many unique key/value heads as possible along with the corresponding queries in stage-0. In the subsequent stages, we only communicate the next queries of the corresponding groups, reusing the key/value tensors from stage-0 until stage-G, where G is the group size. generate output for head 0, and perform out all to all. In the second stage… view at source ↗
Figure 5
Figure 5. Figure 5: Llama3-8B: Peak GPU memory usage and throughput (normalized w.r.t USP-Hybrid) comparison of UPipe and USP￾Hybrid at different sequence lengths on 16×H100s. Our method significantly outperforms USP-Hybrid in terms of memory effi￾ciency, and allows a maximum context size of 8M, improving upon USP-Hybrid (6M tokens) by 33%, while maintaining throughput. for sequences ≥ 2M. Notably, UPipe always outperforms FP… view at source ↗
Figure 6
Figure 6. Figure 6: Llama3-8B: Ablation analysis of UPipe’s head-chunk size U, with 512K context size on C = 4 GPUs. Smaller chunk size yields better memory efficiency at the cost of throughput and vice versa, allowing a trade-off between memory and throughput. Next, we present an ablation on U discussing the associated tradeoffs between memory and throughput. 5.4. Ablation on the head-chunk size U As shown in [PITH_FULL_IMA… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

33 extracted references · 25 linked inside Pith

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Gqa: Training generalized multi-query transformer models from multi-head checkpoints, 2023

    Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y., Lebrón, F., and Sanghai, S. Gqa: Training generalized multi-query transformer models from multi-head checkpoints, 2023. URL https://arxiv.org/abs/2305.13245

    Pith/arXiv arXiv 2023

  3. [3]

    Arctic long sequence training: Scalable and efficient training for multi-million token sequences, 2025

    Bekman, S., Rajbhandari, S., Wyatt, M., Rasley, J., Ruwase, T., Yao, Z., Qiao, A., and He, Y. Arctic long sequence training: Scalable and efficient training for multi-million token sequences, 2025. URL https://arxiv.org/abs/2506.13996

    Pith/arXiv arXiv 2025

  4. [4]

    Minimax-m1: Scaling test-time compute efficiently with lightning attention

    Chen, A., Li, A., Gong, B., Jiang, B., Fei, B., Yang, B., Shan, B., Yu, C., Wang, C., Zhu, C., et al. Minimax-m1: Scaling test-time compute efficiently with lightning attention. arXiv preprint arXiv:2506.13585, 2025

    Pith/arXiv arXiv 2025

  5. [5]

    K., Cheng, L., Chan, H

    Chia, Y. K., Cheng, L., Chan, H. P., Liu, C., Song, M., Aljunied, S. M., Poria, S., and Bing, L. M-longdoc: A benchmark for multimodal super-long document understanding and a retrieval-aware tuning framework, 2024. URL https://arxiv.org/abs/2411.06176

    Pith/arXiv arXiv 2024

  6. [6]

    Y., Ermon, S., Rudra, A., and R \'e , C

    Dao, T., Fu, D. Y., Ermon, S., Rudra, A., and R \'e , C. Flash A ttention: Fast and memory-efficient exact attention with IO -awareness. In Advances in Neural Information Processing Systems (NeurIPS), 2022

    2022

  7. [7]

    and Zhao, S

    Fang, J. and Zhao, S. Usp: A unified sequence parallelism approach for long context generative ai, 2024. URL https://arxiv.org/abs/2405.07719

    Pith/arXiv arXiv 2024

  8. [8]

    Gemini 3.0: A new era of intelligence with gemini 3

    Gemini Team . Gemini 3.0: A new era of intelligence with gemini 3. Google DeepMind, 2025

    2025

  9. [9]

    Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., Yang, A., Fan, A., Goyal, A., Hartshorn, A., Yang, A., Mitra, A., Sravankumar, A., Korenev, A., Hinsvark, A., Rao, A., Zhang, A., Rodriguez, A., Gregerson, A., Spataru, A., Roziere, B., Biron, B., Tang, B., Chern, B., Caucheteux...

    Pith/arXiv arXiv 2024

  10. [10]

    Unsloth, 2023

    Han, D., Han, M., and team, U. Unsloth, 2023. URL https://github.com/unslothai/unsloth

    2023

  11. [11]

    Hori, T., Moritz, N., Hori, C., and Roux, J. L. Advanced long-context end-to-end speech recognition using context-expanded transformers. ArXiv, abs/2104.09426, 2021. URL https://api.semanticscholar.org/CorpusID:233296591

    Pith/arXiv arXiv 2021

  12. [12]

    Liger-kernel: Efficient triton kernels for LLM training

    Hsu, P.-L., Dai, Y., Kothapalli, V., Song, Q., Tang, S., Zhu, S., Shimizu, S., Sahni, S., Ning, H., Chen, Y., and Wang, Z. Liger-kernel: Efficient triton kernels for LLM training. In Championing Open-source DEvelopment in ML Workshop @ ICML25, 2025. URL https://openreview.net/forum?id=36SjAIT42G

    2025

  13. [13]

    Qwen2.5-coder technical report, 2024

    Hui, B., Yang, J., Cui, Z., Yang, J., Liu, D., Zhang, L., Liu, T., Zhang, J., Yu, B., Lu, K., Dang, K., Fan, Y., Zhang, Y., Yang, A., Men, R., Huang, F., Zheng, B., Miao, Y., Quan, S., Feng, Y., Ren, X., Ren, X., Zhou, J., and Lin, J. Qwen2.5-coder technical report, 2024. URL https://arxiv.org/abs/2409.12186

    Pith/arXiv arXiv 2024

  14. [14]

    A., Tanaka, M., Zhang, C., Zhang, M., Song, S

    Jacobs, S. A., Tanaka, M., Zhang, C., Zhang, M., Song, S. L., Rajbhandari, S., and He, Y. Deepspeed ulysses: System optimizations for enabling training of extreme long sequence transformer models, 2023. URL https://arxiv.org/abs/2309.14509

    Pith/arXiv arXiv 2023

  15. [15]

    Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D

    Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., de las Casas, D., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., Lavaud, L. R., Lachaux, M.-A., Stock, P., Scao, T. L., Lavril, T., Wang, T., Lacroix, T., and Sayed, W. E. Mistral 7b, 2023. URL https://arxiv.org/abs/2310.06825

    Pith/arXiv arXiv 2023

  16. [16]

    Longrag: Enhancing retrieval-augmented generation with long-context llms, 2024

    Jiang, Z., Ma, X., and Chen, W. Longrag: Enhancing retrieval-augmented generation with long-context llms, 2024. URL https://arxiv.org/abs/2406.15319

    Pith/arXiv arXiv 2024

  17. [17]

    Kimi Team , Bai, Y., Bao, Y., Chen, G., Chen, J., Chen, N., Chen, R., Chen, Y., Chen, Y., Chen, Y., Chen, Z., Cui, J., Ding, H., Dong, M., Du, A., Du, C., Du, D., Du, Y., Fan, Y., Feng, Y., Fu, K., Gao, B., Gao, H., Gao, P., Gao, T., Gu, X., Guan, L., Guo, H., Guo, J., Hu, H., Hao, X., He, T., He, W., He, W., Hong, C., Hu, Y., Hu, Z., Huang, W., Huang, Z....

    Pith/arXiv arXiv 2025

  18. [18]

    Reducing activation recomputation in large transformer models, 2022

    Korthikanti, V., Casper, J., Lym, S., McAfee, L., Andersch, M., Shoeybi, M., and Catanzaro, B. Reducing activation recomputation in large transformer models, 2022. URL https://arxiv.org/abs/2205.05198

    Pith/arXiv arXiv 2022

  19. [19]

    B., Zi, Y., Muennighoff, N., Kocetkov, D., Mou, C., Marone, M., Akiki, C., Li, J., Chim, J., Liu, Q., Zheltonozhskii, E., Zhuo, T

    Li, R., Allal, L. B., Zi, Y., Muennighoff, N., Kocetkov, D., Mou, C., Marone, M., Akiki, C., Li, J., Chim, J., Liu, Q., Zheltonozhskii, E., Zhuo, T. Y., Wang, T., Dehaene, O., Davaadorj, M., Lamy-Poirier, J., Monteiro, J., Shliazhko, O., Gontier, N., Meade, N., Zebaze, A., Yee, M.-H., Umapathi, L. K., Zhu, J., Lipkin, B., Oblokulov, M., Wang, Z., Murthy, ...

    Pith/arXiv arXiv 2023

  20. [20]

    Sequence parallelism: Long sequence training from system perspective, 2022

    Li, S., Xue, F., Baranwal, C., Li, Y., and You, Y. Sequence parallelism: Long sequence training from system perspective, 2022. URL https://arxiv.org/abs/2105.13120

    Pith/arXiv arXiv 2022

  21. [21]

    Sequence parallelism: Long sequence training from system perspective

    Li, S., Xue, F., Baranwal, C., Li, Y., and You, Y. Sequence parallelism: Long sequence training from system perspective. In Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 2391--2404, Toronto, Canada, July 2023 b . Association for Com...

    doi:10.18653/v1/2023.acl-long.134 2023

  22. [22]

    Torchtitan: One-stop pytorch native solution for production ready LLM pretraining

    Liang, W., Liu, T., Wright, L., Constable, W., Gu, A., Huang, C.-C., Zhang, I., Feng, W., Huang, H., Wang, J., Purandare, S., Nadathur, G., and Idreos, S. Torchtitan: One-stop pytorch native solution for production ready LLM pretraining. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=SFN6Wm7YBI

    2025

  23. [23]

    Ring attention with blockwise transformers for near-infinite context, 2023

    Liu, H., Zaharia, M., and Abbeel, P. Ring attention with blockwise transformers for near-infinite context, 2023. URL https://arxiv.org/abs/2310.01889

    Pith/arXiv arXiv 2023

  24. [24]

    Sand.ai , Teng, H., Jia, H., Sun, L., Li, L., Li, M., Tang, M., Han, S., Zhang, T., Zhang, W. Q., Luo, W., Kang, X., Sun, Y., Cao, Y., Huang, Y., Lin, Y., Fang, Y., Tao, Z., Zhang, Z., Wang, Z., Liu, Z., Shi, D., Su, G., Sun, H., Pan, H., Wang, J., Sheng, J., Cui, M., Hu, M., Yan, M., Yin, S., Zhang, S., Liu, T., Yin, X., Yang, X., Song, X., Hu, X., Zhang...

    Pith/arXiv arXiv 2025

  25. [25]

    Flashattention-3: Fast and accurate attention with asynchrony and low-precision, 2024

    Shah, J., Bikshandi, G., Zhang, Y., Thakkar, V., Ramani, P., and Dao, T. Flashattention-3: Fast and accurate attention with asynchrony and low-precision, 2024. URL https://arxiv.org/abs/2407.08608

    Pith/arXiv arXiv 2024

  26. [26]

    Roformer: Enhanced transformer with rotary position embedding, 2023

    Su, J., Lu, Y., Pan, S., Murtadha, A., Wen, B., and Liu, Y. Roformer: Enhanced transformer with rotary position embedding, 2023. URL https://arxiv.org/abs/2104.09864

    Pith/arXiv arXiv 2023

  27. [27]

    Wan: Open and advanced large-scale video generative models, 2025

    Team Wan , Wang, A., Ai, B., Wen, B., Mao, C., Xie, C.-W., Chen, D., Yu, F., Zhao, H., Yang, J., Zeng, J., Wang, J., Zhang, J., Zhou, J., Wang, J., Chen, J., Zhu, K., Zhao, K., Yan, K., Huang, L., Feng, M., Zhang, N., Li, P., Wu, P., Chu, R., Feng, R., Zhang, S., Sun, S., Fang, T., Wang, T., Gui, T., Weng, T., Shen, T., Lin, W., Wang, W., Wang, W., Zhou, ...

    Pith/arXiv arXiv 2025

  28. [28]

    N., Kaiser, L., and Polosukhin, I

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need. In Advances in Neural Information Processing Systems (NeurIPS), 2017

    2017

  29. [29]

    Hunyuanvideo 1.5 technical report, 2025

    Wu, B., Zou, C., Li, C., Huang, D., Yang, F., Tan, H., Peng, J., Wu, J., Xiong, J., Jiang, J., Linus, Patrol, Zhang, P., Chen, P., Zhao, P., Tian, Q., Liu, S., Kong, W., Wang, W., He, X., Li, X., Deng, X., Zhe, X., Li, Y., Long, Y., Peng, Y., Wu, Y., Liu, Y., Wang, Z., Dai, Z., Peng, B., Li, C., Gong, G., Xiao, G., Tian, J., Lin, J., Liu, J., Zhang, J., L...

    Pith/arXiv arXiv 2025

  30. [30]

    Qwen3 technical report, 2025

    Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., Zheng, C., Liu, D., Zhou, F., Huang, F., Hu, F., Ge, H., Wei, H., Lin, H., Tang, J., Yang, J., Tu, J., Zhang, J., Yang, J., Yang, J., Zhou, J., Zhou, J., Lin, J., Dang, K., Bao, K., Yang, K., Yu, L., Deng, L., Li, M., Xue, M., Li, M., Zhang, P., Wang, P., Zhu, Q...

    Pith/arXiv arXiv 2025

  31. [31]

    Yang, D., Wang, D., Guo, H., Chen, X., Wu, X., and Meng, H. M. Simplespeech: Towards simple and efficient text-to-speech with scalar latent transformer diffusion models. ArXiv, abs/2406.02328, 2024. URL https://api.semanticscholar.org/CorpusID:270226637

    Pith/arXiv arXiv 2024

  32. [32]

    A., Tanaka, M., Ruwase, O., Subramoni, H., and Panda, D

    Yao, J., Jacobs, S. A., Tanaka, M., Ruwase, O., Subramoni, H., and Panda, D. K. Training ultra long context language model with fully pipelined distributed transformer, 2025. URL https://arxiv.org/abs/2408.16978

    Pith/arXiv arXiv 2025

  33. [33]

    Pytorch fsdp: Experiences on scaling fully sharded data parallel, 2023

    Zhao, Y., Gu, A., Varma, R., Luo, L., Huang, C.-C., Xu, M., Wright, L., Shojanazeri, H., Ott, M., Shleifer, S., Desmaison, A., Balioglu, C., Damania, P., Nguyen, B., Chauhan, G., Hao, Y., Mathews, A., and Li, S. Pytorch fsdp: Experiences on scaling fully sharded data parallel, 2023. URL https://arxiv.org/abs/2304.11277

    Pith/arXiv arXiv 2023