Pith. sign in

REVIEW 12 cited by

Reducing Activation Recomputation in Large Transformer Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2205.05198 v1 pith:JB2RKQ6Z submitted 2022-05-10 cs.LG cs.CL

Vijay Korthikanti , Jared Casper , Sangkug Lym , Lawrence McAfee , Michael Andersch , Mohammad Shoeybi , Bryan Catanzaro This is my paper
classification cs.LGcs.CL
keywords activationrecomputationmemorymodelslargereducingtrainingtransformer
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Training large transformer models is one of the most important computational challenges of modern AI. In this paper, we show how to significantly accelerate training of large transformer models by reducing activation recomputation. Activation recomputation is commonly used to work around memory capacity constraints. Rather than storing activations for backpropagation, they are traditionally recomputed, which saves memory but adds redundant compute. In this work, we show most of this redundant compute is unnecessary because we can reduce memory consumption sufficiently without it. We present two novel yet very simple techniques: sequence parallelism and selective activation recomputation. In conjunction with tensor parallelism, these techniques almost eliminate the need to recompute activations. We evaluate our approach on language models up to one trillion parameters in scale and show that our method reduces activation memory by 5x, while reducing execution time overhead from activation recomputation by over 90%. For example, when training a 530B parameter GPT-3 style model on 2240 NVIDIA A100 GPUs, we achieve a Model Flops Utilization of 54.2%, which is 29% faster than the 42.1% we achieve using recomputation. Our implementation will be available in both Megatron-LM and NeMo-Megatron.

Discussion (0). Sign in to comment.

Forward citations

Cited by 12 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. OpenAlex reports about 55 citations worldwide. Full citation record

  1. Explaining Data Mixing Scaling Laws

    cs.LG 2026-06 unverdicted novelty 7.0 of 10

    Under a shared-head/disjoint-tail assumption, multi-domain loss decomposes into a capacity-competition term c_i x_i^*(h)^{-b_i} plus a per-domain noise term A_i(Dh_i)^{-a_i}, and the fitted law extrapolates optimal mi...

  2. Unlock the Potential of Fine-grained LLM Serving via Dynamic Module Scaling

    cs.DC 2025-07 conditional novelty 7.0 of 10

    CoCoServe dynamically replicates and migrates individual LLM modules across GPUs, claiming up to 46% cost reduction and 1.16x-4x throughput gains over HFT and vLLM.

  3. A Training-Memory Regression in MLA Sequence Parallelism: Why Megatron-Core Forbids Absorption, and LAGA -- a Communication-Efficient Fix

    cs.DC 2026-07 conditional novelty 6.0 of 10

    LAGA replaces MLA training's per-head K/V all-to-all with a latent all-gather and local up-projection, matching explicit-form memory while cutting collective communication ~1.98x.

  4. The Cost and Network Limits of Space-Based AI Compute

    cs.DC 2026-07 conditional novelty 6.0 of 10

    Orbital laser-mesh networks have ~10,000x less bisection bandwidth than terrestrial Clos networks, making LEO training of frontier LLMs 100x+ more expensive while single-satellite inference remains plausible.

  5. Untied Ulysses: Memory-Efficient Context Parallelism via Headwise Chunking

    cs.LG 2026-02 conditional novelty 6.0 of 10

    UPipe chunks attention by head so QKV and all-to-all buffers scale with a small tunable chunk size rather than head count, enabling 5M-token Llama3-8B training on one 8×H100 node with throughput close to Ulysses.

  6. Scalable Synthesis of distributed LLM workloads through Symbolic Tensor Graphs

    cs.DC 2025-11 conditional novelty 6.0 of 10

    STAGE synthesizes high-fidelity Chakra-format execution graphs for distributed LLM workloads from symbolic tensor definitions, validated against real 128-GPU H100 traces and scaled to 32K GPUs.

  7. RoboBrain 2.0 Technical Report

    cs.RO 2025-07 conditional novelty 6.0 of 10

    RoboBrain 2.0, a 7B/32B embodied vision-language model built on Qwen2.5-VL, reports state-of-the-art or near-top scores on several spatial and temporal reasoning benchmarks for robotics.

  8. Arctic Long Sequence Training: Scalable And Efficient Training For Multi-Million Token Sequences

    cs.LG 2025-06 conditional novelty 6.0 of 10

    ALST combines tiled loss and MLP compute, Ulysses-style sequence parallelism for Hugging Face GQA and MQA models, and activation-checkpoint CPU offload to train Llama-8B at up to 500K tokens on one H100 and 15M tokens...

  9. QLPO: Quadrant-weighted Sampling for Length-aware Policy Optimization

    cs.AI 2026-07 conditional novelty 5.0 of 10

    QLPO resamples GRPO training groups to favor short correct and long incorrect responses, cutting reasoning length substantially while keeping accuracy roughly unchanged.

  10. LongStraw: Long-Context RL Beyond 2M Tokens under a Fixed GPU Budget

    cs.LG 2026-07 reject novelty 5.0 of 10

    LongStraw caches a long prompt without autograd and replays response suffixes with gradients to fit 2M+ token GRPO-like training into fixed GPU budgets, but it never proves the resulting update equals the true RL gradient.

  11. Efficient Scaling of LLM Training with Flexible Context Parallelism

    cs.DC 2026-02 reject novelty 5.0 of 10

    Regrouping context-parallel workers per micro-batch with non-power-of-two group sizes, chosen by a fitted cost model plus dynamic programming, is claimed to speed up multimodal LLM training by up to 1.36x (1.46x per t...

  12. Photonic Fabric Platform for AI Accelerators

    cs.PF 2025-07 conditional novelty 4.0 of 10

    A vendor paper projects large LLM speedups and energy savings from a photonic shared-memory appliance, using a simulator validated only on conventional GPUs, not on the appliance itself.

Pith tools