Pith. sign in

REVIEW 2 minor 158 cited by

Training Deep Nets with Sublinear Memory Cost

T0 review · 0 major / 2 minor · reviewed 2026-05-12 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read An algorithm trains an n-layer deep network using O(sqrt(n)) memory at the cost of one extra forward pass.

desk verdict Key takeaway: O(sqrt(n)) memory for deep net training via graph segmentation and recomputation, with solid experiments. read the letter →

arxiv 1604.06174 v2 pith:OPXDY4I7 submitted 2016-04-21 cs.LG

Tianqi Chen , Bing Xu , Chiyuan Zhang , Carlos Guestrin This is my paper · ORCID
classification cs.LG
keywords deepneuralnetworktrainingmemoryoptimizationcheckpointingcomputationgraphanalysissublinearresidualnetworksrecurrentGPUreduction
checked against Foundation.LedgerCanonicalityFoundation.SimplicialLedger
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces a method to train deep neural networks using only the square root of the number of layers in memory. It works by storing checkpoints at regular intervals and recomputing the missing activations during the backward pass. A sympathetic reader would care because many state-of-the-art models are limited by GPU memory, and this allows deeper and more complex models without additional hardware. The approach uses computation graph analysis for automatic in-place operations and memory sharing. Experiments show large reductions such as training a 1000-layer residual network with far less memory.

What carries the argument

The checkpointing strategy that segments the computation graph into sqrt(n) intervals, storing activations only at boundaries and recomputing forwards inside each interval during backpropagation.

What would settle it

Running the algorithm on a 1000-layer residual network and measuring whether peak memory usage scales as O(sqrt(n)), total runtime increases by about 30 percent, and the resulting gradients match those from full-storage training.

Watch

Extended reading notes

Core claim

We design an algorithm that costs O(sqrt(n)) memory to train a n layer network, with only the computational cost of an extra forward pass per mini-batch. As many of the state-of-the-art models hit the upper bound of the GPU memory, our algorithm allows deeper and more complex models to be explored. We focus on reducing the memory cost to store the intermediate feature maps and gradients during training. Computation graph analysis is used for automatic in-place operation and memory sharing optimizations. We show that it is possible to trade computation for memory - giving a more memory efficient training algorithm with a little extra computation cost. In the extreme case, our analysis also 7G

Load-bearing premise

The computation graph can be cleanly segmented into sqrt(n) intervals where recomputing forward passes inside each interval is both correct and cheaper than storing all intermediate activations.

Editorial extensions

If this is right

  • A 1000-layer residual network trains with memory reduced from 48G to 7G and only 30 percent extra running time on ImageNet.
  • Complex recurrent neural networks become trainable on very long sequences with substantially lower memory.
  • State-of-the-art models no longer hit GPU memory limits as quickly, enabling exploration of deeper architectures.
  • An extreme variant reduces memory to O(log n) at the cost of O(n log n) extra forward computation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The approach could lower hardware barriers for training large models and make advanced deep learning more accessible on modest GPUs.
  • Adaptive checkpoint intervals based on per-layer compute cost might improve the compute-memory trade-off further.
  • The method pairs naturally with model parallelism to scale to even larger networks without changing the core algorithm.
  • Systems with high compute throughput relative to memory bandwidth would see the smallest effective overhead from the extra forward passes.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

0 major / 2 minor

Summary. The manuscript presents an algorithm to train deep neural networks with O(sqrt(n)) memory cost for an n-layer network, incurring only the cost of one extra forward pass per mini-batch. This is achieved through computation graph analysis, segmenting the network into intervals, storing boundary activations, and recomputing forward passes within segments during backpropagation. The approach is extended to O(log n) memory with O(n log n) extra computation, and validated on ImageNet with a 1000-layer ResNet (48G to 7G memory) and long-sequence RNNs.

Significance. If the claims hold, this is a significant contribution to deep learning training efficiency, allowing exploration of deeper models on memory-constrained hardware like GPUs. The systematic use of DAG properties for memory optimization, combined with empirical validation showing memory reduction with modest time overhead and correct gradients, provides a practical tool for advancing DL research. The parameter-free derivation from standard graph segmentation is a strength.

minor comments (2)
  1. [Abstract] Abstract: the O(sqrt(n)) claim would be clearer if it explicitly stated the segmentation assumption (clean intervals where recomputation is correct and cheaper than storing all activations) that underpins the bound.
  2. [Experiments] The 30% extra time cost for the 1000-layer ResNet is reported, but a per-component breakdown (recomputation vs. original forward/backward) would make the compute-memory trade-off more transparent.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for the positive and accurate summary of our work, the assessment of its significance, and the recommendation to accept the manuscript. No major comments requiring response or revision were raised.

read point-by-point responses

  1. Referee: No specific major comments were listed in the report.

    Authors: We appreciate the referee's recognition that the algorithm provides a systematic, parameter-free approach to memory reduction via graph segmentation and recomputation, with empirical validation on large models. The description of the O(sqrt(n)) memory bound, the O(log n) extension, and the ImageNet/ResNet and RNN experiments matches our claims exactly. revision: no

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; derivation is self-contained

full rationale

The O(sqrt(n)) memory bound is obtained by partitioning the n-layer computation DAG into sqrt(n) segments, retaining only the sqrt(n) boundary activations, and performing one recomputation of each segment during back-propagation; the total extra work equals one forward pass by direct operation counting on the graph. This counting argument relies only on standard properties of feed-forward and recurrent DAGs plus the in-place/memory-sharing optimizations described in the paper; no parameters are fitted to data, no result is defined in terms of itself, and no load-bearing step reduces to a self-citation. The reported ImageNet and RNN experiments serve as empirical confirmation rather than definitional inputs.

Assumptions & free parameters 0 free parameters · 1 assumptions · 0 invented entities

The paper introduces no new free parameters, axioms beyond standard DAG properties, or invented entities; the contribution is purely algorithmic.

assumptions (1)
  • standard math The forward computation graph is a directed acyclic graph whose nodes correspond to layer activations.
    Invoked when analyzing memory storage and recomputation segments.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Training Deep Nets with Sublinear Memory Cost." pith.science (2026). https://pith.science/paper/OPXDY4I7

@misc{pith2026160406174,
  author       = {Pith},
  title        = {Pith review of: Training Deep Nets with Sublinear Memory Cost},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OPXDY4I7}},
  note         = {Machine review of arXiv:1604.06174}
}
read the original abstract

We propose a systematic approach to reduce the memory consumption of deep neural network training. Specifically, we design an algorithm that costs O(sqrt(n)) memory to train a n layer network, with only the computational cost of an extra forward pass per mini-batch. As many of the state-of-the-art models hit the upper bound of the GPU memory, our algorithm allows deeper and more complex models to be explored, and helps advance the innovations in deep learning research. We focus on reducing the memory cost to store the intermediate feature maps and gradients during training. Computation graph analysis is used for automatic in-place operation and memory sharing optimizations. We show that it is possible to trade computation for memory - giving a more memory efficient training algorithm with a little extra computation cost. In the extreme case, our analysis also shows that the memory consumption can be reduced to O(log n) with as little as O(n log n) extra cost for forward computation. Our experiments show that we can reduce the memory cost of a 1,000-layer deep residual network from 48G to 7G with only 30 percent additional running time cost on ImageNet problems. Similarly, significant memory cost reduction is observed in training complex recurrent neural networks on very long sequences.

Discussion (0). Continue with ORCID to comment.

Forward citations

Showing 60 of 158 Pith papers that cite this

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. OpenAlex reports about 540 citations worldwide. See all 158 Pith citations

  1. UltraEP: Unleash MoE Training and Inference on Rack-Scale Nodes with Near-Optimal Load Balancing

    cs.DC 2026-06 unverdicted novelty 8.0 of 10

    UltraEP is the first exact-load real-time expert balancer for large-EP MoE training and serving on rack-scale nodes, reaching 94.3% of ideal throughput and 1.49x over no-balancing.

  2. Systematic Discovery of Semantic Attacks in Online Map Construction through Conditional Diffusion

    cs.CV 2026-05 unverdicted novelty 8.0 of 10

    MIRAGE discovers semantic attacks on online HD map construction via conditional diffusion, enabling boundary removal and injection that degrade AV performance while passing as realistic environmental changes.

  3. Differentiate the Solver, Not the Equation: Reverse-Sweep Adjoints for Block Implicit Simulation

    cs.GR 2026-08 conditional novelty 7.0 of 10

    A reverse sweep of local 3x3 adjoint solves gives machine-precision gradients through the exact finite-depth Vertex Block Descent solver without forming any global system.

  4. D2PO: Optimizing Diffusion Samplers via Dynamic Preference

    cs.LG 2026-07 conditional novelty 7.0 of 10

    D2PO learns better low-NFE diffusion timestep schedules and CFG weights via DPO on a score-based energy with a dynamic denser-schedule preference target.

  5. A matrix-free, differentiable PyTorch solver for phase-field fracture: Formulation, benchmarks, and inverse analysis

    cs.CE 2026-06 unverdicted novelty 7.0 of 10

    A matrix-free, GPU-compatible PyTorch implementation of phase-field fracture with explicit dynamics, custom differentiable implicit damage solve, benchmarks on dynamic and quasi-static cases, and inverse recovery of f...

  6. Diffusion-Driven State Space Models

    stat.ML 2026-06 unverdicted novelty 7.0 of 10

    DDSSM replaces Gaussian transitions in SSMs with diffusion models to jointly train autoencoders and diffusion on sequential data, outperforming standard deep SSMs on simulated multimodal time series.

  7. ITNet: A Learnable Integral Transform That Subsumes Convolution, Attention, and Recurrence

    cs.AI 2026-06 unverdicted novelty 7.0 of 10

    ITNet unifies convolution, self-attention, and autoregressive recurrence as special cases of a single learnable integral transform implemented via an MLP kernel and matches specialized models on ImageNet, GLUE, ModelN...

  8. RATrain: A Resource-Aware Training Runtime for Large Language Models on Bandwidth-Constrained Heterogeneous Supercomputing Platforms

    cs.DC 2026-06 unverdicted novelty 7.0 of 10

    RATrain introduces a resource-aware scheduler and MT-3000-specific backend for 1F1B LLM training that achieves 1.35x speedup and 97% scaling efficiency while preserving training correctness.

  9. Learn Where Outcomes Diverge: Efficient VLA RL via Probabilistic Chunk Masking

    cs.LG 2026-05 unverdicted novelty 7.0 of 10

    PCM uses success-failure action variance to probabilistically select and mask chunks for gradient updates in GRPO, matching standard success rates with 2.38x wall-clock speedup and 60% lower memory on LIBERO benchmarks.

  10. Efficient and provably convergent end-to-end training of deep neural networks with linear constraints

    math.OC 2026-05 unverdicted novelty 7.0 of 10

    An efficiently computable HS-Jacobian acts as a conservative mapping for projections onto polyhedral sets, supporting provably convergent Adam-based end-to-end training of linearly constrained deep neural networks.

  11. Locking Pretrained Weights via Deep Low-Rank Residual Distillation

    cs.LG 2026-05 unverdicted novelty 7.0 of 10

    DLR-Lock locks open-weight LLMs against unauthorized fine-tuning by swapping MLPs for deep low-rank residual networks that inflate backprop memory and complicate optimization, yet preserve original capabilities via mo...

  12. Finite Volume-Informed Neural Network Framework for 2D Shallow Water Equations: Rugged Loss Landscapes and the Importance of Data Guidance

    cs.LG 2026-05 unverdicted novelty 7.0 of 10

    Data-guided finite-volume PINNs for 2D shallow water equations avoid trivial low-momentum collapse via sparse measurements, achieving up to 22x error reduction on benchmarks and accurate surrogates on real river data.

  13. FlatLands: Generative Floormap Completion From a Single Egocentric View

    cs.CV 2026-03 conditional novelty 7.0 of 10

    A new multi-source real indoor benchmark shows conditional generative models outperform deterministic and ensemble baselines at single-view BEV floor completion, with uncertainty concentrated at layout boundaries.

  14. 4D-LRM: Large Space-Time Reconstruction Model From and To Any View at Any Time

    cs.CV 2025-06 conditional novelty 7.0 of 10

    4D-LRM is a transformer that maps sparse posed frames scattered across time to a cloud of 4D Gaussians and renders any query view at any query time in under 1.5 seconds.

  15. DGS-LRM: Real-Time Deformable 3D Gaussian Reconstruction From Monocular Videos

    cs.GR 2025-06 conditional novelty 7.0 of 10

    A single feed-forward transformer predicts per-pixel deformable 3D Gaussians with dense scene flow from a posed monocular video, enabling real-time dynamic view synthesis and 3D tracking.

  16. Training-Free Inference for High-Resolution Sinogram Completion

    cs.CV 2025-06 unverdicted novelty 7.0 of 10

    HRSino is a training-free inference scheme that completes high-resolution sinograms with adaptive patch skipping and step allocation, cutting memory and time without losing accuracy.

  17. SlimPipe: Memory-Thrifty and Efficient Pipeline Parallelism for Long-Context LLM Training

    cs.LG 2025-04 conditional novelty 7.0 of 10

    A slice-level pipeline-parallel schedule with attention-work redistribution that cuts activation memory roughly by the pipeline size and reduces pipeline bubbles for long-context LLM training.

  18. GME: Improving Universal Multimodal Retrieval by Multimodal LLMs

    cs.CL 2024-12 unverdicted novelty 7.0 of 10

    GME achieves state-of-the-art results in universal multimodal retrieval by training on a balanced synthetic multimodal dataset.

  19. GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection

    cs.LG 2024-03 conditional novelty 7.0 of 10

    GaLore performs full-parameter LLM training with up to 65.5% less optimizer memory by projecting gradients onto a low-rank subspace at each step, matching full-rank performance on LLaMA pre-training and RoBERTa fine-tuning.

  20. Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers for Generative Recommendations

    cs.LG 2024-02 unverdicted novelty 7.0 of 10

    HSTU-based generative recommenders with 1.5 trillion parameters scale as a power law with compute up to GPT-3 scale, outperform baselines by up to 65.8% NDCG, run 5-15x faster than FlashAttention2 on long sequences, a...

  21. Moonwalk: Inverse-Forward Differentiation

    cs.LG 2024-02 unverdicted novelty 7.0 of 10

    Moonwalk enables memory-efficient training of deep networks via mixed-mode gradient computation with vector-inverse-Jacobian products for submersive layers and fragmental checkpointing otherwise, matching backprop run...

  22. Ring Attention with Blockwise Transformers for Near-Infinite Context

    cs.CL 2023-10 unverdicted novelty 7.0 of 10

    Ring Attention uses blockwise computation and ring communication to let Transformers process sequences up to device-count times longer than prior memory-efficient methods.

  23. Efficient Memory Management for Large Language Model Serving with PagedAttention

    cs.LG 2023-09 conditional novelty 7.0 of 10

    PagedAttention achieves near-zero waste in LLM key-value cache memory and enables 2-4x higher serving throughput than prior systems.

  24. LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale

    cs.LG 2022-08 conditional novelty 7.0 of 10

    LLM.int8() performs 8-bit inference for transformers up to 175B parameters with no accuracy loss by combining vector-wise quantization for most features with 16-bit mixed-precision handling of systematic outlier dimensions.

  25. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness

    cs.LG 2022-05 accept novelty 7.0 of 10

    FlashAttention reduces GPU high-bandwidth memory accesses in self-attention via tiling, delivering exact attention with lower IO complexity, 2-3x wall-clock speedups on models like GPT-2, and the ability to train on s...

  26. ZeRO: Memory Optimizations Toward Training Trillion Parameter Models

    cs.LG 2019-10 accept novelty 7.0 of 10

    ZeRO removes memory redundancies in parallel training to scale deep learning models to over a trillion parameters with high throughput on current hardware.

  27. ALBERT: A Lite BERT for Self-supervised Learning of Language Representations

    cs.CL 2019-09 accept novelty 7.0 of 10

    ALBERT reduces BERT parameters via embedding factorization and layer sharing, adds inter-sentence coherence pretraining, and reaches SOTA on GLUE, RACE, and SQuAD with fewer parameters than BERT-large.

  28. On the Acceleration of Deep Learning Model Parallelism with Staleness

    cs.LG 2019-09 reject novelty 7.0 of 10

    DSP decouples forward and backward passes in model-parallel deep CNN training by giving each layer block a preset staleness, yielding speedups up to 4.8x with comparable or better accuracy.

  29. ReconSpan: Reconstruction-Guided Adaptive Latent Tokenization

    cs.CL 2026-08 conditional novelty 6.0 of 10

    Reconstruction-guided chunking creates adaptive latent tokens that beat random boundaries on native reconstruction, but downstream readers recover topic information while losing exact lexical details.

  30. ElastiCo: Elastic Configuration and Interference-Aware Orchestration for GPU Clusters

    cs.DC 2026-08 conditional novelty 6.0 of 10

    A Kubernetes-native scheduler jointly reshapes job configurations, prices cluster resources, and predicts GPU-sharing interference, reporting up to 2.94x lower average JCT and 2.02x higher throughput in testbed and si...

  31. LoCA: Forward-Only LLM Tuning after One-Shot Calibration with Local Credit Assignment

    cs.AI 2026-08 conditional novelty 6.0 of 10

    LoCA tunes adapters with one calibrated map per layer and closed-form ridge solves, avoiding repeated backpropagation after calibration and roughly matching LoRA quality on tested tasks.

  32. TextNCA: Neural Cellular Automata for Language Modeling via Hierarchical Local Attention

    cs.CL 2026-08 conditional novelty 6.0 of 10

    A hierarchical local-attention NCA for language modeling shows that the narrow-to-wide window schedule explains most of its behavior, with iteration adding a small bounded benefit, and the model underperforms transformers.

  33. Backpropagation-Free Trunk Training via the Split Forward Gradients

    stat.ML 2026-07 conditional novelty 6.0 of 10

    Split-FG splits a network into an exactly trained head and a forward-gradient-estimated trunk, reducing variance and reaching 387 perplexity on WikiText-103 with a 16M GPT-2-style model.

  34. AsySplat: Efficient Asymmetric 3D Gaussian Splatting for Long-Sequence Scene Modeling

    cs.CV 2026-07 accept novelty 6.0 of 10

    An asymmetric geometry-appearance architecture for generalizable 3DGS reallocates computation so smaller models match optimization-based NVS quality at ~800× speedup on 32-view 960P inputs while improving zero-shot results.

  35. Architecture Generalization with MetaNCA

    cs.LG 2026-07 conditional novelty 6.0 of 10

    A learned local rule (Weight Transformer) iteratively self-organizes task-network weights from local graph neighborhoods and generalizes across unseen MLP, CNN, and ResNet architectures up to ~2M parameters.

  36. Weave of Formal Thought

    cs.CL 2026-06 unverdicted novelty 6.0 of 10

    Weave of Formal Thought presents a complete constrained decoder via speculative-lexing GLR and an RWS latent fine-tuning method that reduces per-token cross-entropy by 14.3% on StarCoder2-3B for Python relative to tex...

  37. FoMoE: Breaking the Full-Replica Barrier with a Federation of MoEs

    cs.LG 2026-06 unverdicted novelty 6.0 of 10

    FoMoE partitions expert layers across workers in MoE LLMs, skips non-resident experts, and reports up to 1.42x lower communication than baselines plus 1.4x throughput gains while maintaining stable routing.

  38. Qwen-RobotWorld Technical Report: Unifying Embodied World Modeling through Language-Conditioned Video Generation

    cs.CV 2026-06 unverdicted novelty 6.0 of 10

    Qwen-RobotWorld is a language-conditioned video world model using Double-Stream MMDiT, an 8.6M-frame embodied corpus, and progressive curriculum training that ranks first on EWMBench and DreamGen Bench.

  39. CD-RCM: Generalizable Continuous-Depth Novel View Synthesis for Reflectance Confocal Microscopy

    cs.CV 2026-06 unverdicted novelty 6.0 of 10

    CD-RCM is a feedforward neural model for novel view synthesis that predicts unseen depths in reflectance confocal microscopy stacks to produce isotropic 3D volumes for arbitrary sectioning.

  40. Policy-based Foveated Imaging and Perception

    cs.CV 2026-06 unverdicted novelty 6.0 of 10

    A task-aware policy learned via reinforcement learning allocates high-resolution pixels on dual-stream sensors in real time, outperforming fixed or non-predictive baselines under tight pixel budgets in both simulation...

  41. Schedule-Level Shared-Prefix Reuse for LLM RL Training

    cs.DC 2026-05 unverdicted novelty 6.0 of 10

    Schedule-level shared-prefix reuse decouples prefix and suffix passes in GRPO training to compute shared prefixes once, delivering up to 4.395x speedup and 59.1% HBM reduction while preserving numerical equivalence.

  42. StressDream: Steering Video World Models for Robust Policy Evaluation and Improvement

    cs.CV 2026-05 unverdicted novelty 6.0 of 10

    StressDream optimizes initial noise in diffusion video world models using VLM semantic and plausibility objectives to steer generations toward specified high-impact outcomes for improved policy evaluation.

  43. ChunkFT: Byte-Streamed Optimization for Memory-Efficient Full Fine-Tuning

    cs.LG 2026-05 conditional novelty 6.0 of 10

    ChunkFT enables full-parameter fine-tuning of Llama 3-8B on one 24 GB GPU and Llama 3-70B on two 80 GB GPUs by streaming gradients over dynamically activated sub-tensors.

  44. Towards Understanding Self-Pretraining for Sequence Classification

    cs.LG 2026-05 unverdicted novelty 6.0 of 10

    Self-pretraining improves Transformer sequence classification by enabling learning of proximity-biased attention from positional encodings that label supervision alone cannot easily acquire from random starts.

  45. STELLAR: Scaling 3D Perception Large Models for Autonomous Driving

    cs.CV 2026-05 unverdicted novelty 6.0 of 10

    STELLAR trains up to 500M-parameter multi-modal models on 50M driving scenes and reports empirical scaling trends plus new state-of-the-art results on the Waymo Open Dataset.

  46. Njord: A Probabilistic Graph Neural Network for Ensemble Ocean Forecasting

    cs.LG 2026-05 unverdicted novelty 6.0 of 10

    Njord is a probabilistic GNN model using latent variables and adaptive K-means meshes that produces ensemble forecasts and outperforms deterministic ML baselines on global OceanBench and Baltic Sea domains.

  47. AGoQ: Activation and Gradient Quantization for Memory-Efficient Distributed Training of LLMs

    cs.CL 2026-05 unverdicted novelty 6.0 of 10

    AGoQ delivers up to 52% lower memory use and 1.34x faster training for 8B-32B LLaMA models by using near-4-bit adaptive activations and 8-bit gradients while preserving pretraining convergence and downstream accuracy.

  48. MCMit: Hardware-Software Co-Design for Mid-Circuit Measurement Error Mitigation

    quant-ph 2026-04 unverdicted novelty 6.0 of 10

    MCMit proposes a constant-latency multi-control branch instruction, transformer and CNN discriminators, plus static MCM elimination and stochastic branching, evaluated on Qubic with QPU traces to cut latency by 70% an...

  49. SIEVES: Selective Prediction Generalizes through Visual Evidence Scoring

    cs.CV 2026-04 unverdicted novelty 6.0 of 10

    SIEVES improves selective prediction coverage up to 3x on OOD VQA benchmarks by training a selector on visual localization quality, generalizing across datasets and proprietary reasoners without specific adaptation.

  50. An Efficient Heterogeneous Co-Design for Fine-Tuning on a Single GPU

    cs.DC 2026-03 conditional novelty 6.0 of 10

    SlideFormer uses layer-sliding async offloading, pre-allocated heterogeneous memory, and fused Triton kernels to fine-tune 123B+ models on one RTX 4090 with 1.4–6.3× higher throughput and roughly half the memory of pr...

  51. GeoPT: Scaling Physics Simulation via Lifted Geometric Pre-Training

    cs.LG 2026-02 unverdicted novelty 6.0 of 10

    GeoPT pre-trains on over one million geometry samples augmented with synthetic dynamics to improve neural physics simulators on fluid and solid mechanics benchmarks while reducing labeled data needs by 20-60% and acce...

  52. CausalFlip: A Benchmark for LLM Causal Judgment Beyond Semantic Matching

    cs.AI 2026-02 conditional novelty 6.0 of 10

    A new benchmark and training strategy show LLMs trained to internalize causal reasoning steps are less fooled by semantically similar, label-flipped questions than models using explicit chain-of-thought.

  53. Solving Inverse Problems with Flow-based Models via Model Predictive Control

    eess.IV 2026-01 conditional novelty 6.0 of 10

    MPC-Flow applies model predictive control to guide pretrained flow models through inverse problems, with a single-step variant that avoids backpropagation and scales to 32B-parameter models on consumer hardware.

  54. PhyGDPO: Physics-Aware Groupwise Direct Preference Optimization for Physically Consistent Text-to-Video Generation

    cs.CV 2025-12 conditional novelty 6.0 of 10

    PhyGDPO uses groupwise direct preference optimization with real videos as winners to make text-to-video models generate more physically plausible videos.

  55. Multi-view Pyramid Transformer: Look Coarser to See Broader

    cs.CV 2025-12 conditional novelty 6.0 of 10

    MVP uses a two-level hierarchy of attention windows and token resolutions to reconstruct large 3D scenes from up to 256 input views in a single feed-forward pass, beating Long-LRM and iLRM on DL3DV and several zero-sh...

  56. MTraining: Distributed Dynamic Sparse Attention for Efficient Ultra-Long Context Training

    cs.CL 2025-10 conditional novelty 6.0 of 10

    MTraining scales LLM training to 512K-token contexts on 32 A100 GPUs by integrating dynamic sparse training patterns with balanced and hierarchical sparse ring attention, achieving up to 6x throughput gains without ac...

  57. OctoPipe: Reducing Pipeline Bubbles for Heterogeneous Models via Co-Optimizing Partitioning, Placement, and Scheduling

    cs.DC 2025-09 reject novelty 6.0 of 10

    Co-optimizing model partition, placement, and workload scheduling for pipeline-parallel LLM training is claimed to improve throughput by 1.15 to 1.44x (abstract) or up to 2.14x (body).

  58. CR-Net: Scaling Parameter-Efficient Training with Cross-Layer Low-Rank Structure

    cs.LG 2025-09 unverdicted novelty 6.0 of 10

    CR-Net uses cross-layer low-rank residuals in a dual-path network plus specialized recomputation to outperform prior low-rank methods on 60M-7B model pre-training while using less compute and memory.

  59. MetaEmbed: Scaling Multimodal Retrieval at Test-Time with Flexible Late Interaction

    cs.IR 2025-09 unverdicted novelty 6.0 of 10

    MetaEmbed trains fixed learnable Meta Tokens to produce granularity-organized multi-vector embeddings that support test-time scaling in multimodal retrieval.

  60. SmartSwap: Swap-Based Memory Optimization for LLM Training under Varying Operator Sequences

    cs.DC 2025-09 conditional novelty 6.0 of 10

    Chameleon is a swap-based memory optimizer that handles changing operator sequences in eager-mode LLM training, enabling models up to 4x larger than device memory.

See all 158 Pith citations

Reference graph

Works this paper leans on

19 extracted references · 19 canonical work pages · cited by 158 Pith papers (see all)

  1. [1]

    Mart ´ın Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Good- fellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Man ´e, Rajat Monga, Sherry Moore, Derek Murra...

    work page 2015

  2. [2]

    Seltzer, Malcolm Slaney, Andreas Stolcke, Yongqiang Wang, Huaming Wang, Kaisheng Yao, Dong Yu, Yu Zhang, and Geoffrey Zweig

    Amit Agarwal, Eldar Akchurin, Chris Basoglu, Guoguo Chen, Scott Cyphers, Jasha Droppo, Adam Eversole, Brian Guenter, Mark Hillebrand, Ryan Hoens, Xuedong Huang, Zhiheng Huang, Vladimir Ivanov, Alexey Kamenev, Philipp Kranen, Oleksii Kuchaiev, Wolfgang Manousek, Avner May, Bhaskar Mitra, Olivier Nano, Gaizka Navarro, Alexey Orlov, Marko Padmilac, Hari Part...

    work page 2014

  3. [3]

    Aho, Ravi Sethi, and Jeffrey D

    Alfred V . Aho, Ravi Sethi, and Jeffrey D. Ullman. Compilers: Principles, Techniques, and Tools. Addison-Wesley Longman Publishing Co., Inc., Boston, MA, USA, 1986

    work page 1986

  4. [4]

    Goodfellow, Arnaud Bergeron, Nicolas Bouchard, and Yoshua Bengio

    Fr ´ed´eric Bastien, Pascal Lamblin, Razvan Pascanu, James Bergstra, Ian J. Goodfellow, Arnaud Bergeron, Nicolas Bouchard, and Yoshua Bengio. Theano: new features and speed improve- ments. Deep Learning and Unsupervised Feature Learning NIPS 2012 Workshop, 2012

    work page 2012

  5. [5]

    Theano: a CPU and GPU math expression compiler

    James Bergstra, Olivier Breuleux, Fr ´ed´eric Bastien, Pascal Lamblin, Razvan Pascanu, Guil- laume Desjardins, Joseph Turian, David Warde-Farley, and Yoshua Bengio. Theano: a CPU and GPU math expression compiler. In Proceedings of the Python for Scientific Computing Conference (SciPy), June 2010. Oral Presentation

    work page 2010

  6. [6]

    MXNet: A flexible and efficient machine learning library for heterogeneous distributed systems

    Tianqi Chen, Mu Li, Yutian Li, Min Lin, Naiyan Wang, Minjie Wang, Tianjun Xiao, Bing Xu, Chiyuan Zhang, , and Zheng Zhang. MXNet: A flexible and efficient machine learning library for heterogeneous distributed systems. In Neural Information Processing Systems, Workshop on Machine Learning Systems (LearningSys’15), 2015

    work page 2015

  7. [7]

    Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Quoc V

    Jeffrey Dean, Greg S. Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Quoc V . Le, Mark Z. Mao, MarcAurelio Ranzato, Andrew Senior, Paul Tucker, Ke Yang, and Andrew Y . Ng. Large scale distributed deep networks. In NIPS, 2012

    work page 2012

  8. [8]

    Deep learning

    Ian Goodfellow, Yoshua Bengio, , and Aaron Courville. Deep learning. Book in preparation for MIT Press, 2016

    work page 2016

Show all 19 references
  1. [9]

    Algorithm 799: Revolve: An implementation of checkpointing for the reverse or adjoint mode of computational differentiation

    Andreas Griewank and Andrea Walther. Algorithm 799: Revolve: An implementation of checkpointing for the reverse or adjoint mode of computational differentiation. ACM Trans. Math. Softw., 26(1):19–45, March 2000

    2000

  2. [10]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. arXiv preprint arXiv:1512.03385, 2015

    2015 arXiv

  3. [11]

    Identity mappings in deep residual networks

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. arXiv preprint arXiv:1603.05027, 2016

    2016

  4. [12]

    Long short-term memory

    Sepp Hochreiter and J ¨urgen Schmidhuber. Long short-term memory. Neural Comput. , 9(8):1735–1780, November 1997. 11

    1997

  5. [13]

    Batch normalization: Accelerating deep network training by reducing internal covariate shift

    Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of the 32th International Conference on Machine Learning (ICML’15), 2015

    2015

  6. [14]

    Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems 25 , pages 1097–1105. 2012

    2012

  7. [15]

    Gradient-based learning applied to document recognition

    Yann LeCun, L ´eon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. In S. Haykin and B. Kosko, editors, Intelligent Signal Pro- cessing, pages 306–351. IEEE Press, 2001

    2001

  8. [16]

    Virtualizing deep neural networks for memory-efficient neural network design.arXiv preprint arXiv:1602.08124, 2016

    Minsoo Rhu, Natalia Gimelshein, Jason Clemons, Arslan Zulfiqar, and Stephen W Keckler. Virtualizing deep neural networks for memory-efficient neural network design.arXiv preprint arXiv:1602.08124, 2016

    2016

  9. [17]

    Senior, and Franc ¸oise Beaufays

    Hasim Sak, Andrew W. Senior, and Franc ¸oise Beaufays. Long short-term memory recur- rent neural network architectures for large scale acoustic modeling. In INTERSPEECH 2014, 15th Annual Conference of the International Speech Communication Association, Singapore, September 14-...

    2014

  10. [18]

    Training very deep networks

    Rupesh Kumar Srivastava, Klaus Greff, and J¨urgen Schmidhuber. Training very deep networks. arXiv preprint arXiv:1507.06228, 2015

    2015

  11. [19]

    Highway long short-term memory rnns for distant speech recognition

    Yu Zhang, Guoguo Chen, Dong Yu, Kaisheng Yao, Sanjeev Khudanpur, and James Glass. Highway long short-term memory rnns for distant speech recognition. arXiv preprint arXiv:1510.08983, 2015. A Search over Budget B Alg. 3 allows us to generate an optimized memory plan given a sin...

    2015

Pith tools

Reviewed May 12, 2026 · model on record in the stance chip above.