BladeDISC++: Memory Optimizations Based On Symbolic Shape

Ang Wang; Jie Zhang; Wei Lin; Wenting Shen; Xiafei Qiu; Xiulong Yuan; Xu Yan; Yong Li

REVIEW 4 major objections 4 minor 17 references

BladeDISC++: Memory Optimizations Based On Symbolic Shape

T0 review · 4 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A dynamic-shape compiler can optimize memory by comparing symbolic shape expressions instead of exact tensor sizes, keeping peak memory within a few percent of static-shape training.

desk verdict Plausible symbolic-shape memory optimization idea, but the evaluation is too thin to show the symbolic mechanism is what carries the gains. read the letter →

arxiv 2412.16985 v1 pith:3Q3DMGWS submitted 2024-12-22 cs.DC

Xiulong Yuan , Xu Yan , Wenting Shen , Xiafei Qiu , Ang Wang , Jie Zhang , Yong Li , Wei Lin This is my paper · ORCID

classification cs.DC

keywords dynamicshapecompilationsymbolicshapesmemoryoptimizationoperatorschedulingrematerializationrecomputationGPUdeeplearningcompiler

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

BladeDISC++ sets out to prove that memory optimization does not require exact tensor shapes: a compiler can compare the memory impact of scheduling choices and recomputation subgraphs using symbolic expressions over shape variables. The paper builds a global symbolic shape graph from op semantics, derives algebraic relations among dimensions, and uses those relations to simplify and order memory-impact expressions. For rematerialization, it moves the eviction decision to runtime because peak memory varies across runs, while the search for candidates and regeneration subgraphs happens at compile time. On fine-tuning a 1B-parameter transformer with variable-length batches, the resulting peak GPU memory is close to static-shape training while avoiding recompilation and padding overhead. If correct, this makes dynamic-shape compilation more practical for large-model training.

What carries the argument

The symbolic shape graph is the mechanism: a global structure whose nodes are symbolic dimensions ($S_0, S_1, \dots$) of tensor shapes and whose edges record algebraic equations inferred from operators, e.g. reshape implies equality of element counts. On top of it, memory impacts are expressed as SymbolicExprs, such as $11008 \cdot S_1$ or $1024 \cdot S_0$, and comparisons are made by substituting equalities (e.g. replacing $S_0$ with $12 S_1$) and simplifying. This same comparison machinery drives both the op scheduler's choice among ready operations and the rematerialization search's decision whether a recomputation subgraph reduces or raises peak memory; the runtime monitor then bridges the residual gap between symbolic knowledge and the actual shapes seen in a given run.

What would settle it

Construct a dynamic graph with two legal op orders whose relative memory depends on a non-affine shape relation, such as $S_0 = \lfloor S_1 / 2 \rfloor$, and let BladeDISC++ choose; if its symbolic comparison cannot derive the relation and it selects the order with higher peak memory than the exact-shape optimum, the claim that symbolic shapes suffice for memory optimization is falsified.

Watch

Extended reading notes

Core claim

The paper's central claim is that a fixed-topology graph with unknown tensor shapes still carries enough information to make memory-conscious decisions, once those shapes are represented symbolically and tied together by semantic equations. Each unknown dimension becomes a symbolic variable, and op semantics produce equations such as $S_0 = 12 S_1$ from a dynamic reshape; tensor sizes then become symbolic expressions, and two expressions can be compared after simplification through the graph. The op scheduler uses these comparisons to choose, at each step, the ready operation with the smallest memory footprint, and falls back to a graph-topology heuristic only when expressions are not reducible to a comparable form. For rematerialization, the compiler enumerates eviction candidates and generates recomputation/reload subgraphs whose memory impacts are assessed symbolically, then inserts runtime decision points where actual memory pressure determines which tensor to evict and how to regenerate it. The reported peak-memory numbers—35.76, 37.89, and 39.18 GiB for batch sizes 14, 16, and 18 versus 35.75, 37.71, and 38.92 GiB in static-shape training—are offered as evidence that symbolic-shape optimization is on par with exact-shape optimization.

Load-bearing premise

The method assumes the symbolic shape graph captures every relation needed to compare the memory impact of any two op sequences or recomputation subgraphs that matter; if a relation is missing or non-affine, the comparison falls back to a heuristic without a guarantee.

Editorial extensions

If this is right

Dynamic-shape training can avoid shape specialization and input bucketing while still getting static-shape-level peak memory, cutting recompilation overhead.
Rematerialization becomes a two-phase decision: exhaustive candidate and regeneration-subgraph search at compile time, with a cheap eviction choice at runtime using monitored memory pressure.
Memory-impact comparison works across expressions with disjoint symbol sets whenever the symbolic graph supplies enough equations to relate them.
The approach generalizes to any dynamic workload whose graph topology is fixed but whose tensor sizes vary, such as variable-batch or variable-length inference.
When expressions cannot be compared, the method falls back to a topology-based heuristic, preserving a practical optimization even without proof of optimality.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Beyond the paper, a natural extension is applying the same symbolic comparison to op fusion choices and to host-device offloading, where memory impact is likewise a symbolic function of shape.
The runtime eviction decision opens the door to online re-scheduling as actual shapes arrive: if the compile-time symbolic ranking is inconclusive, runtime could re-rank by exact sizes without recompiling.
The completeness assumption will matter most on workloads with non-affine shape relations, such as sizes derived from data-dependent sampling or geometric transforms; our reading is that those cases would degrade to the heuristic fallback, and it is untested how often that occurs in practice.
Because the evaluation covers one 1B-parameter model and one dataset, the claim of parity with static-shape memory is a proof of concept; the symbolic-comparison mechanism itself seems model-agnostic, though runtime decision overhead and fallback frequency should be measured on more workloads.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

Quick take: this paper has a real idea, and the main weakness is not the idea but the evidence attached to it.

What is new: using a symbolic shape graph to compare memory impacts of ops and recomputation subgraphs at compile time, then deferring the actual eviction decision to runtime through pre-enumerated branches. That is a genuine extension over static-shape schedulers and rematerializers, and over runtime-only systems like DTR and Delta. The shape relation derivation from op semantics (e.g., @S0 = 12 * @S1) is coherent, and I don't see circularity: the reported memory numbers are benchmarked against static shape training, not fitted. The self-citations to BladeDISC/DISC are appropriate substrate.

Where it goes soft: the empirical section is one model, one dataset, one GPU, one table, with no variance, no repeats, and no ablation. The stress-test note lands: the paper itself admits symbolic comparisons are best-effort and falls back to a graph-topology heuristic when expressions cannot be compared, but it never reports how often that fallback triggers. Without that, the central claim that symbolic-shape reasoning produces memory comparable to static-shape training is not backed up; the gains could come from the topology heuristic or from runtime exact-shape bookkeeping. The rematerialization example also has a confusing sign convention (-11007 * @S1 vs +1 * @S1) and the signs are never defined. That is a presentation issue, not a load-bearing flaw. The absence of an artifact is a real cost in a systems paper.

Who this is for: compiler folks working on dynamic-shape memory optimization. The idea deserves a serious referee, but the current evaluation would not convince me to adopt the technique without more evidence. My recommendation: send it to review with a request for a stronger evaluation; specifically, an ablation isolating the symbolic comparison from the fallback, statistics on comparison success rates, more than one workload, and a code release.

Referee Report

4 major / 4 minor

Summary. The paper proposes BladeDISC++, a compiler extension for dynamic shape graphs that performs memory optimization without exact tensor shapes. It constructs a symbolic shape graph expressing algebraic relations among shape dimensions, then uses symbolic expressions of memory impact to guide operation scheduling and to enumerate/evaluate rematerialization candidates at compile time, with final eviction decisions deferred to runtime. The evaluation reports peak memory and throughput for fine-tuning a Llama-2-1b model on CodeAlpaca-20K on one GPU, arguing that BladeDISC++ achieves memory consumption comparable to static-shape training while supporting dynamic shapes.

Significance. If the claims hold, the paper addresses a real and increasingly important gap: memory optimization for dynamic shape workloads without padding or recompilation. The core idea of deriving and comparing memory-impact expressions from a symbolic shape graph is concrete, falsifiable, and not circular: the memory expressions are derived from op semantics and the symbolic shape graph, not fitted to the reported measurements. The compile-time/runtime split for rematerialization is also a sensible design for shape dynamics. However, the current evidence is narrow—one model, one dataset, one GPU, one table with no variance or ablation—and the presentation of the comparison mechanism contains ambiguities that must be resolved before the central claim is fully supported.

major comments (4)

[Section 2.3] The sign convention for the SymbolicExpr "memory impact" is never defined, and the two uses of the term appear inconsistent. In Section 2.2, the DotOp's impact is reported as 10996 * @S1 because scheduling it allocates %3 (11008 * @S1) and deallocates %2 (12 * @S1), which reads as allocated-minus-freed. In Section 2.3, the recomputation subgraph impacts are listed as -11007 * @S1, -11 * @S1, and +1 * @S1, and the +1 case is called "memory-efficient," which reads as freed-minus-allocated (or positive savings). Since the entire scheduling and rematerialization selection is a comparison of these values, the paper must state the sign convention explicitly and use it consistently; otherwise the example cannot be checked and the load-bearing comparison mechanism is under-specified.
[Section 2.2, Listing 1] The illustrative scheduling example is internally inconsistent. The text says "The DynamicReshapeOp's memory impact, on the other hand, is 4096 * @S0 because scheduling it only involves allocation for %1," but DynamicReshapeOp produces %2, whose element count is 12 * @S1, not 4096 * @S0; %1 is produced by the preceding broadcast operation. This makes the sole worked example of symbolic comparison ambiguous and should be corrected to identify the op whose output is %1 and whose allocation is 4096 * @S0.
[Section 3, Table 1] The evaluation does not isolate the contribution of the symbolic comparison mechanism. The reported peak memory and throughput are end-to-end numbers, and the paper neither reports how often the symbolic comparison path succeeds versus the graph-topology fallback described in Section 2.2, nor includes an ablation with the symbolic comparison disabled. Without such an ablation or success-rate statistics, the claim that symbolic-shape reasoning, rather than the topology heuristic or runtime exact-shape bookkeeping, produces the memory comparable to static-shape training is not supported. I would also ask for repeated runs or variance information, since differences as small as 0.01 GiB (batch size 14) are reported without any measure of run-to-run variability.
[Sections 2.3 and 3] The runtime rematerialization decision is described only as following "a similar approach as outlined in [10]" (Delta). Because the proposed method's novelty depends on the combination of compile-time symbolic subgraph search and runtime decisions, the paper should specify what symbolic information is used in the runtime branch and how the runtime decision incorporates the SymbolicExpr-based memory impacts, rather than deferring entirely to a prior system.

minor comments (4)

[Section 2.1] The text switches between "exp1" and "expr1" for the same symbolic expression; please use one spelling consistently.
[Section 2.2] The sentence "the last consumer of %2" is followed immediately by "// The last consumer of %3" in the listing without a clear narrative link; consider labeling each op in the example with its name and output tensor to make the scheduling discussion easier to follow.
[Section 3] Table 1 would be clearer if it reported the GPU memory limit explicitly and if the OOM entries were marked with the batch size at which the limit was exceeded, rather than only showing "OOM" with no throughput.
[References] Several references are cited by URL with no version or commit information (e.g., [4], [11], [13], [14]), which makes reproducibility harder; please add versioned citations or DOIs where available.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the memory-impact comparisons are derived from op semantics and symbolic algebra, and the evaluation is an external benchmark against static-shape training.

full rationale

The paper's derivation chain is self-contained. Section 2.1 constructs the symbolic shape graph by deriving relations from op semantics (for example, @S0 = 12 * @S1 from DynamicReshapeOp), and Sections 2.2 and 2.3 use only those relations plus arithmetic on SymbolicExprs to compare memory impacts, such as simplifying 4096*@S0 to 49152*@S1 to rank the DynamicReshapeOp against the DotOp. No parameter is fitted to the reported throughput or peak-memory numbers, and the comparison to static-shape training in Table 1 is an external benchmark rather than a consequence of the symbolic-shape definition. The self-citations to BladeDISC and DISC scope the op-fusion substrate and do not carry the load of the scheduling or rematerialization claims. The evaluation's failure to quantify how often the symbolic-comparison path succeeds versus the graph-topology fallback, and the undefined sign convention for recomputation-subgraph memory impact, are evidence-quality limitations rather than circular reductions. An apparent arithmetic slip in the expr1 simplification (11008*@S1 said to become 132096*@S0) is a correctness issue, not a definitional loop.

Assumptions & free parameters 0 free parameters · 4 assumptions · 3 invented entities

The paper introduces compiler-internal abstractions but no fitted scientific parameters. The central results rest on the completeness of symbolic shape relations and on the runtime eviction policy borrowed from prior work, both listed above.

assumptions (4)

domain assumption The symbolic shape graph captures enough algebraic relationships between dynamic dimensions to compare memory impacts of candidate ops and recomputation subgraphs.
Invoked in sections 2.1 and 2.2; if relations such as @S0 = 12*@S1 are missing or inexpressible, the symbolic comparison falls back to heuristics and may misorder decisions.
domain assumption Memory impact of an op can be modeled as the difference between bytes freed and allocated, expressed as a SymbolicExpr.
Used throughout sections 2.2 and 2.3; ignores allocator fragmentation, alignment, and kernel workspace overhead.
ad hoc to paper When expressions cannot be compared, graph-topology-based lifetime selection is a safe fallback.
Stated in section 2.2: when it is unfeasible to compare two memory impact SymbolicExprs, the system resorts to selecting the op that results in smaller overall tensor lifetimes based on graph topology. This is a heuristic, not derived.
domain assumption Runtime eviction decisions can follow the policy of Delta with no loss of optimality.
Section 2.3 states that final runtime decisions are made following a similar approach as outlined in reference [10]; the details are not reproduced in this paper.

invented entities (3)

SymbolicDim op
purpose: Binds a symbolic value to a tensor dimension so shape relationships can be represented in the graph.
Introduced in section 2.1 as compiler IR; no external falsifiable handle beyond its role inside BladeDISC++.
Remat::EvictOp
purpose: Runtime checkpoint inserted after each op to trigger eviction decisions when memory pressure appears.
Introduced in section 2.3 as a compiler-internal instruction; its behavior is defined only by the BladeDISC++ implementation.
Remat::RegenerateOp
purpose: Queries eviction decisions and triggers reload or recomputation branches.
Introduced in section 2.3 as a compiler-internal instruction; no independent empirical evidence is provided outside the BladeDISC++ system.

how reviews work

0 comments

Cite this review

Pith. "Pith review of BladeDISC++: Memory Optimizations Based On Symbolic Shape." pith.science (2026). https://pith.science/paper/3Q3DMGWS

@misc{pith2026241216985,
  author       = {Pith},
  title        = {Pith review of: BladeDISC++: Memory Optimizations Based On Symbolic Shape},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3Q3DMGWS}},
  note         = {Machine review of arXiv:2412.16985}
}

read the original abstract

Recent deep learning workloads exhibit dynamic characteristics, leading to the rising adoption of dynamic shape compilers. These compilers can generate efficient kernels for dynamic shape graphs characterized by a fixed graph topology and uncertain tensor shapes. However, memory optimization, although particularly crucial in this large model era, remains relatively underexplored for dynamic shape graphs. The fundamental challenge lies in the lack of precise tensor shapes which are essential in conventional methods such as operation scheduling(op scheduling) and rematerialization. To address this challenge, we propose op scheduling and rematerialization approaches based on symbolic shapes and developed BladeDISC++. Besides, since rematerialization decisions cannot be made solely at compile time when tensor shapes are unknown, BladeDISC++ employs a compilation-runtime combined strategy to optimally address shape dynamics. Evaluations indicate that BladeDISC++ effectively reduces memory usage for dynamic shape graphs, achieving memory consumption comparable to optimizations using precise shapes, thereby promoting the broader adoption of dynamic shape compilers.

Figures

Figures reproduced from arXiv: 2412.16985 by the authors.

**Figure 2.** OpScheduler algorithm main loop Op scheduling tries to find a memory-efficient op sequence from the original computation graph. Existing scheduling algorithms[8] often traverse the computation graph and select an op from a ReadySet (including ops whose predecessors have already been scheduled) at each step. The selection is mainly based on comparing different ops’ memory impact, which is determined by the differen… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

17 extracted references · 13 canonical work pages

[10]

Delta: Dynamically optimizing gpu memory beyond tensor recomputation, 2022

Yu Tang, Chenyu Wang, Yufan Zhang, Yuliang Liu, Xingcheng Zhang, Linbo Qiao, Zhiquan Lai, and Dongsheng Li. Delta: Dynamically optimizing gpu memory beyond tensor recomputation, 2022

work page 2022
[1]

Accessed: December 24, 2024

Codealpaca-20k dataset, 2024. Accessed: December 24, 2024

work page 2024
[2]

Magis: Memory optimization via coordinated graph transformation and scheduling for dnn

Renze Chen, Zijian Ding, Size Zheng, Chengrui Zhang, Jingwen Leng, Xuanzhe Liu, and Yun Liang. Magis: Memory optimization via coordinated graph transformation and scheduling for dnn. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, V olume 3, ASPLOS ’24, page 607–621, New Yor...

work page 2024
[3]

Training deep nets with sublinear memory cost, 2016

Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost, 2016

2016
[4]

Alibaba cloud ecs.gn7-c12g1.3xlarge instance, 2024

Alibaba Cloud. Alibaba cloud ecs.gn7-c12g1.3xlarge instance, 2024. Accessed: December 24, 2024

work page 2024
[5]

Swapadvisor: Pushing deep learning beyond the gpu memory limit via smart swapping

Chien-Chin Huang, Gu Jin, and Jinyang Li. Swapadvisor: Pushing deep learning beyond the gpu memory limit via smart swapping. In Proceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating Systems , ASPLOS ’20, page 1341–1355, New York, NY , USA, 2020. Association for Computing Machinery

work page 2020
[6]

Gonzalez

Paras Jain, Ajay Jain, Aniruddha Nrusimha, Amir Gholami, Pieter Abbeel, Kurt Keutzer, Ion Stoica, and Joseph E. Gonzalez. Checkmate: Breaking the memory wall with optimal tensor rematerialization, 2020

2020
[7]

Dynamic tensor rematerialization

Marisa Kirisame, Steven Lyubomirsky, Altan Haan, Jennifer Brennan, Mike He, Jared Roesch, Tianqi Chen, and Zachary Tatlock. Dynamic tensor rematerialization. CoRR, abs/2006.09616, 2020

arXiv 2006

Show all 17 references

[8]

Xla : Compiling machine learning for peak performance, 2020

Amit Sabne. Xla : Compiling machine learning for peak performance, 2020

2020
[9]

Olla: Optimizing the lifetime and location of arrays to reduce the memory usage of neural networks, 2022

Benoit Steiner, Mostafa Elhoushi, Jacob Kahn, and James Hegarty. Olla: Optimizing the lifetime and location of arrays to reduce the memory usage of neural networks, 2022

2022
[11]

Bladedisc github repository, 2021

BladeDISC Team. Bladedisc github repository, 2021. Accessed: December 24, 2024

2021
[12]

meta-llama/llama-2-7b, 2024

Meta Llama Team. meta-llama/llama-2-7b, 2024. Accessed: December 24, 2024

2024
[13]

Dynamic shape in modular, 2024

Modular Team. Dynamic shape in modular, 2024. Accessed: December 24, 2024

2024
[14]

Dynamic shape in pytorch, 2023

Pytorch Team. Dynamic shape in pytorch, 2023. Accessed: December 24, 2024

2023
[15]

Hierarchical memory- constrained operator scheduling of neural architecture search networks

Zihan Wang, Chengcheng Wan, Yuting Chen, Ziyi Lin, He Jiang, and Lei Qiao. Hierarchical memory- constrained operator scheduling of neural architecture search networks. In Proceedings of the 59th ACM/IEEE Design Automation Conference, DAC ’22, page 493–498, New York, NY , USA, ...

2022
[16]

Bladedisc: Optimizing dynamic shape machine learning workloads via compiler approach

Zhen Zheng, Zaifeng Pan, Dalin Wang, Kai Zhu, Wenyi Zhao, Tianyou Guo, Xiafei Qiu, Minmin Sun, Junjie Bai, Feng Zhang, et al. Bladedisc: Optimizing dynamic shape machine learning workloads via compiler approach. Proceedings of the ACM on Management of Data , 1(3):1–29, 2023

2023
[17]

Disc: A dynamic shape compiler for machine learning workloads

Kai Zhu, WY Zhao, Zhen Zheng, TY Guo, PZ Zhao, JJ Bai, Jun Yang, XY Liu, LS Diao, and Wei Lin. Disc: A dynamic shape compiler for machine learning workloads. In Proceedings of the 1st Workshop on Machine Learning and Systems , pages 89–95, 2021. 5

2021

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [10]

Delta: Dynamically optimizing gpu memory beyond tensor recomputation, 2022

Yu Tang, Chenyu Wang, Yufan Zhang, Yuliang Liu, Xingcheng Zhang, Linbo Qiao, Zhiquan Lai, and Dongsheng Li. Delta: Dynamically optimizing gpu memory beyond tensor recomputation, 2022

work page 2022

[2] [1]

Accessed: December 24, 2024

Codealpaca-20k dataset, 2024. Accessed: December 24, 2024

work page 2024

[3] [2]

Magis: Memory optimization via coordinated graph transformation and scheduling for dnn

Renze Chen, Zijian Ding, Size Zheng, Chengrui Zhang, Jingwen Leng, Xuanzhe Liu, and Yun Liang. Magis: Memory optimization via coordinated graph transformation and scheduling for dnn. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, V olume 3, ASPLOS ’24, page 607–621, New Yor...

work page 2024

[4] [3]

Training deep nets with sublinear memory cost, 2016

Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost, 2016

2016

[5] [4]

Alibaba cloud ecs.gn7-c12g1.3xlarge instance, 2024

Alibaba Cloud. Alibaba cloud ecs.gn7-c12g1.3xlarge instance, 2024. Accessed: December 24, 2024

work page 2024

[6] [5]

Swapadvisor: Pushing deep learning beyond the gpu memory limit via smart swapping

Chien-Chin Huang, Gu Jin, and Jinyang Li. Swapadvisor: Pushing deep learning beyond the gpu memory limit via smart swapping. In Proceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating Systems , ASPLOS ’20, page 1341–1355, New York, NY , USA, 2020. Association for Computing Machinery

work page 2020

[7] [6]

Gonzalez

Paras Jain, Ajay Jain, Aniruddha Nrusimha, Amir Gholami, Pieter Abbeel, Kurt Keutzer, Ion Stoica, and Joseph E. Gonzalez. Checkmate: Breaking the memory wall with optimal tensor rematerialization, 2020

2020

[8] [7]

Dynamic tensor rematerialization

Marisa Kirisame, Steven Lyubomirsky, Altan Haan, Jennifer Brennan, Mike He, Jared Roesch, Tianqi Chen, and Zachary Tatlock. Dynamic tensor rematerialization. CoRR, abs/2006.09616, 2020

arXiv 2006