REVIEW 3 major objections 6 minor 20 references
Adaptive Matrix Multiplication for Dynamic Shapes on Ascend NPUs
T0 review · 3 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read The paper claims that decoupling MatMul into a hardware-aware 2D tiling taxonomy and a closed-form pipeline model allows O(1) dispatch with 1.85x mean speedup over the vendor library.
desk verdict Good engineering, credible speedup on the tested shapes, but the paper hasn't shown the model generalizes beyond the shapes it was calibrated on. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the 2D tiling template space plus the closed-form pipeline latency model. The X-axis is spatial parallelism: whether the spatial task count P_m · P_n reaches the number of AI cores. The Y-axis is the L0C accumulation regime: whether the m1 × n1 FP32 output tile fits in the L0C buffer. Four quadrants give four templates that trade GM read volume (Eq. 1) against write volume (Eqs. 2–5). The latency model (Eqs. 7–10) decomposes each kernel into Prologue, Main Loop, and Epilogue, models the MTE2→MTE1→Cube→FixPipe pipeline with per-stage costs t_{r,v}=n_{r,v}λ_r + D_{r,v}/(BW_r η_{r,v}), and computes an initiation interval that captures buffer-reuse hazards; optimizations are evaluated by how they change data volume, instruction count, or dependency exposure.
What would settle it
Take the published 80,000-shape dataset, hold out every shape whose dimension falls in a randomly selected 10% of the M,N,K ranges, recalibrate λ_r and η_{r,v} on the remaining 90%, run the offline selection on the held-out shapes, and measure whether the mean speedup over ACLNN remains near 1.85x; if it drops substantially, the claim of universal generalization is falsified.
Extended reading notes
Core claim
The central claim is that the optimal MatMul implementation for a dynamic shape can be determined offline, without runtime search, by a deterministic two-stage model. Given a shape (M,N,K), the 2D taxonomy selects one of four tiling templates—Common, SingleCoreSplitK, MultiCoreSplitK, HybridSplitK—based on whether the AICore task count saturates the cores and whether the FP32 accumulation tile fits in L0C. Then a composable optimization library (padding, L1 resident, ShuffleK, multi-buffering, preloading, scalar elimination) is evaluated by an analytical latency model in which each hardware stage has a per-instruction overhead and a bandwidth-efficiency factor; the configuration with minimal predicted latency is compiled and cached under a 64-bit TilingKey. The paper reports that on 80,000 industrial shapes this model-guided selection matches the exhaustive oracle on 63.86% of shapes and still beats ACLNN on 91.3% of shapes, yielding a 1.85x mean speedup and 5.47x over the CATLASS baseline.
Load-bearing premise
The analytical performance model assumes that the per-instruction overheads (λ_r) and hardware efficiencies (η_{r,v}) measured on a calibration set, together with the deterministic pipeline formulation, rank the candidate kernels correctly for every dynamic shape encountered at runtime, including shapes outside the calibration distribution and combinations of optimizations that interact with one another.
Editorial extensions
If this is right
- If AdaptCore is correct, a single framework can replace hand-tuned static MatMul kernels for dynamic-shape workloads on Ascend NPUs, sustaining the same kernel-selection quality without runtime search overhead.
- The 1.85x mean speedup over ACLNN translates into 1.09x–1.48x end-to-end speedups on five recommendation models (MMOE, DLRM, DCN V2, ESMM, RankMixer), since GEMM is the dominant bottleneck.
- Because selection is done offline and dispatch is O(1) via a 64-bit TilingKey, AdaptCore is deployable in latency-critical inference servers with no auto-tuning stalls.
- The MultiCoreSplitK template scales near-ideally along K: a tall-skinny shape (M=3, N=256, K=87087) runs 21.9x faster at 24 cores than the ACLNN baseline, which uses only one core.
- The analytical model lets developers add new optimizations to the library and immediately rank them against existing ones, without re-running search on every shape.
Reading between the lines
- The 2D taxonomy could generalize to other explicitly controlled SIMD accelerators, provided the buffer capacities and instruction-set stages are re-parameterized; the template names would change but the two-axis decision (core saturation vs accumulation residency) is hardware-agnostic.
- The model's assumption that optimizations interact only through the variables in Equations 8–10 may miss cross-effects such as ShuffleK's TLB pressure interacting with multi-buffering; a testable extension is to add pairwise interaction terms and measure whether they close the gap between the 63.86% oracle-match rate and 100%.
- The offline calibration of λ_r and η_{r,v} is the main portability cost; an online calibration pass that refreshes these constants during idle time could make the framework self-tuning on new hardware revisions.
- For MoE workloads, where gating changes shapes at every token batch, the O(1) dispatch makes the framework a drop-in replacement that eliminates the per-batch shape-adaptation penalty, not just the per-shape search penalty.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. AdaptCore targets the dynamic-shape MatMul problem on Huawei Ascend NPUs. It decomposes operator optimization into a 2D tiling taxonomy (four templates: Common, SingleCoreSplitK, MultiCoreSplitK, HybridSplitK) and an analytical pipeline model (Prologue/Main Loop/Epilogue; MTE2–MTE1–Cube–FixPipe stages), combined with a composable optimization library that includes padding, L1 residency, ShuffleK, multi-buffering, preloading, and scalar elimination. An offline phase explores legal configurations, evaluates them with the analytical model, and caches the selected kernel in a launch_map keyed by a 64-bit TilingKey, enabling O(1) runtime dispatch. Evaluation on about 80,000 shapes reports a 1.85x mean speedup over ACLNN and 5.47x over basic CATLASS, with 91.3% of shapes faster than ACLNN, a 63.86% oracle-configuration match rate, a 21.9x multi-core scaling result on one skewed shape, and 1.09x–1.48x end-to-end speedups on five recommendation models.
Significance. The paper addresses a real and under-served platform: Ascend NPUs have an explicitly managed SIMD architecture where GPU-oriented auto-tuning frameworks and static template libraries both struggle with dynamic shapes. The proposed tiling taxonomy and the closed-form pipeline equations are concrete and mechanically applicable, and the evaluation is large-scale: 80,000 shapes, an ablation with monotonic gains, a scaling study, end-to-end workloads, and an oracle-match analysis. If the analytical model generalizes beyond its calibration set, AdaptCore is a practical contribution with a clean offline-selection/online-dispatch workflow. The significance is conditional, however, on resolving the evaluation-overlap and cache-miss concerns below; the current manuscript does not yet substantiate the words 'universal' and 'O(1)' for arbitrary runtime shapes.
major comments (3)
- [§3.2, §5.1, §6.6] The paper does not disclose the relationship between the shapes used to calibrate the model constants λ_r and η_{r,v} in Eq. (8) and the 80,000-shape evaluation set. Section 5.1 states that exploration is 'performed offline for the target shape set,' and Section 6.6 measures oracle-match accuracy 'on the 80,000-shape benchmark.' If calibration and evaluation overlap, the reported 63.86% oracle-match rate and the 1.85x mean speedup quantify in-distribution fit rather than generalization to unseen dynamic shapes. The authors should describe how λ_r and η_{r,v} are measured, report the calibration/evaluation split, and add a held-out shape test to support the claimed universality.
- [§3.2, §5.1] The O(1) runtime dispatch claim is incomplete because the paper never specifies what happens when a runtime shape is missing from the precompiled launch_map. Since the motivation includes MoE-style shapes that 'fluctuate unpredictably on the fly' (Section 2.2), the runtime input space can exceed any finite offline target set. The authors should either restrict the universality claim to the target shape set, or describe and measure a fallback path (e.g., nearest-key reuse, online JIT compilation, or a generic template) and include its overhead in the reported dispatch cost.
- [§6.1, §6.2, §6.6] The evaluation under-reports measurement and workload-distribution details. No error bars or repeated-run counts are given for any latency figure, and the 80,000 shapes are described only by a 3D scatter plot and a 'derived from real-world industrial input ranges and distributions' remark. Because a uniform mean over 80,000 shapes can be dominated by a small number of large or highly skewed shapes, the authors should disclose the shape sampling distribution, report per-template speedup statistics (e.g., median and geometric mean), and provide confidence intervals or at least run-to-run variance for the headline speedups.
minor comments (6)
- [§4.2, Eqs. (1)–(5)] The units in the data-volume formulas are inconsistent: Eq. (1) includes a factor of 2 that is explained if D_r is expressed in bytes for FP16 operands, but the write-volume formulas use 2·MN where an FP32 output tile should contribute 4·MN bytes. Please state the units explicitly or switch consistently to element counts.
- [§2.3] The last sentence contains a typo: 'an challenge' should be 'a challenge.'
- [§6.1] The exclusion of Triton-Ascend is justified only qualitatively; please report its measured latency on a few representative shapes so readers can assess the baseline choice.
- [Figure 8] The P10–P90 shading is computed by arithmetic-intensity binning, but the bin counts and the number of runs per shape are not given; please add this information so the percentiles are interpretable.
- [§5.3] The text says Equations 12–14 are substituted into the latency model, but the connection between these specialized formulas and the general Δ_{dep,v} in Eq. (9) is not fully spelled out; a short derivation of Δ_{dep,v} for the baseline, double-buffered, and preloaded schedules would help.
- [Table 2] ShuffleK is listed with a 'benefit is shape-dependent' note, but the analytical model never shows how ShuffleK enters Eq. (8) or Eq. (9); please indicate which model variable it changes.
Circularity Check
No significant circularity: the headline speedup is an independent empirical comparison, and the calibrated cost model is used only for offline candidate ranking.
full rationale
The central claim, a 1.85x mean speedup over ACLNN across 80,000 shapes, is an empirical latency comparison of generated kernels against a vendor baseline. It does not reduce to the model's inputs: the model selects configurations offline, and the measured kernels are then executed and timed against ACLNN and CATLASS. The analytical model in Eq. 8 does contain empirically calibrated constants (lambda_r and eta_r,v obtained through offline latency profiling), so its latency estimates are not parameter-free derivations. However, this is standard cost-model calibration rather than circularity: the model does not incorporate the benchmark oracle or the ACLNN results as inputs, and the oracle-match figure (63.86%) is presented as a validation of ranking quality, not as the source of the speedup. The tiling taxonomy in Section 4.2 is definitional in that templates are characterized by capacity and parallelism inequalities, but this is an organizational scheme for known tiling strategies, and the paper does not claim to derive a new mathematical result from those definitions. No load-bearing self-citation chain or imported uniqueness theorem appears; the Ascend/CATLASS/ACLNN citations describe hardware and vendor libraries, not a uniqueness argument. The undisclosed calibration/evaluation overlap for lambda_r and eta_r,v is a reporting limitation and a correctness risk, but without evidence that calibration was performed on the same 80,000 shapes, it cannot be shown that the reported speedup is forced by construction. The O(1) dispatch claim's dependence on precompiled launch_map entries for unseen shapes is another correctness gap, not a circular step. Overall, the derivation chain is self-contained and its headline result rests on direct measurement.
Assumptions & free parameters
free parameters (2)
- per-instruction overhead lambda_r =
not disclosed
- hardware efficiency eta_r,v =
not disclosed
assumptions (4)
- domain assumption The latency of a MatMul kernel equals the sum of independent Prologue, Main Loop, and Epilogue latencies, with the maximum across cores as the completion time (Equation 7).
- domain assumption The steady-state initiation interval is the maximum of recurring hardware stage times plus an exposed dependency latency (Equation 9).
- ad hoc to paper The four tiling templates form a complete design space for balancing L0C residency and spatial parallelism.
- ad hoc to paper Offline profiling of lambda_r and eta_r,v on Ascend hardware generalizes to the 80,000-shape benchmark.
Cite this review
Pith. "Pith review of Adaptive Matrix Multiplication for Dynamic Shapes on Ascend NPUs." pith.science (2026). https://pith.science/paper/CRUWQ4AL
@misc{pith2026260810803,
author = {Pith},
title = {Pith review of: Adaptive Matrix Multiplication for Dynamic Shapes on Ascend NPUs},
year = {2026},
howpublished = {\url{https://pith.science/paper/CRUWQ4AL}},
note = {Machine review of arXiv:2608.10803}
}
read the original abstract
Matrix Multiplication (MatMul) faces a "generalization crisis" driven by highly dynamic tensor shapes. This crisis is particularly acute on Ascend NPUs, where explicitly controlled architectures and strict physical constraints render existing GPU-centric optimizations ineffective. To resolve this, we propose AdaptCore, an adaptive framework for universally high-performance MatMul on Ascend NPUs. AdaptCore systematically decouples operator optimization into spatial tiling and instruction orchestration. It first maps dynamic shapes into a hardware-aware 2D tiling taxonomy to balance on-chip capacity limits and multi-core parallelism. Furthermore, it integrates a composable optimization library with a deterministic analytical performance model. By mathematically evaluating hardware state mutations, AdaptCore proactively selects and caches optimal implementations, enabling O(1) overhead runtime dispatching. Evaluations demonstrate that AdaptCore delivers a remarkable 1.85x mean speedup across 80,000 input shapes, and achieves up to a 1.48x acceleration in representative end-to-end models over the highly-tuned native vendor library (ACLNN).
Figures
Figures from the paper (5 more)
Reference graph
Works this paper leans on
-
[1]
Weinan Dai, Hanlin Wu, Qiying Yu, Huan ang Gao, Jiahao Li, Chengquan Jiang, Weiqiang Lou, Yufan Song, Hongli Yu, Jiaze Chen, Wei-Ying Ma, Ya-Qin Zhang, Jingjing Liu, Mingxuan Wang, Xin Liu, and Hao Zhou. 2026. CUDA Agent: Large-Scale Agentic RL for High- Performance CUDA Kernel Generation. arXiv:2602.24286 [cs.LG] doi:10.48550/arXiv.2602.24286
-
[2]
DeepSeek AI. 2025. DeepGEMM: Clean and Efficient FP8 GEMM Kernels with Fine-Grained Scaling.https://github.com/deepseek-ai/ DeepGEMM. JIT-compiled GEMM library for Hopper (SM90) and Blackwell (SM100) GPUs
work page 2025
-
[3]
William Fedus, Barret Zoph, and Noam Shazeer. 2022. Switch Trans- formers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity. arXiv:2101.03961 [cs.LG] doi:10.48550/arXiv.2101.03961
-
[4]
Huawei. 2019.Ascend 910 AI Processor Technical White Paper.https: //www.hiascend.com/documentAccessed on 2026-04-09
work page 2019
-
[5]
Huawei. 2026. CANN: Compute Architecture for Neural Networks. https://www.hiascend.com/cann. Accessed on 2026-04-09
work page 2026
-
[6]
Huawei CANN Community. 2026. catlass.https://gitcode.com/cann/ catlass. Accessed on 2026-04-09; version v1.4.0
work page 2026
-
[7]
Changho Hwang, Wei Cui, Yifan Xiong, Ziyue Yang, Ze Liu, Han Hu, Zilong Wang, Rafael Salas, Jithin Jose, Prabhat Ram, Joe Chau, Peng Cheng, Fan Yang, Mao Yang, and Yongqiang Xiong. 2023. Tutel: Adaptive Mixture-of-Experts at Scale. arXiv:2206.03382 [cs.DC] doi:10. 48550/arXiv.2206.03382
-
[8]
Heng Liao, Jiajin Tu, Jing Xia, Hu Liu, Xiping Zhou, Honghui Yuan, and Yuxing Hu. 2021. Ascend: a Scalable and Unified Architecture for Ubiquitous Deep Neural Network Computing : Industry Track Paper. In2021 IEEE International Symposium on High-Performance Computer Architecture (HPCA). 789–801. doi:10.1109/HPCA51647.2021.00071
arXiv 2021
Show all 20 references
- [9]
- [10]
-
[11]
NVIDIA Corporation. 2026. cuTile-python.https://github.com/nvidia/ cutile-python. Accessed on 2026-04-09
2026
-
[12]
NVIDIA Corporation. 2026. CUTLASS: CUDA Templates for Linear Algebra Subroutines.https://github.com/NVIDIA/cutlass. Accessed on 2026-04-09
2026
-
[13]
Philippe Tillet, H. T. Kung, and David Cox. 2019. Triton: an interme- diate language and compiler for tiled neural network computations. InProceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages(Phoenix, AZ, USA) (MAPL 2019). Ass...
2019
-
[14]
Triton community. 2026. Triton-Ascend.https://github.com/triton- lang/triton-ascend. Accessed on 2026-08-04
2026
-
[15]
Ruoxi Wang, Rakesh Shivanna, Derek Cheng, Sagar Jain, Dong Lin, Lichan Hong, and Ed Chi. 2021. DCN V2: Improved Deep & Cross Network and Practical Lessons for Web-scale Learning to Rank Sys- tems. InProceedings of the Web Conference 2021(Ljubljana, Slovenia) (WWW ’21). Associa...
2021
-
[16]
Abel, Xu Guo, Jianbing Dong, Ji Shi, and 11 Kunlun Li
Zehuan Wang, Yingcan Wei, Minseok Lee, Matthias Langer, Fan Yu, Jie Liu, Shijie Liu, Daniel G. Abel, Xu Guo, Jianbing Dong, Ji Shi, and 11 Kunlun Li. 2022. Merlin HugeCTR: GPU-accelerated Recommender System Training and Inference. InProceedings of the 16th ACM Con- ference on ...
2022
- [17]
- [18]
-
[19]
Ruiqi Zheng, Liang Qu, Bin Cui, Yuhui Shi, and Hongzhi Yin. 2023. AutoML for Deep Recommender Systems: A Survey.ACM Trans. Inf. Syst.41, 4, Article 101 (March 2023), 38 pages. doi:10.1145/3579355
2023 doi
-
[20]
Jie Zhu, Zhifang Fan, Xiaoxie Zhu, Yuchen Jiang, Hangyu Wang, Xin- tian Han, Haoran Ding, Xinmin Wang, Wenlin Zhao, Zhen Gong, Huizhi Yang, Zheng Chai, Zhe Chen, Yuchao Zheng, Qiwei Chen, Feng Zhang, Xun Zhou, Peng Xu, Xiao Yang, Di Wu, and Zuotao Liu. 2025. RankMixer: Scaling...
2025
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.