REVIEW 3 major objections 4 minor 21 references
XAMBA: Enabling Efficient State Space Models on Resource-Constrained Neural Processing Units
T0 review · 3 major / 4 minor · reviewed 2026-08-08 · deepseek-v4-flash
Pith's one-line read XAMBA turns Mamba-2's serial sums and activations into parallel matrix operations, slashing NPU latency by up to 4.8x while keeping quality within 1.5 points.
desk verdict Genuinely useful first result for SSMs on commercial NPUs; CumBA/ReduBA hold up, but ActiBA's speedup and accuracy are measured on two different functions. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the lower-triangular CumBA mask $M_{\mathrm{CumBA}}$ with $M_{\mathrm{CumBA}}(i,j)=1$ iff $j \le i$, turning the row-wise cumulative sum of $X \in \mathbb{R}^{m\times n}$ into the matrix product $C = M_{\mathrm{CumBA}}X$; ReduBA uses an all-ones vector $M_{\mathrm{ReduBA}}$ so that row-summing becomes $R = M_{\mathrm{ReduBA}}X$. This identity swaps a serial $m$-cycle DSP loop for a parallel MatMul/MVM on the MAC array. The second mechanism is the NPU's Piecewise Linear Unit with its configurable lookup table, which stores slopes and intercepts so that $f(x) \approx m_k x + c_k$ on each interval $[x_k, x_{k+1}]$, letting Swish and Softplus run during the drain phase without a sequential DSP kernel. ZVC compression exploits the mask's ~50% sparsity to reduce memory traffic.
What would settle it
Run the actual PLU-configured Swish/Softplus kernels (not ReLU-substituted) on the same Intel Core Ultra Series 2 NPU with the same 4-token prefill/decoding harness and compare wall-clock latency per token against the reported 260 tokens/s; if the real kernels run measurably slower than the ReLU emulation, the ActiBA speedup is not representative.
Extended reading notes
Core claim
The paper claims that the dominant bottlenecks of Mamba-2 and Mamba on an NPU are not their matrix multiplications but their sequential operators: the CumSum b inside the SSD step-1 of Mamba-2 accounts for over 99.9% of CumSum time on a 256x256 matrix, and Swish/Softplus dominate Mamba's runtime when executed as DSP loops. XAMBA's central discovery is that these serial operators can be reformulated as data-parallel matrix operations that exactly match the NPU's MAC-array strengths. CumBA computes the cumulative sum as C = M_CumBA * X with a lower-triangular binary mask, ReduBA computes ReduceSum as a matrix-vector product with an all-ones mask, and both exploit the resulting sparsity and reuse to cut latency and memory traffic. ActiBA stores piecewise-linear slopes and intercepts in the PLU's configurable lookup table so Swish and Softplus evaluate during the drain phase, effectively for free. Measured on a commercial Intel Core Ultra Series 2 NPU, CumBA alone reduces Mamba-2 130M latency by 2.7x, ReduBA by 1.2x, and the combination by 4.8x, while ActiBA reduces Mamba 130M latency by up to 2.6x; the accuracy table shows a maximum average-accuracy drop of 1.36 percentage points on the smallest Mamba model and essentially no change on Mamba-2 variants.
Load-bearing premise
ActiBA's speedup is measured with ReLU standing in for Swish/Softplus in the latency tests, while the accuracy table uses the real piecewise-linear PLU mapping, so the 2.6x transfer depends on those two implementations costing the same on the NPU.
Editorial extensions
If this is right
- Mamba-2's CumSum bottleneck drops from over 50% of baseline latency to a minor component after CumBA, directly improving tokens per second.
- Because CumBA's mask is half zeros, ZVC and sparsity-based compute skipping turn the triangular mask into a memory and compute saving, not just a data layout change.
- ActiBA's PLU approximations keep average accuracy within 1.5 percentage points for the smallest 130M models and within 0.1 points for most larger variants.
- The Mamba-130M decoding throughput rises from 100 to 260 tokens/s, clearing the 50 tokens/s KPI defined for client-side responsiveness.
- The techniques are described as architecture-generic: any NPU with parallel MAC units plus a DSP can adopt the matrix-mask and PLU-LUT mappings.
Reading between the lines
- The prefix-sum-as-matmul identity is classical; XAMBA's contribution is showing that on an MPU+DSP NPU the matrix form wins over the serial DSP loop. A natural next test is whether the same mask trick helps other sequential primitives such as cumprod or selective-scan state updates, where the mask would be non-triangular.
- The 4.8x figure comes from a single-block, 4-token Mamba-2 measurement. Longer sequences increase the matmul cost (m rows in the mask), so the speedup could shrink or grow depending on tile sizes; that tradeoff is not explored in the paper.
- ActiBA trades accuracy for speed, and the paper reports the knob: adding PLU segments near the origin recovers accuracy. An automatic per-layer segment-count search could push throughput further while meeting a quality floor.
- The reported gains are specific to Intel's NPU software stack; whether AMD or Qualcomm NPUs with different PLU and sparsity support receive the same factors is an open empirical question.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. XAMBA presents a three-step methodology for running Mamba and Mamba-2 state-space models on commercial Intel Core Ultra Series 2 NPUs: first enabling the models on the NPU via ONNX/OpenVINO conversion, then replacing sequential CumSum and ReduceSum with masked matrix operations (CumBA, ReduBA), and finally approximating SiLU and Softplus activations with piecewise-linear C-LUT mappings on the NPU's PLU (ActiBA). The paper reports up to 4.8x latency reduction for Mamba-2 130M from CumBA plus ReduBA, and up to 2.6x latency reduction for Mamba 130M from ActiBA, with Table 1 reporting 'negligible' quality loss for PLU-based approximations across several Mamba and Mamba-2 model sizes.
Significance. If the claims hold, XAMBA is a useful practical contribution: it is, to my knowledge, the first reported framework for deploying SSMs on a commercial COTS NPU, and it demonstrates a convincing 4.8x speedup for Mamba-2 through two mathematically exact reformulations. The CumBA and ReduBA ideas are simple, elegant, and appear exact, and the use of measured performance on a real Intel NPU against an OpenVINO baseline is a strength. The public code availability is also a plus. The ActiBA contribution, however, has a load-bearing experimental gap between the latency measurements and the quality measurements, and the quality numbers themselves are not uniformly 'negligible.' These issues are fixable with additional experiments and careful reporting, so the paper does not warrant rejection, but it needs a major revision.
major comments (3)
- [Section 3 and Section 4 (Fig. 4(c), Table 1)] The ActiBA latency result and the ActiBA quality result are measured on two different approximations. Section 3 states 'ActiBA is emulated by replacing activation functions with ReLU,' and Figure 4(c) labels the measured variants 'Si-Sp-Rl,' yet Table 1 reports accuracy for 'PLU SiLU & Softplus.' ReLU is a one-segment clamp; the PLU/C-LUT datapath in Figure 2(e) includes segment-range comparison, slope/intercept lookup, and saturation, so there is no evidence that the ReLU-emulated execution time equals the PLU execution time. Consequently, the paper's joint claim of 'up to 2.6x with negligible quality loss' for ActiBA is not established: the 2.6x speedup was measured on a substitute operator, while the quality numbers were obtained with the actual PLU approximation. Please run the latency experiment with the exact PLU configuration used for Table 1, or provide a rigorous argument that a ReLU datapath has identical cost, and report both metrics for the same variant.
- [Section 2.2 and Table 1] The accuracy of the PLU approximation and its hardware cost both depend on the number and placement of linear segments, but the paper never reports the C-LUT configuration used to produce Table 1. Section 2.2 states that 'increasing the number of linear segments... can further reduce this loss,' which makes the segment count and breakpoints a first-order parameter of the ActiBA tradeoff. Without this information, the Table 1 results are not reproducible, and one cannot assess whether the 'negligible' accuracy loss is an artifact of using very fine segmentation that would invalidate the 2.6x latency claim. Please specify the segment counts, boundaries, slopes/intercepts (or a reference to the exact C-LUT configuration), and state how the PLU was emulated for the quality evaluation.
- [Table 1 and Section 4] The claim of 'negligible quality loss' is weakened by component-level accuracy drops that the paper does not discuss. For Mamba-130M, the PLU variant drops from 35.25 to 31.83 on PIQA and from 64.47 to 59.19 on Winogrande, even though the average accuracy drops only from 42.03 to 40.67. The paper's statement that the 'largest accuracy drop' is 1.36% refers to the average and obscures these benchmark-specific degradations. If the intended claim is that the average degradation is small, it should be stated as such; otherwise, 'negligible' is too strong. Please report and discuss the per-benchmark variance, and explain why these component-wise drops are acceptable for the intended deployment scenarios.
minor comments (4)
- [Section 3 and Figure 4] The experimental methodology should state whether reported latencies are means or medians, how many benchmark iterations were run, whether warm-up was performed, and the exact input shape and batch size used beyond 'fixed input tokens of 4.' Figure 4(a) says 'average inference latency' while Figure 4(c) says 'first inference latency'; these metrics should be defined and used consistently.
- [Figure 4(a)] The combined 4.8x speedup from CumBA (2.7x) and ReduBA (1.2x) does not follow trivially from the individual speedups, since 2.7x and 1.2x would combine to about 3.2x if applied independently. Please clarify how the optimizations interact and why the combined speedup is 4.8x.
- [Table 1] The LAMBADA PPL entry for Mamba-130M-PLU is reported as '—'; either provide the missing value or explain why it is unavailable, since omitting one of the main benchmarks for the model most affected by ActiBA weakens the quality claim.
- [Throughout] There are typos in Table 1 and Figure 5: 'Mmaba2-1.3B' and 'Mmaba2-2.7B' should be 'Mamba2-1.3B' and 'Mamba2-2.7B,' and 'Swish' is sometimes written inconsistently. These should be corrected before publication.
Circularity Check
No circularity in XAMBA's derivation: speedups are measured against an external OpenVINO baseline; ActiBA's ReLU/PLU emulation is a validity gap, not a circular reduction.
full rationale
The central claims are empirical, not derived by construction. CumBA ('CumSum computed as C = MCumBA · X' with a lower-triangular mask) and ReduBA ('R = MReduBA · X' with an all-ones vector) are exact algebraic identities for CumSum and ReduceSum; the reported 2.7X, 1.2X, and 4.8X speedups are measured on an Intel Core Ultra Series 2 NPU using the OpenVINO benchmark app against an external baseline, so no prediction reduces to its input. ActiBA's accuracy results (Table 1) are for PLU piecewise-linear SiLU/Softplus, while Section 3 states 'ActiBA is emulated by replacing activation functions with ReLU.' This mismatch means the joint '2.6X with negligible quality loss' claim is not fully verified, but it is a correctness/validity gap rather than circularity: the ReLU latency measurement is not derived from the PLU accuracy table, and no parameter is fitted to force the speedup. The only self-citation is the NPU architecture description 'inspired by Raha et al. (2024)' (FlexNN), whose authors overlap with the present paper; however, this architectural model is motivational, and the performance conclusions are validated against real hardware, so the self-citation is not load-bearing. No equation is equivalent to another by construction in a way that assumes the claimed result, and no fitted quantity is renamed as a prediction. Score 1 reflects the minor, non-load-bearing self-citation; the ActiBA ReLU-emulation issue is a limitation to be weighed under correctness, not circularity.
Assumptions & free parameters
free parameters (1)
- PLU C-LUT segment slopes/intercepts for SiLU and Softplus =
Not disclosed
assumptions (2)
- domain assumption The Intel Core Ultra Series 2 NPU provides a Piecewise Linear Unit (PLU) with a Configurable Lookup Table as described in Figure 2(e).
- domain assumption The output-stationary MPU architecture illustrated in Figure 2(a), including Zero Value Compression and sparsity-aware compute skipping, matches the behavior of the commercial NPU used in the experiments.
Cite this review
Pith. "Pith review of XAMBA: Enabling Efficient State Space Models on Resource-Constrained Neural Processing Units." pith.science (2026). https://pith.science/paper/CO57YG7U
@misc{pith2026250206924,
author = {Pith},
title = {Pith review of: XAMBA: Enabling Efficient State Space Models on Resource-Constrained Neural Processing Units},
year = {2026},
howpublished = {\url{https://pith.science/paper/CO57YG7U}},
note = {Machine review of arXiv:2502.06924}
}
read the original abstract
State-Space Models (SSMs) have emerged as efficient alternatives to transformers for sequential data tasks, offering linear or near-linear scalability with sequence length, making them ideal for long-sequence applications in NLP, vision, and edge AI, including real-time transcription, translation, and contextual search. These applications require lightweight, high-performance models for deployment on resource-constrained devices like laptops and PCs. Designing specialized accelerators for every emerging neural network is costly and impractical; instead, optimizing models for existing NPUs in AI PCs provides a scalable solution. To this end, we propose XAMBA, the first framework to enable and optimize SSMs on commercial off-the-shelf (COTS) state-of-the-art (SOTA) NPUs. XAMBA follows a three-step methodology: (1) enabling SSMs on NPUs, (2) optimizing performance to meet KPI requirements, and (3) trading accuracy for additional performance gains. After enabling SSMs on NPUs, XAMBA mitigates key bottlenecks using CumBA and ReduBA, replacing sequential CumSum and ReduceSum operations with matrix-based computations, significantly improving execution speed and memory efficiency. Additionally, ActiBA enhances performance by approximating expensive activation functions (e.g., Swish, Softplus) using piecewise linear mappings, reducing latency with minimal accuracy loss. Evaluations on an Intel Core Ultra Series 2 AI PC show that XAMBA achieves up to 4.8X speed-up over the baseline. Our implementation is available at https://github.com/arghadippurdue/XAMBA.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
Tri Dao and Albert Gu. Transformers are SSMs: generalized models and efficient algorithms through structured state space duality . In ICML, 2024
work page 2024
-
[2]
NITRO: LLM Inference on Intel Laptop NPUs
Anthony Fei and Mohamed S. Abdelfattah. NITRO: LLM Inference on Intel Laptop NPUs , 2024. URL https://arxiv.org/abs/2412.11053
work page Pith review arXiv 2024
-
[3]
Mamba: Linear-Time Sequence Modeling with Selective State Spaces
Albert Gu and Tri Dao. Mamba: Linear-Time Sequence Modeling with Selective State Spaces . In COLM, 2024
work page 2024
-
[4]
HiPPO: Recurrent Memory with Optimal Polynomial Projections
Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher R \'e . HiPPO: Recurrent Memory with Optimal Polynomial Projections . In NeurIPS, 2020
work page 2020
-
[5]
Efficiently Modeling Long Sequences with Structured State Spaces
Albert Gu, Karan Goel, and Christopher Re. Efficiently Modeling Long Sequences with Structured State Spaces . In ICLR, 2022
work page 2022
-
[6]
Intel \ Distribution of OpenVINO \ Toolkit
Intel. Intel \ Distribution of OpenVINO \ Toolkit . URL https://www.intel.com/content/www/us/en/developer/tools/openvino-toolkit/overview.html. Accessed: Feb. 07, 2025
work page 2025
-
[7]
Intel® Core™ Ultra series mobile processors product brief , 2024 a
Intel Corporation . Intel® Core™ Ultra series mobile processors product brief , 2024 a . URL https://www.intel.com/content/www/us/en/products/docs/processors/core-ultra/core-ultra-series-2-mobile-product-brief.html. Accessed: Feb. 07, 2025
work page 2024
-
[8]
Intel® Core™ Ultra series 1 product brief , 2024 b
Intel Corporation . Intel® Core™ Ultra series 1 product brief , 2024 b . URL https://www.intel.com/content/www/us/en/products/docs/processors/core-ultra/core-ultra-series-1-product-brief.html. Accessed: Feb 07, 2025
work page 2024
Show all 21 references
-
[9]
J. Li, S. Huang, J. Xu, J. Liu, L. Ding, N. Xu, and G. Dai. MARCA: Mamba Accelerator with ReConfigurable Architecture . In arXiv, 2024. URL https://arxiv.org/abs/2409.11440
2024 arXiv
-
[10]
MobileLLM: optimizing sub-billion parameter language models for on-device use cases
Zechun Liu, Changsheng Zhao, Forrest Iandola, Chen Lai, Yuandong Tian, Igor Fedorov, Yunyang Xiong, Ernie Chang, Yangyang Shi, Raghuraman Krishnamoorthi, Liangzhen Lai, and Vikas Chandra. MobileLLM: optimizing sub-billion parameter language models for on-device use cases . In ...
2024
-
[11]
OpenVINO IR Format: Operation Sets and Specifications
OpenVINO Documentation . OpenVINO IR Format: Operation Sets and Specifications . https://docs.openvino.ai/2024/documentation/openvino-ir-format/operation-sets/operation-specs.html, 2024. Accessed: Feb. 07, 2025
2024
-
[12]
Mamba-360: Survey of State Space Models as Transformer Alternative for Long Sequence Modelling: Methods, Applications, and Challenges
Badri Narayana Patro and Vijay Srinivas Agneeswaran. Mamba-360: Survey of State Space Models as Transformer Alternative for Long Sequence Modelling: Methods, Applications, and Challenges . In arXiv, 2024. URL https://arxiv.org/abs/2404.16112
2024 arXiv
-
[13]
Mathaikutty, Soumendu K
Arnab Raha, Deepak A. Mathaikutty, Soumendu K. Ghosh, and Shamik Kundu. FlexNN: A Dataflow-aware Flexible Deep Learning Accelerator for Energy-Efficient Edge Devices . In arXiv, 2024. URL https://arxiv.org/abs/2403.09026
2024 arXiv
-
[14]
Flex-SFU: Accelerating DNN Activation Functions by Non-Uniform Piecewise Approximation
Enrico Reggiani, Renzo Andri, and Lukas Cavigelli. Flex-SFU: Accelerating DNN Activation Functions by Non-Uniform Piecewise Approximation . In DAC, 2023
2023
-
[15]
Minsoo Rhu, Mike O'Connor, Niladrish Chatterjee, Jeff Pool, Youngeun Kwon, and Stephen W. Keckler. Compressing DMA Engine: Leveraging Activation Sparsity for Training Deep Neural Networks . In HPCA, 2018
2018
-
[16]
Wang et al
X. Wang et al. Efficient Inference of Recurrent Neural Networks on Neural Processors . IEEE Transactions on Neural Networks and Learning Systems, 2018
2018
-
[17]
Kinsy, Nanning Zheng, and Pengju Ren
Tao Yang, Yadong Wei, Zhijun Tu, Haolun Zeng, Michel A. Kinsy, Nanning Zheng, and Pengju Ren. Design Space Exploration of Neural Network Activation Function Circuits . IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 2019
2019
-
[18]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
-
[19]
@esa (Ref
\@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...
-
[20]
\@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...
-
[21]
jU HA ZUu^ޚz HcYqYIcv/IO[x r@z V = <V Lj +2Ǥ=Fiͬ1 d>v qC ѨM@ ȡ6q 6( [Aeiw w'8 H : Z w |o [ zj <>uwj^\ ҄;?հ#FO2&=̹ڈ
@open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...
1999
Reviewed August 8, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.