MambaLite-Micro: Memory-Optimized Mamba Inference on MCUs

Hongjun Xu; Junxi Xia; Stephen Xia; Weisi Yang; Yueyuan Sui

REVIEW 3 major objections 4 minor 12 references

MambaLite-Micro: Memory-Optimized Mamba Inference on MCUs

T0 review · 3 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Mamba inference runs on a microcontroller with 83% less peak RAM

desk verdict Genuine engineering first—Mamba in C on an MCU—but the conclusion oversells fidelity as bit-level and the artifacts aren't public; send to review with requests for code and tighter claims. read the letter →

arxiv 2509.05488 v1 pith:4GQE4FPO submitted 2025-09-05 cs.LG cs.AIcs.OS

Hongjun Xu , Junxi Xia , Weisi Yang , Yueyuan Sui , Stephen Xia This is my paper

classification cs.LGcs.AIcs.OS

keywords Mambastatespacemodelsmicrocontrollerinferencememoryoptimizationoperatorfusionkeywordspottinghumanactivityrecognitionruntime-freedeployment

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

MambaLite-Micro claims to be the first true microcontroller deployment of a Mamba sequence model. It reimplements the Mamba forward pass in plain C, compiles the weights directly into firmware, and restructures the selective state-space recurrence so that the large discretized tensors are never built. The result is an 83.0% peak-RAM reduction versus the paper's own unfused baseline, with average output error near 1.7e-5 relative to PyTorch. On keyword-spotting and human-activity-recognition test sets, every MCU classification matches the PyTorch baseline. If correct, this removes a major obstacle to running modern sequence models on battery-powered devices.

What carries the argument

The key object is the fused selective-SSM update: instead of forming the explicit 4D discretization tensors exp(delta*A) and delta*(B*u), the engine evaluates them on the fly for each time step as x <- exp(delta_i .* A) .* x + delta_i .* (B .* u), with y <- einsum of x and C. This turns the Mamba layer into a streaming computation with O(BDN) intermediate memory, and lifetime-aware buffer allocation then reuses memory across non-overlapping tensor lifetimes.

What would settle it

Take the trained models and search for inputs whose top-two logit values differ by less than 1.31e-3, the reported worst-case L-infinity error, then compare the MCU and PyTorch class labels. A single disagreement on such a near-boundary input would show that 100% consistency is a property of the tested samples, not of the implementation in general.

Watch

Extended reading notes

Core claim

The central discovery is that the memory bottleneck in deploying Mamba is avoidable: the 4D tensors A-bar and B-bar-u, which the reference implementation materializes at O(B,D,L,N) storage, can be computed element-wise inside each recurrence step. MambaLite-Micro fuses those computations into the selective SSM recurrence so that only O(B,D,N) state is held, and it streams along the sequence length instead of storing every step. Combined with lifetime-aware buffer reuse, this makes a Mamba layer run in 230 KB peak RAM on an ESP32S3 and 276 KB on an STM32H7, at an average numerical error of 1.7e-5 and with zero classification mismatches on the evaluated tasks.

Load-bearing premise

The result leans on the trained PyTorch model as the ground truth: '100% consistency' is measured only on the evaluated test sets and assumes no input has a decision margin smaller than the numerical error between the C and PyTorch outputs.

Editorial extensions

If this is right

Peak RAM for a Mamba layer drops from O(BDLN) to O(BDN), so sequence length no longer multiplies the memory footprint.
Mamba can run on MCUs without vendor-specific runtimes or ONNX export because the entire forward pass is plain C compiled into firmware.
Keyword spotting and human activity recognition fit in 29-276 KB peak RAM and run in 95-1133 ms in fp32, making the implementation competitive with int8 attention baselines on the same hardware.
The same pipeline reproduces PyTorch Mamba outputs to roughly 1e-5 average error and passes the evaluated test sets with identical labels.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The fusion trick is not specific to this exact model: any selective state-space layer that materializes a per-step discretization tensor could likely reuse the same on-the-fly computation to shrink memory.
Because the recurrence streams over sequence length, chunked or windowed inference is a natural next step that the paper does not explore but its machinery permits.
The 100% consistency result is demonstrated only on the supplied test sets; inputs whose logits are within the worst-case error band of about 1e-3 could in principle flip a label, so a margin-aware evaluation would make the claim stronger.
Quantization is deliberately left out; combining this memory layout with int8 or fixed-point arithmetic could reduce flash and RAM further, at the cost of rechecking numerical consistency.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague—

The useful part of this paper is the engineering, not the sales pitch. It is, to my knowledge, a genuine first: a Mamba model running in C on an MCU, with no runtime, on two boards. The fusion trick in Eqs. (3)–(4) is a textbook loop fusion—don't materialize A_bar and B_bar, compute their entries inside the recurrence step—and the memory accounting adds up: 1,384,472 B to 235,620 B is 83.0%, and the lifetime-aware buffer reuse is a reasonable way to get there. The confusion matrices in Appendix B show exact label agreement with the PyTorch reference on the test sets, and the latency numbers are plausible for fp32 on an ESP32S3. If I needed to deploy a small Mamba model on a microcontroller, this pipeline would be my starting point.

The soft spots are in how the results are described, not in the core implementation. “Bit-level fidelity” in the conclusion is simply wrong: Table 1b reports errors up to 1.31e-3, and bit-level would mean exact equality. That should be corrected to “small numerical error” or similar. The “100% consistency” claim is also stronger than the evidence. The tables show identical labels on the test sets, which is good evidence, but it is not proof that classification is fully preserved on all inputs: with worst-case L∞ error of 1.31e-3, a sample whose top-two logits differ by less than the local error can flip. Reporting logit margins or a sensitivity analysis would close this gap. I would not call this a load-bearing flaw—the test-set agreement is real, and the margin critique is standard reviewer material—but the wording should be tightened.

A more practical concern is reproducibility. The paper says code will be released, but as far as I can tell nothing is public, and there are no repeated measurements, no standard deviations, no power or energy readings. For a systems paper about MCU deployment, the absence of artifact release makes the headline numbers hard to verify. The arithmetic is consistent, so I don't doubt the result, but an editor should ask for the code and data before accepting anything.

The citation base is thin—one reference for “embedded Mamba” prior art, and a lot of standard TinyML cites—but the self-claim is properly hedged with “to our knowledge.” No circularity concerns.

My take: this is a solid, useful systems result with an overreach in the conclusion and a reproducibility gap. It deserves proper peer review, with requests for artifacts and tighter language. I'd cite it if I worked on Mamba-on-MCU, and I'd bring it to a reading group focused on TinyML systems.

Referee Report

3 major / 4 minor

Summary. The paper presents MambaLite-Micro, a fully C-based, runtime-free inference engine for deploying a PyTorch Mamba model on microcontrollers. The pipeline exports model weights to C arrays, fuses the Mamba discretization step into the SSM recurrence, and applies lifetime-aware buffer allocation. On the ESP32S3 and STM32H7, the authors report an 83.0% peak-RAM reduction, an average numerical error of 1.7×10^-5, worst-case L∞ error up to 1.31×10^-3, and 100% label consistency with the PyTorch baseline on KWS and HAR test sets. The paper claims to be the first actual MCU deployment of a Mamba-based architecture.

Significance. If properly scoped, this is a valuable engineering contribution: it shows that a selective state-space model can run on commodity MCUs without a runtime, with a concrete memory optimization recipe and cross-platform validation. The ablation in Table 1(a) and the confusion matrices in Appendix B support the claimed memory reduction and exact test-set agreement. The paper is also candid in reporting numerical discrepancies, which is a strength because it lets the reader identify exactly where the conclusions overreach.

major comments (3)

[Section 4 and Conclusion] Table 1(b) reports average L∞ errors in the range 10^-5–10^-4 and worst-case L∞ up to 1.31×10^-3. The Conclusion's phrase 'preserving bit-level fidelity with the PyTorch Mamba layer outputs' is directly contradicted by these numbers: bit-level fidelity means exact bitwise equality, and any non-zero error excludes it. Please remove 'bit-level' and restate the claim as 'numerical fidelity within the reported error bounds'.
[Section 4 / Abstract] The claim that 'downstream classification accuracy is fully preserved' and that the implementation achieves '100% consistency' is supported only for the specific test sets evaluated. Because the worst-case L∞ error is 1.31×10^-3 (10-class KWS), a sample whose top-1/top-2 logit margin is smaller than the local error band could flip. The paper does not report logit margins or an error-propagation sensitivity study. Please either add such an analysis or qualify the claim to 'zero disagreements on the evaluated test sets'.
[Eq. (3)–(4), §2.2] The fused recurrence is written with inconsistent shapes and indices. In Algorithm 1, B is (B,L,N) and u is the input sequence, but Eq. (3) writes δ[:,:,i]⊙(B⊙u), which does not broadcast to the required (B,D,N) at time step i. The einsum in Eq. (1), 'bdl,dn,bdl->bdln', also does not match the Mamba reference's use of B as (B,L,N). Please rewrite the fusion with explicit per-step indexing (e.g., δ_i, B_i, u_i and outer products) so that the core memory-saving equivalence is checkable.

minor comments (4)

[§1, Introduction] The 'first deployment' claim should be accompanied by a concrete discussion of the cited prior work [10] (and any other Mamba edge deployments). Currently [10] is dismissed in a single sentence; if the distinction is 'actual MCU execution,' state that explicitly and explain why [10] does not satisfy it.
[Table 1(a)] The row 'w/o Lifetime' is not defined in the table caption or the text. Clarify that this is the variant without lifetime-aware allocation (and state whether operator fusion is also omitted). Also spell out 'MambaLM KWS' as 'MambaLite-Micro KWS' for readability.
[Appendix B] The confusion matrices support the claim of identical test-set predictions, but the PyTorch baseline accuracy is not stated separately. Add the reference accuracy under each table so the reader can verify that the C engine preserves it without having to recompute from the ground-truth matrix.
[Abstract and typography] Format the error as 1.7×10^-5 (not '1.7x10^-5'), and consider renaming the Table 1(b) header 'Avg. L∞' to 'Average L∞ error' for consistency with §3.1.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the fused Mamba recurrence is an algebraic rearrangement of the original update, and the fidelity/consistency claims are empirical porting comparisons, not fitted predictions.

full rationale

The paper's central technical derivation is the operator fusion in Section 2.2. The original Mamba recurrence materializes 4D tensors ¯A and ¯Bu (Eq. 1) and updates x ← ¯A⊙x + ¯Bu (Eq. 2). The proposed fused implementation computes the same quantities on the fly: ¯A[:, :, i, :] = exp(δ[:, :, i] ⊙ A) and ¯Bu[:, :, i, :] = δ[:, :, i] ⊙ (B ⊙ u) (Eq. 3), yielding x ← exp(δ ⊙ A)x + δ ⊙ (B ⊙ u) (Eq. 4). This is exactly a substitution of the definitions of ¯A and ¯Bu into the original recurrence; it does not introduce new fitted parameters, and it does not define the target result in terms of itself. The memory reduction follows directly from not materializing the (B, D, L, N) intermediates and is measured independently. The '100% consistency' claim is an empirical agreement between two independently written implementations (C engine vs. PyTorch reference) on held-out test sets, not a prediction forced by a fitted parameter. No load-bearing self-citation appears: the cited Mamba references are the original external architecture papers, and the datasets are public benchmarks. The only self-referential aspect is that the PyTorch baseline is the authors' own training pipeline, but using that baseline as a porting oracle is a standard and legitimate methodology; it does not make the numerical error or label-agreement results circular. A legitimate correctness concern remains—the reported worst-case L∞ error of 1.31e-3 (Table 1b) means 'fully preserving classification accuracy' is stated more strongly than the margin analysis supports—but that is an evidential/robustness issue, not circular reasoning. Therefore the circularity score is 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No free parameters or invented entities are introduced in this work. The trained weights and model configurations are inputs from an external training procedure, not quantities fit by this paper. The remaining assumptions concern the correctness reference, the algebraic equivalence of the fused recurrence, and the representativeness of the MCU measurements.

assumptions (3)

domain assumption The PyTorch Mamba implementation is a correct reference for Mamba semantics and for the trained task labels.
Sections 3.1 and 4 use the authors' PyTorch baseline as ground truth; no independent correctness oracle or logit-margin analysis is provided.
standard math The fused recurrence in Section 2.2 is algebraically equivalent to the original materialized recurrence, including broadcast semantics.
Eq. (3) to Eq. (4) present the fusion as a short rearrangement; this is not machine-checked but appears mathematically straightforward.
domain assumption Observed MCU memory and latency measurements are representative of the actual devices and toolchains.
Table 1 reports single values with no raw logs, binaries, or repeated-run variance; the measurements cannot be independently audited from the preprint.

how reviews work

0 comments

Cite this review

Pith. "Pith review of MambaLite-Micro: Memory-Optimized Mamba Inference on MCUs." pith.science (2026). https://pith.science/paper/4GQE4FPO

@misc{pith2026250905488,
  author       = {Pith},
  title        = {Pith review of: MambaLite-Micro: Memory-Optimized Mamba Inference on MCUs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4GQE4FPO}},
  note         = {Machine review of arXiv:2509.05488}
}

read the original abstract

Deploying Mamba models on microcontrollers (MCUs) remains challenging due to limited memory, the lack of native operator support, and the absence of embedded-friendly toolchains. We present, to our knowledge, the first deployment of a Mamba-based neural architecture on a resource-constrained MCU, a fully C-based runtime-free inference engine: MambaLite-Micro. Our pipeline maps a trained PyTorch Mamba model to on-device execution by (1) exporting model weights into a lightweight format, and (2) implementing a handcrafted Mamba layer and supporting operators in C with operator fusion and memory layout optimization. MambaLite-Micro eliminates large intermediate tensors, reducing 83.0% peak memory, while maintaining an average numerical error of only 1.7x10-5 relative to the PyTorch Mamba implementation. When evaluated on keyword spotting(KWS) and human activity recognition (HAR) tasks, MambaLite-Micro achieved 100% consistency with the PyTorch baselines, fully preserving classification accuracy. We further validated portability by deploying on both ESP32S3 and STM32H7 microcontrollers, demonstrating consistent operation across heterogeneous embedded platforms and paving the way for bringing advanced sequence models like Mamba to real-world resource-constrained applications.

Figures

Figures reproduced from arXiv: 2509.05488 by the authors.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

12 extracted references · 6 canonical work pages

[1]

Mobilenetv2: Inverted residuals and linear bottlenecks

Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 4510–4520, 2018

2018
[2]

Deep compression: Compressing deep neural net- works with pruning, trained quantization and huffman coding.arXiv preprint arXiv:1510.00149, 2015

Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural net- works with pruning, trained quantization and huffman coding.arXiv preprint arXiv:1510.00149, 2015

arXiv 2015
[3]

Benchmarking tinyml systems: Challenges and direction.arXiv preprint arXiv:2003.04821, 2020

Colby R Banbury, Vijay Janapa Reddi, Max Lam, William Fu, Amin Fazel, Jeremy Holleman, Xinyuan Huang, Robert Hurtado, David Kanter, Anton Lokhmotov, et al. Benchmarking tinyml systems: Challenges and direction.arXiv preprint arXiv:2003.04821, 2020

arXiv 2003
[4]

Mcunet: Tiny deep learning on iot devices.Advances in neural information processing systems, 33:11711–11722, 2020

Ji Lin, Wei-Ming Chen, Yujun Lin, Chuang Gan, Song Han, et al. Mcunet: Tiny deep learning on iot devices.Advances in neural information processing systems, 33:11711–11722, 2020

work page 2020
[5]

Tensorflow lite micro: Embedded machine learning for tinyml systems.Proceedings of machine learning and systems, 3:800–811, 2021

Robert David, Jared Duke, Advait Jain, Vijay Janapa Reddi, Nat Jeffries, Jian Li, Nick Kreeger, Ian Nappier, Meghna Natraj, Tiezhen Wang, et al. Tensorflow lite micro: Embedded machine learning for tinyml systems.Proceedings of machine learning and systems, 3:800–811, 2021

work page 2021
[6]

Improving the efficiency of transformers for resource-constrained devices

Hamid Tabani, Ajay Balasubramaniam, Shabbir Marzban, Elahe Arani, and Bahram Zonooz. Improving the efficiency of transformers for resource-constrained devices. In2021 24th Euromicro Conference on Digital System Design (DSD), pages 449–456, 2021. doi: 10.1109/ DSD53832.2021.00074

work page arXiv 2021
[7]

Machine learning for microcontroller-class hardware: A review.IEEE Sensors Journal, 22(22):21362–21390, 2022

Swapnil Sayan Saha, Sandeep Singh Sandha, and Mani Srivastava. Machine learning for microcontroller-class hardware: A review.IEEE Sensors Journal, 22(22):21362–21390, 2022. doi: 10.1109/JSEN.2022.3210773

work page arXiv 2022
[8]

Mamba: Linear-time sequence modeling with selective state spaces

Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023

arXiv 2023

Show all 12 references

[9]

Transformers are ssms: Generalized models and efficient algorithms through structured state space duality.arXiv preprint arXiv:2405.21060, 2024

Tri Dao and Albert Gu. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality.arXiv preprint arXiv:2405.21060, 2024

2024 arXiv
[10]

Optimis- ing tinyml with quantization and distillation of transformer and mamba models for indoor localisation on edge devices.Scientific Reports, 15(1):10081, 2025

Thanaphon Suwannaphong, Ferdian Jovan, Ian Craddock, and Ryan McConville. Optimis- ing tinyml with quantization and distillation of transformer and mamba models for indoor localisation on edge devices.Scientific Reports, 15(1):10081, 2025

2025
[11]

P. Warden. Speech Commands: A Dataset for Limited-V ocabulary Speech Recognition.ArXiv e-prints, April 2018. URLhttps://arxiv.org/abs/1804.03209

2018 arXiv
[12]

A public domain dataset for human activity recognition using smartphones

Davide Anguita, Alessandro Ghio, Luca Oneto, Xavier Parra, Jorge Luis Reyes-Ortiz, et al. A public domain dataset for human activity recognition using smartphones. InEsann, volume 3, pages 3–4, 2013. 5 Appendix A Experiment Setup A.1 Hardware & toolchain Experiments were condu...

2013

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.

[1] [1]

Mobilenetv2: Inverted residuals and linear bottlenecks

Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 4510–4520, 2018

2018

[2] [2]

Deep compression: Compressing deep neural net- works with pruning, trained quantization and huffman coding.arXiv preprint arXiv:1510.00149, 2015

Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural net- works with pruning, trained quantization and huffman coding.arXiv preprint arXiv:1510.00149, 2015

arXiv 2015

[3] [3]

Benchmarking tinyml systems: Challenges and direction.arXiv preprint arXiv:2003.04821, 2020

Colby R Banbury, Vijay Janapa Reddi, Max Lam, William Fu, Amin Fazel, Jeremy Holleman, Xinyuan Huang, Robert Hurtado, David Kanter, Anton Lokhmotov, et al. Benchmarking tinyml systems: Challenges and direction.arXiv preprint arXiv:2003.04821, 2020

arXiv 2003

[4] [4]

Mcunet: Tiny deep learning on iot devices.Advances in neural information processing systems, 33:11711–11722, 2020

Ji Lin, Wei-Ming Chen, Yujun Lin, Chuang Gan, Song Han, et al. Mcunet: Tiny deep learning on iot devices.Advances in neural information processing systems, 33:11711–11722, 2020

work page 2020

[5] [5]

Tensorflow lite micro: Embedded machine learning for tinyml systems.Proceedings of machine learning and systems, 3:800–811, 2021

Robert David, Jared Duke, Advait Jain, Vijay Janapa Reddi, Nat Jeffries, Jian Li, Nick Kreeger, Ian Nappier, Meghna Natraj, Tiezhen Wang, et al. Tensorflow lite micro: Embedded machine learning for tinyml systems.Proceedings of machine learning and systems, 3:800–811, 2021

work page 2021

[6] [6]

Improving the efficiency of transformers for resource-constrained devices

Hamid Tabani, Ajay Balasubramaniam, Shabbir Marzban, Elahe Arani, and Bahram Zonooz. Improving the efficiency of transformers for resource-constrained devices. In2021 24th Euromicro Conference on Digital System Design (DSD), pages 449–456, 2021. doi: 10.1109/ DSD53832.2021.00074

work page arXiv 2021

[7] [7]

Machine learning for microcontroller-class hardware: A review.IEEE Sensors Journal, 22(22):21362–21390, 2022

Swapnil Sayan Saha, Sandeep Singh Sandha, and Mani Srivastava. Machine learning for microcontroller-class hardware: A review.IEEE Sensors Journal, 22(22):21362–21390, 2022. doi: 10.1109/JSEN.2022.3210773

work page arXiv 2022

[8] [8]

Mamba: Linear-time sequence modeling with selective state spaces

Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023

arXiv 2023