MAUPITI: On-Device Prototype-Based Learning on a Smart Infrared Sensor

Alessio Burrello; Alfio Basile; Beatrice Alessandra Motetti; Daniele Jahier Pagliari; Enrico Macii; Francesco Daghero; Marco Castellano; Massimo Poncino; Matteo Risso; Tanguy Dugas du Villard

REVIEW 2 major objections 4 minor 19 references

MAUPITI: On-Device Prototype-Based Learning on a Smart Infrared Sensor

T0 review · 2 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A low-resolution infrared sensor with a RISC-V microcontroller learns new poses and gestures on-device, without backpropagation, using a frozen CNN encoder plus nearest-class-mean prototypes; the paper shows accuracy on par with a…

desk verdict Solid engineering demonstration of on-device NCM continual learning on a 16x16 IR sensor; the main gap is a missing explicit statement that accuracy results use the deployed shift-based update, though power-of-two shots make that update exact. read the letter →

arxiv 2608.07192 v1 pith:ILT5UNTH submitted 2026-08-07 cs.LG

Beatrice Alessandra Motetti , Tanguy Dugas du Villard , Matteo Risso , Alessio Burrello , Francesco Daghero , Enrico Macii , Massimo Poncino , Marco Castellano

show 2 more authors

Alfio Basile Daniele Jahier Pagliari

This is my paper · ORCID

classification cs.LG

keywords infraredarraysensorson-devicelearningcontinualnearestclassmeanprototype-basedtinyMLquantization-awaretraininggesturerecognition

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper shows that a tiny infrared sensor—a 16x16 thermal pixel array with a RISC-V microcontroller and under 32 kB of memory—can learn new classes of poses and gestures on the device, without backpropagation or replay buffers. The approach freezes a CNN encoder trained offline to embed each frame into a low-dimensional space, then classifies by distance to class-mean prototypes that are updated in streaming mode as new labelled examples arrive. On two datasets (five poses and nine hand gestures) the authors report accuracy close to a conventional CNN classifier, with a total latency overhead below 0.29% for both classification and prototype update. If it holds, the result matters because it means sensor nodes can be adapted in the field, to new users or new environments, simply by showing them a handful of examples.

What carries the argument

The machinery is the class prototype $m_c = \frac{1}{|D_c|}\sum_{i\in D_c} z_i$ (Eq. 1), nearest-prototype prediction $\hat{c}=\arg\min_c d(z,m_c)$ (Eq. 2), and the streaming update $m_c \leftarrow m_c + \frac{1}{N_c}(z - m_c)$ (Eq. 3). The encoder $f_\theta$ is trained offline with triplet loss (Euclidean distance, margin 100) and quantization-aware training so everything runs in INT8 on the sensor; the division by $N_c$ becomes a bit shift because shot counts are kept at powers of two. Prototypes are stored as INT32 vectors, and with $p=64$ and five classes they add just over 1 kB to a 16 kB data memory, while the whole classification plus update stays below 0.29% extra latency.

What would settle it

Train the encoder offline on four poses, then add a fifth pose that is a small variation of an existing one (e.g., 'right arm half-raised' when 'right arm raised' is already a class). With 4–16 shots on-device, if the new class's prototype falls inside the old class's cluster and accuracy on the new class stays near chance while a softmax fine-tuned variant succeeds, the core assumption of a fixed separable embedding fails.

Watch

Extended reading notes

Core claim

The central claim is that a Nearest Class Mean (NCM) classifier acting on embeddings from a frozen, quantized CNN encoder is sufficient for accurate and continually adaptable pose and gesture recognition on a low-power IR sensor, matching a conventional softmax classifier while avoiding the memory and compute costs of gradient-based fine-tuning. New classes are added by computing the mean embedding over a few labelled frames; the prototype is updated in constant time per sample. The authors report average online accuracy of about 88% when one class is added on-device and 68% when two are added, against roughly 94% for the fully offline softmax model, and they verify on the MAUPITI sensor that the total latency overhead of classification plus prototype update is below 0.29%.

Load-bearing premise

The load-bearing assumption is that the embedding space produced by the offline-trained, frozen encoder keeps never-seen classes sufficiently separated, so a single mean prototype per class is a good summary; the 68% accuracy when adding two classes shows the assumption can be fragile.

Editorial extensions

If this is right

Devices with only a few tens of kilobytes of memory can be taught new classes in the field from a handful of examples, with no cloud connection and no stored raw frames.
Because the encoder never changes on-device, the cost per training sample is a distance computation plus a mean update, so online learning can keep pace with the sensor's 10 FPS frame rate.
The accuracy gap to a full backprop-trained model is about 2–3 percentage points offline, and NCM beats fine-tuning approaches that lack a replay buffer in continual-learning settings.
Adding multiple new classes at once (two or three) degrades accuracy, mapping out a clear operating limit for the fixed embedding space.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The distance to the nearest prototype could be surfaced as an uncertainty score, letting the sensor say 'I do not know' instead of forcing a class—the paper does not evaluate this, but the NCM head provides it for free.
Storing only class means means no individual thermal frames are ever kept, which makes the already privacy-focused IR modality private-by-construction in the learning phase too.
The power-of-two shot counts are a hardware convenience; swapping the bit-shift for a fixed-point divide would allow arbitrary shot counts with negligible cost and smoother updates.
A stronger stress test than the two-class experiment would be adding a new class that is visually close to an existing one; if the single-mean summary fails there, storing multiple prototypes per class or lightly adapting the encoder would be the direct remedy.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this is a well-executed engineering paper. It combines known components—NCM classifier, metric learning, QAT, and a RISC-V smart sensor—into a working on-device continual learning system with measured latency and memory, and it ships code and data. That integration is the contribution, and it is a legitimate one for the TinyML / smart-sensing crowd.

What it does well: the experimental setup is careful. They compare against three online baselines with and without replay, report variance over seeds, and include a Bayes-optimized architecture search. The measured overhead numbers on MAUPITI are concrete: <0.29% total latency overhead, fits in 16 kB data memory. The public artifact helps. The limitation sections honestly note that novel classes must fit in a fixed latent space, and the 2C accuracy drop to ~68% shows this is the real constraint.

The soft spots are minor but real. First, the shift-based prototype update in Sec. II-C: the paper never explicitly says that the accuracy curves in Fig. 2 and Fig. 5 were generated using that deployed code. In practice, since the experiments use power-of-two shot counts, the shift is an exact integer mean (sum >> log2N), so the concern is not load-bearing—but the authors should state this in one sentence. Second, the “on par with a conventional classifier” claim in the abstract is only true against the constrained online baselines; offline, NCM loses 2-3 points to a softmax head. Worth rewording. Third, offline accuracy results lack error bars and dataset sizes are not given. These are easy to fix.

The stress-test note about the shift approximation does not actually land on reading the paper carefully: the text in Sec. III-A explicitly says power-of-two shots allow replacing divisions with shifts, which ties the experiments to the deployed update. It is a reporting clarity issue, not a correctness issue.

Who this is for: anyone doing on-device learning or low-resolution IR sensing. It deserves a serious peer review and should be accepted with minor revisions. I would bring it to a reading group for the measured overhead analysis, and would cite it if I were writing about prototype-based on-device learning.

Recommendation: engage, ask for the clarification and missing error bars, then accept.

Referee Report

2 major / 4 minor

Summary. The paper describes MAUPITI, a smart infrared sensor integrating a 16x16 TMOS array with a RISC-V core, and proposes an on-device prototype-based learning framework. A CNN encoder is trained offline with triplet metric learning and quantization-aware training; classification is performed by a Nearest Class Mean (NCM) head whose class prototypes are updated on the sensor in streaming mode without backpropagation or replay buffers. The authors evaluate pose and gesture recognition on two datasets, comparing against several continual-learning baselines, and report that the NCM approach achieves accuracy comparable to a conventional softmax classifier while adding less than 0.29% latency overhead for classification and prototype updates combined, all within 16 kB data memory and about 1.5 mW.

Significance. If the empirical claims hold, the paper demonstrates a practically useful embedded continual-learning solution for low-resolution IR sensing: it avoids backpropagation and replay memory, fits under tight 32 kB memory, and is evaluated on real hardware with multiple baselines and seed variability. The public release of code and datasets is a concrete strength that supports reproducibility. Conceptually, NCM is a well-known method, so the novelty lies in the system integration, quantization, and measurement of on-device overheads rather than in a new learning algorithm; for a letter in an applied sensors venue, that level of novelty is appropriate. The main risk is a gap between the exact update rule used in the accuracy experiments and the shift-based update described for the deployed firmware, which the authors need to close before the central claim is fully supported.

major comments (2)

[Sec. II-C vs Sec. III-A] The manuscript never reports an accuracy measurement for the deployed shift-based prototype update. Eq. 3 defines the exact running-mean update m_c <- m_c + (1/N_c)(z - m_c), but Sec. II-C states that the embedded implementation replaces the division in Eq. 3 with a shift by accumulating z values until N_c is a power of two before subtracting and normalizing. All accuracy results in Fig. 2, Fig. 5, and Table I are presented as NCM accuracy after metric learning, with no statement that they were produced using the deployed shift update, and Sec. III-B reports only latency and memory on the real sensor, not accuracy. If those curves were generated with the exact Eq. 3 arithmetic while the firmware uses the shift version, the paper's claim that the deployed system achieves accuracy on par with a conventional classifier is unsupported. Please either rerun the continual-learning experiments with the exact deployed update or provide a convincing equivalence argument and empirical validation for the settings tested.
[Sec. II-C] The description of the shift-based update is too imprecise to reproduce or assess. 'Accumulating z values until N_c is a power of two' does not specify whether updates are deferred until the count reaches the next power of two, whether intermediate samples are accumulated or discarded, what happens when a new class is created with a non-power-of-two count, or how fixed-point arithmetic avoids overflow in the accumulated sums. Since this update is the exact code running on MAUPITI, please provide pseudocode or a formal algorithmic definition and measure or prove its closeness to Eq. 3 under the INT8 quantization used on the device.

minor comments (4)

[Sec. III-A] The offline accuracy comparison would be more informative if the standard deviation over seeds were reported for the softmax baseline (94.81%) and for the offline NCM range (91.72%-92.76%), rather than only for the online continual-learning results in Fig. 2.
[Sec. III-B] The text says that each prototype update requires O(N_c * p) operations, but Eq. 3 is O(p) per sample; please clarify whether the O(N_c * p) refers to creating an initial prototype from N_c shots rather than to the incremental update of Eq. 3.
[Sec. II-B] The triplet-loss setup is described only by the final choice (Euclidean distance, margin 100); for reproducibility, please state the number of epochs, batch size, and whether online triplet mining or another sampling strategy was used.
[Fig. 4] The y-axis of Fig. 4 includes negative latency-overhead values, which is surprising for an overhead plot; the text explains that inference is slightly faster for p<64 due to loop overheads, so please relabel the axis as a signed relative latency difference or otherwise clarify the sign convention.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central accuracy and latency claims are empirical comparisons against independent softmax and continual-learning baselines, not derivations from the method's own fitted quantities.

full rationale

The paper's central claims are empirical: NCM accuracy is compared with a conventional softmax classifier and with continual-learning baselines (full retraining, last-layer retraining, new-neuron, with and without replay), and the reported latency/memory figures are measured on the MAUPITI sensor. Hyperparameters such as the triplet margin, prototype size, and encoder architecture are selected on validation data and then evaluated on held-out, session-split data; this is ordinary model fitting, not a prediction forced by construction. The self-citations to prior MAUPITI work [9] provide the hardware platform, the blueprint architecture, and the session-wise split, but they do not themselves assert the NCM learning result, so they are building blocks rather than load-bearing circular evidence. Equations (1)-(3) define the NCM prototypes and update rule, and the reported accuracies are empirical outcomes of those definitions, not restatements of them. The one notable validation gap is that the deployed power-of-two shift update introduced in Sec. II-C is not separately accuracy-tested against the exact Eq. (3) update; however, this is an experimental completeness concern, not circularity, and the paper explicitly restricts the accuracy experiments to power-of-two shot counts, where division by a power of two is exactly a shift. Overall, no fitted parameter is renamed as a prediction and no load-bearing conclusion reduces to a self-citation chain.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claim is an empirical engineering feasibility claim. It rests on standard NCM mathematics and on the domain assumption that a frozen metric-learned embedding separates base and novel classes. The unverified shift-based update and the validation-selected hyperparameters are the main dependencies for which the reader does not get independent evidence.

free parameters (3)

Triplet loss margin and distance metric = margin=100, Euclidean distance
Sec. II-B: margin 100 and Euclidean distance were selected because they yielded the best validation results. This shapes the embedding geometry on which the NCM claim depends.
Prototype size p = 64 for continual-learning experiments
p is swept over {8,16,32,64} for offline accuracy and fixed to 64 for the reported continual learning and latency results. The accuracy and memory tradeoffs depend on this choice.
Encoder architecture hyperparameters = Optimized (M) model with 1,323 parameters
Filter counts, kernel sizes, and pooling sizes are chosen by Bayesian optimization on validation accuracy in Sec. II-B. The deployed model is one point on the resulting Pareto front.

assumptions (3)

domain assumption The embedding space learned offline from base classes remains discriminative for classes introduced online, with no encoder adaptation.
Sec. III-A explicitly says novel classes must be accommodated within a fixed latent space. The 2C accuracy drop to 68% shows this assumption is only partially satisfied.
standard math A single class mean prototype is a sufficient summary of each class embedding distribution.
This is the standard NCM assumption from Mensink et al. [11], used in Eq. 1. The paper does not test non-mean summaries such as multiple prototypes per class.
ad hoc to paper The shift-based prototype update with power-of-two counts reproduces the exact running mean in Eq. 3.
Sec. II-C replaces the division in Eq. 3 with accumulating z values and shifting when the class count reaches a power of two. No experiment measures the accuracy of this deployed variant versus the exact update.

how reviews work

0 comments

Cite this review

Pith. "Pith review of MAUPITI: On-Device Prototype-Based Learning on a Smart Infrared Sensor." pith.science (2026). https://pith.science/paper/ILT5UNTH

@misc{pith2026260807192,
  author       = {Pith},
  title        = {Pith review of: MAUPITI: On-Device Prototype-Based Learning on a Smart Infrared Sensor},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ILT5UNTH}},
  note         = {Machine review of arXiv:2608.07192}
}

abstract

Low-resolution infrared (IR) array sensors represent an interesting solution for privacy-preserving human sensing in embedded systems. In this letter, we describe a smart multi-pixel IR sensor integrating a 16$\times$16 thermal MOSFET (TMOS) array and a RISC-V microcontroller extended with low-precision SIMD instructions, capable of on-device learning and continual adaptation for pose and gesture recognition tasks under tight memory and power constraints ($<$32kB on-chip memory, $\approx$1.5mW). To avoid the memory overheads of backpropagation and replay buffers, we adopt a prototype-based Nearest Class Mean (NCM) classifier in which a simple Convolutional Neural Network (CNN) encoder is trained and quantized offline, while class prototypes are stored and updated on the device in streaming mode. With experiments on two datasets, we show that this approach yields accuracy on par with a conventional classifier, with negligible latency overheads in both the classification and the prototype update ($<$0.29% considering both phases), effectively enabling online adaptation of the perception framework.

Figures

Figures reproduced from arXiv: 2608.07192 by the authors.

**Figure 2.** 5-class accuracy of the NCM classifier and the considered baselines, [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 4.** Latency overhead with respect to a traditional classifier (M) model from Tab. I, i.e., the one achieving the best accuracy versus size trade-off. When quantizing the models, the test accuracy drop with respect to floating point versions is equal to 0.47 percentage points for 1C and 1.70 for 2C [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

19 extracted references · 18 canonical work pages

[1]

Efficient deep learning models for privacy-preserving people counting on low-resolution infrared arrays,

C. Xieet al., “Efficient deep learning models for privacy-preserving people counting on low-resolution infrared arrays,”IEEE Internet of Things Journal, vol. 10, no. 15, pp. 13 895–13 907, 2023

work page 2023
[2]

Low-resolution infrared array sensor for counting and localizing people indoors: When low end technology meets cutting edge deep learning techniques,

M. Bouaziziet al., “Low-resolution infrared array sensor for counting and localizing people indoors: When low end technology meets cutting edge deep learning techniques,”Information, vol. 13, no. 3, 2022

work page 2022
[3]

Cross-domain human activity recognition using low- resolution infrared sensors,

G. Diazet al., “Cross-domain human activity recognition using low- resolution infrared sensors,”Sensors, vol. 24, no. 19, 2024

work page 2024
[4]

Human activity recognition with low-resolution infrared array sensor using semi-supervised cross-domain neural networks for indoor environment,

C. Yinet al., “Human activity recognition with low-resolution infrared array sensor using semi-supervised cross-domain neural networks for indoor environment,”IEEE Internet of Things Journal, vol. 10, no. 13, pp. 11 761–11 772, 2023

work page 2023
[5]

Deep learning with edge computing: A review,

J. Chen and X. Ran, “Deep learning with edge computing: A review,” Proceedings of the IEEE, vol. 107, no. 8, pp. 1655–1674, 2019

2019
[6]

A novel hybrid deep learning model for activity detection using wide-angle low-resolution infrared array sensor,

K. A. Muthukumaret al., “A novel hybrid deep learning model for activity detection using wide-angle low-resolution infrared array sensor,” IEEE Access, vol. 9, pp. 82 563–82 576, 2021

work page 2021
[7]

Low-cost thermal camera-based counting occupancy meter facilitating energy saving in smart buildings,

M. Kraftet al., “Low-cost thermal camera-based counting occupancy meter facilitating energy saving in smart buildings,”Energies, vol. 14, no. 15, 2021

work page 2021
[8]

Edge computing with embedded ai: Thermal image analysis for occupancy estimation in intelligent buildings,

A. Metwalyet al., “Edge computing with embedded ai: Thermal image analysis for occupancy estimation in intelligent buildings,” inINTESA 2019

work page 2019

Show all 19 references

[9]

Hw-sw optimization of dnns for privacy-preserving people counting on low-resolution infrared arrays,

M. Rissoet al., “Hw-sw optimization of dnns for privacy-preserving people counting on low-resolution infrared arrays,” inDATE 2024

2024
[10]

Prototypical networks for few-shot learning,

J. Snellet al., “Prototypical networks for few-shot learning,” inNeurIPS, 2017, p. 4080–4090

2017
[11]

Distance-based image classification: Generalizing to new classes at near-zero cost,

T. Mensinket al., “Distance-based image classification: Generalizing to new classes at near-zero cost,”IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 35, no. 11, pp. 2624–2637, 2013

2013
[12]

Plinio: A user-friendly library of gradient-based methods for complexity-aware dnn optimization,

D. Jahier Pagliariet al., “Plinio: A user-friendly library of gradient-based methods for complexity-aware dnn optimization,” inFDL, 2023

2023
[13]

Pact: Parameterized clipping activation for quantized neural networks,

J. Choiet al., “Pact: Parameterized clipping activation for quantized neural networks,” 2018

2018
[14]

Optuna: A next-generation hyperparameter optimization framework,

T. Akibaet al., “Optuna: A next-generation hyperparameter optimization framework,” inSIGKDD, 2019

2019
[15]

End-to-end incremental learning,

F. M. Castroet al., “End-to-end incremental learning,” inECCV, 2018

2018
[16]

Latent replay for real-time continual learning,

L. Pellegriniet al., “Latent replay for real-time continual learning,” in IROS, 2020, pp. 10 203–10 209

2020
[17]

Train-on-request: An on-device continual learning work- flow for adaptive real-world brain machine interfaces,

L. Meiet al., “Train-on-request: An on-device continual learning work- flow for adaptive real-world brain machine interfaces,” inBioCAS, 2024

2024
[18]

Tinyol: Tinyml with online-learning on microcontrollers,

H. Renet al., “Tinyol: Tinyml with online-learning on microcontrollers,” inIJCNN, 2021

2021
[19]

Overcoming catastrophic forgetting in incremental few- shot learning by finding flat minima,

G. Shiet al., “Overcoming catastrophic forgetting in incremental few- shot learning by finding flat minima,” inNeurIPS, 2021

2021

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [1]

Efficient deep learning models for privacy-preserving people counting on low-resolution infrared arrays,

C. Xieet al., “Efficient deep learning models for privacy-preserving people counting on low-resolution infrared arrays,”IEEE Internet of Things Journal, vol. 10, no. 15, pp. 13 895–13 907, 2023

work page 2023

[2] [2]

Low-resolution infrared array sensor for counting and localizing people indoors: When low end technology meets cutting edge deep learning techniques,

M. Bouaziziet al., “Low-resolution infrared array sensor for counting and localizing people indoors: When low end technology meets cutting edge deep learning techniques,”Information, vol. 13, no. 3, 2022

work page 2022

[3] [3]

Cross-domain human activity recognition using low- resolution infrared sensors,

G. Diazet al., “Cross-domain human activity recognition using low- resolution infrared sensors,”Sensors, vol. 24, no. 19, 2024

work page 2024

[4] [4]

Human activity recognition with low-resolution infrared array sensor using semi-supervised cross-domain neural networks for indoor environment,

C. Yinet al., “Human activity recognition with low-resolution infrared array sensor using semi-supervised cross-domain neural networks for indoor environment,”IEEE Internet of Things Journal, vol. 10, no. 13, pp. 11 761–11 772, 2023

work page 2023

[5] [5]

Deep learning with edge computing: A review,

J. Chen and X. Ran, “Deep learning with edge computing: A review,” Proceedings of the IEEE, vol. 107, no. 8, pp. 1655–1674, 2019

2019

[6] [6]

A novel hybrid deep learning model for activity detection using wide-angle low-resolution infrared array sensor,

K. A. Muthukumaret al., “A novel hybrid deep learning model for activity detection using wide-angle low-resolution infrared array sensor,” IEEE Access, vol. 9, pp. 82 563–82 576, 2021

work page 2021

[7] [7]

Low-cost thermal camera-based counting occupancy meter facilitating energy saving in smart buildings,

M. Kraftet al., “Low-cost thermal camera-based counting occupancy meter facilitating energy saving in smart buildings,”Energies, vol. 14, no. 15, 2021

work page 2021

[8] [8]

Edge computing with embedded ai: Thermal image analysis for occupancy estimation in intelligent buildings,

A. Metwalyet al., “Edge computing with embedded ai: Thermal image analysis for occupancy estimation in intelligent buildings,” inINTESA 2019

work page 2019