arxiv: 2604.22834 · v1 · submitted 2026-04-20 · 💻 cs.CV · cs.LG

Recognition: unknown

WebSerial Vision Training for Microcontrollers: A Browser-Based Companion to On-Device CNN Training

Jeremy Ellis

Authors on Pith no claims yet

Pith reviewed 2026-05-10 05:25 UTC · model grok-4.3

classification 💻 cs.CV cs.LG

keywords TinyMLBrowser-based MLESP32WebSerialVision trainingLocal CNNMicrocontroller deploymentZero-install

0 comments

The pith

A browser application enables fully local training and deployment of vision models on microcontrollers in under ten minutes.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces a single-file browser application that acts as a companion to on-device firmware for training custom vision classifiers on the XIAO ESP32-S3 Sense hardware. It manages the entire pipeline from firmware flashing via WebSerial, image capture using webcam or the device's camera, CNN training with TensorFlow.js, weight export, and live activation visualization, all without any installation or external data transmission. This setup allows educators, small businesses, and researchers to create task-specific models under their precise deployment conditions. Browser-side training completes a three-class example in about one minute compared to nine minutes on-device, shortening the full collect-train-deploy cycle to under ten minutes while keeping everything private and local.

Core claim

The paper claims that webmcu-vision-web provides a private, fully local machine learning pipeline from firmware flashing through image collection, CNN training, weight export, and live activation visualization for the Seeed Studio XIAO ESP32-S3 Sense, enabling a complete collect-train-deploy cycle in under 10 minutes with browser training completing a three-class run in approximately 1 minute versus 9 minutes on-device.

What carries the argument

webmcu-vision-web, the single-file browser application that uses WebSerial for device communication and TensorFlow.js for in-browser CNN training to coordinate the full local pipeline including config.json live-sync and weight export.

If this is right

Hyperparameters can be adjusted live through config.json synchronization without any firmware recompilation.
Weights export as myWeights.bin and myWeights.h files ready for direct microcontroller deployment.
A confusion matrix and live Conv2 activation heatmap streamed from the device provide immediate performance feedback.
Five-run evaluation on the 0Blank, 1Cup, 2Pen reference problem shows stable convergence with reported mean accuracy and standard deviation.
The single-file design supports zero-install use in any Chromium browser for fully local operation.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The repository template could support rapid adaptation to new microcontrollers or tasks through similar browser-based interfaces.
This local pipeline reduces reliance on cloud services for small-scale embedded vision projects.
Educators might integrate the system into hands-on machine learning lessons without requiring complex setup.
Extensions could add support for additional model architectures while preserving the under-10-minute cycle.

Load-bearing premise

The system assumes that the referenced on-device Arduino firmware operates correctly alongside the browser companion and that WebSerial and TensorFlow.js function reliably in Chromium-based browsers on the target hardware.

What would settle it

A direct test on the XIAO ESP32-S3 Sense that fails to complete firmware flashing, image collection, browser training, export, and deployment in under 10 minutes with stable accuracy on the three-class reference task would falsify the central claim.

Figures

Figures reproduced from arXiv: 2604.22834 by Jeremy Ellis.

**Figure 1.** Figure 1: Data flow between webmcu-vision-web and the ESP32-S3 firmware. All communication passes over a single USB serial connection at 115200 baud via the browser WebSerial API. Six channels are multiplexed over this connection: camera frames (inbound), SD file operations (bidirectional), weight and config files (outbound), heatmap activation frames (inbound), and firmware flash (outbound). No data leaves the loc… view at source ↗

**Figure 2.** Figure 2: The XIAO ML Kit showing the ESP32-S3 Sense board with integrated OV2640 camera, [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗

**Figure 3.** Figure 3: The complete webmcu-vision-web interface (index.html). All sections are visible on a single scrolling page with no navigation or routing. The six numbered sections (Flash Firmware, WebSerial & SD Browser, Camera & Data Collection, Training Progress, ESP32 Weight Export, and Activity Log) are self-contained and independently usable. The entire source is auditable in the browser developer tools. 9 [PITH_FUL… view at source ↗

**Figure 4.** Figure 4: Section 1: In-browser firmware flashing via [PITH_FULL_IMAGE:figures/full_fig_p010_4.png] view at source ↗

**Figure 5.** Figure 5: The config.json textarea in Section 5, kept live in sync with all training fields. The config.json file can be edited in the SD card browser (Section 2) and saved back to the device, or downloaded to the PC project folder and copied to the microSD card manually. The same JSON is read by the ESP32 firmware at boot, making it the single shared configuration source between the browser and the device. 12 [PIT… view at source ↗

**Figure 6.** Figure 6: Section 3: Camera data collection. The practitioner captures images of their actual [PITH_FULL_IMAGE:figures/full_fig_p013_6.png] view at source ↗

**Figure 7.** Figure 7: Example training images from the three-class reference dataset (0Blank, 1Cup, 2Pen) at [PITH_FULL_IMAGE:figures/full_fig_p014_7.png] view at source ↗

**Figure 8.** Figure 8: Confusion matrix from a three-class training run (0Blank, 1Cup, 2Pen) at 64 [PITH_FULL_IMAGE:figures/full_fig_p015_8.png] view at source ↗

**Figure 9.** Figure 9: Section 5: Weight and configuration export. [PITH_FULL_IMAGE:figures/full_fig_p016_9.png] view at source ↗

**Figure 10.** Figure 10: Live Conv2 activation heatmap streamed from the ESP32 during inference. Red = high [PITH_FULL_IMAGE:figures/full_fig_p018_10.png] view at source ↗

**Figure 11.** Figure 11: Training loss curves across five independent runs (100 epochs each, [PITH_FULL_IMAGE:figures/full_fig_p020_11.png] view at source ↗

**Figure 12.** Figure 12: Validation accuracy across five runs. The validation split is held out at the start of each [PITH_FULL_IMAGE:figures/full_fig_p020_12.png] view at source ↗

**Figure 13.** Figure 13: Epoch at which each run first reaches a stable convergence threshold. Early convergence [PITH_FULL_IMAGE:figures/full_fig_p021_13.png] view at source ↗

**Figure 14.** Figure 14: Validation accuracy stability (variance across the last 10 epochs of each run) across [PITH_FULL_IMAGE:figures/full_fig_p021_14.png] view at source ↗

**Figure 15.** Figure 15: Validation accuracy (computed on held-out training-set images) versus real-world in [PITH_FULL_IMAGE:figures/full_fig_p022_15.png] view at source ↗

**Figure 16.** Figure 16: Live inference on the cup class under normal lighting: raw camera image (left), Conv2 [PITH_FULL_IMAGE:figures/full_fig_p023_16.png] view at source ↗

**Figure 17.** Figure 17: Live inference on the cup class under very low ambient light: raw camera image (left), [PITH_FULL_IMAGE:figures/full_fig_p023_17.png] view at source ↗

read the original abstract

This paper presents webmcu-vision-web, a single-file, zero-install browser application for end-to-end TinyML vision model training and deployment on the Seeed Studio XIAO ESP32-S3 Sense (XIAO ML Kit, $15--40 USD). Acting as a browser-based companion to the on-device Arduino firmware of Paper 1 [1], it provides a private, fully local machine learning pipeline, from firmware flashing through image collection, CNN training, weight export, and live activation visualization, without any software installation beyond a Chromium-based browser. The system targets educators, small businesses, and researchers who need to train task-specific visual classifiers under their exact deployment conditions. Key capabilities include: in-browser firmware flashing via esptool-js; an SD card file browser with image preview and inline editing; config.json live-sync for zero-recompile hyperparameter adjustment; webcam and ESP32 OV2640 camera image capture; TensorFlow.js CNN training completing a three-class run (~30 images per class, 20 epochs) in approximately 1 minute browser-side versus 9 minutes on-device, enabling a complete collect-train-deploy cycle in under 10 minutes; weight export as myWeights.bin and myWeights.h; confusion matrix; and a live Conv2 activation heatmap streamed from the ESP32 during inference. No data leaves the local machine at any stage. A five-run consistency evaluation on the three-class reference problem (0Blank, 1Cup, 2Pen) demonstrates stable convergence with mean accuracy and standard deviation reported; all artefacts are released at the repository link below. The repository is a living template for LLM-assisted adaptation to new hardware and tasks. All source code is MIT-licensed at https://github.com/webmcu-ai/webmcu-vision-web.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

This is a practical single-file browser tool that speeds up local collect-train-deploy cycles for a specific ESP32 board, but it is mostly integration work on top of the author's prior firmware.

read the letter

The main point is a zero-install web app that handles firmware flashing via esptool-js, image capture from webcam or the board's camera, TensorFlow.js CNN training, weight export as .bin and .h files, and live Conv2 activation streaming back from the device. It keeps everything private and local, with config.json sync so you avoid recompiles, and it claims a full three-class cycle in under 10 minutes (1 minute browser training versus 9 on-device). The five-run consistency check on the 0Blank/1Cup/2Pen task shows stable mean accuracy with reported standard deviation, and the MIT-licensed repo is framed as a template for adapting to new hardware or tasks with LLM help. That combination of live visualization, SD card browser, and seamless export is the concrete deliverable that works as described in the abstract. The implementation is complete enough to be immediately usable for the target audience. The soft spots are that the on-device training logic and firmware come from the referenced prior paper, so the novelty is in the browser glue and UI rather than new algorithms or theory. The evaluation stays narrow—just five runs on one simple problem with no full hyperparameter details, architecture diagram, or comparison to other TinyML flows. It assumes WebSerial and the XIAO ESP32-S3 Sense behave reliably across browsers and lighting conditions, which the open repo helps mitigate but does not fully prove. This is aimed at educators, makers, and small teams who want a quick private pipeline for custom vision classifiers on that $15-40 board. Readers looking for a ready template will get value; those seeking advances in on-device learning methods will not. I would send it to peer review in an applied tools or education track so the implementation can be checked against real hardware use.

Referee Report

1 major / 2 minor

Summary. The paper presents webmcu-vision-web, a single-file, zero-install browser application for end-to-end TinyML vision model training and deployment on the Seeed Studio XIAO ESP32-S3 Sense. It acts as a companion to prior on-device Arduino firmware, providing local firmware flashing, image collection, TensorFlow.js CNN training, weight export, confusion matrix, and live activation visualization, with a claimed complete collect-train-deploy cycle in under 10 minutes for a three-class task, supported by a five-run consistency evaluation showing stable convergence.

Significance. If the functionality and performance claims hold, the work provides a valuable practical tool for educators, small businesses, and researchers to develop task-specific visual classifiers on affordable microcontrollers in a fully private, local manner. The release of all artifacts under MIT license, including the repository as a living template for LLM-assisted adaptation, enhances reproducibility and extensibility, addressing a gap in accessible TinyML workflows.

major comments (1)

[Five-run consistency evaluation (abstract and results)] The five-run consistency evaluation reports mean accuracy and standard deviation for the three-class reference problem (0Blank, 1Cup, 2Pen) but provides no details on the experimental protocol, including image collection procedure, train/test splits, hyperparameter settings, or sources of variance across runs. This information is required to substantiate the stability and timing claims that support the central end-to-end pipeline assertion.

minor comments (2)

[Abstract] The reference to 'Paper 1 [1]' for the on-device firmware should include a complete bibliographic entry in the references section to enable independent access to the foundational implementation.
[Repository and artifacts] The manuscript would benefit from specifying the exact commit or release version of the repository corresponding to the reported five-run results and timing measurements.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for the positive evaluation of our work and the recommendation for minor revision. We address the single major comment below.

read point-by-point responses

Referee: [Five-run consistency evaluation (abstract and results)] The five-run consistency evaluation reports mean accuracy and standard deviation for the three-class reference problem (0Blank, 1Cup, 2Pen) but provides no details on the experimental protocol, including image collection procedure, train/test splits, hyperparameter settings, or sources of variance across runs. This information is required to substantiate the stability and timing claims that support the central end-to-end pipeline assertion.

Authors: We agree that the manuscript lacks a detailed description of the experimental protocol supporting the five-run consistency evaluation. In the revised version we will add a dedicated subsection in the Results section that specifies the image collection procedure (using the browser webcam and ESP32 OV2640 interfaces to acquire the reference images), the train/test split strategy, the TensorFlow.js CNN hyperparameters, and the sources of variance (different random seeds for initialization and shuffling) across the five independent runs. This addition will allow readers to reproduce the reported mean accuracy and standard deviation and will strengthen the claims regarding pipeline stability and timing. revision: yes

Circularity Check

0 steps flagged

No significant circularity detected

full rationale

The manuscript is a systems description of an open-source browser tool implementing local flashing, TF.js training, weight export, and visualization. Its core claims (under-10-minute end-to-end cycle, five-run consistency on the reference three-class task) rest on direct empirical timing measurements and released code rather than any mathematical derivation, fitted parameter renamed as prediction, or uniqueness theorem. The single self-reference to Paper 1 simply identifies the complementary on-device firmware the web tool is designed to work with; that citation is not load-bearing for the new tool's architecture, performance numbers, or privacy guarantees, all of which are independently implemented and tested here. No equations, ansatzes, or predictions reduce to the paper's own inputs by construction.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The paper is a system description relying on standard web technologies rather than new mathematical derivations; no free parameters or invented entities are introduced.

axioms (1)

domain assumption Chromium-based browsers reliably support WebSerial API, esptool-js, and TensorFlow.js for local operations without external dependencies.
Invoked for the zero-install flashing, training, and visualization pipeline to function as described.

pith-pipeline@v0.9.0 · 5625 in / 1314 out tokens · 60401 ms · 2026-05-10T05:25:23.652721+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

11 extracted references · 3 canonical work pages · 1 internal anchor

[1]

On-Device Vision Training, Deployment, and Inference on a Thumb-Sized Microcon- troller,

J. Ellis, “On-Device Vision Training, Deployment, and Inference on a Thumb-Sized Microcon- troller,” arXiv preprint, April 2026.https://github.com/webmcu-ai/on-device-vision-ai

2026
[2]

Adam: A Method for Stochastic Optimization

D. P. Kingma and J. Ba, “Adam: A Method for Stochastic Optimization,”Proc. Int. Conf. Learning Representations (ICLR), 2015.https://arxiv.org/abs/1412.6980

work page internal anchor Pith review Pith/arXiv arXiv 2015
[3]

Edge Impulse: Machine Learning for Embedded Systems,

Edge Impulse Inc., “Edge Impulse: Machine Learning for Embedded Systems,” 2024.https: //edgeimpulse.com

2024
[4]

Teachable Machine,

Google, “Teachable Machine,” 2024.https://teachablemachine.withgoogle.com

2024
[5]

Arduino IDE,

Arduino, “Arduino IDE,” 2024.https://www.arduino.cc/en/software

2024
[6]

TensorFlow Lite for Microcontrollers,

TensorFlow Authors, “TensorFlow Lite for Microcontrollers,” 2024.https://www. tensorflow.org/lite/microcontrollers

2024
[7]

Web Serial API,

W3C Web Incubator Community Group, “Web Serial API,” 2024.https://wicg.github. io/serial/

2024
[8]

TinyML4D: Scaling Embedded Machine Learn- ing Education in the Developing World,

B. Plancher, S. B¨ uttrich, J. Ellis, et al., “TinyML4D: Scaling Embedded Machine Learn- ing Education in the Developing World,”Proc. AAAI Symposium Series, pp. 508–515, 2024. https://doi.org/10.1609/aaaiss.v3i1.31265

work page doi:10.1609/aaaiss.v3i1.31265 2024
[9]

TensorFlow.js: Machine Learning for the Web and Beyond,

M. Smilkov et al., “TensorFlow.js: Machine Learning for the Web and Beyond,” arXiv preprint arXiv:1901.05350, 2019.https://arxiv.org/abs/1901.05350

work page arXiv 1901
[10]

esptool-js: JavaScript implementation of esptool,

Espressif Systems, “esptool-js: JavaScript implementation of esptool,” GitHub, 2024.https: //github.com/espressif/esptool-js

2024
[11]

ESP-DL: Espressif Deep Learning Library,

Espressif Systems, “ESP-DL: Espressif Deep Learning Library,” GitHub, 2024.https:// github.com/espressif/esp-dl 30

2024