Benchmarking TinyML Systems: Challenges and Direction
read the original abstract
Recent advancements in ultra-low-power machine learning (TinyML) hardware promises to unlock an entirely new class of smart applications. However, continued progress is limited by the lack of a widely accepted benchmark for these systems. Benchmarking allows us to measure and thereby systematically compare, evaluate, and improve the performance of systems and is therefore fundamental to a field reaching maturity. In this position paper, we present the current landscape of TinyML and discuss the challenges and direction towards developing a fair and useful hardware benchmark for TinyML workloads. Furthermore, we present our four benchmarks and discuss our selection methodology. Our viewpoints reflect the collective thoughts of the TinyMLPerf working group that is comprised of over 30 organizations.
This paper has not been read by Pith yet.
Forward citations
Cited by 5 Pith papers
-
AdvScan: Black-Box Adversarial Example Detection at Runtime through Power Analysis
AdvScan detects adversarial examples in black-box TinyML on ARM Cortex-M devices via one-sample t-test on runtime power signatures against a benign baseline, reporting 99.984% detection with 40 false negatives and zer...
-
Running hardware-aware neural architecture search on embedded devices under 512MB of RAM
A HW-NAS framework executable on resource-limited embedded devices generates optimized CNNs for low-end MCUs and reports state-of-the-art human-recognition accuracy on the Visual Wake Word dataset.
-
Fully Autonomous Z-Score-Based TinyML Anomaly Detection on Resource-Constrained MCUs Using Power Side-Channel Data
A Z-score anomaly detector trained and inferred fully on an STM32 microcontroller using power side-channel RMS data achieves perfect precision and recall on a 14-day fridge dataset with low memory and latency.
-
Network-Adaptive Cloud Processing for Visual Neuroprostheses
Network-adaptive encoding reduces end-to-end latency in cloud-based visual preprocessing for neuroprostheses during congestion while preserving global scene structure at the cost of sharper boundary degradation.
-
ArrythML: An Autoencoder-Based TinyML Approach for On-Device Arrhythmia Detection on Resource-Constrained Embedded Systems
Reports INT8 autoencoder TinyML models for on-device arrhythmia detection from ECG, achieving 84% recall and 9 ms latency on ESP32 after filtering ambiguous cases.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.