REVIEW 4 major objections 8 minor 46 references
Inertial classification accuracy grows as a log of training size, so small pilot runs can predict how much data you need.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
Classification accuracy on inertial HAR and SLR tasks follows a consistent logarithmic growth with training-set size, enabling a MAPD-based stability-point metric that often saturates far below traditional heuristics.
T0 review reviewed 2026-07-13 challenge →
load-bearing objection Solid multi-dataset empirical regularity for inertial HAR/SLR learning curves, with a usable but internally defined stability metric that is not independently predictive. the 4 major comments →
Data-Efficient Deep Learning: Empirical Guidelines for Training Set Size Estimation in Inertial Sensor Classification
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
Core claim
Classification accuracy on inertial sensor tasks follows a consistent logarithmic growth pattern with training-set size, independent of whether the problem is binary or multi-class and across six diverse real-world datasets; a MAPD-defined stability point therefore lets researchers extrapolate asymptotic performance from a short sequence of pilot runs.
What carries the argument
The logarithmic learning-curve model Accuracy(n) = a log(n) + b together with the stability point N*, the smallest number of sample-size increments whose fitted curve stays inside a pre-chosen MAPD tolerance of the reference curve fitted on almost all points.
Load-bearing premise
That a log curve fitted on a few early pilot points, using fixed MAPD tolerances chosen in advance, will still describe the true asymptotic accuracy of a new inertial classification task whose class mix or sensors differ from the six datasets studied.
What would settle it
Train the same fixed architecture on a fresh inertial classification dataset never used in the paper; if the first handful of sample-size points produce a log fit whose MAPD to the eventual full curve exceeds the paper's stated 1–5 % tolerances, the claimed generalizability fails.
If this is right
- HAR and SLR campaigns can be sized from small pilot studies instead of collecting the entire corpus up front.
- Recording effort can be stopped once the stability point is reached, avoiding redundant annotation cost.
- The same log-plus-MAPD procedure can be reused to compare data efficiency of different sensor placements or label granularities.
- Traditional fixed-ratio heuristics (e.g., ten samples per parameter) can be replaced by an empirical, task-specific stability threshold.
Where Pith is reading between the lines
- If the log regime is truly data-driven rather than architecture-driven, the same pilot procedure should transfer to other bounded-accuracy sensor tasks such as gesture recognition or gait mode detection.
- A natural next measurement is whether the fitted slope a itself predicts how many more classes a dataset can support before the stability point jumps.
- The framework could be inverted: given a target accuracy and a known slope from a pilot, solve directly for the minimal n that meets a chosen MAPD budget.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper empirically studies how classification accuracy scales with training-set size for inertial HAR and SLR. Across six public datasets (102.7 h, 142 subjects), a fixed CNN-BiLSTM (plus a limited 3-layer CNN ablation) is trained on a shared grid of subsample sizes under both binary and multi-class labelings. The authors report that test accuracy is well described by Accuracy(n)=a log(n)+b, introduce a MAPD-based “stability point” N* relative to a reference log fit on the same grid, and argue that this lets practitioners extrapolate total data needs from small pilot runs, often with fewer samples than classical heuristics suggest.
Significance. If the reported log regime and pilot-to-full-curve stability hold more broadly, the work would give HAR/SLR practitioners a concrete, low-cost planning tool for recording campaigns—an area where labeled IMU data are expensive and sample-size guidance is currently heuristic. Strengths include the multi-dataset scope, parallel binary/multi-class design, fixed architecture/protocol that isolates n, public data, and an explicit (if limited) architecture ablation that preserves the qualitative pattern. The contribution is empirical and engineering-oriented rather than theoretical; its value rests on how reliably the pilot procedure predicts performance at larger n and on new tasks.
major comments (4)
- Abstract and §3.4 define the stability point via deviation from an “asymptotic maximum,” yet the working model Accuracy(n)=a log(n)+b (Eq. 1 / Eq. 19) is unbounded and has no asymptote. For large n the fit can (and in several multi-class tables would) exceed 100%. The operational reference is only the log fit on the first nine grid points of the same experiment, not a saturating limit. Either replace the log form with a bounded model (e.g., A−B/n^α or a logistic-in-log form) or redefine the target as “accuracy at the largest observed n / full training pool” and drop asymptotic language throughout.
- §3.4 and Eq. (20): MAPD is computed between a candidate log fit on the first N grid points and a reference log fit on the first nine points of the same run, evaluated only on those same ten n-values. This certifies internal consistency of successive log fits, not that a pilot-derived curve predicts true accuracy at held-out larger n or on a new dataset. The central claim that total data requirements can be extrapolated from small pilots therefore lacks an independent validation design (e.g., fit on n≤n_pilot, evaluate on n>n_pilot or on a held-out subject/dataset split). Without that, N* should be framed as “when the pilot log fit stabilizes relative to the full-grid log fit,” not as a validated estimator of the task’s data needs.
- §3 states that power-law, exponential, and polynomial forms “failed to accurately capture” growth once model-parameter dependence was removed, while the log form was best, but no quantitative comparison (R², residual plots, AIC/BIC, or MAPD tables for alternative forms) appears in the results. Given that the log choice underpins every stability table and the extrapolation framework, the paper needs a short, reproducible model-selection section (at least on the six multi-class curves) showing why log is preferred over saturating alternatives commonly used for bounded accuracy.
- The claim that models “reach practical stability with substantially fewer samples than traditional heuristics suggest” (abstract, highlights, conclusions) is not quantified against any stated baseline. The Rule of 10 is mentioned in the introduction but never applied to the CNN-BiLSTM parameter count or to the reported window counts. Either compute an explicit comparison (e.g., parameters × 10 vs. observed N* sample counts) or soften the claim to “stability often occurs well below the full training pool on these grids.”
minor comments (8)
- §3.3.1 heading “Data perpetration” is a typo for “Data preparation.”
- §3.2 item 2: “Conventional frontend” should be “Convolutional frontend.”
- Abstract opening: “Deep learning models dependency” → “models’ dependency” (or “dependence”).
- Fitted coefficients a,b are reported without standard errors or confidence intervals; even bootstrap CIs over the R subsampling runs would strengthen Tables 2–15.
- Ablation (§4.3) covers only UCI-HAR and PAMAP2 multi-class with a 3-layer CNN; a one-sentence caveat that architecture coverage remains narrow would match the conclusions’ own limitation language.
- Sample-size grids (binary ~10k–100k, multi-class ~35k–350k) are described as “e.g.” fractions of the smallest dataset; state the exact n_i list once (table or appendix) so others can reproduce the MAPD analysis.
- Figures 2–15 would be easier to read if the reference curve and the N* curve were highlighted distinctly from the intermediate N fits (many panels are dense).
- Related-work citations on learning curves / scaling laws are appropriate; a brief pointer to saturating learning-curve models used in clinical tabular work (beyond Silvey & Liu) would help situate the log choice.
Circularity Check
Stability point is defined by MAPD of pilot log-fits against a reference log-fit on the same ten-point grid, so the claimed pilot-to-asymptote extrapolation is an internal consistency check, not an independent prediction.
specific steps
-
fitted input called prediction
[Section 3.4, Eqs. (19)–(20) and definition of N*]
"The reference curve y*(n) is therefore fitted on the first nine points … For each prefix length N∈{2,…,10}, a candidate curve ŷ_N(n) is fitted using only the first N points. Both ŷ_N and y* are then evaluated at all ten sample sizes … MAPD(ŷ_N,y*)=… The stability point N* is the smallest N for which the MAPD falls within a predefined tolerance."
N* is declared the sample size at which a pilot already “predicts” the full learning curve, yet the sole success criterion is agreement (MAPD) with a reference log-fit constructed from (almost) the identical ten accuracy points. No larger held-out n, no external dataset, and no true asymptotic accuracy independent of the same log ansatz is ever used; the pilot-to-full extrapolation is therefore statistically forced by the internal consistency of successive fits on the observed grid.
-
self definitional
[Abstract & Section 3.4 (stability-point definition)]
"we propose a quantitative stability point metric, defined as the sample size required for the learning curve to stabilize within a predefined mean absolute percentage deviation of its asymptotic maximum."
The “asymptotic maximum” against which stability is measured is itself obtained from the logarithmic fit Accuracy(n)=a log(n)+b performed on the same experimental grid; the metric therefore compares early fits of the assumed form to a late fit of the same form, rendering the claimed saturation point definitional rather than an independent observation of true performance saturation.
full rationale
The paper is an honest empirical curve-fitting study: it selects the log form after comparing candidates on the six public datasets, reports that Accuracy(n)=a log(n)+b fits well for both binary and multi-class tasks, and shows that successive prefix fits converge quickly under fixed MAPD tolerances. No first-principles derivation or uniqueness theorem is claimed, and the six corpora plus the CNN ablation supply independent empirical content. The only circularity is methodological and local to the stability metric itself. Section 3.4 explicitly constructs the reference curve y* from the first nine of the same ten grid points used for the candidates, then defines N* as the smallest prefix whose MAPD to that reference falls below a pre-chosen τ. Consequently the statement “a pilot of size N* already predicts the full learning curve” is true by construction of MAPD on the observed grid; it does not constitute an out-of-sample test of the extrapolated asymptote on larger n or on a held-out dataset. Because the paper never pretends the log form is theoretically forced and openly notes that the tenth point is not an independent check, the circularity remains partial and does not collapse the central empirical claim. Score 3 reflects one clear fitted-input-called-prediction step that is load-bearing for the “extrapolate from pilots” slogan yet does not render the whole result tautological.
Axiom & Free-Parameter Ledger
free parameters (3)
- a, b (per dataset/scenario log fit)
- MAPD tolerance thresholds (τ_lenient/strict)
- Sample-size grids (binary 10 k–100 k, multi-class 35 k–350 k)
axioms (3)
- ad hoc to paper Bounded classification accuracy on inertial windows is adequately described by Accuracy(n) = a log(n) + b once model-parameter dependence is removed.
- domain assumption A fixed CNN-BiLSTM (or 3-layer CNN) architecture and Adam protocol isolate the effect of training-set size.
- domain assumption Public HAR/SLR datasets (PAMAP2, MotionSense, MobilePos, REALDISP, UCI-HAR, WISDM) are representative of the broader inertial classification domain.
invented entities (1)
-
Stability point N* (MAPD-based)
no independent evidence
Cite this review
Pith. "Pith review of Data-Efficient Deep Learning: Empirical Guidelines for Training Set Size Estimation in Inertial Sensor Classification." pith.science (2026). https://pith.science/paper/A75QFQLD
@misc{pith2026260709402,
author = {Pith},
title = {Pith review of: Data-Efficient Deep Learning: Empirical Guidelines for Training Set Size Estimation in Inertial Sensor Classification},
year = {2026},
howpublished = {\url{https://pith.science/paper/A75QFQLD}},
note = {Machine review of arXiv:2607.09402}
}
read the original abstract
Deep learning models dependency on large-scale inertial datasets presents a significant bottleneck in inertial sensor-based classification tasks, such as human activity recognition and smartphone location recognition. In these domains, data collection requires massive recording campaigns that are complex, time-consuming, and difficult to scale. Currently, data-driven guidelines for determining the minimum sample size required to reach a desired accuracy level do not exist. To address this gap, this study presents a systematic empirical evaluation of learning curve convergence rates in inertial classification. We introduce a unified framework that analyzes classification performance under both binary and multi-class scenarios, and derive an empirical formula to estimate performance relative to dataset size. Testing across six diverse, real-world datasets totaling 102.7 hours of inertial measurements demonstrates that accuracy follows a consistent logarithmic growth pattern, regardless of task complexity. Leveraging this finding, we propose a quantitative stability point metric, defined as the sample size required for the learning curve to stabilize within a predefined mean absolute percentage deviation of its asymptotic maximum. Our analysis reveals that models often reach practical stability with substantially fewer samples than traditional heuristics suggest. Ultimately, we offer a generalizable framework to extrapolate total data requirements from small-scale pilot studies, optimizing the tradeoff between recording effort and model reliability. These findings shift the prevailing paradigm from maximizing data volume toward optimizing data efficiency, offering concrete, data-backed guidelines for planning recording campaigns in inertial sensing applications.
Figures
Reference graph
Works this paper leans on
-
[1]
Cortes, Corinna and Jackel, L. D. and Solla, Sara A. and Vapnik, Vladimir and Denker, John S. , title =. Advances in Neural Information Processing Systems , year =
-
[2]
arXiv preprint arXiv:2102.04074 , year =
Hutter, Marcus , title =. arXiv preprint arXiv:2102.04074 , year =
-
[3]
BMC bioinformatics , volume=
Evaluation of a decided sample size in machine learning applications , author=. BMC bioinformatics , volume=. 2023 , publisher=
2023
-
[4]
Sample size in machine learning and artificial intelligence , author=. Uab. edu. Available online: https://sites. uab. edu/periop-datascience/2021/06/28/sample-size-in-machine-learning-and-artificial-intelligence/(accessed on 30 May 2022) , year=
2021
-
[5]
arXiv preprint arXiv:1909.12673 , year=
A constructive prediction of the generalization error across scales , author=. arXiv preprint arXiv:1909.12673 , year=
Pith/arXiv arXiv 1909
-
[6]
BMC Medical Research Methodology , volume=
Sample size requirements are not being considered in studies developing prediction models for binary outcomes: a systematic review , author=. BMC Medical Research Methodology , volume=. 2023 , publisher=
2023
-
[7]
Journal of Medical Internet Research , volume=
Sample size requirements for popular classification algorithms in tabular clinical data: empirical study , author=. Journal of Medical Internet Research , volume=. 2024 , publisher=
2024
-
[8]
arXiv preprint arXiv:2211.14061 , year=
A survey of learning curves with bad behavior: or how more data need not lead to better performance , author=. arXiv preprint arXiv:2211.14061 , year=
-
[9]
2012 , howpublished =
Reiss, Attila , title =. 2012 , howpublished =
2012
-
[10]
Proceedings of the European conference on computer vision (ECCV) , pages=
RIDI: Robust IMU double integration , author=. Proceedings of the European conference on computer vision (ECCV) , pages=
-
[11]
Proceedings of the international conference on internet of things design and implementation , pages=
Mobile sensor data anonymization , author=. Proceedings of the international conference on internet of things design and implementation , pages=
-
[12]
2015 International conference on electrical engineering and information communication technology (ICEEICT) , pages=
Human activity recognition from wearable sensors using extremely randomized trees , author=. 2015 International conference on electrical engineering and information communication technology (ICEEICT) , pages=. 2015 , organization=
2015
-
[13]
Sensors , volume=
Silva do Monte Lima, Jo. Sensors , volume=. 2019 , publisher=
2019
-
[14]
Proceedings of the IEEE , volume=
Efficient processing of deep neural networks: A tutorial and survey , author=. Proceedings of the IEEE , volume=. 2017 , publisher=
2017
-
[15]
Towards Data Sci , volume=
Activation functions in neural networks , author=. Towards Data Sci , volume=
-
[16]
Proceedings of the 27th international conference on machine learning (ICML-10) , pages=
A theoretical analysis of feature pooling in visual recognition , author=. Proceedings of the 27th international conference on machine learning (ICML-10) , pages=
-
[17]
Neural computation , volume=
Long short-term memory , author=. Neural computation , volume=. 1997 , publisher=
1997
-
[18]
Framewise Phoneme Classification with Bidirectional
Graves, Alex and Schl. Framewise Phoneme Classification with Bidirectional. Proceedings of the IEEE International Joint Conference on Neural Networks (IJCNN) , year =
-
[19]
The journal of machine learning research , volume=
Dropout: a simple way to prevent neural networks from overfitting , author=. The journal of machine learning research , volume=. 2014 , publisher=
2014
-
[20]
, title =
Bridle, John S. , title =. Neurocomputing: Algorithms, Architectures and Applications , year =
-
[21]
arXiv preprint arXiv:1412.6980 , year=
Adam: A method for stochastic optimization , author=. arXiv preprint arXiv:1412.6980 , year=
-
[22]
Physical Therapy Reviews , volume=
Cuesta-Vargas, Antonio I and Gal. Physical Therapy Reviews , volume=. 2010 , publisher=
2010
-
[23]
2018 , publisher=
Camomilla, Valentina and Bergamini, Elena and Fantozzi, Silvia and Vannozzi, Giuseppe , journal=. 2018 , publisher=
2018
-
[24]
Sensors , volume=
Smartphone location recognition: A deep learning-based approach , author=. Sensors , volume=. 2019 , publisher=
2019
-
[25]
ACM Computing Surveys (CSUR) , volume=
A survey on deep learning for human activity recognition , author=. ACM Computing Surveys (CSUR) , volume=. 2021 , publisher=
2021
-
[26]
Information Fusion , volume=
Multi-sensor information fusion based on machine learning for real applications in human activity recognition: State-of-the-art and research challenges , author=. Information Fusion , volume=. 2022 , publisher=
2022
-
[27]
Machine Learning and Knowledge Extraction , volume=
A Comprehensive Survey on Deep Learning Methods in Human Activity Recognition , author=. Machine Learning and Knowledge Extraction , volume=. 2024 , publisher=
2024
-
[28]
IEEE Sensors Journal , volume=
A perspective on human activity recognition from inertial motion data , author=. IEEE Sensors Journal , volume=. 2023 , publisher=
2023
-
[29]
Journal of Ambient Intelligence and Humanized Computing , year=
State-of-the-art in human activity recognition based on inertial measurement unit sensors: survey and applications , author=. Journal of Ambient Intelligence and Humanized Computing , year=
-
[30]
Sensors , volume=
Human activity recognition: review, taxonomy and open challenges , author=. Sensors , volume=. 2022 , publisher=
2022
-
[31]
IEEE Communications Magazine , volume=
Information theory, complexity, and neural networks , author=. IEEE Communications Magazine , volume=. 1989 , publisher=
1989
-
[32]
IEEE Transactions on Pattern Analysis and Machine Intelligence , volume=
Small sample size effects in statistical pattern recognition: Recommendations for practitioners , author=. IEEE Transactions on Pattern Analysis and Machine Intelligence , volume=. 1991 , publisher=
1991
-
[33]
Canadian Association of Radiologists Journal , volume=
Sample-size determination methodologies for machine learning in medical imaging research: a systematic review , author=. Canadian Association of Radiologists Journal , volume=. 2019 , publisher=. doi:10.1016/j.carj.2019.06.002 , note=
-
[34]
UCI Machine Learning Repository: WISDM Smartphone and Smartwatch Activity and Biometrics Dataset Data Set , volume=
Wisdm smartphone and smartwatch activity and biometrics dataset , author=. UCI Machine Learning Repository: WISDM Smartphone and Smartwatch Activity and Biometrics Dataset Data Set , volume=. 2019 , publisher=
2019
-
[35]
, author=
A public domain dataset for human activity recognition using smartphones. , author=. Esann , volume=
-
[36]
Results in Engineering , pages=
Inertial navigation meets deep learning: A survey of current trends and future directions , author=. Results in Engineering , pages=. 2024 , publisher=
2024
-
[37]
2025 IEEE/ION Position, Location and Navigation Symposium (PLANS) , pages=
On Neural Inertial Classification Networks for Pedestrian Activity Recognition , author=. 2025 IEEE/ION Position, Location and Navigation Symposium (PLANS) , pages=. 2025 , organization=
2025
-
[38]
Results in Engineering , pages=
Enhancement of neural inertial regression networks: A data-driven perspective , author=. Results in Engineering , pages=. 2026 , publisher=
2026
-
[39]
IEEE Journal of Indoor and Seamless Positioning and Navigation , year=
Optimizing Neural Inertial Classification: A Benchmark Study of Data-Driven Techniques , author=. IEEE Journal of Indoor and Seamless Positioning and Navigation , year=
-
[40]
Applied sciences , volume=
Impact of dataset size on classification performance: an empirical evaluation in the medical domain , author=. Applied sciences , volume=. 2021 , publisher=
2021
-
[41]
Computers and electronics in agriculture , volume=
Impact of dataset size and variety on the effectiveness of deep learning and transfer learning for plant disease classification , author=. Computers and electronics in agriculture , volume=. 2018 , publisher=
2018
-
[42]
Results in Engineering , volume=
Pedestrian inertial navigation: An overview of model and data-driven approaches , author=. Results in Engineering , volume=. 2025 , publisher=
2025
-
[43]
IEEE Sensors Journal , volume=
Attitude adaptive estimation with smartphone classification for pedestrian navigation , author=. IEEE Sensors Journal , volume=. 2021 , publisher=
2021
-
[44]
Malihe Mardanpour and Majid Sepahvand and Fardin Abdali-Mohammadi and Mahya Nikouei and Homeyra Sarabi , keywords =. Human activity recognition based on multiple inertial sensors through feature-based knowledge distillation paradigm , journal =. 2023 , issn =. doi:https://doi.org/10.1016/j.ins.2023.119073 , url =
-
[45]
Machine Learning and Knowledge Extraction , VOLUME =
Kaseris, Michail and Kostavelis, Ioannis and Malassiotis, Sotiris , TITLE =. Machine Learning and Knowledge Extraction , VOLUME =. 2024 , NUMBER =
2024
-
[46]
and et al
Abbas, I. and et al. , title =. PMC , year =
This paper was first reviewed by grok-4.5 on July 13, 2026.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.