{"id":"ba603bb1-0adf-4b61-9b47-c5903b035713","arxiv_id":"2608.13316","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Foundation models for accelerometer data do not consistently beat supervised baselines on standard activity recognition, but some lead on fall/stress detection and as frozen feature extractors.","lead":"This paper benchmarks four open-source accelerometer foundation models against supervised baselines across 19 tasks in activity recognition, fall detection, sleep, stress, and demographics. It finds no consistent foundation-model advantage on standard activity recognition, while some models win on fall/stress and as frozen feature extractors.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline 'no consistent advantage' rests on per-window bootstrap ranks that ignore subject-level clustering; with 8–38 subjects, ties and rank differences may be artifacts.","rationale":"The reader identified the uniform fine-tuning protocol as the weakest assumption. That is a valid secondary concern: UniMTS's last-place fine-tuning rank could indeed be a hyperparameter artifact, and a simple learning-rate or warmup sweep would settle it. However, the single most load-bearing assumption is the statistical ranking procedure itself, because every headline statement—'no consistent advantage,' 'FMs lead on stress,' 'statistically tied'—is formulated in terms of significant ranks. The paper's own text says these ranks come from pairwise bootstrapping on the test set, which, given the small number of subjects per dataset and the patient-respecting splits, amounts to resampling windows within a handful of test subjects. That can produce artificially narrow confidence intervals and false confidence in both differences and ties. The benchmark itself, the released code, and the representation analyses are valuable and independent contributions; the issue is specifically that the central negative conclusion about HAR is an absence-of-evidence claim that requires an equivalence or non-inferiority analysis with subject-level uncertainty. I therefore keep the conditional verdict (UNCHANGED) and recommend adding a subject-level cluster bootstrap plus an explicit equivalence margin before the 'no consistent advantage' claim is accepted.","tokens_in":20902,"tokens_out":6388,"duration_ms":67202,"concrete_test":"Re-run the Section II.C ranking procedure on the released code using cluster bootstrap by subject (resample subjects, keep all their windows) for all 19 tasks, and compare the resulting ranks and ties with published Tables V and VI. Then perform a non-inferiority test on the HAR category with a pre-registered margin (e.g., top-FM minus top-baseline AUROC within -0.01 is considered equivalent) and report subject-level 95% confidence intervals. If rank ties reorder or the intervals include both meaningful FM advantage and disadvantage, the 'no consistent advantage / baselines are sufficient' conclusion is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—'supervised models remain competitive with FMs on HAR, with no consistent advantage for either' (Abstract)—is operationalized through the rank table in Section II.C: 'model rankings within each task were determined through pairwise bootstrapping on the test set (1000 iterations)… shared its rank if the 95% bootstrap interval of the score difference included zero.' This procedure treats windows as independent units. But the datasets have only 8–38 subjects and are split into 8:1:1 or 3:1:1 folds while 'respecting patient assignments,' so a test set can contain just 1–4 subjects. Resampling windows within those few subjects massively underestimates between-subject variability of AUROC/MAE; reported 'statistically significant' rank differences and 'statistical ties' are therefore not established at the subject level. For example, HAR70+ has 18 subjects; with a 3:1:1 split, the test set has roughly 2 subjects, so the 0.915 vs 0.943 differences in Table VI carry huge subject-level uncertainty. The conclusion that supervised baselines are 'sufficient' (Section III-C) is a non-inferiority or equivalence claim, but the paper only reports failure to reject the null of no difference; with small test sets that failure has little evidential weight. Additionally, no correction is applied across the many pairwise model comparisons, inflating the chance of false 'significant' differences. Thus the central 'no consistent advantage' result—and the 'statistical tie' that underlies it—may not survive a valid subject-level analysis.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper benchmarks four open-source accelerometer foundation models (NormWear, UniMTS, Oxford SSL, ElderNet) against four supervised baselines (TinyHAR, Inception1D, XResNet1D, S4) on 19 tasks spanning human activity recognition, fall detection, sleep staging, stress detection, and demographics, using 10 public datasets. Each model is evaluated in linear, frozen, and fine-tuned modes, with additional analyses of sensor-position robustness, computational efficiency, layerwise representational similarity (CKA/CBA), and concept discovery. The central claims are that foundation models show no consistent advantage over supervised baselines on standard HAR tasks, that selected FMs lead on fall and stress detection, that frozen FM representations are strongest in specific categories (notably UniMTS), and that supervised baselines remain sufficient and more efficient for well-defined activity classes. The paper concludes with scenario-based deployment recommendations and suggests activity-profile inference as a promising research direction.","tokens_in":21214,"tokens_out":7026,"duration_ms":69493,"significance":"If the conclusions are sustained, this would be a valuable benchmark: it is a like-for-like comparison of multiple open accelerometer FMs with strong supervised baselines across a broad task set, with publicly available code and detailed efficiency and representation analyses. The central 'no consistent advantage' claim and the deployment recommendations are directly relevant to practitioners. The main weaknesses are statistical: the ranking procedure ignores subject-level clustering, which undermines the significance-based claims, and some headline statements overstate the equivalence evidence. Because the benchmark infrastructure and the bulk of the descriptive results remain useful, the issues are fixable within a revision rather than fatal.","major_comments":[{"comment":"The pairwise bootstrap resamples test-set windows rather than subjects. With per-dataset test sets of roughly 1–4 subjects (e.g., HAR70+ has 18 subjects and an 8:1:1 split, leaving about 2 test subjects), window-level resampling ignores between-subject variability and yields artificially narrow confidence intervals for AUROC and MAE differences. Consequently, the statements that 'differences in rank correspond to statistically significantly different performance' and the many 'statistically tied' entries in Tables III, V, VI, VII, and VIII are not established at the subject level. This directly affects the central conclusion of 'no consistent advantage' for HAR and specific rank claims such as 'Oxford SSL and ElderNet which both rank first on four datasets.' Please replace the window bootstrap with a subject-level or cluster bootstrap, apply a correction for the multiple pairwise comparisons, and either provide subject-level confidence intervals or remove significance-based language from the summaries.","section":"Section II-C, 'Metrics and statistical significance'"},{"comment":"The statement that 'supervised baselines such as Inception1D and XResNet1D are sufficient' is an equivalence or non-inferiority claim, but the paper only reports failure to reject the null of no difference, and that failure is based on the invalid window-level bootstrap. With very small test sets, absence of evidence has little evidential weight. Please support the 'sufficient' claim with a proper equivalence test (e.g., two one-sided tests) or rephrase to 'we did not find consistent evidence that FMs outperform supervised baselines on these tasks.'","section":"Section III-C, 'Comparative assessment and recommendations for practitioners'"},{"comment":"The uniform fine-tuning protocol (AdamW with constant learning rate 0.0003, 100 epochs, early stopping) is applied to all models. If UniMTS, or NormWear, requires different optimization settings, its last-place fine-tuning rank—and the resulting 'no consistent advantage' conclusion for HAR—could be artifacts of the protocol rather than model capability. The paper itself notes UniMTS's sensitivity to fine-tuning in Section III-B1, but no sensitivity analysis is provided. Please run a minimal hyperparameter sensitivity check (e.g., vary the learning rate for UniMTS and one strong baseline) or explicitly qualify all conclusions that depend on the fine-tuning rankings.","section":"Section II-C, 'Training', and Table VI"}],"minor_comments":[{"comment":"The demographic task counts are inconsistent: Figure 1 labels Age (n=5), Weight (n=4), and Height (n=3), which sum to 12, while Table VI lists only 4 age, 3 weight, and 2 height tasks (9 demographic tasks total) and the abstract states 19 tasks overall. Please clarify which counts are correct and align the figure labels with the tables.","section":"Figure 1 and Table VI"},{"comment":"The header 'GPU Memory [MB]↑' uses an upward arrow as if higher memory is better, but the text and standard usage treat lower memory as better. The arrow direction should be inverted to '↓'.","section":"Table IV"},{"comment":"The sentence 'UniMTS ranks last across the HAR task' is not apparent from Table VI point estimates, where UniMTS is not the lowest-scoring model on several datasets (e.g., PAMAP2, HAR70+, USC-HAD). Please rephrase to refer to the aggregated rank rather than per-dataset AUROC, or provide the rank table that supports the statement.","section":"Section III-A1"},{"comment":"The text states that 'NLMCD discovers 34 concepts' from ElderNet's last-layer representations, while the Figure 4 caption says 'based on 107 concepts discovered by NLMCD.' These numbers should be reconciled.","section":"Section III-B3 and Figure 4"},{"comment":"Some references are incomplete, for example reference [8] lacks full bibliographic details (journal volume, pages, year), and reference [27] is missing the year and venue. Please complete the bibliographic entries.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The benchmark effort is substantive and the code release is commendable, but the statistical inference needs to be overhauled before the central claims can be trusted. The subject-level clustering issue is the key blocker. I see no circularity problem in the performance rankings; the use of the authors' own NLMCD-ALIGN framework is confined to the interpretability analyses and does not affect the benchmark results. The paper would also benefit from a more cautious framing of the 'sufficient baselines' conclusion."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, this is the first like-for-like comparison of four open accelerometer FMs against proper supervised baselines across 19 tasks, with code, efficiency numbers, sensor-position experiments, and representation analysis. That is real value. Second, the central claim—no consistent FM advantage on HAR—is plausible from the point estimates, but the paper's own significance machinery does not support the strength of the claim as written.\n\nWhat's new and good: The benchmark design is mostly sensible: three evaluation modes (linear, frozen, finetune), multi-label soft targets, patient-respecting folds, available code, and a useful efficiency table. The results are interesting even if not field-shaking: UniMTS is the best frozen extractor yet the worst finetuned FM, Oxford SSL/ElderNet are efficient and lead on falls/stress, and baselines hold their own on sleep and demographics. The layerwise CBA/CKA analysis is a nice addition, though not the main event.\n\nWhere it goes soft: (1) The bootstrap ranking procedure resamples windows, not subjects. With test sets of 1–4 subjects, the 95% intervals are far too tight, so 'statistically tied' and 'significant difference' ranks are not established at the subject level. This is load-bearing because the 'no consistent advantage' conclusion uses the rank table. The fix is a subject-level bootstrap or mixed-effects model, plus a multiple-testing correction. Without that, the paper should stop using 'differences in rank correspond to statistically significantly different performance' and instead report raw scores with subject-level variability. (2) Some narrative claims contradict the tables: the text says Inception1D outperforms all FMs on demographics, but Table VI shows Oxford SSL with MAE 5.155 vs Inception1D 30.754 on USC-HAD weight. That needs fixing. (3) The uniform finetuning protocol (fixed LR, epochs) may disadvantage some FMs; the UniMTS finetune/frozen reversal is a warning sign and deserves a sensitivity check. (4) Task-count inconsistencies between abstract, Fig. 1 and tables are sloppy but minor.\n\nBottom line: The benchmark deserves serious referee time—the field needs this kind of independent evaluation. But the statistics must be redone at the subject level before the conclusions can be trusted, and the internal contradictions cleaned up. I would send it to review with major revision, not desk reject.","headline":"A genuinely useful accelerometer-FM benchmark, but the window-level bootstrap under the central 'no consistent advantage' claim is not subject-level and needs a major statistical fix.","tokens_in":21704,"tokens_out":3054,"would_cite":true,"duration_ms":30754,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Four accelerometer foundation models show no consistent advantage over supervised baselines on standard activity recognition in a 19-task benchmark.","keywords":["foundation models","accelerometer data","human activity recognition","wearable sensors","benchmarking","self-supervised learning","stress detection","sleep staging"],"falsifier":"Re-run the 19-task benchmark with per-model hyperparameter search (for example, learning rates from 1e-4 to 1e-3) and give UniMTS and NormWear their native modalities (gyroscope and multimodal physiological signals); if UniMTS's fine-tuning rank moves from last to first, the no-consistent-advantage claim would need to be revised.","tokens_in":20724,"feed_emoji":"⌚","tokens_out":7207,"duration_ms":72227,"temperature":0.7,"pith_summary":"This paper asks whether pretrained foundation models for accelerometer data are ready to replace or beat ordinary supervised models in health-monitoring and activity-recognition settings. To answer, it runs a like-for-like benchmark of four open-source movement foundation models against four supervised baselines across 19 tasks in five categories, using three evaluation modes and statistical significance testing. The central finding is task-dependent: on standard human activity recognition the foundation models show no consistent advantage, while they lead on fall and stress detection and on robustness to sensor placement; as frozen feature extractors, UniMTS is strongest. The authors conclude that for well-defined activity classes, cheap supervised baselines remain sufficient, and that foundation models are currently most useful in specific niches rather than as general replacements. The work matters because practitioners need evidence about when paying the pretraining and compute cost is worth it.","feed_headline":"Accelerometer foundation models show no edge over simple baselines","feed_subtitle":"A 19-task benchmark finds supervised models still match or beat pretrained AI on standard activity recognition.","key_machinery":"The central mechanism is a standardized benchmark protocol that makes heterogeneous models comparable: all models are fed triaxial acceleration windows, adapted through wrappers to their native input lengths and sampling rates, and evaluated in three modes—linear probing (frozen features plus a linear head), frozen probing (frozen features plus a query-attention head), and full fine-tuning under a single uniform training recipe (AdamW, constant learning rate 0.0003, 100 epochs, early stopping). Statistical significance is decided by pairwise bootstrapping on the test set, so rank differences in the radar plots are not treated as noise. The representational analysis rests on cluster-based concept alignment and centered kernel alignment (CKA), which measure how similar different layers are within each model and how well discovered concepts line up with activity labels.","core_discovery":"On the paper's own terms, the discovery is that the promise of accelerometer foundation models as general-purpose feature extractors is not yet realized: across 19 tasks, no model, foundation or supervised, dominates, and on standard human activity recognition the four foundation models are statistically tied with or beaten by supervised baselines as often as they win. The exceptions are specific: Oxford SSL and ElderNet lead on fall and stress detection and across most sensor positions under fine-tuning; UniMTS provides the best frozen representations, the only foundation model that surpasses supervised baselines without fine-tuning; NormWear is strongest for demographics when paired with a non-linear head but costs roughly two orders of magnitude more than the efficient foundation models. Sleep staging stays near chance for every model, which the authors read as a modality limitation rather than a model-choice problem. Internally, all foundation models show high representational similarity across layers, suggesting their representations are less specialized than they could be.","pith_inferences":["The uniformly high inter-layer similarity in all four foundation models implies that future pretraining objectives should push layers to specialize; this is a concrete, model-agnostic direction the paper leaves implicit.","UniMTS's dramatic gap between best frozen probing and last-place fine-tuning suggests its representations are fragile under full fine-tuning, so a two-stage fine-tuning strategy is a direct, testable remedy.","If concept-based activity profiles prove valid against video annotations, foundation-model representations could move beyond fixed activity labels toward continuous behavior characterization, a research direction the paper identifies but does not develop.","Giving NormWear and UniMTS their native modalities (multimodal physiological signals for NormWear, gyroscope data for UniMTS) could change the rankings; the acceleration-only design is the conservative worst case for those two models."],"forward_implications":["For standard, well-defined activity classes, supervised CNN baselines such as Inception1D and XResNet1D match foundation-model performance at a fraction of the compute, so defaulting to a foundation model for human activity recognition is not currently justified.","When labeled data are available and the task is fall or stress detection, Oxford SSL and ElderNet are the strongest choices, with efficiency on par with lightweight baselines.","When labels are scarce and the model is used frozen, UniMTS is the best ready-to-use extractor overall, and NormWear is best for demographic attributes, but both need a non-linear read-out head to reach their potential.","For deployments where sensor position cannot be controlled, foundation models, led by ElderNet, outperform supervised baselines across six of seven body locations.","Sleep staging should not currently be attempted from accelerometer data alone with any of these models; all stay near chance, suggesting the bottleneck is the modality rather than the model choice."],"supporting_citations":[{"why":"This reference supplies the UniMTS pretrained model and its pretraining method, one of the four foundation models under test.","marker":"[37]"},{"why":"This reference supplies the NormWear pretrained model and its multimodal pretraining data, one of the four foundation models under test.","marker":"[38]"},{"why":"This reference supplies the Oxford SSL model pretrained on roughly 700,000 person-days of UK Biobank accelerometer data.","marker":"[39]"},{"why":"This reference supplies the ElderNet model and checkpoint built on the Oxford SSL foundation model.","marker":"[41]"},{"why":"This reference supplies the TinyHAR supervised baseline architecture.","marker":"[42]"},{"why":"This reference supplies the XResNet1D supervised baseline that performs best under linear probing.","marker":"[43]"},{"why":"This reference supplies the Inception1D supervised baseline that outperforms all foundation models on demographic prediction.","marker":"[45]"},{"why":"This reference supplies the S4 state-space supervised baseline that is strong on HAR and sleep tasks.","marker":"[46]"},{"why":"This reference supplies the RealWorld dataset with seven sensor positions, which underpins the sensor-placement robustness rankings.","marker":"[11]"},{"why":"This reference supplies the WESAD stress-detection dataset on which Oxford SSL and ElderNet lead.","marker":"[15]"}],"fun_headline_variants":["Accelerometer foundation models show no consistent edge over supervised baselines","Supervised models still match foundation models on activity recognition","Benchmark: foundation models for movement data lack decisive wins","No clear winner between foundation and supervised models on 19 tasks","Foundation models for accelerometer data fail to outshine simple baselines"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The rankings assume that one fixed fine-tuning recipe (learning rate 0.0003, AdamW, 100 epochs, early stopping) and a single input modality (triaxial acceleration) are fair to all eight models; if a model needs different optimization settings or its native sensors, its rank could change.","fun_headline_variants_meta":{"raw":{"variants":["Accelerometer foundation models show no consistent edge over supervised baselines","Supervised models still match foundation models on activity recognition","Benchmark: foundation models for movement data lack decisive wins","No clear winner between foundation and supervised models on 19 tasks","Foundation models for accelerometer data fail to outshine simple baselines"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000246,"raw_usage":{"total_tokens":1540,"prompt_tokens":949,"completion_tokens":591,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":565,"completion_tokens_details":{"reasoning_tokens":506}},"tokens_in":565,"tokens_out":591,"duration_ms":6542,"temperature":1.0,"reasoning_tokens":506,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T13:47:41.402016+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the 19-task benchmark with per-model hyperparameter search (for example, learning rates from 1e-4 to 1e-3) and give UniMTS and NormWear their native modalities (gyroscope and multimodal physiological signals); if UniMTS's fine-tuning rank moves from last to first, the no-consistent-advantage claim would need to be revised.","supporting_citations":[{"cited_title":"Toward Foundation Model for Multivariate Wearable Sensing of Physiological Signals,","cited_arxiv_id":null,"evidence_quote":"This reference supplies the NormWear pretrained model and its multimodal pretraining data, one of the four foundation models under test."},{"cited_title":"Self-supervised learning for human activity recognition using 700,000 person-days of wearable data,","cited_arxiv_id":null,"evidence_quote":"This reference supplies the Oxford SSL model pretrained on roughly 700,000 person-days of UK Biobank accelerometer data."},{"cited_title":"Self-supervised learning of wrist-worn daily living accelerometer data improves the automated detection of gait in older adults,","cited_arxiv_id":null,"evidence_quote":"This reference supplies the ElderNet model and checkpoint built on the Oxford SSL foundation model."},{"cited_title":"Deep learning for ecg analysis: Benchmarks and insights from ptb-xl,","cited_arxiv_id":null,"evidence_quote":"This reference supplies the XResNet1D supervised baseline that performs best under linear probing."},{"cited_title":"Efficiently modeling long sequences with structured state spaces,","cited_arxiv_id":null,"evidence_quote":"This reference supplies the S4 state-space supervised baseline that is strong on HAR and sleep tasks."},{"cited_title":"On-body localization of wearable devices: An investigation of position-aware activity recognition,","cited_arxiv_id":null,"evidence_quote":"This reference supplies the RealWorld dataset with seven sensor positions, which underpins the sensor-placement robustness rankings."}],"review_version":1}