{"id":"86b30103-43f6-448a-b78c-44574945c45d","arxiv_id":"2412.09037","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A multi-model audit of six HAR benchmarks identifies 'intersect of false classifications' windows that no tested model can classify, argues they reveal annotation and recording ambiguities, and provides a trinary mask for filtering them.","lead":"This paper examines six public human-activity-recognition datasets and finds stretches of sensor data that none of six machine learning models can label correctly. It argues these stretches are often ambiguous or mislabeled rather than just hard, and offers a mask to mark them clean, minor, or major.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The inference from 'six models fail' to 'data are inherently ambiguous' is the load-bearing step; an additional model family or window size could shrink the IFC and undercut the central claim.","rationale":"The paper is a useful, clearly motivated dataset audit, and the authors are transparent about the main limitation. The IFC itself is well-defined, and the proposed mask could still be valuable as a benchmark-specific filter. However, the central claim—that IFC segments are inherently ambiguous and benchmark accuracies are capped by data quality—rests on the unmeasured representativeness of six models. The reader's weakest_assumption identifies exactly this, and the paper's own Section 7.2 admits it. A single additional model family or window-size ablation would settle whether the IFC is a stable data property or a model-family artifact. The table inconsistency in Oppo-Gest (percentages sum to 101.20 in Table 4) is minor and does not change the verdict, but it supports the need for code and mask release so numbers can be independently verified. Since the concern is already the basis of the CONDITIONAL verdict, no change is needed.","tokens_in":24808,"tokens_out":6242,"duration_ms":56649,"concrete_test":"Train an architecturally distinct model (e.g., a 1D ResNet with residual blocks, or a dilated temporal convolutional network) on all six datasets using the identical preprocessing, Group-K-Fold splits, and hyperparameter-selection protocol, and recompute the IFC fractions in Table 4. If the IFC decreases by more than 10% relative on any dataset (e.g., PAMAP2 from 11.74% to below 10.57%), the intersection is not stable across model families and the inference from model failure to dataset ambiguity fails. As a secondary check, rerun with a 400-sample window (200-sample stride) to test window-size dependence; material IFC shrinkage would show the 'ambiguous segments' are partly an artifact of the chosen temporal segmentation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing step is the inference in Section 3.3: 'we assume that the origin of the wrong classification has to lay in the dataset itself if the whole set of applied machine learning models cannot classify it properly.' The IFC is the intersection of failures of six models, but all six share a common paradigm: window-based discriminative deep networks (CNN, GRU, LSTM, ConvLSTM, TinyHAR, and CPC-pretrained encoder) trained with Adam, cross-entropy, the same 200/100 sliding-window preprocessing, and the same normalization. They differ in architecture but not in the fundamental inductive bias that a fixed window of a few seconds must contain enough information for a single label. Section 7.2 concedes this: 'we couldn't establish a clear measure of our set of selected machine learning models sufficiently representing an appropriate intersection.' If a different family (e.g., a 1D ResNet, a dilated TCN, or a classical feature-based SVM) or a different window size classifies some IFC windows, then the IFC is an artifact of the model family, not an inherent property of the data. The paper's central contribution—a 'realistic accuracy target' and a trinary mask for filtering 'ambiguous' segments—depends on this causal attribution. The abstract's wording 'cannot be correctly classified' and the conclusion's 'cannot be correctly classified by any existing ML models' go beyond what the experiment supports.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper inspects six popular HAR benchmark datasets (PAMAP2, Opportunity locomotion and gesture tracks, MM-Fit, MHealth, MotionSense, WISDM) by training six model architectures (CNN, GRU, LSTM, ConvLSTM, TinyHAR, and a CPC-pretrained encoder) under Group-K-Fold cross-validation, repeated four times. It defines the Intersect of False Classifications (IFC) as the set of windows that none of the six models classifies correctly, reports the IFC percentage for each dataset, analyzes confused classes and false-classification durations, visually attributes IFC segments to three causes (ambiguous annotations, recording irregularities, and transition periods), and proposes a trinary clean/minor/major mask for filtering problematic segments. The stated goal is to show that benchmark accuracies are partly capped by dataset-inherent ambiguities rather than by model capacity alone.","tokens_in":25156,"tokens_out":4736,"duration_ms":48749,"significance":"If the central inference were valid, the paper would offer a practically useful quantification of annotation ambiguity across widely used HAR benchmarks and a concrete mask artifact for dataset patching. The work has genuine strengths: it is systematic (six models, four repeated training runs, grouped person/session cross-validation), it provides detailed model and preprocessing descriptions, it documents a large amount of visual inspection in the appendix, and it explicitly acknowledges its main limitation in Section 7.2. However, the significance of the quantitative IFC numbers and of the trinary mask depends on two load-bearing points that are not currently established: the correctness of the IFC calculation in Eq. (1), and the representativeness of the six-model family for the claim that some windows are inherently unclassifiable. As written, the headline claim that some segments 'cannot be correctly classified by any existing ML models' goes beyond the evidence.","major_comments":[{"comment":"The IFC calculation appears to overcount the intersection of false classifications. Table 4 reports 'Common Ground' and six 'Single Contributions' (percentages of windows correctly classified by only one model), and Eq. (1) computes IFC = 100% - Common Ground - sum(Single Contributions). If 'Common Ground' means windows correctly classified by all six models, then windows correctly classified by two, three, four, or five models are neither in Common Ground nor in any Single Contribution, and Eq. (1) incorrectly assigns them to the IFC. The percentages sum to 100 only if no such multi-model-correct windows exist, which is unlikely. The authors should report the union of correct classifications, or the full Venn counts, and recompute the IFC values; as they stand, the reported IFC percentages (e.g., 11.74% for PAMAP2) are upper bounds rather than the claimed intersection.","section":"Section 3.3, Eq. (1), Table 4"},{"comment":"The inference from 'none of the six applied models can classify a window' to 'the cause lies in the dataset itself' is an unvalidated assumption, as the authors themselves concede in Section 7.2 ('we couldn't establish a clear measure of our set of selected machine learning models sufficiently representing an appropriate intersection'). All six evaluated methods are window-based discriminative deep networks trained with Adam and cross-entropy on the same 200-sample window with 100-sample stride and the same normalization; they share strong inductive biases. The abstract's and conclusion's phrasing that such segments 'cannot be correctly classified by any existing ML models' is not supported by the experiments. The authors should either test a more architecturally diverse set (e.g., classical feature-based SVM, dilated TCN, different window sizes and strides, or a different training protocol) or reframe the IFC as 'windows that none of the six evaluated models could classify'.","section":"Section 3.3 and Section 7.2"},{"comment":"The trinary mask is a central contribution, but the major/minor categorization is based on an ad hoc and unvalidated rule: a window is labeled 'major' if the largest gap in the sorted predicted probabilities lies between the first two probabilities, and 'minor' otherwise. The authors state in Section 6 that the automated categorization was only 'plausibly checked' through visual inspection. Since Table 6 and the proposed downstream use of the mask depend on this split, the rule should be validated quantitatively, for example by comparing mask labels with independent human re-annotation, with label noise estimates, or with the behavior of held-out models. Without such validation, the clean/minor/major percentages cannot be interpreted as measuring dataset ambiguity.","section":"Section 6, Table 6"},{"comment":"The probability-based analysis is described only at a high level: Section 4.1 says the authors compute the 'mean calculation of false classified probability distributions across the trained models' and then select the confused class by maximum probability. It is not specified how probability distributions are aligned across models with different class orders or different hyperparameter runs, nor how the IFC windows are combined with the probability gap rule in Section 6. This makes the major/minor mask difficult to reproduce and should be clarified with precise formulas and aggregation steps.","section":"Section 4.1 and Section 6"}],"minor_comments":[{"comment":"The phrase 'maximum portability selection' should be 'maximum probability selection.'","section":"Section 4.1"},{"comment":"The opening sentence contains a duplicated article: 'we only stressed the the ambiguities' should be 'we only stressed the ambiguities.'","section":"Section 5"},{"comment":"The text refers to 'MHealth and MotionSense in Fig. 10d and Fig. 10e,' but in Fig. 10 the subfigures are labeled (d) MM-FIT, (e) MHEALTH, and (f) MotionSense; the references should be adjusted accordingly.","section":"Section 4.2"},{"comment":"The sentence 'the origin of the wrong classification has to lay in the dataset itself' should use 'lie' instead of 'lay.'","section":"Section 3.3"},{"comment":"Dataset names are used inconsistently: 'MM-Fit' vs. 'MM-FIT' and 'MHealth' vs. 'MHEALTH' appear in different parts of the paper; please standardize the notation.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The authors' own Section 7.2 limitation statement is the crux of the matter: the central claim is acknowledged to rest on an unproven representativeness assumption. I do not recommend rejection because the experimental effort and the mask artifact could still be valuable if the claims are reframed as statements about the evaluated model zoo and if Eq. (1) is corrected. The revision should also address the arithmetic issue in the IFC computation, which may materially change the headline percentages."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a worthwhile, well-executed dataset audit whose headline claim outruns its evidence. The IFC (intersection of false classifications across six models) is a useful empirical object, and the per-dataset trinary mask is a genuinely useful artifact—but the leap from 'six models fail' to 'the data are inherently ambiguous' is not supported, and the paper itself admits as much in Section 7.2. The conclusion then overstates it again by saying the windows 'cannot be correctly classified by any existing ML models.'\n\nWhat's actually new: a systematic, multi-dataset quantification of multi-model false-classification overlap; class-confusion analysis with chord diagrams and duration histograms; and a proposed mask to patch datasets. The experimental setup is reasonably careful: four training runs, group-k-fold, leave-one-person/session-out, best hyperparameter selection per model/dataset. That's real work and largely reproducible—though the promised mask and code are still missing.\n\nThe soft spots are proportionate. The central inference is an assumption, and the paper knows it. All six model architectures share the same window-based, discriminative, cross-entropy-trained paradigm, so the IFC could shrink with a different window size or model family. Section 7.2 states this candidly. The fix is to reframe IFC as 'the intersection of failures of this model ensemble' rather than 'inherent dataset ambiguity'—the artifact is still useful, especially as a benchmark floor. The causal categories (ambiguous annotation, recording irregularities, transitions) rest on visual inspection of selected windows, not systematic quantification. That's fine for hypothesis generation, but not as a load-bearing taxonomy.\n\nOne minor numeric inconsistency: in Table 4, the Oppo-Gesture row doesn't satisfy Eq. (1); the contributions sum to about 10.9% but the table lists 12.06%. The other rows check out, so it looks like a typo, but it should be fixed.\n\nThe major/minor mask derived from probability gaps is a reasonable heuristic, but it's validated only by plausibility checks. And the mask isn't actually released yet, which limits the contribution's current usability.\n\nBottom line: this deserves a serious referee. I'd send it to review with a request to soften the impossibility framing, release the mask and code, and add a short sensitivity analysis (e.g., one alternative window size or a classical SVM) to test how fragile the IFC is.","headline":"A solid multi-dataset HAR audit with a useful mask artifact, but the 'inherently ambiguous data' claim outruns the evidence; reframe the IFC as ensemble-relative and release the mask.","tokens_in":25662,"tokens_out":2656,"would_cite":false,"duration_ms":24453,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Across six widely used human-activity-recognition benchmark datasets, the paper identifies sensor windows that none of six state-of-the-art models can classify, and traces them to dataset-inherent problems rather than model limitations.","keywords":["human activity recognition","dataset quality","label ambiguity","intersect of false classifications","trinary mask","wearable sensors","benchmark datasets","cross-validation"],"falsifier":"Take the IFC windows from any of the six datasets and train a substantially different classifier, one with a different window size or stride, a model that sees surrounding context, or an oracle that uses synchronized video of the recording session; if any such system correctly labels a large share of IFC windows, the inference from model failure to dataset ambiguity is refuted. A second check: have independent human annotators re-label the IFC segments without knowing the original labels; if they consistently agree on a single class, those segments are not inherently ambiguous.","tokens_in":24612,"feed_emoji":"📊","tokens_out":7699,"duration_ms":73596,"temperature":0.7,"pith_summary":"This paper tries to establish that on six widely used human-activity-recognition (HAR) benchmark datasets, a measurable share of sensor windows is mislabeled or ambiguous in a way that no classifier can be expected to learn, and that this data-level ceiling, not model expressiveness, is what keeps reported accuracies below near-perfect levels. The authors train six milestone model architectures under grouped leave-out cross-validation, intersect the windows every model gets wrong into the Intersect of False Classifications (IFC), and then inspect those windows at signal level to identify the causes: ambiguous annotations, recording execution irregularities, and misaligned transition periods. For a reader, the payoff is a concrete way to separate 'model failure' from 'data failure' in HAR benchmarks, plus a trinary mask that marks each window clean, minor, or major so datasets can be patched before being reused.","feed_headline":"Six popular HAR datasets contain windows no model can classify","feed_subtitle":"A cross-model intersection of errors points to ambiguous labels and transitions, not architectures, as the accuracy ceiling.","key_machinery":"The load-bearing object is the Intersect of False Classifications (IFC): the set of sliding windows that none of the six models in the panel classifies correctly, computed after training each model under grouped leave-out cross-validation and merging the false classifications across models. It is defined as $\\mathrm{IFC} = 100\\% - \\text{common ground} - \\sum_i \\text{single contribution}_i$, where common ground is the share of windows all models classify correctly and single contributions are windows only one model gets right. The IFC converts a set of model errors into a dataset property, and its subdivision into a trinary mask carries the practical argument: the mask labels each window clean, minor, or major by taking the gap between the highest predicted probability and the next highest, so confidently wrong prediction windows are marked major and uncertain ones minor.","core_discovery":"The paper's central claim is that the IFC segments exist in every one of the six datasets and are caused by properties of the dataset itself rather than by weak models: PAMAP2 has an IFC share of 11.74%, Opportunity gesture labels 12.06%, Opportunity locomotion 5.31%, MHealth 4.58%, WISDM 1.61%, while MM-Fit and MotionSense stay below 0.5%. Signal-level inspection attributes the persistent errors to annotation design where classes are not mutually exclusive (e.g., ironing containing standing), to null classes that absorb heterogeneous movements, to recording irregularities such as loose sensors or periodic noise, and to transitions between activities being labeled with too coarse a granularity. The paper concludes that benchmark accuracy on these datasets is capped by data quality, and offers the trinary mask as a dataset patch and a set of collection guidelines as a preventive measure.","pith_inferences":["Editorial inference: because IFC is defined relative to a fixed panel of window-based discriminative models, the reported percentages are lower-bound estimates of ambiguity; a panel with different inductive biases could shrink them, so the numbers are evidence of dataset problems rather than exact measurements of them.","Editorial inference: the trinary mask could be turned into a training regularizer that downweights or ignores major segments and treats minor segments as uncertain, which would probably raise clean-window accuracy; the paper suggests this use but does not test it.","Editorial inference: the intersection-of-false-classifications audit is a general data-quality method that transfers to other sensor benchmarks and to any labeled time-series dataset where leave-out cross-validation is feasible.","Editorial inference: if the mask becomes common practice, benchmark results with and without IFC filtering will no longer be directly comparable, so dataset versions and mask versions will need to be tracked in publications."],"forward_implications":["Reported accuracies on these benchmarks should be read as ceilings set partly by data quality: for PAMAP2, 11.74% of windows are in the IFC and cannot be learned by any of the tested models.","Practitioners can use the trinary mask to filter major segments before training, or to integrate the mask labels into the loss function so models become aware of demanding sections.","The contrast between datasets with fine-grained labels (MotionSense, MM-Fit) and those with coarse protocol-based labels (PAMAP2) indicates that annotation design, not IMU sensing in general, drives most of the residual error.","Model comparison in HAR should report the IFC or common-ground share alongside accuracy and F1, because two models with the same accuracy can fail on very different windows.","Future HAR data collections should adopt the proposed guidelines, especially mutually exclusive label tracks, fine-grained transition labeling, and synchronized reference video for auditability."],"supporting_citations":[{"why":"Supplies the PAMAP2 benchmark, whose coarse protocol-based annotations produce the largest IFC share in the study.","marker":"[66]"},{"why":"Supplies the Opportunity benchmark, with its dual locomotion and gesture label tracks on the same sensor streams.","marker":"[15]"},{"why":"Supplies the MM-Fit exercise dataset, whose dominant null class shapes the confusion analysis.","marker":"[72]"},{"why":"Supplies the MHealth dataset, whose null-labeled rest periods contain the periodic noise that becomes false classifications.","marker":"[9]"},{"why":"Supplies the MotionSense dataset, whose fine-grained labeling anchors the low-IFC end of the comparison.","marker":"[57]"},{"why":"Supplies the WISDM smartphone dataset, whose class imbalance drives the observed relative confusion.","marker":"[49]"},{"why":"Defines TinyHAR, the transformer-based architecture that achieves the best accuracy in the model panel.","marker":"[94]"},{"why":"Defines Contrastive Predictive Coding for HAR, the self-supervised pre-training method included in the panel.","marker":"[36]"},{"why":"Establishes that the CPC encoder should be unfrozen during fine-tuning, the protocol used for that model.","marker":"[25]"},{"why":"Documents human factors in IMU data annotation, motivating the paper's ambiguity analysis.","marker":"[42]"}],"fun_headline_variants":["Dataset flaws, not models, cap HAR benchmark accuracy","No model can classify these HAR windows across six datasets","HAR accuracy ceiling traced to ambiguous labels and transitions","Six HAR benchmarks share unclassifiable data windows"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument collapses if the six chosen models share enough inductive biases that their joint failure says nothing about other classifiers; the paper itself concedes in Section 7.2 that it could not establish a clear measure that this set of models sufficiently represents an appropriate intersection.","fun_headline_variants_meta":{"raw":{"variants":["Dataset flaws, not models, cap HAR benchmark accuracy","No model can classify these HAR windows across six datasets","HAR accuracy ceiling traced to ambiguous labels and transitions","Six HAR benchmarks share unclassifiable data windows"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000388,"raw_usage":{"total_tokens":2024,"prompt_tokens":901,"completion_tokens":1123,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":517,"completion_tokens_details":{"reasoning_tokens":1061}},"tokens_in":517,"tokens_out":1123,"duration_ms":9189,"temperature":1.0,"reasoning_tokens":1061,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T17:21:49.724448+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the IFC windows from any of the six datasets and train a substantially different classifier, one with a different window size or stride, a model that sees surrounding context, or an oracle that uses synchronized video of the recording session; if any such system correctly labels a large share of IFC windows, the inference from model failure to dataset ambiguity is refuted. A second check: have independent human annotators re-label the IFC segments without knowing the original labels; if they consistently agree on a single class, those segments are not inherently ambiguous.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the MM-Fit exercise dataset, whose dominant null class shapes the confusion analysis."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the MotionSense dataset, whose fine-grained labeling anchors the low-IFC end of the comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the WISDM smartphone dataset, whose class imbalance drives the observed relative confusion."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines TinyHAR, the transformer-based architecture that achieves the best accuracy in the model panel."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes that the CPC encoder should be unfrozen during fine-tuning, the protocol used for that model."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents human factors in IMU data annotation, motivating the paper's ambiguity analysis."}],"review_version":1}