{"id":"70d05710-14c5-41c1-ab3b-2649d2240ef2","arxiv_id":"2505.21918","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"An n-dimensional numerical Transformer with linear embedding, bin-based discretization, and parallel output heads improves human activity recognition accuracy by 10-15% over a tokenized vanilla Transformer.","lead":"Researchers trained Transformer models on three-axis accelerometer data by replacing the usual word embedding with a linear layer, discretizing the continuous values into bins, and using masked-language-model style pre-training. The resulting activity recognition model beat a tokenized vanilla Transformer by about 10-15% accuracy on five benchmark datasets and beat a ResNet baseline on three of them.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Undisclosed initialization confounds the 10–15% improvement claim; without specifying whether models start from NLP-pretrained weights, the reported gains cannot be attributed to the proposed method.","rationale":"The reader's conditional verdict correctly identifies the missing initialization disclosure as the weakest point. The paper's abstract states that the authors 'built a pretrained language model' and 'leverage this pretrained model,' which strongly suggests the transformer blocks are initialized from the official DistilBERT/GPT-2 checkpoints. However, Section 4.1 and Algorithm 2 only describe self-supervised pre-training on capture24; they never say what weights are loaded before that pre-training, and the experimental section does not state the initialization of the vanilla Transformer or the 'without pre-training' variants. Without this information, the headline 10–15% improvement over vanilla Transformer cannot be attributed to the proposed linear embedding, binning, or parallel output heads, because a language-pretrained transformer may provide a strong prior that is absent from a from-scratch baseline. The parameter counts in Table 4 are consistent with the official checkpoints but do not prove initialization. This is a single, concrete, testable omission, and it directly affects the validity of the strongest claim. I therefore agree with the reader: the paper should be accepted only conditionally, after the authors disclose initialization and ideally demonstrate the result with controlled baselines.","tokens_in":14771,"tokens_out":10699,"duration_ms":108757,"concrete_test":"Obtain the training configuration or released code and verify whether each model's weights are initialized from the official `transformers` checkpoint for DistilBERT/GPT-2. As an independent check, re-run Table 2's central comparison twice: once with all models initialized from the same official NLP-pretrained checkpoint (resizing the vanilla baseline's token embedding to its vocabulary) and once with all models initialized from scratch. If the 10–15% margin over the vanilla Transformer does not persist under controlled initialization, the central claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the proposed combination of linear embedding, binning, and parallel output heads with sensor-data pre-training yields a 10–15% accuracy improvement over a vanilla Transformer. This attribution requires controlled initialization: all compared models must start from the same weights so that the only difference is the proposed components. The paper never states this. The abstract says 'we built a pretrained language model... By leveraging this pretrained model,' implying the transformer layers inherit NLP-pretrained weights. Yet Section 4.1 (Pre-Training Setup) and Algorithm 2 describe only the sensor pre-training on capture24 and never say what weights are loaded before that step. Table 4 lists parameter counts consistent with official DistilBERT/GPT-2 checkpoints, but parameter count does not establish initialization. If the proposed models start from NLP-pretrained weights while the 'Vanilla Transformer(DistilBERT)' baseline is trained from scratch, the 10–15% gap could be caused by language transfer, not by the proposed embedding/binning/heads. The 'without Pre-Training' rows do not resolve the ambiguity: it is unclear whether those runs are also initialized from NLP weights or from scratch. Thus the reported improvement cannot be assigned to the proposed method without this disclosure.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an adaptation of NLP Transformer models (DistilBERT and GPT-2) to multi-dimensional accelerometer sensor data for human activity recognition. The method has three components: a linear embedding layer that projects n-dimensional sensor readings into the model dimension, a binning preprocessing step that discretizes continuous values into classification labels for self-supervised pretraining, and parallel linear output heads, one per sensor dimension. Pretraining is performed on the capture24 dataset with three tasks (reconstruction, masked language modeling, and next-token prediction), followed by downstream activity classification on ADL, Opportunity, PAMAP2, REAL WORLD, and WISDM. The paper reports that the proposed models improve accuracy by 10%--15% over a vanilla Transformer, that the masked-language-model variant performs best, and that the proposed DistilBERT-based model outperforms SENvT variants on all five datasets while ResNet remains stronger on two of the five datasets.","tokens_in":14899,"tokens_out":4870,"duration_ms":46729,"significance":"If the attribution question is resolved, this is a useful empirical recipe: it shows that off-the-shelf NLP Transformer architectures can be adapted to multi-dimensional sensor data with a simple linear embedding, binning, and parallel output heads, and it evaluates the recipe on five public datasets against several baselines. The paper also provides parameter counts and training/inference time and memory measurements, which are helpful for practitioners assessing deployability. The main value is the comparative evaluation; however, the central 10%--15% claim is only as strong as the experimental controls, and the manuscript currently leaves a key control (model initialization) undisclosed and reports single runs without error bars or ablations.","major_comments":[{"comment":"The manuscript never states whether the DistilBERT and GPT-2 models are initialized from their official NLP-pretrained checkpoints or from randomly initialized weights before the capture24 pretraining loop. Algorithm 2 only describes the sensor-data training loop, and Section 4.2 says 'the pretrained model's weights, excluding the classification head, are loaded,' without saying what weights were loaded before pretraining. Table 4 lists parameter counts consistent with official DistilBERT/GPT-2 sizes, but parameter count does not establish initialization. If the proposed models inherit language-pretrained transformer weights while the vanilla Transformer baseline or the from-scratch rows do not, the 10%--15% gap could be caused by language transfer rather than by the proposed embedding, binning, and parallel output heads. Please state the exact initialization used for every row of Table 2, including the 'without Pre-Training' rows, and, if language-pretrained weights are used, include a from-scratch control for the proposed architecture so the comparison is clean.","section":"4.1, Algorithm 2, Table 4"},{"comment":"The headline claim rests on single runs: the paper reports no error bars, no standard deviations over multiple seeds, and no significance tests. Some test sets are small (e.g., ADL has 127 test samples), so differences of 1--2% between proposed variants may be within noise. Moreover, the paper evaluates many configurations (2 bin sizes, 3 pretraining tasks, 3 model sizes) on the same test sets and selects the best configuration (MLM, bin size 1000) for the conclusion; this selection procedure can inflate apparent gains relative to the non-selected configurations. Please report means and standard deviations over at least three seeds, apply a paired significance test or confidence intervals where appropriate, and state how configuration choices were made to avoid post-hoc selection on the test sets.","section":"Table 2, Section 5.2.2"},{"comment":"The three claimed contributions are never ablated. The proposed model differs from the vanilla Transformer by (i) linear embedding, (ii) binning, and (iii) parallel output heads, but the paper does not run variants that remove each component, nor a variant with linear embedding only (the Soft Sensing Transformer idea already mentioned in Section 2.1). As a result, the title and abstract attribute the 10%--15% improvement to the proposed combination, yet the evidence only shows the difference between the full proposed system and a token-ID baseline. Please add ablations such as 'linear embedding only', 'linear embedding + binning', and 'full model' so that the contribution of each component is isolated.","section":"3.1--3.3, Table 2"},{"comment":"The central comparison is against a vanilla Transformer that is forced to convert continuous min-max scaled values into integer token IDs (vocabulary sizes 10000 and 30000) after concatenating axes into a length-900 token sequence. This is a weak representation for numerical sensor data, and it is unclear from Section 5.1 whether this baseline is also pretrained on capture24 or whether it is trained from scratch at the downstream stage. Since Soft Sensing Transformer [31] already proposed linear transformation of sensor inputs without tokenization, comparing against that prior adaptation (or a vanilla Transformer with a linear input layer) would help isolate the contribution of binning and parallel output heads. Please add such a baseline or explicitly state in the contributions paragraph that the 10%--15% figure is relative to the token-ID vanilla Transformer only.","section":"5.1, Table 2"}],"minor_comments":[{"comment":"There are several typos: 'Oppotunity' in the Section 5.2.3 heading, and Figure 3 and Figure 4 captions both incorrectly say 'Confusion matrix for Oppotunity' even though Figure 3 is for REAL WORLD and Figure 4 shows pretraining loss curves. Please correct the captions and headings.","section":"5.2.3, 5.2.4, Figure 3, Figure 4"},{"comment":"Section 5.2.3 refers to 'the performance of the pre-trained ResNet model', but Section 6 states that ResNet18 was trained only on the downstream task without any pretraining. This inconsistency should be resolved by saying 'baseline ResNet' or clarifying how ResNet was pretrained.","section":"5.2.3, Section 6"},{"comment":"The heading 'Assumptions' for the experimental setup is not descriptive; consider renaming it to 'Experimental Setting' or 'Common Experimental Conditions'.","section":"5.1"},{"comment":"The note on memory usage is ambiguous: it says 'Memory Usage is the amount of memory used when measured by inference in the CPU', but it is unclear whether this is peak memory, which tool measured it, and whether the value includes only inference or also pretraining. Please clarify.","section":"Table 5"},{"comment":"Some references have formatting issues, such as trailing commas in [13] and [28], and the reference list omits some publication details (e.g., volume and pages for [1]). Please check the reference formatting against the journal style.","section":"Section 2.2 and References"},{"comment":"No code, random seed, or detailed hyperparameter settings for the baselines are provided. For reproducibility, please include code or at least specify the random seeds, optimizer settings for all baselines, and the exact protocol used for the 'without Pre-Training' rows.","section":"Reproducibility"}],"recommendation":"major_revision","confidential_remarks":"The paper is a plausible empirical contribution but the missing initialization disclosure is a genuine blocker for the headline attribution claim. I would be willing to accept after the authors clarify initialization, add from-scratch controls, report variability across seeds, and add basic ablations. I would also note to the editor that the paper's scope is narrow and the writing is rough; a careful revision is needed, but the underlying idea is worth publishing if the controls are added."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know: this is a solid, unglamorous empirical paper that takes DistilBERT and GPT-2, swaps the token embedding for a linear layer, adds binning and parallel output heads, and reports accuracy on five HAR benchmarks. The components are not new individually, but the combination is, and the paper's real contribution is a systematic comparison of three self-supervised pretraining tasks (reconstruction, MLM, next-token) plus a careful look at parameter counts, inference time, and memory. It also beats SENvT, a related sensor transformer, on all five datasets, and beats ResNet on three. Those are useful data points for anyone adapting NLP transformers to multivariate time series.\n\nWhat the paper does well: the method section is concrete enough to re-implement, the experiments cover a reasonable range of conditions, and the discussion of edge deployment is honest about the cost of transformer models. The authors also acknowledge when their pretraining loss does not predict downstream performance, which is a nice touch.\n\nThe soft spots, in proportion: first, the paper never states whether the transformer layers are initialized from the official NLP checkpoints or trained from scratch. The parameter counts in Table 4 are consistent with loading pretrained weights and replacing the word embedding with a linear layer, but the text never says so. This matters because if the proposed models inherit language-pretrained weights while the vanilla Transformer baseline is trained from scratch, part of the gain could come from language transfer rather than the proposed embedding, binning, and output-head design. The “without Pre-Training” rows do not resolve the ambiguity, since those runs could also be initialized from NLP weights. This is a real confound, and it is fixable with one sentence. Second, the vanilla Transformer baseline is weak: it converts continuous sensor values into integer token IDs by scaling and truncating, which destroys the continuity that the proposed linear embedding preserves. Beating that baseline is expected; the stronger comparisons are against ResNet and SENvT. Third, there are no error bars or multiple seeds, and many configurations are compared on the same test sets with the best ones highlighted. That is selection on the test data, though the pattern is consistent enough that it is unlikely to be pure noise. Finally, three of the four figures have duplicate or wrong captions (“Confusion matrix for Oppotunity” appears in all three), which is sloppy but minor.\n\nWho this is for: people working on HAR or on adapting NLP models to non-linguistic data will get a practical, honest benchmark. It deserves a serious referee. The missing initialization disclosure, the weak baseline, and the lack of variance reporting should be addressed, but none of these sink the paper. After those revisions, the empirical claim would be much easier to trust.\n\nRecommendation: send it to peer review, and ask the authors to state the initialization explicitly, add ablations that isolate each component, and report results over multiple seeds.","headline":"A workmanlike empirical study adapting DistilBERT and GPT-2 to sensor data, but a missing initialization disclosure and a weak vanilla baseline leave the headline 10–15% gain not fully attributable.","tokens_in":15537,"tokens_out":2275,"would_cite":false,"duration_ms":24902,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper's central claim is that a linear-embedding Transformer with binning and parallel output heads lifts sensor-based activity recognition accuracy by 10–15% over a vanilla Transformer.","keywords":["human activity recognition","Transformer","self-supervised learning","sensor data","masked language modeling","linear embedding","binning","DistilBERT"],"falsifier":"Retrain the proposed model twice with identical data and hyperparameters, once from the official DistilBERT/GPT-2 checkpoints and once from random initialization, and compare downstream accuracy on all five datasets. If the random-init version reproduces the 10–15% edge over the vanilla Transformer, the architecture claim is confirmed; if it falls to the Table 2 'without Pre-Training' levels (roughly 0.82–0.90), the reported gains are transfer from language pretraining rather than from the proposed components.","tokens_in":14444,"feed_emoji":"🏃","tokens_out":11196,"duration_ms":102060,"temperature":0.7,"pith_summary":"This paper tries to show that language-model Transformers can be adapted to multi-dimensional sensor data without a custom attention architecture. The recipe is to feed each n-dimensional sensor reading through a linear embedding layer, turn continuous values into k-bin labels for self-supervised pretraining, and give each sensor axis its own linear output head. Pretrained on unlabeled accelerometer data and fine-tuned for activity recognition, the model reports 10–15% higher accuracy than a vanilla Transformer across five datasets and beats ResNet on three of them. The authors' wider point is that the input/output interface, not the attention mechanism, is what makes NLP Transformers work for numerical sensor signals.","feed_headline":"Sensor Transformer lifts activity recognition 10–15% over vanilla","feed_subtitle":"DistilBERT with linear embedding and binning beats ResNet on three datasets and all SENvT variants.","key_machinery":"The central object is the n-dimensional numerical processing Transformer, an NLP Transformer with three swapped parts. The embedding is a linear layer $h = Wx + b$ that maps each n-axis reading $x\\in\\mathbb{R}^n$ to a hidden vector, replacing the token embedding; the pretraining target is produced by binning each dimension into k labels with $y_i = \\min(\\lfloor k \\cdot x^{\\mathrm{scaled}}_i \\rfloor, k-1)$; and the output is a set of n parallel linear heads $\\hat{y}_i = f_i(H) = W_i H + b_i$, one per sensor axis. This machinery turns continuous multi-axis sensor streams into a token-like sequence for self-supervised pretraining, and the pretrained encoder is then reused for downstream activity classification with a new head.","core_discovery":"The central discovery, as the paper states it, is empirical: replacing a vanilla Transformer's token embedding with a linear layer, discretizing sensor values into bins to create self-supervised labels, and using parallel output heads for each sensor axis yields a 10–15% accuracy improvement over a vanilla Transformer on human activity recognition. On the ADL, Opportunity, and PAMAP2 datasets the proposed models also outperform ResNet18 and Random Forest. The DistilBERT variant pretrained with masked language modeling beats SENvT-u4 and SENvT-contrastive on all five datasets, and masking outperforms reconstruction and next-token pretraining even though reconstruction has the lowest pretraining loss. The paper also reports that bin size (100 vs 1000) had no clear effect and that ResNet remains stronger on REAL WORLD and WISDM.","pith_inferences":["Inference — because the linear embedding layer is the only place sensor dimensionality enters, the same recipe should extend to 6-axis and 9-axis IMU data, a step the paper motivates but does not run.","Inference — the paper's bin sizes (100 and 1000) and single masking ratio (25%) cover only a small slice of the design space; a sweep over bin count and mask ratio would map where the 10–15% gain is largest.","Inference — if the transformers were initialized from official language checkpoints, the 'without Pre-Training' rows in Table 2 (roughly 0.82–0.90 accuracy) suggest most of the gain is transfer from language pretraining rather than from the embedding, binning, and parallel-head design, so the paper should be read with that ambiguity in mind.","Inference — the same tokenization problem applies to multivariate time-series forecasting, where continuous values are often discretized; the parallel-head and binning design could be tested as an alternative to patch-based encoders in that setting."],"forward_implications":["The 10–15% accuracy gap over the vanilla Transformer indicates that integer tokenization of continuous sensor values loses information that a linear embedding preserves.","Masked-language-model pretraining transfers to sensor data better than reconstruction or next-token prediction, despite converging with the noisiest loss curve.","The proposed model surpasses ResNet on ADL, Opportunity, and PAMAP2, so pretrained transformers are competitive with convolutional baselines on some HAR benchmarks.","Beating SENvT-u4 and SENvT-contrastive on all five datasets suggests per-timestep linear projection can outperform one-dimensional patch embeddings for sensor inputs.","GPU inference meets the 30 fps real-time constraint (4.5–6.3 ms per sample), while CPU inference does not, so deployment choices depend on hardware."],"supporting_citations":[{"why":"Supplies the capture24 accelerometer corpus used for all pretraining.","marker":"[3]"},{"why":"Provides the DistilBERT encoder backbone that the linear embedding, binning, and parallel heads are attached to.","marker":"[23]"},{"why":"Defines the vanilla Transformer baseline and the multi-head attention machinery the proposed model keeps.","marker":"[26]"},{"why":"Provides the ResNet18 baseline that the proposed models beat on ADL, Opportunity, and PAMAP2.","marker":"[8]"},{"why":"Defines SENvT-u4 and SENvT-contrastive, the signal transformers that DistilBERT (MLM) outperforms on all five datasets.","marker":"[19]"},{"why":"Earlier linear-transformation embedding for sensor inputs that the proposed model extends with binning and parallel heads.","marker":"[31]"},{"why":"Provides the GPT-2 decoder backbone used for the next-token prediction variant.","marker":"[20]"},{"why":"Provides the Random Forest baseline used in the comparison.","marker":"[1]"}],"fun_headline_variants":["Transformer with binning boosts sensor activity recognition 10-15%","Self-supervised sensor Transformer beats vanilla by 10-15% accuracy","Pretrained Transformer for sensors: 10-15% better activity recognition","Linear embedding and binning improve sensor Transformer accuracy","Self-supervised Transformer lifts sensor HAR accuracy by 10-15%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper never states whether the DistilBERT and GPT-2 models start with weights already trained on text or with random weights; if they carry language pretraining, the reported 10–15% gain may come from that prior training rather than from the proposed embedding, binning, and parallel output heads.","fun_headline_variants_meta":{"raw":{"variants":["Transformer with binning boosts sensor activity recognition 10-15%","Self-supervised sensor Transformer beats vanilla by 10-15% accuracy","Pretrained Transformer for sensors: 10-15% better activity recognition","Linear embedding and binning improve sensor Transformer accuracy","Self-supervised Transformer lifts sensor HAR accuracy by 10-15%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000165,"raw_usage":{"total_tokens":1193,"prompt_tokens":829,"completion_tokens":364,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":445,"completion_tokens_details":{"reasoning_tokens":274}},"tokens_in":445,"tokens_out":364,"duration_ms":3838,"temperature":1.0,"reasoning_tokens":274,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:19:39.907671+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the proposed model twice with identical data and hyperparameters, once from the official DistilBERT/GPT-2 checkpoints and once from random initialization, and compare downstream accuracy on all five datasets. If the random-init version reproduces the 10–15% edge over the vanilla Transformer, the architecture claim is confirmed; if it falls to the Table 2 'without Pre-Training' levels (roughly 0.82–0.90), the reported gains are transfer from language pretraining rather than from the proposed components.","supporting_citations":[{"cited_title":"Chan Chang, R","cited_arxiv_id":null,"evidence_quote":"Supplies the capture24 accelerometer corpus used for all pretraining."},{"cited_title":"Towards llms for sensor data: Multi-task self-supervised learning","cited_arxiv_id":null,"evidence_quote":"Defines SENvT-u4 and SENvT-contrastive, the signal transformers that DistilBERT (MLM) outperforms on all five datasets."},{"cited_title":"Soft sensing transformer: hun- dreds of sensors are worth a single word","cited_arxiv_id":null,"evidence_quote":"Earlier linear-transformation embedding for sensor inputs that the proposed model extends with binning and parallel heads."},{"cited_title":"A random forest guided tour.Test, 25:197–227, 2016","cited_arxiv_id":null,"evidence_quote":"Provides the Random Forest baseline used in the comparison."}],"review_version":1}