{"id":"b0656c39-3c4e-4160-b745-815082172964","arxiv_id":"2504.17735","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"A hierarchical model trained only on long, high-level activity labels learns motion embeddings that also classify short low-level activities like stationary, walking, and running from a head-mounted IMU.","lead":"EgoCHARM is a small neural network that recognizes both long activities such as cooking and short actions such as walking from a single motion sensor in smart glasses. It is designed to run cheaply enough for always-on use, which matters for battery-powered wearables and context-aware assistants.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Model selection used the test F1 as the optimization target (§3.7), so the reported 0.826/0.855 test scores are selection-biased maxima, not unbiased estimates; a held-out validation split is required before the headline claim is supported.","rationale":"I read the paper as making a resource-efficiency claim: a 22k-parameter low-level encoder and a 63k-parameter high-level model can classify 9 high-level and 3 low-level activities from a head-mounted IMU, with the low-level encoder trained semi-supervised and probed for low-level recognition. The architecture details, parameter counts, and FLOPs are clearly reported, and the PCA clustering provides a qualitative check of embedding separability. However, the central empirical support depends on test F1 numbers that were used to select the model. Section 3.7 explicitly optimizes the test F1 score, so the numbers in Table 2 and the abstract are not trustworthy as unbiased generalization estimates. This is a load-bearing flaw because the strongest claim is exactly those test scores. The reader's rationale identifies this issue, and I agree with the REJECT verdict. I mark partial agreement with the reader's weakest_assumption field because that field names the low-level label reannotation noise rather than the test-set selection; the label noise is real but secondary. The verdict should remain REJECT until a validation-based rerun is provided. If the rerun reproduces roughly 0.82/0.85 F1, the paper would be acceptable; if not, the central claim is unsupported.","tokens_in":14830,"tokens_out":4164,"duration_ms":43678,"concrete_test":"Re-run the EgoCHARM pipeline with a strict three-way split at the participant level: train, validation, and test. Use the validation set for all SAASBO/Sobol model selection (200 runs per architecture) and touch the test set exactly once to report the final model's F1. Compare this unbiased test F1 to the paper's 0.826; if it falls below 0.80 or by more than about 0.03, the headline claim fails. For the low-level probing, re-select the probing layer and report a nested-CV F1 on data not used to select the encoder, or at minimum report the validation F1 of the chosen probing layer.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.7 states that for every architecture combination the authors 'conduct 200 runs and use Sobol sampling... in addition to a Bayesian optimization method, SAASBO... optimizing over the test F1 score.' With 15 combinations this is roughly 3,000 evaluations of the same test set, and the final model is the best of that selection. The reported test F1 of 0.826 (high level) and 0.855 (low level) are therefore maxima of a selection process over the test data, not unbiased estimates of generalization. Even with participant-level stratification, best-of-N selection inflates expected performance; with 9 classes and thousands of trials the inflation can be nontrivial. The low-level result inherits the same bias because the frozen encoder was selected using the high-level test F1, and the low-level probing is then evaluated on separate low-level folds. A secondary concern is the coarse 30-second text summaries propagated to 1-second windows for low-level labels, but the primary load-bearing issue is the test-set selection. As submitted, the paper cannot support the claim 'achieving 0.826 test F1... and 0.855 test F1'.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces EgoCHARM, a hierarchical activity-recognition architecture that uses a single head-mounted IMU. A small CNN-GRU encoder processes 1-second windows of 6-channel IMU data and outputs motion embeddings; these embeddings are aggregated over a 30-second window and passed to a GRU high-level classifier that recognizes 9 long activities. The encoder is trained with only high-level labels, then frozen and probed with a linear layer to recognize 3 low-level activities (stationary, walking, running). Data are taken from Ego-Exo4D and Nymeria, with participant-disjoint train/test stratification. The paper reports a test F1 of 0.826 for high-level recognition and 0.855 for low-level probing, with 21,868 low-level and 63,369 high-level parameters, plus a sensitivity analysis over sample counts, sampling frequency, and window size.","tokens_in":15083,"tokens_out":4442,"duration_ms":44567,"significance":"The architecture idea is attractive: a sub-25k-parameter low-level encoder is genuinely deployable on current IMU chips, and the semi-supervised training from high-level labels is a practical way to obtain low-level embeddings without dense annotation. The paper is also honest about the limitations of head-mounted IMU signals, and the confusion-matrix discussion is useful. The comparison against several architecture combinations and the sensitivity analysis are valuable empirical contributions, provided the evaluation protocol is sound. However, the central quantitative claims are not currently supported because the hyperparameter search directly optimized the reported test F1, making the headline numbers selection-biased maxima rather than unbiased estimates.","major_comments":[{"comment":"The hyperparameter search is explicitly described as 'optimizing over the test F1 score.' With 200 runs per architecture combination and 15 evaluated combinations, the final high-level model is selected as the best of roughly 3,000 evaluations of the same test set. The reported test F1 of 0.826 and accuracy of 82.86% in Table 2 are therefore maxima of a selection procedure over the test data, not unbiased estimates of generalization. The low-level probing result (0.855) is also affected, because the frozen encoder was chosen using high-level test performance, so the embedding quality is not validated independently. The paper needs a validation split (or nested cross-validation) used for all architecture and hyperparameter choices, with the test set evaluated only once. This is a load-bearing issue for the central claims in the abstract and Section 4.1.","section":"Section 3.7"},{"comment":"The low-level labels are derived by manually choosing 30-second windows from the Nymeria text summaries and assigning that single label to every 1-second window inside the chosen segment, with video refinement only for the running class. If a chosen segment contains brief stops, pauses, or other actions, the per-second labels are wrong. This label noise is not measured or modeled, and without per-second verification the 0.855 low-level probing F1 is not a reliable estimate of true per-second recognition accuracy. Please provide per-second verification (or a second annotator agreement study, or an evaluation that removes boundary seconds from each segment) before treating the low-level result as established.","section":"Section 3.1"},{"comment":"The sensitivity analysis in Fig. 6 and Tables S1–S4 uses the same 200-run protocol with test-F1 optimization described in Section 3.7. The comparisons across sample counts, sampling frequencies, and window sizes are therefore also subject to the same selection bias. The claim in Section 5 that a 15 Hz sampling frequency yields performance comparable to 50 Hz is based on these selection-biased numbers. The resource-efficiency conclusions require an unbiased validation protocol before they can be accepted.","section":"Section 4.2"}],"minor_comments":[{"comment":"In the sentence 'In line with our PCA visualization, Fig 3.5', the reference should be to Figure 5 (the PCA plot), not 'Fig 3.5'.","section":"Section 4.1"},{"comment":"The table header 'LL + HL' followed by repeated column labels for F1, Accuracy, Parameters, and FLOPs is difficult to parse; please restructure with clear subheaders indicating LL, HL, and PL columns.","section":"Table 2"},{"comment":"The manuscript inconsistently uses 'hand picked' and 'hand-picked'; please unify the spelling.","section":"Section 3.3"},{"comment":"After the evaluation protocol is corrected, the reported test F1 values in the abstract, introduction, and sensitivity tables will need to be updated to reflect the unbiased results.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The paper is transparent about the test-set optimization, which is diagnostically useful, but it means the headline numbers cannot be taken at face value. Since the fix (a proper validation split with a single final test evaluation) is straightforward and the architecture contribution is otherwise sensible, I support major revision rather than rejection. Providing code or a detailed protocol for the re-analysis would strengthen the revision significantly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nRead EgoCHARM. The short version: this is a useful engineering contribution that needs a methodological fix before the numbers can be believed. The authors adapt CHARM's hierarchical trick to a head-mounted IMU, replace hand-picked features with a variable-dilation CNN-GRU, and evaluate on two large in-the-wild datasets. That is a real extension, and the resource-efficiency story is well told: 22k low-level parameters, 63k high-level, plus a sensitivity analysis that actually checks sample counts and sampling rate. I give them credit for citing CHARM clearly and for being honest about egocentric IMU limits, like cooking versus bike repair looking nearly identical in the signal.\n\nThe soft spot is load-bearing. Section 3.7 says hyperparameter sweeps \"optimize over the test F1 score\" for every architecture combination, 200 runs each, fifteen combinations. That means the reported 0.826 and 0.855 are maxima of a selection process on the test set, not unbiased estimates. There is no validation split. Even with participant-level stratification, best-of-N selection inflates the expected score, and with roughly 3,000 test-set evaluations the inflation can be nontrivial. The abstract's claim \"achieving 0.826 test F1\" is not supported as stated. The fix is straightforward: hold out a validation split for model selection, re-run, and report the test score once. I expect the performance would drop a few points, but the architecture would likely still look good.\n\nA secondary concern is the low-level label construction. Thirty-second text summaries from Nymeria are assigned to every 1-second window in a chosen segment. That will introduce label noise at segment boundaries and during brief stops. Running labels were refined with video, but stationary and walking were not. The 0.855 probing F1 is probably optimistic for per-second recognition.\n\nNone of this makes the paper a waste of time. The hierarchical semi-supervised idea is sound, the empirical comparisons are transparent, and the sensitivity analysis is a genuine plus. It just cannot ship with the current evaluation protocol. A serious referee should take it, but the decision letter should require a proper validation split and re-reported numbers. I would not cite the headline numbers as they stand.\n\nBring it to reading group? Maybe, if the group wants to discuss evaluation pitfalls. But I would not accept the numbers.","headline":"Useful CHARM-on-the-head extension with a compact architecture, but the headline F1 scores are selection maxima because the hyperparameter search optimizes directly on the test set.","tokens_in":15654,"tokens_out":2324,"would_cite":false,"duration_ms":23691,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A hierarchical network using one head-mounted IMU recognizes nine long activities at 0.826 F1 and, by freezing its 22k-parameter motion encoder, three brief motions at 0.855 F1.","keywords":["human activity recognition","egocentric inertial measurement unit","smartglasses","hierarchical model","semi-supervised learning","motion embeddings","wearables","EgoCHARM"],"falsifier":"Pick a random sample of the reannotated low-level windows, watch the synchronized Aria video frame-by-frame, and assign a per-second label; if the video-derived labels disagree with the assigned uniform label in a nontrivial fraction of seconds, the reported 0.855 probing F1 is an upper bound on real per-second recognition.","tokens_in":14636,"feed_emoji":"🏃","tokens_out":9792,"duration_ms":86485,"temperature":0.7,"pith_summary":"EgoCHARM claims that a single head-mounted inertial measurement unit, paired with a hierarchical network of about 22k low-level and 63k high-level parameters, can recognize nine long everyday activities at 0.826 test F1 and three brief motions at 0.855 test F1. The training scheme deliberately needs only high-level labels: the low-level encoder learns motion embeddings through the high-level task, and a frozen copy plus a 99-parameter probe separates stationary, walking, and running. The point is to make always-on activity recognition on smartglasses cheap enough to run on the IMU chip itself, saving power and memory on the main processor. If true, smartglasses could track daily routines and feed context-aware assistants from motion alone, without continuous camera processing.","feed_headline":"22k parameters distinguish walking, running, and 9 daily activities","feed_subtitle":"Head-mounted motion embeddings trained only on high-level labels separate stationary, walking, and running.","key_machinery":"The carrying mechanism is the low-level motion encoder: a CNN-GRU that consumes six raw IMU channels (3-axis accelerometer, 3-axis gyroscope) at 50 Hz in 1-second windows and emits a 32-dimensional embedding per second. Its 1-D CNN layers run several parallel kernels with different dilation rates so the same small network can catch both fast and slow periodic patterns in head motion; the GRU then folds the temporal sequence into a single embedding. During training only high-level labels supervise the stack, so the encoder is forced to preserve motion distinctions useful over 30-second horizons. At inference, the same frozen encoder can be paired with a single tunable probing layer, which is what makes the 0.855 F1 result a test of embedding generalizability rather than of a separately trained classifier.","core_discovery":"On the paper's own terms, EgoCHARM establishes that one head-mounted IMU is enough to recognize both coarse and fine activities with a model of only tens of thousands of parameters. The system splits each 30-second high-level window into 1-second windows, encodes each second into a 32-dimensional motion embedding with a CNN-GRU using variable-dilation convolutions, and feeds the sequence of embeddings into a high-level GRU. Training uses only high-level labels, yet the frozen encoder, probed by one 99-parameter layer, separates stationary, walking, and running with 0.855 F1 (90.64% accuracy), and the full model classifies nine high-level activities with 0.826 F1 (82.86% accuracy). The reported parameter counts are 21,868 for the low-level encoder and 63,369 for the high-level head, with about 1.04M and 1.86M FLOPs respectively. Sensitivity experiments further claim that 15 Hz sampling, 20-second windows, and a few hundred training samples per class each preserve most of the performance.","pith_inferences":["An extension the paper leaves implicit: push the frozen-encoder probe to low-level classes beyond stationary, walking, and running; the PCA separation suggests the embeddings carry general motion structure, but only three probe classes are tested.","The FLOP counts are estimates, not measured power; the decisive resource test is deployment on a real IMU chip with on-chip compute at 15 Hz, measuring current draw and latency.","The mechanism is not fundamentally head-specific, so the same high-level-label-only recipe could be tested on wrist- or body-worn IMUs; success would extend the sample-efficiency claim beyond egocentric sensing."],"forward_implications":["The low-level encoder fits within the 25k-parameter budget of current IMU chips with on-chip compute, so the frequent 1-second inference can run on the sensor itself while the main processor handles only the 30-second high-level pass.","Dropping sampling to 15 Hz keeps high-level F1 at 0.807 and cuts low-level encoder FLOPs roughly 3.5x, so an always-on system can be made substantially cheaper with minimal accuracy loss.","A 20-second high-level window reaches 0.808 F1, close to the 30-second peak of 0.826, so high-level labels can be produced with lower latency than the training window.","Only about 500 high-level samples per class are enough for 0.748 F1, suggesting that useful egocentric recognizers can be trained with far less data than current large-scale datasets.","The frozen encoder's PCA clusters already separate stationary, walking, and running without seeing their labels, so the same embeddings are plausible inputs for other on-device tasks such as context for an assistant or triggering higher-power sensors."],"supporting_citations":[{"why":"Supplies the hierarchical CHARM design that EgoCHARM adapts: train low-level embeddings using only high-level labels, with hand-picked features replaced by raw IMU input and a CNN-GRU encoder.","marker":"[36]"},{"why":"Provides seven of the nine high-level activity classes and the large multi-participant data pool used to train and evaluate the high-level recognizer.","marker":"[14]"},{"why":"Provides the two Nymeria high-level classes and the 30-second activity text summaries from which the low-level stationary, walking, and running labels are reannotated.","marker":"[25]"},{"why":"Defines the CNN+GRU encoder style and the frozen-encoder probing evaluation used for low-level recognition, and serves as the large-parameter baseline.","marker":"[28]"},{"why":"Provides a second pretrained IMU encoder baseline with about 1.4M parameters that motivates the resource-efficiency comparison.","marker":"[8]"},{"why":"Defines the on-chip compute budget of current IMU chips, and motivates the under-25k-parameter constraint for the low-level encoder.","marker":"[4]"},{"why":"Documents another IMU with on-chip compute, supporting the claim that a 22k-parameter encoder can be deployed directly on existing sensor hardware.","marker":"[39]"},{"why":"Describes the Aria smartglasses whose single left IMU supplies the six accelerometer and gyroscope channels.","marker":"[9]"}],"fun_headline_variants":["63k parameters classify 9 daily activities from smartglasses","Head-mounted IMU uses 63k params for 9-activity recognition","EgoCHARM: 22k-parameter motion encoder with 0.85 F1","Tiny model on smartglasses tells walking, running, more","9 activities, 63k params, single head IMU—resource-efficient"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The per-second low-level labels are inferred from 30-second text summaries by assuming each chosen segment is uniformly stationary, walking, or running; if a segment hides brief stops or other motions, the probing F1 overstates true per-second accuracy.","fun_headline_variants_meta":{"raw":{"variants":["63k parameters classify 9 daily activities from smartglasses","Head-mounted IMU uses 63k params for 9-activity recognition","EgoCHARM: 22k-parameter motion encoder with 0.85 F1","Tiny model on smartglasses tells walking, running, more","9 activities, 63k params, single head IMU—resource-efficient"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000541,"raw_usage":{"total_tokens":2610,"prompt_tokens":980,"completion_tokens":1630,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":596,"completion_tokens_details":{"reasoning_tokens":1530}},"tokens_in":596,"tokens_out":1630,"duration_ms":11169,"temperature":1.0,"reasoning_tokens":1530,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:32:07.138886+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Pick a random sample of the reannotated low-level windows, watch the synchronized Aria video frame-by-frame, and assign a per-second label; if the video-derived labels disagree with the assigned uniform label in a nontrivial fraction of seconds, the reported 0.855 probing F1 is an upper bound on real per-second recognition.","supporting_citations":[{"cited_title":"CHARM: A Hierarchical Deep Learning Model for Classification of Complex Human Activities Using Motion Sensors","cited_arxiv_id":"2207.07806","evidence_quote":"Supplies the hierarchical CHARM design that EgoCHARM adapts: train low-level embeddings using only high-level labels, with hand-picked features replaced by raw IMU input and a CNN-GRU encoder."},{"cited_title":"Grauman, A","cited_arxiv_id":null,"evidence_quote":"Provides seven of the nine high-level activity classes and the large multi-participant data pool used to train and evaluate the high-level recognizer."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the two Nymeria high-level classes and the 30-second activity text summaries from which the low-level stationary, walking, and running labels are reannotated."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides a second pretrained IMU encoder baseline with about 1.4M parameters that motivates the resource-efficiency comparison."},{"cited_title":"Bhi360: Inertial measurement unit (imu)","cited_arxiv_id":null,"evidence_quote":"Defines the on-chip compute budget of current IMU chips, and motivates the under-25k-parameter constraint for the low-level encoder."},{"cited_title":"Lsm6dso16is","cited_arxiv_id":null,"evidence_quote":"Documents another IMU with on-chip compute, supporting the claim that a 22k-parameter encoder can be deployed directly on existing sensor hardware."}],"review_version":1}