{"id":"845bf3e6-de55-45f9-a147-31c864aeba41","arxiv_id":"2505.18754","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"A hybrid Euclidean-distance and LLM-relevance example selector for few-shot sensor classification reports a small, statistically fragile gain over distance-only selection on a fatigue detection dataset.","lead":"This paper introduces a way to choose the examples shown to a large language model when classifying accelerometer data, by first finding numerically similar candidates and then asking the model to score which candidates are contextually relevant. Tested on fatigue detection in runners, the method edges out simpler selection, but the improvement is small and the evaluation setup weakens the claim.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The few-shot protocol leaks per-user labels: support examples are drawn from all other labeled samples of the same user for each test instance, making the evaluation transductive and not genuinely few-shot; the paper's own Nemenyi test also shows the 2.3% margin over distance-only is not…","rationale":"The reader's weakest assumption identified the transductive few-shot protocol, and my reading of Section 4.1 confirms that this is the single most load-bearing issue. For each test sample, the two support examples are selected from all other labeled samples of the same user, so the method has access to per-user label information that scales with the size of the user's data, not with the nominal 2-shot setting. This invalidates the paper's framing as few-shot optimization and means the comparison with the ML baseline (trained on exactly two examples) is not a controlled few-shot comparison. The statistical inconsistency between Table 4 (Nemenyi p=0.537) and Table 5 (t-test p=0.027) for the key HED-LM #ParamA vs Distance comparison further undermines the claim that the 2.3% gain is genuine; at best it is unproven, and at worst it is an artifact of an invalid protocol. The paper has useful engineering components—clear method description, reproducible dataset, and honest per-user reporting—but the central contribution as stated does not withstand scrutiny. My verdict adjustment is UNCHANGED because the reader's REJECT is appropriate; the concern I raise reinforces rather than redirects the rejection.","tokens_in":30461,"tokens_out":2533,"duration_ms":23028,"concrete_test":"Re-run the complete experiment under an inductive few-shot protocol: fix a single support set of two labeled examples (one per class) per user before seeing any test instance, never reveal labels of other instances during selection or scoring, and test on all remaining instances. Compare HED-LM, distance-only, and random under this protocol. If HED-LM's margin over distance-only shrinks, reverses, or becomes statistically insignificant (e.g., Wilcoxon signed-rank p>0.05 on the 19 user-level macro-F1 scores), the reported gains are an artifact of transductive per-user label access. Independently, resolve the statistical contradiction by pre-registering one significance test (e.g., Wilcoxon signed-rank) for HED-LM #ParamA vs Distance and reporting it as the primary evidence; if p>0.05, the 2.3% relative improvement claim should be withdrawn.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.1 states that for each test instance, 'exactly two support examples drawn from the same user's data, strictly excluding the test instance itself' are used. This means the selection mechanism can examine the labels of the entire per-user pool (e.g., roughly 327 labeled instances for user10) before choosing the two support examples. The LLM re-ranking in Eq. 7 explicitly uses label synergy, so HED-LM can exploit per-user label distribution and class boundaries rather than demonstrating few-shot generalization from a fixed, small labeled set. Random and distance baselines use the same protocol, so the comparison is internally consistent, but the central claim—that HED-LM improves few-shot prompting—is not supported because the evaluation is not few-shot: the number of labeled instances available during inference is not limited to two. Additionally, the paper's Table 4 Nemenyi test reports p=0.537 for Distance vs HED-LM #ParamA, while Table 5 reports a t-test p=0.027 for the same comparison; this unresolved contradiction means the headline 2.3% relative gain over distance-only cannot be taken as established even under the paper's own protocol. The contribution of fitted domain rules (Appendix A.2 derives thresholds from the same dataset) further compounds the difficulty of attributing gains to the proposed selection mechanism.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes HED-LM, a two-stage example-selection method for few-shot prompting on sensor data: candidates are first filtered by Euclidean distance in a 30-dimensional feature space, then re-ranked by an LLM relevance score that combines numeric similarity with label synergy, and finally a 2-shot prompt is used to classify accelerometer fatigue vs. non-fatigue. Experiments on a 19-user running dataset report mean macro F1 of 69.13 for HED-LM #ParamA against 59.30 for random selection and 67.61 for distance-only selection, which the authors describe as relative improvements of 16.6% and 2.3%. The paper also studies the influence of LLM-generated domain knowledge and reports Friedman/Nemenyi and t/Wilcoxon significance tests.","tokens_in":30759,"tokens_out":6573,"duration_ms":51159,"significance":"The idea of combining a cheap numerical similarity filter with an LLM-based contextual re-ranker for in-context example selection is reasonable and potentially useful for sensor-data classification. The manuscript is transparent in several respects: it provides full prompt templates in the appendix, per-user results, confusion matrices, and a statistical analysis. However, the central claim is not supported as stated. The evaluation protocol is not a few-shot protocol, the paper's own Nemenyi test contradicts its pairwise significance claim for the main 2.3% margin, the domain knowledge is generated from the same dataset's feature values and then applied back to that dataset, and the HED-LM prompt is enriched with LLM score/reason fields that the baselines may not receive. These are load-bearing issues, not presentation fixes.","major_comments":[{"comment":"The evaluation is not few-shot. The protocol states that for each test instance, 'exactly two support examples drawn from the same user’s data, strictly excluding the test instance itself' are used, meaning the candidate pool for selection is the entire remaining labeled set of that user (e.g., 327 samples for user10). The method therefore has access to the labels of nearly the whole per-user set at inference time, and the LLM re-ranking in Eq. (7) explicitly uses label synergy to choose support examples. This does not demonstrate few-shot generalization from a small, fixed labeled set; it demonstrates per-test-instance retrieval from a large labeled pool. The headline claim of 'few-shot optimization' and the comparisons framed as 'few-shot' are consequently invalidated. The authors should either adopt a genuine few-shot protocol with a fixed support set drawn from a disjoint training split, or reframe the contribution as large-pool example retrieval and remove the few-shot claims.","section":"§4.1, Evaluation Protocol and Prompt Construction"},{"comment":"The paper contains a direct statistical contradiction. Table 4 reports a Nemenyi post-hoc p-value of 0.5369701 for Distance vs. HED-LM #ParamA, which is not significant, and the text explicitly states that Distance, HED-LM #ParamA, and HED-LM #ParamB 'are potentially in roughly equal performance clusters.' Yet Table 5 reports a t-test p-value of 0.02716 and a Wilcoxon p-value of 0.00240 for the same comparison, and the text claims that HED-LM #ParamA is 'significantly different (and likely superior) to Distance.' These two analyses cannot both be correct as presented. The Nemenyi test is the more conservative, multiple-comparison-corrected procedure, and on its basis the 2.3% mean improvement over distance-only is not statistically significant. The authors must resolve this inconsistency and revise the abstract and Section 1 claims accordingly, since the 2.3% relative improvement is a central quantitative contribution.","section":"§5.1, Tables 4 and 5"},{"comment":"The domain knowledge used in LLM scoring and in the few-shot prompts is generated by GPT-4o from the dataset's feature values (Figure A2: 'Based on the attached file, please analyze carefully to create a domain of knowledge'), and the resulting thresholds (e.g., 'Mean Acceleration: Values above 0.44 are indicative of fatigue') are then applied to score and classify the same dataset. This is circular: the scoring rules are fitted to the evaluation data, including its label-related feature distributions, so improvements attributed to 'domain knowledge' may simply reflect test-set information. To support the claim that domain knowledge improves performance, the thresholds must be derived from an independent training split or from external physiological sources, and the test subjects' feature values must not be used to generate the knowledge.","section":"Appendix A.2 and §3.4"},{"comment":"The HED-LM few-shot prompt includes, for each selected example, a 'Relevance Score and Reason' produced by the LLM scoring stage for that test instance. The Random and Distance baselines, which perform no LLM scoring, cannot include such fields, and the paper does not state that they receive equivalent score/reason information. The comparison is therefore confounded: part of HED-LM's gain may come from the additional prompt content (self-referential relevance scores and justifications) rather than from the example-selection mechanism itself. The authors should either provide the baselines with comparable scored examples (e.g., random or distance-based scores with reasons) or run an ablation of HED-LM that removes the score/reason fields from the prompt.","section":"§3.6 and Appendix A.3"},{"comment":"The hyperparameters distance-K and top-K are tuned on User ID 4 (Figure 4), and the same user's result in Table 3 (67.70% for #ParamA) is then reported as part of the aggregate mean. This introduces selection bias for that user and makes the mean over the 19 users not a clean evaluation of a fixed configuration. The parameters should be chosen on a validation set disjoint from the test users, or nested cross-validation should be used, with the selection procedure stated explicitly.","section":"§4.2.1 and Table 3"}],"minor_comments":[{"comment":"The text says the dataset 'consists of 6006 labeled subjects'; this should be '6006 labeled instances' or 'samples,' since the dataset has 19 participants.","section":"§3.1"},{"comment":"The abbreviation list defines HED-LM as 'Hybrid Edit Distance - Language Model,' but the paper consistently uses 'Hybrid Euclidean Distance with Large Language Models'; the abbreviation definition should match the body.","section":"Abbreviations"},{"comment":"There is a typo: 'melt detection' should be 'fatigue detection'.","section":"§5.1"},{"comment":"The sentence 'Using a consistent n = 2 across both ML and LLM paradigms ensures a balanced and realistic few-shot comparison' is misleading, because the ML baseline trains on exactly 2 samples while the LLM methods select support examples from a per-user pool of hundreds of labeled samples; the two paradigms do not use the same amount of label information.","section":"§4.1"},{"comment":"The Friedman test is reported as 'F-statistic = 54.55'; the Friedman test typically yields a chi-square statistic, and the Iman-Davenport F statistic is a different quantity. Please specify which test was used and report the appropriate statistic with degrees of freedom.","section":"§5.1, Table 4"},{"comment":"The conclusion repeats the claim that 'Post-hoc Nemenyi showed that the HED-LM approach (both #ParamA and #ParamB) formed a superior group significantly different from Random and traditional ML methods,' while the body of Section 5.1 also states that Distance, HED-LM #ParamA, and HED-LM #ParamB are not significantly different from each other; the conclusion should be consistent with the corrected post-hoc reading.","section":"Conclusion"}],"recommendation":"reject","confidential_remarks":"The manuscript is in scope for a sensor-data/LLM venue, and the authors have been unusually transparent about their prompts and per-user results. However, the combination of an invalid few-shot protocol, a self-contradictory significance analysis that undermines the headline 2.3% improvement, and circular dataset-derived domain knowledge means the central claims cannot be accepted. In my view these are not local fixes; they require redoing the experimental protocol and likely weaken the stated contribution substantially. I would not invite a major revision unless the authors are expected to reframe the paper as a large-pool retrieval method and re-run the evaluation with a proper train/test split."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth your time as a cautionary example, not as a result to build on.\n\nThe paper proposes HED-LM: filter candidate examples by Euclidean distance, re-rank with LLM relevance scoring, then feed the top two into a 2-shot prompt. The method description is clear, the appendices are thorough, and they use a public dataset. That part is fine.\n\nThe load-bearing problem is the evaluation. The paper calls it few-shot, but for each test instance the two support examples are drawn from all other labeled samples of the same user (per-user pool of roughly 327 instances). The selector has access to the entire per-user label set during selection. That is transductive, not few-shot. The random and distance baselines use the same protocol, so the comparison is internally consistent, but the headline claim about few-shot improvement is not supported.\n\nThe statistical reporting is also inconsistent. Table 4 (Nemenyi) says Distance vs HED-LM #ParamA is not significant (p=0.537), while Table 5 (t-test and Wilcoxon) claims significance (p=0.027 and p=0.002). Those cannot both be right. The paper never acknowledges the discrepancy.\n\nThere is a further leakage issue: Appendix A.2 shows GPT-4o derives domain-knowledge thresholds directly from the dataset's feature values, and those thresholds are then used in scoring and prompting the same data. That is fitted rules, not domain expertise, and it inflates the apparent benefit of the LLM component.\n\nThe new contribution is narrow: applying a known hybrid selection idea to accelerometer fatigue data. That is a valid application-level study, but the central empirical claim doesn't survive scrutiny. The improvement over distance-only is small, non-significant by their own Nemenyi test, and confounded by the label-rich selection protocol and threshold fitting.\n\nWho gets value: someone designing LLM prompts for sensor classification might appreciate the detailed prompt templates in the appendices. But as a research claim, the paper needs a proper few-shot protocol (fixed small labeled support set, held-out test), a corrected statistical account, and a discussion of the domain-knowledge leakage.\n\nMy recommendation: do not send to peer review as is. If the authors redo the evaluation with a genuinely few-shot setup and fix the stats, it could become a modest but acceptable contribution. I would not cite it as evidence for few-shot improvement.","headline":"HED-LM is a clear application of distance-filtering plus LLM re-ranking for sensor data, but the evaluation leaks per-user labels and the paper's own stats contradict whether the 2.3% gain over distance-only is real.","tokens_in":31319,"tokens_out":3288,"would_cite":false,"duration_ms":29872,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"For accelerometer fatigue detection, a hybrid selector that filters by Euclidean distance then re-ranks by LLM relevance reaches 69.13% mean macro F1, a relative improvement of 16.6% over random example selection.","keywords":["Few-shot prompting","Example selection","LLM scoring","Fatigue detection","Accelerometer data","Euclidean distance","HED-LM","Sensor data classification"],"falsifier":"Re-run the HED-LM comparison under a strict inductive split: choose two fixed support examples per user from a separate training portion (one fatigue, one non-fatigue) with no access to test-instance labels, and compare HED-LM, distance-only, and random selection. If the reported 16.6% relative gain over random shrinks or reverses under that split, the same-user label access is carrying the result.","tokens_in":30169,"feed_emoji":"📈","tokens_out":8280,"duration_ms":64135,"temperature":0.7,"pith_summary":"This paper proposes HED-LM, a two-stage example selector for few-shot prompting of large language models on sensor data. The first stage keeps the labeled samples closest to the test instance by Euclidean distance in a 30-dimensional feature space; the second stage has an LLM score each candidate for numeric similarity and label alignment, re-ranks them, and hands the top two to a 2-shot prompt. The paper argues that this pairing addresses the weakness of each ingredient alone: distance ignores whether a numerically close sample carries the right label, while LLMs handle contextual reasoning but read raw numeric signals poorly. On accelerometer data from 19 runners, HED-LM achieves a mean macro F1 score of 69.13%, a relative improvement of 16.6% over random example selection and 2.3% over distance-only filtering. The intended conclusion is that numeric and contextual similarity are complementary criteria for choosing few-shot examples.","feed_headline":"Hybrid example picker lifts few-shot fatigue scores 16.6%","feed_subtitle":"Pairing Euclidean distance with LLM relevance scoring beats random and distance-only example choice on accelerometer data.","key_machinery":"The load-bearing mechanism is the HED-LM selection pipeline. Each raw 180-sample acceleration trace is windowed into three 60-sample segments, low-pass filtered, and reduced to 30 features (ten per segment, including mean, RMS, skewness, kurtosis, dominant frequency, and low-band energy). A test instance's feature vector is compared against all labeled samples by Euclidean distance; the distance-K closest candidates are then scored by an LLM on a 0-to-1 relevance scale that combines numeric feature differences with label synergy, and the top-K are re-ranked for prompt construction. The prompt itself is 2-shot, carrying one fatigue and one non-fatigue example, plus domain-knowledge threshold rules (for example, 'RMS values above 0.5 in segments 2 and 3 typically indicate fatigue') that the LLM uses during scoring and final classification. The design is what makes the argument run: the distance stage supplies numeric proximity cheaply, and the LLM stage supplies the contextual judgment the distance metric lacks.","core_discovery":"The paper's central claim, stated in its own terms, is that combining a Euclidean-distance prefilter with LLM-based contextual relevance scoring improves few-shot prompting on high-variability sensor data. On a dataset of 6006 accelerometer strides from 19 runners labeled fatigue or non-fatigue, HED-LM with distance-K=5 and top-K=3 reaches a mean macro F1 of 69.13±10.71%, compared with 59.30±10.13% for random selection and 67.61±11.39% for distance-only filtering. The comparison is run per user, with each test instance's 2-shot prompt built from the same user's other labeled samples, and the paper reports that HED-LM forms a statistically distinct, higher-performing cluster than random selection and a Random Forest baseline, while its post-hoc tests place distance-only and HED-LM in the same performance cluster. The intended moral is that the hybrid selector is robust across users and that domain-knowledge thresholds embedded in the prompts sharpen the LLM's relevance scores from mid-range to decisive values.","pith_inferences":["My reading of the paper's own statistics is that distance-only filtering deserves most of the credit: the paper's post-hoc tests place distance-only and HED-LM in the same high-performance cluster, so a reader should expect the LLM re-ranking to add a small, user-dependent margin rather than a step change.","The evaluation is transductive: support examples for each test instance are drawn from the same user's labeled pool, so the method is really selecting which labeled neighbors accompany a query, not learning a two-example training set that transfers to new users; a strict inductive few-shot split would likely narrow the gap over random selection.","A cheap falsifying experiment follows from that transductive setting: compare HED-LM against a plain k-nearest-neighbor classifier on the same 30 features; if k-NN matches or beats HED-LM, the LLM scoring stage is not adding information beyond the distance metric.","The hand-written domain-knowledge thresholds are a likely source of user-specific sensitivity (the paper's own User-10 failure), so a natural extension is to learn those thresholds from a held-out slice per user and test whether gray-zone cases become less frequent."],"forward_implications":["Under the paper's protocol, random example selection and a traditional Random Forest trained on two samples form a lower-performing cluster, so few-shot prompting on sensor data is not automatically better than a simple baseline.","Most of the hybrid gain is carried by the distance prefilter: the paper's mean improvement over distance-only filtering is 2.3% relative, and its own significance tests put distance-only and HED-LM in the same performance cluster.","Two well-chosen examples can beat a full-shot prompt: the paper reports 2-shot HED-LM at 67.70% macro F1 on User 4 versus 52.68% for its full-shot variant, with roughly 7.7 times lower inference time.","Domain-knowledge thresholds are doing real work: when they are removed, LLM relevance scores cluster in the 0.4–0.6 range and re-ranking loses power, so the reported gains depend on injecting those rules into the prompts.","Per-user evaluation matters for sensor few-shot tasks: macro F1 across individual users in the paper ranges from about 52% to 90% depending on user, so pooled numbers alone hide which subjects benefit from selection."],"supporting_citations":[{"why":"Supplies the public accelerometer fatigue dataset from 19 runners with 6006 stride-level instances used in all experiments.","marker":"[33]"},{"why":"Prior work on this task showing random example selection can underperform zero-shot prompting, motivating the structured selector.","marker":"[11]"},{"why":"Supports the premise that well-aligned examples improve few-shot accuracy, cited as up to 30% over generic examples.","marker":"[6]"},{"why":"Cited for the difficulty LLMs have interpreting temporal sensor data, motivating the feature-summary prompt format.","marker":"[13]"},{"why":"Supports the claim that LLMs struggle to capture subtle variations in sensor readings for human activity recognition.","marker":"[15]"},{"why":"Provides the survey basis for choosing Euclidean distance and for the alternative distance metrics discussed in limitations.","marker":"[36]"}],"fun_headline_variants":["Hybrid example picker lifts few-shot fatigue F1 by 16.6%","LLM rerank on top of distance improves few-shot fatigue","Fatigue detection few-shot: hybrid selection beats random by 16.6%","Euclidean prefilter plus LLM rerank wins few-shot example picks","Combining distance and LLM boosts fatigue few-shot to 69% F1"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that selecting each test instance's two support examples from the same user's other labeled samples is a legitimate few-shot protocol; if that transductive access to nearly all per-user labels is treated as leakage rather than few-shot learning, the headline comparisons against random and distance-only selection would not transfer to a strict train/test split.","fun_headline_variants_meta":{"raw":{"variants":["Hybrid example picker lifts few-shot fatigue F1 by 16.6%","LLM rerank on top of distance improves few-shot fatigue","Fatigue detection few-shot: hybrid selection beats random by 16.6%","Euclidean prefilter plus LLM rerank wins few-shot example picks","Combining distance and LLM boosts fatigue few-shot to 69% F1"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000614,"raw_usage":{"total_tokens":2894,"prompt_tokens":1027,"completion_tokens":1867,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":643,"completion_tokens_details":{"reasoning_tokens":1765}},"tokens_in":643,"tokens_out":1867,"duration_ms":13520,"temperature":1.0,"reasoning_tokens":1765,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:26:23.524625+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the HED-LM comparison under a strict inductive split: choose two fixed support examples per user from a separate training portion (one fatigue, one non-fatigue) with no access to test-instance labels, and compare HED-LM, distance-only, and random selection. If the reported 16.6% relative gain over random shrinks or reverses under that split, the same-user label access is carrying the result.","supporting_citations":[{"cited_title":"Generating Explanations to understand Fatigue in Runners Using Time Series Data from Wearable Sensors","cited_arxiv_id":null,"evidence_quote":"Supplies the public accelerometer fatigue dataset from 19 runners with 6006 stride-level instances used in all experiments."},{"cited_title":"Leveraging Large Language Models to Enhance Understanding of Accelerometer Data on Physical Fatigue Detection Question Answering","cited_arxiv_id":null,"evidence_quote":"Prior work on this task showing random example selection can underperform zero-shot prompting, motivating the structured selector."},{"cited_title":"A survey on distance and similarity measures for time-series data analysis","cited_arxiv_id":null,"evidence_quote":"Provides the survey basis for choosing Euclidean distance and for the alternative distance metrics discussed in limitations."}],"review_version":1}