{"id":"4bc05354-bdb1-4d07-8416-1b346f51266e","arxiv_id":"2508.10210","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A k-nearest-neighbors classifier on handcrafted neck-accelerometer features (156-sample window, 39-sample step) reaches about 0.94 test accuracy and 0.99 AUC for four merged dairy cow behavior classes.","lead":"This paper trains machine-learning models to classify four dairy cow behaviors from neck-worn 3-axis accelerometer data, with a k-nearest-neighbors model reaching about 0.94 test accuracy and a 0.99 test AUC. The intended practical use is continuous farm monitoring, with SHAP-based explanations that tell a farmer which sensor signal drove each prediction.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported 0.99 test AUC for KNN is likely inflated by temporal leakage: random split over 75%-overlapping windows with lag features generated before splitting.","rationale":"The reader's weakest assumption is precisely the same load-bearing concern: random splitting of overlapping windows with precomputed lag features violates train/test independence and inflates the reported 0.99 test AUC. I agree with that assessment and with the CONDITIONAL verdict. My stress-test does not add a new concern; it confirms the most critical one. I did consider whether annotation ground truth or the small number of cows (12) is a separate fatal flaw, but those are secondary: even if the labels were perfect, the evaluation leak alone makes the headline performance unverifiable. Conversely, a leakage-free re-evaluation could preserve the result, so the appropriate verdict remains conditional, not rejection. The concrete test I propose would settle the issue: a temporal, per-cow split with feature generation done intra-segment, which removes both the overlap and the lag-feature leakage. If the authors can provide such an evaluation, the central claim would be credible; until then, the reported numbers should not be taken at face value. The paper is otherwise readable and the feature-engineering/SHAP work is potentially useful, so I see no reason to shift the verdict to REJECT or ACCEPT on the current evidence.","tokens_in":11089,"tokens_out":2389,"duration_ms":27022,"concrete_test":"Re-run the entire pipeline with a temporal, cow-blocked split: for each of the 12 cows, divide the raw accelerometer stream chronologically into 60% training, 20% validation, and 20% test segments. Generate sliding-window features (window=156, step=39) and lag features separately within each segment, ensuring no window or lag crosses the split boundary. Train KNN with the reported hyperparameters (n_neighbors=3, p=1, weights='distance') and evaluate on the test segments. If the test AUC drops materially below 0.99 (e.g., <0.95) or test accuracy below 0.90, the reported result is inflated by temporal leakage.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is the KNN test performance (Tables 4–5; Figures 4c–4d): AUC 0.99, accuracy 0.9389 on the test set. This number is load-bearing because it drives model selection and the abstract's headline. The evaluation protocol in §2.5.2 Step 2 splits the dataset randomly into 60/20/20 while preserving class ratios, but the feature matrix is built by sliding a 156-sample window with a step of 39 (75% overlap) and then generating lag features up to 5 previous data points on the full time series before the split (Algorithm 1, lines 16–17). Consequently, adjacent rows in the feature matrix share most raw accelerometer samples, and lagged features reference earlier rows that may fall in a different split partition. A random split therefore places near-duplicate windows in both training and test, so the test set is not independent of training data. This inflates all reported metrics. The fact that the test AUC (0.99) exceeds the training AUC (0.98 ± 0.0026) is a further red flag, as test performance should not systematically exceed training performance under a clean protocol. The same issue affects the SHAP feature-stability analysis, which is derived from the same leaked feature matrix. Thus, the central performance claim is not substantiated until the evaluation is leakage-free.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents an end-to-end pipeline for classifying dairy-cow behavior from neck-mounted tri-axial accelerometer data. Ten annotated behaviors are merged into four classes (STN, RUS, REL, ETC). Features are extracted from sliding windows (window 156 samples, step 39 for the best configuration) and include statistical, wavelet, Savitzky-Golay, and lag features. Several machine-learning classifiers are hyperparameter-tuned and evaluated. The headline claim is that k-nearest neighbors achieves an AUC of 0.98 ± 0.0026 on training and 0.99 on the test set, with test accuracy 0.9389, precision 0.9404, recall 0.9322, and F1 0.9360 (Tables 4–5, Figures 4c–4d). SHAP analysis and KS-based feature stability are used to interpret the model. The paper is an application-oriented study emphasizing pre-processing, model selection, and explainability.","tokens_in":1825,"tokens_out":1664,"duration_ms":43606,"significance":"If the reported performance were obtained under a clean evaluation protocol, the paper would provide a useful, low-cost tool for automated cattle behavior monitoring and a concrete example of explainable AI for agricultural IoT. The authors deserve credit for releasing code, specifying the annotation pipeline with multiple cross-checks, reporting training-set standard deviations, and testing several window lengths and classifiers. However, the central quantitative claim is not yet substantiated because the train/test split is performed after constructing heavily overlapping feature windows with lag features, which can leak information across the split. The SHAP and stability analyses inherit the same problem. The contribution is therefore conditional on a leakage-free re-evaluation.","major_comments":[{"comment":"The evaluation protocol is vulnerable to temporal leakage. The dataset is split randomly into 60/20/20 after feature extraction, but the feature matrix is built by sliding a 156-sample window with step 39 (75% overlap) and lag features up to 5 points are generated on the full time series before the split (Algorithm 1, lines 16–17). Consecutive rows of the feature matrix share most raw accelerometer samples, and lagged features reference earlier rows that may fall into a different partition. A random split can therefore place near-duplicate windows in both training and testing, inflating all test metrics. The reported test AUC of 0.99 exceeding the training AUC of 0.98 ± 0.0026 is a diagnostic warning consistent with this leakage. The claim in Step 3 that leakage was prevented is not sufficient given this protocol. The authors should re-run the evaluation using a split that respects tempo","section":"§2.5.2, Step 2; Algorithm 1; Tables 4–5"},{"comment":"Test-set metrics are reported as single point estimates with no standard deviation, confidence interval, or significance testing. Since the best model is selected using these numbers, and the margins between KNN and other models are small (e.g., 0.9389 vs. 0.9327 accuracy), the reader cannot assess whether the reported ordering is meaningful. The authors should report test metrics with uncertainty, e.g., from repeated stratified splits or bootstrapping, or from the cross-validation folds used for hyperparameter selection.","section":"§2.5.2, Table 5"},{"comment":"The SHAP feature-importance analysis and the KS-based feature-stability analysis are performed on the same feature matrix and model used in the leaked train/test evaluation. If the feature matrix is contaminated by overlap and lag-derived leakage, the resulting SHAP rankings and stability categories are not trustworthy as descriptions of a model that generalizes to new cows or new time periods. The SHAP analysis also uses only 100 randomly sampled values (25 per class) with no repeated sampling or variance estimate; a stable ranking should be demonstrated with multiple random seeds.","section":"§2.6, Figures 5a–5d, Table 6"},{"comment":"The paper claims general applicability to 'monitoring animal health' but the data come from 12 cows on one farm, and the evaluation does not include a per-animal or per-farm split. If the same cow's windows appear in both training and test sets, the reported performance may reflect individual-level memorization rather than behavior recognition across animals. A leave-one-cow-out evaluation or an explicit analysis of between-cow variation is needed to support the stated generalization. This is a load-bearing limitation for the practical claim in the abstract and Discussion.","section":"§2.1, §2.4.2; Discussion"}],"minor_comments":[{"comment":"Minor syntax: '0.99 on testing set)' appears in the abstract and is repeated in the Discussion. Also, the phrase 'AUC of mean 0.98' should be 'mean AUC of 0.98'.","section":"Abstract and §3"},{"comment":"Figure 4a and 4b have identical captions ('Model Performance Analysis for Training Data'); one likely corresponds to the test set or to a different model comparison. The captions should be corrected.","section":"Figure 4"},{"comment":"The sentence 'First, we compress a minute’s worth of data into a single data point while extracting base features(converting 80 data points to 1)' is unclear because the sampling rate of the accelerometer is never stated. Without the sampling rate, the reader cannot map window lengths in samples (e.g., 156) to seconds, which is needed to interpret the results and reproduce the pipeline.","section":"§2.4.1"},{"comment":"Algorithm 1 uses 'AcX, AcY, AcZ' in line 9 while other lines use 'AccX, AccY, AccZ'. Please standardize notation. Also, line 16 says 'Generate LAG Features' but the exact data structure shifted (full time series vs. per-window features) should be made explicit.","section":"Algorithm 1"},{"comment":"Table 3 lists hyperparameters for only five model families, while §2.5.3 mentions Logistic Regression, Decision Tree, and Naive Bayes as also evaluated. The results for these models are not shown. Either add them or clarify that they were excluded from the comparison.","section":"Table 3"},{"comment":"Data availability says 'on request' with no clear repository or access policy. Given the reproducibility emphasis, a permanent data DOI or a detailed data-sharing plan is preferable.","section":"§9"}],"recommendation":"major_revision","confidential_remarks":"The paper is an application-oriented ML study with a clear pipeline and accessible code, but the evaluation protocol is the main obstacle. The random split over 75%-overlapping windows with pre-split lag features is a serious methodological flaw that invalidates the headline test metrics as reported. This is fixable: the authors can re-run the evaluation with a temporal or animal-grouped split and recompute all tables and figures. If, after re-evaluation, KNN remains competitive, the paper could be a solid contribution. I would also encourage the editor to ask for test-set uncertainty estimates and a per-animal analysis. The SHAP interpretation section should be clearly labeled as concerning the fitted model, not as causal findings about cow physiology."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is an honest applied ML paper with a useful recipe, but the headline test AUC of 0.99 is probably inflated by temporal leakage. The newness is narrow but real: a window/step comparison, DWT plus lag features, and SHAP/KS feature-stability analysis on a fresh 12-cow Punjab dataset. It is an extension of an established program, not a breakthrough, and the paper does not oversell it.\n\nWhat the paper does well: the data collection and annotation pipeline is the strongest part. Three levels of checks (video, in-person observer, peer review) with explicit behavior definitions is more careful than most of this literature. The model comparison is thorough, hyperparameter tables are complete, and the authors clearly state what they tried (SMOTE, undersampling) and why they merged minority classes. The code repository is linked, though without a hash, and the full dataset is on request.\n\nThe soft spot is the evaluation protocol, and it is load-bearing. Algorithm 1 builds features by sliding a 156-sample window with step 39, so consecutive windows overlap by 75%. The lag features are generated on the full time series before the 60/20/20 split. A random stratified split therefore puts near-duplicate windows and lagged references across the train/test boundary. The test AUC (0.99) exceeding the training AUC (0.98 ± 0.0026) is a red flag consistent with that leak. The manuscript even claims in Step 3 that the test set was segregated to prevent leakage, but that contradicts the algorithm ordering; the feature matrix is built before the split. The same issue contaminates the SHAP and KS stability results, so those \"findings\" are descriptive of the fitted model, not independent evidence. Without the dataset, the leak cannot be quantified, but the design makes it likely.\n\nFor who it is for: practitioners and precision-livestock researchers who want a concrete starting point for neck-mounted accelerometer pipelines. It deserves a serious referee because the empirical claim is fixable and the data-collection effort is valuable. I would accept at a conference or journal only after major revision: re-run the evaluation with a split by cow or by contiguous temporal block, report class-wise test metrics, and then re-draw the SHAP conclusions. Until then, I would not cite the reported AUCs, though I would cite the feature-engineering description if the code is cleaned up.","headline":"A useful applied ML recipe for cow behavior classification, but the reported test AUC of 0.99 is probably inflated by temporal leakage from overlapping windows and pre-split lag features; the data pipeline and model comparison are solid enough to deserve a serious referee, though the numbers need to be re-run.","tokens_in":11915,"tokens_out":2127,"would_cite":false,"duration_ms":24725,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Neck sensors plus KNN read cow behavior at 0.99 test AUC","keywords":["cattle behavior classification","accelerometer","wearable sensors","k-nearest neighbors","explainable AI","SHAP","feature stability","precision livestock farming"],"falsifier":"Retrain and retest the same KNN pipeline using a split by contiguous time blocks (e.g., morning versus afternoon) or by cow identity rather than random windows; if the test AUC falls well below 0.99, overlapping windows rather than a general behavior signal drove the score.","tokens_in":10836,"feed_emoji":"🐄","tokens_out":3700,"duration_ms":39489,"temperature":0.7,"pith_summary":"The paper reports that a k-nearest-neighbor classifier, trained on features extracted from neck-worn tri-axial accelerometer windows, can separate four merged classes of dairy cow behavior—standing/moving, ruminating, resting while lying, and miscellaneous activities—with a test AUC of 0.99 on a 12-cow farm. The authors combine statistical, wavelet, orientation, and lag-based features from sliding windows and use SHAP to explain which accelerometer axes drive each class. They argue this explainable model gives farmers actionable health and activity insights without needing to watch animals continuously. The central claim is that this specific feature-and-classifier configuration achieves near-perfect discrimination on this dataset.","feed_headline":"Neck sensors plus KNN read cow behavior at 0.99 test AUC","feed_subtitle":"Sliding-window accelerometer features separate standing, lying, ruminating, and odd activities on one 12-cow farm.","key_machinery":"The sliding-window feature-extraction pipeline is the central mechanism: raw three-axis accelerometer samples are compressed into minutes, then a window of 156 samples with step 39 yields statistical moments (mean, variance, skewness, kurtosis, quantiles), discrete wavelet transform coefficients, orientation angles (roll, pitch), and lag features up to five previous points. These feature vectors are classified by k-nearest neighbors, and SHAP attributions identify which axis-specific features carry each behavior's signal.","core_discovery":"The paper claims that KNN with 3 neighbors, Manhattan distance, and distance weighting, using a window of 156 samples and a step of 39, achieves a testing AUC of 0.99, accuracy of 0.9389, precision of 0.9404, recall of 0.9322, and F1 of 0.9360 when classifying four behavior classes constructed from ten raw annotated behaviors. The authors find that merging minority classes into an ETC (miscellaneous) class and standing plus moving into STN outperforms both undersampling and SMOTE oversampling. SHAP analysis shows that for standing, X-axis features dominate; for resting while lying, Y-axis features dominate; and for rumination, X- and Y-axis features matter most, aligning with the physical or","pith_inferences":["If the data were split by contiguous time blocks or by cow identity instead of random windows, the test AUC might drop substantially because consecutive windows heavily overlap and share raw samples.","The ETC class merges six different behaviors, so the model's ability to distinguish licking from attacking or drinking is not actually tested.","The prominence of lag features in SHAP suggests temporal dynamics carry much of the signal; evaluating KNN without lag features would isolate their contribution.","The near-perfect result is from one 12-cow farm and should not be read as proof of performance across breeds, seasons, or farm layouts."],"forward_implications":["The four-class scheme makes the problem tractable and outperforms the originally imbalanced ten-class setup.","KNN with the specified window and step sizes generalizes to unseen test windows with AUC 0.99 in this farm setting.","SHAP and KS-stability analysis yield a ranked, axis-specific feature set that can guide practitioners in interpreting accelerometer readings.","The pipeline can be extended to other accelerometer-based activity recognition tasks, as the authors state in conclusion.","Deviations in daily activity ratios could flag cattle health or welfare anomalies for farmers."],"supporting_citations":[{"why":"Provides the SHAP attribution method used to rank feature importance and explain model predictions.","marker":"(Lundberg and Lee, 2017)"},{"why":"Baseline tri-axial accelerometer study classifying housed dairy cow behaviors that this work extends.","marker":"(Vázquez Diosdado et al., 2015)"},{"why":"Source of the in-situ accelerometry classification approach and statistical feature methodology.","marker":"(Arablouei et al., 2021)"},{"why":"Describes SMOTE, the oversampling technique the authors tested and rejected in favor of class merging.","marker":"(Chawla et al., 2002)"},{"why":"Systematic review of accelerometer processing techniques for ruminant behavior that frames the feature-engineering strategy.","marker":"(Riaboff et al., 2022)"},{"why":"Prior evidence on window-length effects in accelerometer behavior classification motivating the window-size exploration.","marker":"(Decandia et al., 2018)"}],"fun_headline_variants":["Explainable AI turns cow motion into health insights","Neck-worn sensors and KNN hit 0.99 AUC for cow activity","Sliding-window features explain cow behavior to farmers","AI clarifies cattle behavior from accelerometer data"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The result assumes that randomly splitting overlapping 156-sample windows into train and test keeps the observations independent; because windows share most raw accelerometer samples, this premise is doubtful and the 0.99 test AUC likely overstates generalization to new continuous recordings.","fun_headline_variants_meta":{"raw":{"variants":["Explainable AI turns cow motion into health insights","Neck-worn sensors and KNN hit 0.99 AUC for cow activity","Sliding-window features explain cow behavior to farmers","AI clarifies cattle behavior from accelerometer data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00025,"raw_usage":{"total_tokens":1428,"prompt_tokens":821,"completion_tokens":607,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":565,"completion_tokens_details":{"reasoning_tokens":540}},"tokens_in":565,"tokens_out":607,"duration_ms":7779,"temperature":1.0,"reasoning_tokens":540,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T20:35:57.493840+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain and retest the same KNN pipeline using a split by contiguous time blocks (e.g., morning versus afternoon) or by cow identity rather than random windows; if the test AUC falls well below 0.99, overlapping windows rather than a general behavior signal drove the score.","supporting_citations":[],"review_version":1}