{"id":"15284a60-4839-4c5a-a3e6-1d626f6c2bc4","arxiv_id":"2501.06219","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"WhACC, a two-stage ResNet50V2 and LightGBM classifier with engineered temporal features, matches expert human whisker-touch labeling and reduces curation effort by over 98%.","lead":"WhACC is a software pipeline that automatically detects when a mouse whisker touches an object in high-speed video using a neural network plus gradient-boosting classifier. It reports expert-human-level agreement with human curators and reduces curation time from about 333 hours to about 6 hours for a 100 million frame dataset.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Retraining-validation claim may be overstated because the holdout sessions have no independently curated ground truth, and the retrained models are compared only against a single curator whose error rate is unknown.","rationale":"The reader's weakest assumption was that the human majority labels are noisy, particularly around touch edges, and that the paper's central metric measures agreement with those annotators. My concern is related but narrower and more concrete: I focus on the 16-session retraining holdout, where there is no majority at all, only a single curator with an acknowledged unknown error rate. The paper's three-curator test set (Section 3.7) is a fair relative comparison between WhACC and individual curators conditioned on the majority. The retraining holdout, in contrast, is the only place where the paper claims validated performance on a new dataset, and it lacks the independent human baseline needed to say whether post-retraining TC-errors are small in an absolute sense. This is an addressable experimental-design gap rather than a fatal flaw: the pre/post comparison still demonstrates improvement, and the abstract's time-savings claim (333 to 6 hours) does not depend on this baseline. I therefore recommend CONDITIONAL with the condition that at least a subset of the holdout be independently re-curated, rather than a stronger verdict change. I agree with the reader partially: the reader identified the majority-label noise issue as the weakest assumption for the central head-to-head claim; my concern is the lack of an independent baseline on the retraining holdout, which is a related but distinct gap in the same argument.","tokens_in":18100,"tokens_out":1558,"duration_ms":13033,"concrete_test":"Have two additional expert curators label a stratified sample (e.g., 5-10 sessions of the 16-session holdout, or at least 100k frames) and compute the three-curator majority. Then recompute WhACC's TC-errors and edge errors against that majority and compare them to the single-curator numbers in Figure 6B. If WhACC's TC-errors against the majority are substantially lower than against the single curator, the retraining claim is conservatively supported; if they are higher, the published numbers overstate accuracy. This mirrors the paper's own Section 1.2 methodology for estimating human error rate.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central comparison in the retraining validation (Figure 6, Section 3.6) treats a single expert curator's labels as ground truth for the 16-session holdout. The paper explicitly states: 'This dataset was curated by only one expert curator, so the human error rate is not known.' This matters because Section 1.2 shows human curators disagree on roughly 1-in-200 frames and make touch-count errors at a measurable rate. If the single curator is noisy or biased on these sessions, the reported post-retraining TC-errors are contaminated by curator error, not pure model error. The arrow of inference is also asymmetric: WhACC's post-retraining TC-errors are quoted against this single curator, whereas the headline 'expert human-level performance' was established on the three-curator majority test set. No three-curator majority exists for the 16-session holdout, so the claim 'retraining yields expert-level performance on new data' has no independent-error baseline on the very data used to demonstrate it. The concern is not that the pre/post retraining comparison is meaningless; it does show improvement. The load-bearing risk is that the magnitude of the post-retraining improvement, and the ability to attribute residual errors to the model rather than the curator, is not established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript describes WhACC, a two-stage pipeline (ResNet50V2 feature extraction followed by LightGBM classification) for detecting whisker-pole contact in head-fixed mouse high-speed video. Training labels are majority decisions of three human curators; inputs are object-centered 61x61 crops with temporal lag frames and augmentation. The authors report that WhACC achieves human-level or better touch-count error on a large test set, that a feature-selection procedure reduces 84,009 features to 2,105, and that a small-sample retraining protocol improves performance on 16 held-out sessions (~4 million frames). The claimed practical benefit is a reduction from about 333 to about 6 curation hours per 100 million frames.","tokens_in":18331,"tokens_out":9118,"duration_ms":89845,"significance":"WhACC addresses a real bottleneck in systems neuroscience, and the paper has several strengths: the model comparison is extensive, the code and model are promised in a public repository, limitation sections are candid, and the retraining idea is pragmatic. If the human-level claim is established on a properly independent benchmark, this would be a useful methods contribution. The main risks are that part of the reported feature-selection gain is evaluated on data used to choose features, and the retraining validation lacks an independent reference label set; both need to be addressed before the headline performance claims can be taken at face value.","major_comments":[{"comment":"The feature-selection procedure is not leakage-free. The manuscript states in §1.4 that \"some test data was used for feature selection\" and in §3.4 that the split for feature selection was made \"based on frame index (as opposed to segment or video)\". Because the engineered features include rolling windows up to 61 frames, shifts, and discrete differences up to ±50 frames, a frame-index split allows temporal context from training frames to enter validation features. This is load-bearing: Table 1 reports the selected-feature model as the best TC-error model (0.202 after smoothing vs. 0.214 for the full-feature model), and the final WhACC model uses the selected features. The reported advantage of feature selection, and possibly the Figure 5 human-level evaluation, is therefore not a clean generalization estimate. The authors should repeat feature selection on a split that respects video identity and does not touch the final test set, and report the independently selected model's performance on a fully untouched holdout.","section":"§1.4 and §3.4"},{"comment":"The retraining validation uses a single expert curator as the reference, and the paper explicitly notes that \"the human error rate is not known\" for this dataset. Yet Figure 6 and the PSTH comparisons treat this single curator as ground truth for TC-error, spike counts, peak response time, and signal-window width. Since §1.2 shows that expert curators disagree on roughly 1 in 200 frames and that human touch-count errors are a measurable fraction of touches, the post-retraining error and the reported 1-2 ms earlier peak response cannot be attributed cleanly to the model. The pre/post retraining comparison does demonstrate improvement, but the claim of \"expert-level performance on new data\" requires a multi-curator consensus on at least a subset of the 16 sessions, or a re-analysis that explicitly accounts for the single curator's error rate.","section":"§3.6 and Figure 6"},{"comment":"The comparison between WhACC and human curators is asymmetric in a way that can favor WhACC. The model is trained on majority labels from the same three curators whose pairwise consensus is used as the reference when estimating human error rates. The human error rate is estimated by holding one curator out from the two-curator consensus, but WhACC has been optimized to reproduce the majority behavior of all three curators. Even on unseen videos, consistent annotator biases can therefore be baked into WhACC's training target, making it easier for WhACC to match a two-curator consensus than a genuinely independent human would be. The comparison would be fairer if WhACC's training labels excluded the pair of curators used as the evaluation consensus, or if an independently generated annotation protocol were used to define the test-set reference.","section":"§1.2 and §1.4"}],"minor_comments":[{"comment":"The caption states that gray outlines indicate sessions from the top row in A and black outlines indicate those from the top row in A; one of these should refer to the bottom row.","section":"Figure 6 caption"},{"comment":"The text refers to the \"original-feature model with 2,408 features,\" but the correct number is 2,048; Table 1 also lists the full feature count as 84,006 while the text consistently says 84,009.","section":"§2.6 and Table 1"},{"comment":"The abstract reports \"pairwise touch classification agreement on 99.5% of video frames\" while the body reports that all three curators agree on 99.46%; please define both quantities and reconcile the numbers.","section":"Abstract and §1.2"},{"comment":"The claim that WhACC made \"fewer TC-errors on average\" than human curators is not accompanied by an inferential test or the per-comparison values; with only three curators, a paired effect-size statement or exact comparisons would be useful.","section":"Figure 5"},{"comment":"The statement that low-frame-rate tests \"show promise\" is anecdotal unless data are provided; please either add quantitative results or clearly label this as untested.","section":"§2.3"},{"comment":"The manuscript has many typographic artifacts (for example, \"T able\", \"T o\", \"T rue\") that should be corrected in copy editing.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"This is a potentially useful methods paper, and the authors are admirably transparent about several limitations. However, the feature-selection leakage and the single-curator holdout validation directly affect the two central claims (expert-level performance and successful retraining). I would like to see a revision that re-runs feature selection without touching the test set and adds independent human consensus labels for at least part of the retraining holdout. If those analyses confirm the current numbers, I would support publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"WhACC is a practical win. The specific combination—ResNet50V2 features, engineered temporal features, LightGBM, median smoothing—is not a new scientific principle, but the system is carefully built and validated at a scale that matters: over a million frames with three curators, plus an independent 16-session holdout. The paper reports that WhACC matches or slightly beats individual human curators on touch-count errors, the metric that most affects downstream electrophysiology, and the retraining workflow cuts curation time from hours to minutes. That claim is plausible and largely supported.\n\nThe strongest part is the evaluation against human curators: three-expert majority as ground truth, with per-curator error rates from leave-one-out consensus, and a test set drawn from different labs and years. The authors are also unusually honest: they state there is no independent ground truth, they admit that some test data was used during feature selection, and they list the conditions (frame rate, multi-whisker, object shape, contrast) where performance is uncertain.\n\nSoft spots. The admitted feature-selection leakage means the reported advantage of the selected-feature model over the full-feature model in Table 1 is not a clean comparison. That is real but bounded—it does not affect the full-feature model or the final WhACC test-set evaluation if the final split is video-level independent, though the paper does not state that split explicitly for WhACC. Bigger: the retraining validation (Figure 6) compares retrained WhACC against a single expert curator on the 16-session holdout, and the paper says \"the human error rate is not known.\" Given that human curators disagree on ~1 in 200 frames, part of the post-retraining TC-error could be curator error rather than model error. The pre/post improvement is still meaningful, but the \"expert-level on new data\" reading should be tempered.\n\nMinor: no code snapshot with commit hash or dataset is provided in the preprint, so independent verification is limited. Also, edge errors are higher for WhACC (more deduct errors), which the authors acknowledge—fine for PSTH onset analysis, but users should be aware.\n\nWho it's for: labs using head-fixed single-whisker pole-touch paradigms at ~1000 fps. It would save them real time. The central argument holds up. I'd send it to review; the authors should be asked to clarify the final split, add an inter-rater baseline for the retraining holdout, and release a versioned code snapshot.","headline":"Well-engineered, honestly reported tool; the human-level claim holds for the main test set, but the retraining validation would be stronger with a human-error baseline.","tokens_in":18885,"tokens_out":2099,"would_cite":true,"duration_ms":19269,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"WhACC, a two-stage hybrid of a convolutional neural network and gradient-boosted trees, labels whisker touches in high-speed video with expert human-level accuracy, and its retraining workflow cuts the human curation of a…","keywords":["WhACC","whisker touch classification","high-speed video","ResNet50V2","LightGBM","touch count error","temporal feature engineering","vibrissal system"],"falsifier":"Mount a force or strain sensor on the pole, synchronized to the video, so physical contact times are known independently; then compare WhACC's and a human curator's touch-count and edge errors against that sensor. If WhACC's touch-count errors relative to the sensor are significantly worse than the human curator's, the 'expert human-level' claim would be refuted.","tokens_in":17876,"feed_emoji":"🐭","tokens_out":6893,"duration_ms":57396,"temperature":0.7,"pith_summary":"The paper claims that WhACC, a two-stage model built from a ResNet50V2 feature extractor and a LightGBM classifier, can label whisker-pole touches in 1,000 fps video as accurately as expert human curators, in a fully automated pass. On test data spanning two labs and eight years, WhACC agreed with expert curators on about 99.5% of frames and made fewer touch-count errors on average than any individual curator, though it committed more edge errors and therefore tended to shorten touches slightly. The authors argue this matters because manual curation of touch events currently costs over three hours per million video frames, and their retraining workflow, validated on 16 held-out sessions of about four million frames, cuts the human time to roughly six hours for a 100-million-frame dataset. The goal is to replace the second-stage manual curation that current tracking tools still require, giving systems neuroscience millisecond-precision touch labels from video alone.","feed_headline":"Whisker touch labeling now rivals expert curators","feed_subtitle":"WhACC automates 1,000 fps video curation: human-grade accuracy, with 333 hours of work cut to about 6.","key_machinery":"The load-bearing mechanism is the two-stage hybrid WhACC pipeline. Stage one is a fully unfrozen ResNet50V2, pretrained on ImageNet and trained on 61x61 pixel crops centered on the object, with three consecutive video frames stacked into the cyan, magenta, and yellow color channels so each prediction has access to times t, t-1, and t-2; training uses random rotation, shift, zoom, brightness, and Gaussian noise augmentation. Stage two feeds the 2,048 penultimate-layer features through engineered temporal transformations, including forward and backward shifts up to five frames, rolling means and standard deviations over windows from 3 to 61 frames, and discrete differences over offsets from -50 to +50, yielding 84,009 candidate features that an ensemble of LightGBM models recursively prunes to 2,105 high-value features. A final LightGBM classifier, tuned by Optuna and followed by five-frame median smoothing, makes the touch or non-touch decision. These two temporal mechanisms, lag-frame color stacking and engineered rolling and difference features, are what let a per-frame CNN classifier behave like a curator who scrolls back and forth to find touch onset and offset.","core_discovery":"The central discovery is that expert-level touch classification does not require a larger or more complex network: a comparatively lightweight CNN trained on small object-centered crops, followed by engineered temporal features and gradient-boosted trees, matches human curation while being fully automatic. The paper demonstrates this by comparing WhACC against three expert curators on a shared test set, using a majority-of-three consensus as ground truth. Measured by touch-count errors (splits, ghosts, misses, joins), the errors that most degrade touch-count and peri-stimulus time histogram analyses, WhACC outperformed the average human curator; measured by edge errors it made more deductions, trimming touches at their start. Pairwise frame agreement was about 99.5%, equal to agreement between human curators. A separate 16-session holdout showed that retraining LightGBM on 100 to 1,000 frames per session rescues poorly performing sessions, and touch-evoked PSTHs from WhACC labels matched a human curator's for touch-responsive neurons.","pith_inferences":["A natural next test is to instrument the pole with a force or strain sensor to create an independent physical ground truth; if human labels are themselves noisy at onset and offset, WhACC's consistency could actually exceed human performance for population-level measures.","The recipe of CNN features plus engineered temporal features plus gradient boosting plus median smoothing is not whisker-specific, and could be reused for other high-speed video tasks where small object-contact events must be labeled with temporal consistency.","The 100-frame retraining result suggests a cheaper active-learning loop: sample frames near predicted touch onsets and offsets, curate only those, and retrain, which could reduce the six-hour estimate further.","Because lag images hurt MobileNetV3 models, putting temporal information in color channels interacts with architecture choice; engineered features on CNN embeddings may be a more transferable way to add temporal context than recurrent layers."],"forward_implications":["For labs using similar single-whisker, pole-contact, 1,000 fps head-fixed setups, WhACC can serve as a fully automated first-pass curator, with touch-count errors at or below the average human curator.","Because WhACC's PSTHs matched human curators for spike counts on seven touch-responsive neurons, its labels can feed standard electrophysiology analyses without manual re-curation, though peak responses ran one to two milliseconds early.","The retraining workflow, validated on 16 held-out sessions of about four million frames, fixed most of the five initially poor sessions with 100 frames per session and produced additional gains with 1,000 frames per session.","Median smoothing of predictions is a simple uniform booster: the paper reports it alone cut touch-count error by 80% in the original-feature model, and combining smoothing with feature engineering reduced touch-count error to 0.202.","For datasets outside the training conditions, such as different frame rates, multi-whisker video, or non-pole objects, performance is not established without retraining."],"supporting_citations":[{"why":"Supplies the whisker tracing and linking that human curators used to establish touch ground truth, and defines the manual curation baseline that WhACC is meant to replace.","marker":"[11]"},{"why":"Provides the ResNet50V2 architecture used as the feature extractor stage of WhACC.","marker":"[17]"},{"why":"Provides the LightGBM gradient-boosted classifier that takes the engineered features and is retrained on small subsets.","marker":"[20]"},{"why":"Supplies the Optuna hyperparameter optimization framework used to tune the final LightGBM model.","marker":"[21]"},{"why":"Motivates the touch-count error metric by showing that the total number of touches is a key behavioral measure in whisker-guided object localization.","marker":"[4]"},{"why":"Establishes the millisecond-precision touch timing requirements that WhACC is designed to meet for somatosensory cortex recordings.","marker":"[8]"},{"why":"Serves as the closest related work, combining gradient boosting with temporal localization in video, and is the comparison point for the method's novelty.","marker":"[32]"}],"fun_headline_variants":["WhACC: expert whisker touch detection, 98% less manual work","Human-level whisker labeling, 333h cut to 6h with WhACC","WhACC automates whisker touch tagging to human standards","WhACC matches expert curators, slashes labeling time 98%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the majority vote of the paper's three expert human curators is an adequate ground truth for touch; if those humans are systematically biased or noisy, especially around touch onset and offset, then 'expert human-level' means agreement with those annotators rather than true detection.","fun_headline_variants_meta":{"raw":{"variants":["WhACC: expert whisker touch detection, 98% less manual work","Human-level whisker labeling, 333h cut to 6h with WhACC","WhACC automates whisker touch tagging to human standards","WhACC matches expert curators, slashes labeling time 98%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000737,"raw_usage":{"total_tokens":3299,"prompt_tokens":956,"completion_tokens":2343,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":572,"completion_tokens_details":{"reasoning_tokens":2263}},"tokens_in":572,"tokens_out":2343,"duration_ms":69389,"temperature":1.0,"reasoning_tokens":2263,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:55:02.189829+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Mount a force or strain sensor on the pole, synchronized to the video, so physical contact times are known independently; then compare WhACC's and a human curator's touch-count and edge errors against that sensor. If WhACC's touch-count errors relative to the sensor are significantly worse than the human curator's, the 'expert human-level' claim would be refuted.","supporting_citations":[{"cited_title":"Automated tracking of whiskers in videos of head fixed rodents","cited_arxiv_id":null,"evidence_quote":"Supplies the whisker tracing and linking that human curators used to establish touch ground truth, and defines the manual curation baseline that WhACC is meant to replace."},{"cited_title":"Identity Mappings in Deep Residual Networks","cited_arxiv_id":null,"evidence_quote":"Provides the ResNet50V2 architecture used as the feature extractor stage of WhACC."},{"cited_title":"Lightgbm: a highly efficient gradient boosting decision tree","cited_arxiv_id":null,"evidence_quote":"Provides the LightGBM gradient-boosted classifier that takes the engineered features and is retrained on small subsets."},{"cited_title":"The sensorimotor basis of whisker-guided anteroposterior object localization in head-fixed mice","cited_arxiv_id":null,"evidence_quote":"Motivates the touch-count error metric by showing that the total number of touches is a key behavioral measure in whisker-guided object localization."},{"cited_title":"Low-noise encoding of active touch by layer 4 in the somatosensory cortex","cited_arxiv_id":null,"evidence_quote":"Establishes the millisecond-precision touch timing requirements that WhACC is designed to meet for somatosensory cortex recordings."},{"cited_title":"Learning to Localize Temporal Events in Large-scale Video Data","cited_arxiv_id":"1910.11631","evidence_quote":"Serves as the closest related work, combining gradient boosting with temporal localization in video, and is the comparison point for the method's novelty."}],"review_version":1}