{"id":"bafae91a-dd20-4f0c-979d-bbf282bc80ae","arxiv_id":"2501.04292","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A shared-task benchmark shows a simple CNN on audible-range spectrograms distinguishes autism-model mice from wild-type mice with UAR around 0.60 at segment level and 0.625 at subject level.","lead":"This paper introduces a new machine-learning challenge for detecting autism in mice from recorded calls, with a baseline system that reaches above-chance accuracy. It is the first INTERSPEECH challenge on animal vocalizations for autism research, and reports that audible-range sounds work better than ultrasound for this task.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Audible-band superiority may be an artifact of non-vocal recording noise, and the reported significance is inflated by testing across random seeds rather than independent subjects.","rationale":"The reader's weakest_assumption identifies the same issue I would stress: the audible-band result is biologically interpretable only if the sub-20 kHz energy is actually vocalization signal. The paper presents no control for non-vocal acoustics, which is conspicuous because the recordings are unsegmented and the acquisition hardware is optimized for ultrasound. This is not an internal inconsistency but an untested alternative explanation. The additional subject-level significance problem strengthens the concern: 16 test subjects with 10 correct is not significant, and the reported t-test across five seeds does not address subject-level generalization. My recommendation is unchanged conditional acceptance: the challenge infrastructure and benchmark remain valuable, but the abstract's 'audible vocalizations' finding and the significance claims should be revised or supplemented with the noise-window test and a subject-level permutation test. The code and data release make this test feasible.","tokens_in":8703,"tokens_out":7556,"duration_ms":85194,"concrete_test":"Using the released dataset and code, extract below-20 kHz spectrograms only from 30-second windows in the test recordings that contain no detectable ultrasonic energy above 25 kHz (i.e., no USV calls), and evaluate the published audi CNN on these noise-only windows. If UAR remains near 0.600, the audible-band classifier is exploiting non-vocal recording artifacts; if UAR drops to chance, the vocalization-signal interpretation is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4 defines the audi feature set as the sub-20 kHz portion of a spectrogram computed from unsegmented 5-minute recordings, and Section 5 interprets its higher UAR as evidence that audible vocalizations contain discriminative ASD information. The load-bearing premise is that below-20 kHz energy is vocalization signal. The paper never tests this: it reports no segmentation, no noise-floor or microphone calibration, no litter/session batch controls, and no silence-only baseline. The Avisoft UltraSoundGate 416H is an ultrasonic acquisition system; without low-frequency calibration, 'audible' features may be dominated by cage or environmental noise that happens to correlate with genotype. The statistical support is also inflated: the one-tailed one-sample t-tests are computed over five random seeds, which are not independent subjects. The test set contains only 16 subjects, and subject-level UAR of 0.625 corresponds to 10/16 subjects correct, which is not significant under a binomial test (one-sided p≈0.23). Thus the claim of above-chance held-out-subject detection is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MADUV, an INTERSPEECH challenge task in which participants classify 30-second clips of mouse vocalizations as coming from wild-type or ASD-model mice. The authors describe a dataset of 84 subjects (40 ASD-model, 44 wild-type) recorded for about 5 minutes each at postnatal day 8, with a held-out subject-independent test set of 16 subjects split into 160 non-overlapping 30-second clips. They present a CNN baseline using three spectrogram-based feature sets (full, ultra, and audi, the latter restricted to below 20 kHz). The main empirical claims are that automated ASD detection is feasible, that the audible-range 'audi' features achieve the best performance (segment-level UAR 0.600, subject-level UAR 0.625 on the test set), and that spectral characteristics of audible vocalizations appear more discriminative for ASD type than ultrasonic features. The paper also describes challenge logistics: EvalAI hosting, submission limits, majority-voting subject-level evaluation, and public release of baseline code, checkpoints, and pre-extracted features.","tokens_in":8876,"tokens_out":3534,"duration_ms":38353,"significance":"If the central claims hold, the paper provides a useful, reproducible benchmark for a non-human vocalization classification task and a bridge between speech-technology methods and biomedical research. The authors have publicly released baseline code, checkpoints, and pre-extracted features, and the evaluation uses a held-out test set with withheld labels, which are strengths. However, the paper's most interesting scientific claim—that audible-range vocalization features are more discriminative than ultrasonic features—is currently supported mainly by a comparison of unsegmented spectrogram features, without controls for non-vocal acoustic content or recording-session confounds. The statistical evidence for above-chance subject-level performance is also weaker than the text implies, because the significance tests are computed across five random seeds rather than across independent subjects. These issues are load-bearing for the headline interpretation, so the paper needs major revision even though the challenge itself is a worthwhile contribution.","major_comments":[{"comment":"The 'audi' feature set is defined as the sub-20 kHz portion of a spectrogram computed from unsegmented 5-minute recordings, and Section 5 interprets its higher UAR as evidence that audible vocalizations contain discriminative ASD information. This is load-bearing, but the paper provides no control to show that below-20 kHz energy is dominated by mouse vocalization rather than by cage noise, handling sounds, ventilation, or recording-session/litter differences that happen to correlate with the mouse line. I request three additions: a vocalization-segmentation-based control (e.g., using a USV detector to retain only vocalization segments), a noise-only or silence-only baseline, and per-litter or per-session ablations or recordings-order analysis. Without such controls, the 'audible-range vocalizations are more discriminative' claim is not established.","section":"§4 Feature Extraction and §5 Results"},{"comment":"The one-tailed one-sample t-tests are computed across five random seeds, but seeds are not independent statistical units; the test set contains only 16 subjects. The subject-level UAR of 0.625 corresponds to 10/16 subjects classified correctly, which is not significantly above chance under a binomial test (one-sided p ≈ 0.23). Please replace the seed-based t-tests with subject-level permutation tests or bootstrap confidence intervals, and report exact p-values. The current phrasing 'above-chance performance' is exaggerated for the subject-level result.","section":"§5 Statistical Significance"},{"comment":"The headline segment-level UAR of 0.600 is the maximum across five seeds, selected as the run with the best validation UAR. The mean segment-level UAR is 0.588 ± 0.016, and no confidence intervals are reported for any test-set number. Reporting the maximum as 'the benchmark' overstates the stability of the result. Please report all five seed-wise test results (or at least median and range), and provide confidence intervals for both segment-level and subject-level UARs.","section":"§5 Table 2"},{"comment":"Training and validation clips are created from 5-minute recordings with 15-second overlap, yielding 19 clips per original recording, while test clips are non-overlapping. This means that segment-level training/validation examples are strongly dependent within a subject, and the validation UARs in Table 2 are not based on independent samples. The paper should either use subject-blocked cross-validation, explicitly account for within-subject correlation, or at least quantify the effective number of independent samples. This is relevant because the baseline's threshold and model selection are based on these validation numbers.","section":"§4 Segmentation and evaluation protocol"}],"minor_comments":[{"comment":"The caption of Table 2 should state unambiguously that the first value in the Validation Set columns is the maximum UAR across five seeds, and the first value in the Test Set columns is the test UAR of the run achieving that maximum validation UAR; the current wording is ambiguous.","section":"§5 and Table 2 caption"},{"comment":"There are formatting slips in references and author names, e.g., 'Marchiet al.' and 'Nakataniet al.' should have a space before 'et al.', and 'PloS one' should be 'PLOS ONE'.","section":"Throughout"},{"comment":"The paper describes five random seeds but does not list the seed values or explain how the seeds were generated; since code is released, please include the seed values or a statement that they are reproducible from the released repository.","section":"§4"},{"comment":"The sentence explaining the spectrogram dimensions should clarify how a 30-second clip with an FFT size of 300,000 and hop length of 150,000 at a 300 kHz sampling rate results in 59 frames, so readers can reproduce the exact feature dimensions.","section":"§4"},{"comment":"The abstract's phrase 'the considered audible-range features' should define 'audible' as the sub-20 kHz band and clarify that the 'full' feature set also includes the audible range, to avoid confusing readers about what is being compared.","section":"Abstract and §5"}],"recommendation":"major_revision","confidential_remarks":"The paper's main scientific claim—that audible-range vocalization features outperform ultrasonic features for ASD detection in mice—is not yet adequately supported because of missing controls for non-vocal acoustic content and recording-session confounds, and the statistical significance testing is based on random seeds rather than independent subjects. The challenge itself and the released baseline resources are valuable, so the manuscript is worth revising rather than rejecting. The scope fits INTERSPEECH, and the authors' prior work on the dataset is a normal relationship for a challenge baseline, not circularity."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The MADUV paper is, at bottom, a challenge announcement with a baseline. The genuine contribution is the dataset: 84 mice, 7 hours of paired high-rate recordings, with a subject-independent train/validation/test split and a plausible evaluation design at both segment and subject level. That is real and useful. The baseline itself is a simple CNN on spectrograms, which is appropriate for a challenge.\n\nWhat is new beyond the prior pilot is the comparison of audible versus ultrasonic bands, and the claim that audible-range spectrograms work better. But this is the soft spot. The 'audi' feature set is just the sub-20 kHz part of a spectrogram computed from unsegmented 5-minute recordings. The microphone is an ultrasonic acquisition system; there is no calibration, no noise-floor analysis, no silence-only control, no per-litter or per-session ablation. So the interpretation that audible vocalizations contain more discriminative features is not supported. It could be cage noise or recording-session artifacts that correlate with genotype. That is a load-bearing flaw for the novelty claim.\n\nThe statistics also need a second look. The t-tests are run across five random seeds, which are not independent samples. Subject-level UAR of 0.625 corresponds to 10/16 subjects correct, which is not significant by a binomial test. The segment-level results are over non-independent clips. So the feasibility claim rests on a fairly weak evidential base, though the held-out test set does rule out simple overfitting.\n\nThe good news is that these are fixable. The challenge benchmark still stands on the dataset alone. A revision should add confound analyses, report confidence intervals or subject-level permutation tests, and temper the biological interpretation.\n\nThis paper deserves a serious referee; the challenge is real and the dataset is a resource. I would not cite it for the audible-vs-ultrasonic result, but I would point people to the dataset. Overall: conditional accept with revisions.","headline":"A useful challenge dataset and baseline, but the headline audible-vs-ultrasonic result needs confound controls and better statistics before being taken as a biological finding.","tokens_in":9442,"tokens_out":2366,"would_cite":false,"duration_ms":23409,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A simple CNN can tell autism-model mice from wild-type mice by their audible squeaks.","keywords":["autism spectrum disorder","mouse ultrasonic vocalization","audible spectrogram","convolutional neural network","bioacoustic classification","unweighted average recall","animal model screening"],"falsifier":"Retrain the audible-band CNN on the same recordings after masking every spectrogram region that contains a detected mouse vocalization, leaving only cage noise and handling sounds; if the resulting classifier still reaches roughly 0.600 UAR, the audible-band finding is an artifact of non-vocal acoustics rather than evidence about vocalizations.","tokens_in":8495,"feed_emoji":"🐭","tokens_out":6774,"duration_ms":66699,"temperature":0.7,"pith_summary":"This paper launches and benchmarks the first speech-technology challenge aimed at detecting autism spectrum disorder in mice from their vocalizations. The baseline shows the task is tractable: a plain CNN fed with spectrograms separates ASD-model from wild-type mice well above chance, with unweighted average recall (UAR) of 0.600 at the segment level and 0.625 at the subject level. Against expectation, the audible portion of the recordings (below 20 kHz) works at least as well as the ultrasound band, and the authors read this as evidence that audible vocalizations carry more class-discriminative spectral information. If correct, this makes a low-cost, standard-microphone pipeline more plausible for mouse-model screening and reinforces bridges between human paralinguistic analysis and animal bioacoustics.","feed_headline":"Simple CNN spots autism-model mice by their audible squeaks","feed_subtitle":"MADUV baseline's below-20-kHz spectrograms hit 60.0% segment, 62.5% subject unweighted recall.","key_machinery":"The load-bearing object is the frequency-split spectrogram: each 30-second clip becomes a 59-by-500 time-frequency image, and three feature variants are cut from the same underlying 300 kHz spectrogram, namely full (all frequencies), ultra (20-150 kHz), and audi (below 20 kHz). A small CNN with two convolutional and two fully connected layers plus sigmoid classification is trained separately on each variant, with the decision threshold tuned on the validation set, and segment predictions are pooled per subject by majority voting to produce subject-level decisions. The evaluation metric is unweighted average recall (UAR), the mean recall over the two classes, chosen because both classes matter, and significance is assessed by one-tailed one-sample $t$-tests across five random seeds.","core_discovery":"The paper's central claim is that automated detection of autism-spectrum-disorder model mice from their vocalizations is feasible with a deliberately simple baseline: a CNN trained on 30-second spectrogram patches reaches a segment-level unweighted average recall (UAR) of 0.600 and a subject-level UAR of 0.625 on held-out mice, both significantly above the 0.500 chance level. The more specific and less expected finding is that the audible-range spectrogram (below 20 kHz) achieves the best and most consistent performance, surpassing the ultrasonic band (20-150 kHz), and the paper interprets this as evidence that audible vocalizations carry spectral cues that are more discriminative for ASD type than the ultrasonic calls emphasized in earlier work. This interpretation reconciles the baseline with prior observations of genotype-dependent ultrasound frequency differences by attributing the CNN's success to different, audible-range spectral characteristics.","pith_inferences":["Extension: a decisive control would separate litter membership or recording session from genotype, because the test subjects comprise a small number of litters and only one recording per subject; a per-litter cross-validation would show whether the model hears genotype acoustics or just the acoustic idiosyncrasies of particular litters.","Extension: the reported audible-band advantage may partly reflect that the 0-20 kHz region contains cage noise, fan hum, and handling transients that differ between genotype groups; testing on noise-only segments would settle whether the biological reading is safe.","Extension: comparing the audible-band features that separate mouse genotypes with low-frequency prosodic features that separate human ASD speech could reveal a shared acoustic phenotype, a testable cross-species hypothesis.","Extension: because each subject was recorded once at postnatal day 8, the design cannot separate developmental stage effects from genotype effects; a longitudinal recording at multiple postnatal days would test whether the audible discriminative cues are stable over development."],"forward_implications":["The MADUV baseline gives entrants a concrete benchmark: a simple spectrogram CNN already beats chance, so improvements must be measured against 0.600 segment-level and 0.625 subject-level UAR.","Because the audible band is at least as informative as the ultrasonic band for the CNN, future feature engineering should include full-band or audible-only inputs instead of assuming ultrasound is the only useful signal.","Subject-level majority voting yields higher and more stable UAR than segment-level predictions, so per-subject aggregation is a natural component of strong submissions.","If audible vocalizations really are the discriminative signal, standard microphones with a sub-20 kHz response could support mouse-model vocalization screening without specialist ultrasound equipment.","The result suggests that cross-species transfer of human speech features to mouse vocalizations is worth pursuing, not only for ultrasound but also for the audible range."],"supporting_citations":[{"why":"Supplies the recorded mouse vocalization dataset and the original biological evidence that ASD-model mice differ from wild-type mice in vocalization behavior.","marker":"[4]"},{"why":"The most similar prior work, a pilot study that first used a pretrained audio network on mouse ultrasonic vocalizations for ASD detection and achieved 66.6 percent UAR, which MADUV extends with a simpler CNN baseline.","marker":"[26]"},{"why":"Demonstrates that a deep neural network can classify mouse sex and strain from ultrasonic vocalizations, providing evidence that spectrogram-based CNN classifiers capture strain-level acoustic information.","marker":"[23]"},{"why":"The earlier challenge on detecting autism from human children's speech whose evaluation setup and paralinguistic framing MADUV adapts to mouse vocalizations.","marker":"[16]"}],"fun_headline_variants":["Autism-model mice give away their condition in audible squeaks","Audible squeaks outperform ultrasound for spotting autism-model mice","CNN hears autism in mice's audible calls, not just ultrasound","Autism-model mice betrayed by audible squeaks, says challenge baseline"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The below-20-kHz signal that drives the best result is dominated by mouse vocalizations rather than by cage noise or recording-session differences that happen to align with which mice are ASD models.","fun_headline_variants_meta":{"raw":{"variants":["Autism-model mice give away their condition in audible squeaks","Audible squeaks outperform ultrasound for spotting autism-model mice","CNN hears autism in mice's audible calls, not just ultrasound","Autism-model mice betrayed by audible squeaks, says challenge baseline"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000501,"raw_usage":{"total_tokens":2417,"prompt_tokens":878,"completion_tokens":1539,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":494,"completion_tokens_details":{"reasoning_tokens":1468}},"tokens_in":494,"tokens_out":1539,"duration_ms":11133,"temperature":1.0,"reasoning_tokens":1468,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:36:10.445487+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the audible-band CNN on the same recordings after masking every spectrogram region that contains a detected mouse vocalization, leaving only cage noise and handling sounds; if the resulting classifier still reaches roughly 0.600 UAR, the audible-band finding is an artifact of non-vocal acoustics rather than evidence about vocalizations.","supporting_citations":[{"cited_title":"Clips of insufficient duration were excluded from the dataset due to their limited information","cited_arxiv_id":null,"evidence_quote":"Supplies the recorded mouse vocalization dataset and the original biological evidence that ASD-model mice differ from wild-type mice in vocalization behavior."},{"cited_title":"The geneva minimalistic acoustic parameter set (gemaps) for voice research and affective computing,","cited_arxiv_id":null,"evidence_quote":"The most similar prior work, a pilot study that first used a pretrained audio network on mouse ultrasonic vocalizations for ASD detection and achieved 66.6 percent UAR, which MADUV extends with a simpler CNN baseline."},{"cited_title":"Distinc- tive prosodic features of people with autism spectrum disorder: a systematic review and meta-analysis study,","cited_arxiv_id":null,"evidence_quote":"Demonstrates that a deep neural network can classify mouse sex and strain from ultrasonic vocalizations, providing evidence that spectrogram-based CNN classifiers capture strain-level acoustic information."},{"cited_title":"Mouse chromosome engi- neering for modeling human disease,","cited_arxiv_id":null,"evidence_quote":"The earlier challenge on detecting autism from human children's speech whose evaluation setup and paralinguistic framing MADUV adapts to mouse vocalizations."}],"review_version":1}