{"id":"da1b0489-2abe-4a2e-bd8c-8e56f3718771","arxiv_id":"2506.20609","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A CNN on log-mel spectrograms classifies five gun types from curated audio with mAP 0.58, beating an SVM baseline (0.39), but falls to 0.35 on noisy web data.","lead":"This paper trains a CNN to sort gunshot recordings into five firearm types, reporting mAP 0.58 on clean curated data versus 0.39 for an SVM baseline. On noisy web audio, performance drops to 0.35, illustrating the gap between controlled and real-world conditions.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported CNN mAP of 0.58 is uninterpretable unless the curated 'clean' corpus is shown to contain real gunshots rather than sound-effects-library artifacts; the paper provides no such validation.","rationale":"The paper's strongest claim is that acoustic gun type classification is feasible, evidenced by CNN mAP 0.58 versus SVM mAP 0.39. That evidence is valid only if the curated audio is a faithful sample of gunshots. The provenance in Section 3 (Spotify playlists, commercial sound-effect library) makes this doubtful, and the paper provides no label validation, no inter-annotator agreement, no source-stratified evaluation, and no release of data. These omissions are not just missing polish: they bear directly on whether the model learned firearm acoustics or library-specific artifacts. I agree with the reader's weakest assumption, but sharpen it: the more immediate risk is that the recordings are not real gunshots at all, and that a random split hides a source confound. Because the paper's conclusion is conditional on a test it has not performed, the conditional verdict should remain; acceptance should require the external real-recording evaluation described in the concrete test.","tokens_in":7700,"tokens_out":6810,"duration_ms":83994,"concrete_test":"Release the dataset and code, or at minimum an independently labeled held-out set of real gunshot recordings captured with commodity devices (e.g., a smartphone at a shooting range with known firearm types), and evaluate the trained CNN on it. If the real-recording mAP falls to near chance (0.2 for five balanced classes) or is substantially below 0.58, the curated sound-effect corpus is not representative and the central feasibility claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central result (abstract and Section 5.3) is a CNN mAP of 0.58 for gun type classification on clean labeled data. For this number to support the paper's stated goal of classifying real gunshots, the curated corpus must be acoustically representative of actual firearms and the labels must identify the true firearm. Section 3 lists Spotify playlists and a commercial Firearm Sound Effects Library as sources; these are largely sound effects, not necessarily live discharges, and no per-file provenance is given. Manual review confirms only that a clip sounds like a gunshot; it cannot establish the firearm type. The random 60/20/20 split of Section 4.3 does not separate by source, so clips from the same library or playlist can appear in both training and test, letting the model exploit source-specific artifacts (reverb, mastering, synthetic signature) instead of firearm acoustics. Section 7's plan to address label noise admits label uncertainty, but no noise-rate estimate or inter-annotator agreement is reported. Without released data or an external validation set, the 0.58 mAP is uninterpretable as evidence for real-world gun type classification.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a study on gunshot detection and firearm type classification from audio recordings. The authors curate a dataset of 3459 recordings across five classes (Rifle, Submachine Gun, Handgun/Pistol, Machine Gun, Shotgun) from sources including a CMU internal collection, Spotify playlists, and a commercial sound-effects library. They extract log mel spectrograms, train an SVM baseline and a multi-task CNN for joint detection and classification, and report results on clean and noisy web data. The central claim is that the CNN achieves a mean average precision (mAP) of 0.58 on clean labeled data, substantially outperforming the SVM baseline (mAP 0.39), while performance drops to mAP 0.35 on noisy web-sourced data.","tokens_in":7896,"tokens_out":2571,"duration_ms":29076,"significance":"If the results are reliable, the paper is a useful empirical contribution to low-cost acoustic gunshot classification: it demonstrates a concrete deep-learning pipeline on a relatively large curated corpus, identifies which firearm categories are acoustically separable (Submachine Gun, Machine Gun) and which are confusable (Shotgun, Rifle), and quantifies a substantial domain shift to noisy web data. The paper does not ship code or data, however, and several load-bearing experimental details are unspecified, so the quantitative claims are currently hard to verify. The core idea—combining detection and classification in a joint model—is sensible and the comparison to an SVM baseline is appropriate in spirit, even though the baseline is under-specified.","major_comments":[{"comment":"The SVM baseline is not adequately specified. The text states that features were 'likely derived' from the audio representations and that the SVM was 'configured for multi-class classification,' but no exact feature set, kernel, hyperparameters, or training procedure is given. Without this information, the reported mAP of 0.39 for the SVM cannot be interpreted or reproduced, and the claim that the CNN 'significantly outperforms' the baseline is not fully supported. Please provide a precise description of the SVM features and configuration, or explicitly reframe the SVM result as an approximate baseline.","section":"§4.2.1 and §4.4"},{"comment":"The 'Relevant' metric is defined only by an assumption: the text says 'we will assume' that Relevant scores are conditioned on correct gunshot detection. This is a load-bearing ambiguity because the per-class F1 scores in Table 4 are discussed mainly through the 'Relevant' column. Please define the exact computation (e.g., whether it uses the detection branch's hard decision, a confidence threshold, or ground-truth detection masks) and report the number of test instances in each subset. Also clarify how 'Overall' scores differ from 'Relevant' scores.","section":"§5.2, Table 4"},{"comment":"The provenance and validity of the curated dataset are not established. Recordings are drawn from Spotify playlists and a commercial Firearm Sound Effects Library, which are largely sound effects rather than verified live gunshot recordings, and the labels come from manual review. No per-file provenance, label-noise estimate, or inter-annotator agreement is provided. In addition, the 60/20/20 random split in §4.3 does not separate by source, so clips from the same playlist or library can appear in both training and test sets; the model may then learn source-specific artifacts (e.g., reverb, mastering, synthetic signatures) rather than firearm acoustics. Please provide a source-disjoint split, per-source performance analysis, and some validation of label correctness.","section":"§3 and §4.3"},{"comment":"The reported mAP values (0.58 for CNN, 0.39 for SVM, 0.35 for noisy web data) are given without error bars, confidence intervals, or significance tests. The paper mentions 5-fold cross-validation in §4.3, but no variance across folds is reported. The noisy web data evaluation is also under-described: the number of clips, their sources, and how labels were obtained are not stated, making the 0.35 mAP difficult to interpret. Please report fold-level statistics and a detailed description of the web evaluation set.","section":"§5.3"}],"minor_comments":[{"comment":"The sentence in the text ends with 'it is correct 82' and omits the percentage sign and the rest of the sentence; the intended value is presumably 82%.","section":"§5.1, Table 3"},{"comment":"Table 1 lists acoustic characteristics (peak frequency, duration, SPL, shockwave) but no citation or measurement methodology is given for these numbers. Please indicate whether the values are from the cited literature or from the authors' own measurements, and, if the latter, how they were obtained.","section":"§2, Table 1"},{"comment":"The description of the BoAW feature is vague ('a dictionary of acoustic words ... was considered'). Since the BoAW features are not used in the reported experiments, please state explicitly that they were not used in the final evaluation, or provide details if they were.","section":"§4.1"},{"comment":"The discussion of per-class performance describes Handgun/Pistol as achieving a 'moderate F1 score (0.54 relevant)' while Table 4 reports 0.64 in the Relevant column; please ensure consistency between the text and the table.","section":"§5.2"}],"recommendation":"major_revision","confidential_remarks":"The paper's central empirical claim is potentially publishable, but the missing SVM feature description, ambiguous metric definition, and lack of dataset validation are too significant for acceptance in the current form. The manuscript also relies heavily on the first author's prior work and thesis for the methodology; this is not inappropriate, but the novelty of the current contribution should be clearly delineated. I recommend major revision rather than rejection because the identified issues are addressable within the scope of the paper: the authors can specify the SVM baseline, define the Relevant metric, add source-disjoint splits and provenance analysis, and report variance estimates."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper is a reasonable empirical study, but the headline number (CNN mAP 0.58) can't be interpreted until we know the 'clean' corpus contains real gunshots rather than sound-effects-library recordings. That's the load-bearing issue, and the paper doesn't address it.\n\nWhat's genuinely useful: the acoustic background section is accurate and well-cited (Maher's work is the right reference), and the joint detection/classification CNN is a sensible architecture choice. The comparison with an SVM baseline, plus the drop to 0.35 mAP on noisy web data, gives a realistic picture of the domain-shift problem. The authors are also candid about label noise and data scarcity in the future-work section.\n\nSoft spots, in rough order of importance:\n\n1. Dataset provenance. Section 3 lists Spotify playlists and a commercial 'Firearm Sound Effects Library' as primary sources. Sound effects are not the same as recorded live fire: they're often synthesized, recorded in studios, or heavily edited. The fact that a human reviewer 'confirms' a clip sounds like a gunshot doesn't establish the firearm type, and the paper provides no per-file provenance, no metadata on recording conditions, and no inter-annotator agreement. Without this, the mAP of 0.58 is evidence about classifying sound-effect clips, not real gunshots.\n\n2. The random 60/20/20 split doesn't separate by source. Clips from the same library or playlist can appear in both train and test, so the model can exploit source-specific artifacts. A grouped split by recording session or source would be a much stronger test.\n\n3. Missing statistical rigor: no error bars, no significance test for the CNN-vs-SVM gap, and the SVM feature description ('likely derived') is vague. The 'Relevant' metric is defined by a guess ('we will assume'), which is not acceptable in a results section.\n\n4. Data and code are not released. That's not fatal by itself, but it makes the headline numbers uncheckable.\n\nThe citation pattern looks honest—it correctly cites Kiktova et al. 2015, Raponi et al. 2022, and Maher's acoustics work, and it doesn't oversell prior art. The paper's own limitation statements (Section 7) admit the label-noise problem, which counts in its favor.\n\nBottom line: the paper deserves a serious referee, mainly because the question is important and the empirical setup is close enough to be salvageable. I'd want major revision: release the data or at least a substantial sample with metadata, add a source-grouped evaluation, quantify label noise, and fix the metric definitions. For my own work, I wouldn't cite the mAP numbers as evidence about real gunshot classification.\n\nBring to reading group? Maybe, as an example of the sound-effects-vs-real-data pitfall.","headline":"A plausible empirical study whose headline mAP is uninterpretable until the 'clean' corpus is shown to contain real gunshots rather than sound-effects-library recordings.","tokens_in":8427,"tokens_out":1847,"would_cite":false,"duration_ms":19993,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A joint gunshot-detection and firearm-classification CNN achieves a mean average precision of 0.58 on clean curated audio, far above an SVM baseline of 0.39, while degrading sharply on noisy web audio.","keywords":["gunshot detection","gun type classification","acoustic event detection","convolutional neural network","mel spectrogram","muzzle blast","shockwave","mean average precision"],"falsifier":"Take a set of firearms of known make and model, record live shots with several phones at different distances, and run the published CNN on those clips without retraining; if per-class mean average precision falls to near chance, or far below 0.58, while the SVM baseline holds up, the paper's central claim about learnable acoustic gun-type hierarchies is disproved.","tokens_in":7499,"feed_emoji":"🔫","tokens_out":5298,"duration_ms":56166,"temperature":0.7,"pith_summary":"The paper claims that a convolutional neural network trained on log mel spectrograms can both detect a gunshot and identify which of five broad firearm categories produced it, and that this works well enough on clean curated audio (mean average precision 0.58) to beat a traditional SVM baseline (0.39). It argues that the acoustic differences between firearm types—muzzle blast duration and spectrum, the presence of a supersonic shockwave, and the rate of fire—are learnable from ordinary recordings, opening a low-cost path to gunshot intelligence on commodity devices. The paper also shows the same system degrades sharply on noisy web audio (mean average precision 0.35), so the claim is specifically about curated conditions, not field deployment.","feed_headline":"CNN identifies firearm type from gunshot audio at 0.58 mAP","feed_subtitle":"The same network also detects gunshots with F1 0.86, beating an SVM baseline of 0.39 on clean data.","key_machinery":"The load-bearing mechanism is the joint detection-and-classification CNN operating on log mel spectrograms. Shared convolutional layers learn low-level time-frequency patterns common to gunshot events; the detection branch uses a sigmoid output for gunshot presence, and the classification branch uses a softmax output over the five firearm categories, applied when the detection branch positively identifies a gunshot. The underlying physical basis is the two-component gunshot acoustic structure—the roughly 3 to 5 millisecond muzzle blast and, for supersonic rounds, the 200 to 400 microsecond ballistic shockwave—whose timing, spectral content, and presence vary by firearm type, ammunition, and shooting direction.","core_discovery":"On its own terms, the central discovery is that a CNN with shared convolutional layers and two task-specific output branches—one binary gunshot/no-gunshot detector and one five-way firearm classifier—extracts enough structure from 128-band log mel spectrograms to classify gun type with a mean average precision of 0.58 on a curated set of 3459 recordings, compared with 0.39 for an SVM baseline. The gunshot-detection branch reaches an F1 score of 0.86 for the gunshot class, and the most reliable firearm categories are submachine guns and machine guns, whose rapid-fire temporal signatures are acoustically distinctive, while rifles and shotguns remain harder to separate.","pith_inferences":["The paper leaves implicit that its five-class hierarchy is broad; a system aimed at forensic identification or court admissibility would need per-caliber or per-model discrimination, which the current data and labels do not support.","The acoustic hierarchy it reports—muzzle blast duration and peak frequency separating shotguns, rifles, and handguns—could be tested on large uncontrolled sound-event datasets to see whether the same features transfer outside curated gunshot libraries.","A testable extension would measure mean average precision as a function of distance-to-microphone and recording device, since the clean curated data does not report those variables.","The reliance on sound-effect libraries as training data suggests that matching the recording conditions of the target deployment, rather than adding more gunshot clips, may be the fastest way to close the 0.58-to-0.35 gap."],"forward_implications":["If the 0.58 mean average precision result reproduces, a commodity-device gunshot detection system could replace expensive fixed infrastructure, such as the roughly $65,000 per square mile per year cited in the paper, with distributed cell-phone microphones.","Gun type labels from audio could give first responders tactical information—distinguishing a handgun from a rifle or automatic weapon—before visual confirmation.","The sharp drop to 0.35 mean average precision on web-sourced audio implies that noise robustness and domain adaptation, not raw architecture, are the next bottleneck for real-world use.","The joint architecture's gunshot-detection branch, with F1 0.86 for gunshot events, can serve as a gate to reduce downstream classification errors in a deployed alerting system."],"supporting_citations":[{"why":"Supplies the acoustic characterization of muzzle blast and shockwave that the feature design and class distinctions rely on.","marker":"[8]"},{"why":"Provides the prior work on deciphering gunshot recordings that motivates the two-component acoustic model.","marker":"[9]"},{"why":"Prior gun type recognition from gunshot audio that this paper extends with a deep learning baseline comparison.","marker":"[5]"},{"why":"Describes the joint detection and classification architecture and imprecise-label learning that underpin the CNN design.","marker":"[15]"},{"why":"Defines the SVM classifier used as the baseline for the mean average precision comparison.","marker":"[19]"},{"why":"Documents a commercial gunshot detection system and its cost benchmark that motivates the low-cost alternative.","marker":"[2]"},{"why":"Supplies the webly labeled audio learning approach used for the noisy-data generalization experiment.","marker":"[21]"}],"fun_headline_variants":["Gunshot audio reveals firearm type via CNN, 0.58 mAP","Deep learning classifies guns from sound at 0.58 mAP","Acoustic gun type detection: CNN beats SVM, mAP 0.58","Firearm classification from gunshot recordings: CNN scores 0.58","Low-cost gunshot analysis identifies weapon type with CNN"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The results stand or fall on whether the 3459 curated clips—drawn from internal collections, public playlists, and commercial sound-effect libraries—are correctly labeled by firearm type and acoustically faithful to real gunshots as heard on phones and field microphones.","fun_headline_variants_meta":{"raw":{"variants":["Gunshot audio reveals firearm type via CNN, 0.58 mAP","Deep learning classifies guns from sound at 0.58 mAP","Acoustic gun type detection: CNN beats SVM, mAP 0.58","Firearm classification from gunshot recordings: CNN scores 0.58","Low-cost gunshot analysis identifies weapon type with CNN"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00051,"raw_usage":{"total_tokens":2488,"prompt_tokens":955,"completion_tokens":1533,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":571,"completion_tokens_details":{"reasoning_tokens":1437}},"tokens_in":571,"tokens_out":1533,"duration_ms":10811,"temperature":1.0,"reasoning_tokens":1437,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T22:43:46.067450+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a set of firearms of known make and model, record live shots with several phones at different distances, and run the published CNN on those clips without retraining; if per-class mean average precision falls to near chance, or far below 0.58, while the SVM baseline holds up, the paper's central claim about learnable acoustic gun-type hierarchies is disproved.","supporting_citations":[{"cited_title":"Acoustical characterization of gunshots,","cited_arxiv_id":null,"evidence_quote":"Supplies the acoustic characterization of muzzle blast and shockwave that the feature design and class distinctions rely on."},{"cited_title":"Deciphering gunshot recordings,","cited_arxiv_id":null,"evidence_quote":"Provides the prior work on deciphering gunshot recordings that motivates the two-component acoustic model."},{"cited_title":"Gun type recognition from gunshot audio recordings,","cited_arxiv_id":null,"evidence_quote":"Prior gun type recognition from gunshot audio that this paper extends with a deep learning baseline comparison."},{"cited_title":"Computational Audition with Imprecise Labels,","cited_arxiv_id":null,"evidence_quote":"Describes the joint detection and classification architecture and imprecise-label learning that underpin the CNN design."},{"cited_title":"Support vector machine,","cited_arxiv_id":null,"evidence_quote":"Defines the SVM classifier used as the baseline for the mean average precision comparison."},{"cited_title":"ShotSpotter ® Gunshot Detection System,","cited_arxiv_id":null,"evidence_quote":"Documents a commercial gunshot detection system and its cost benchmark that motivates the low-cost alternative."},{"cited_title":"Learning Sound Events From Webly Labeled Data","cited_arxiv_id":"1811.09967","evidence_quote":"Supplies the webly labeled audio learning approach used for the noisy-data generalization experiment."}],"review_version":1}