{"id":"2e85d894-a318-4fc8-9eed-a024a967f143","arxiv_id":"2505.14285","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"An integrated underwater acoustic pipeline combining U-Net denoising, ResNet classification, and autoencoder novelty detection reports 71% classification and 91% novelty detection on Deepship data, but the novelty metric is below a trivial baseline.","lead":"AquaSignal is a new software pipeline that cleans, classifies, and flags unusual patterns in underwater recordings of ships. The authors report 71% accuracy at identifying known ship types and 91% accuracy at spotting novel sounds, but the evaluation has gaps that weaken the claims.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's own results contradict its central claim: denoising lowers classification accuracy from 77.1% to 71.3%, so 'improves task performance' is unsupported.","rationale":"The reader's weakest_assumption focuses on whether the autoencoder reconstruction-error threshold generalizes from training background to test tug samples. That is a legitimate concern, but it is not the most load-bearing issue. The paper's own results contain a direct contradiction of the central claim: the denoising stage, which is a core component of the pipeline, reduces classification accuracy from 77.1% to 71.3% (Section 8). This is an internally acknowledged trade-off, yet the abstract claims the pipeline 'improves task performance.' No external consensus or subtle generalization assumption is needed to see that the primary evidence contradicts the headline claim. Likewise, the reported 91% novelty detection accuracy is suspect on its face: with only 1% positive samples, a trivial all-background classifier would achieve 99% accuracy, and the reported precision/recall imply a different accuracy unless a nonstandard definition is used. These issues are more fundamental than the threshold-generalization question because they undermine the validity of the reported metrics themselves. The reader's overall verdict of REJECT is still correct, but the reasoning should center on the internal inconsistency and the misleading metrics rather than on threshold transfer. Therefore, I disagree with the reader's identification of the weakest assumption, while agreeing with the final verdict.","tokens_in":11482,"tokens_out":7106,"duration_ms":62482,"concrete_test":"Re-run the classification experiment of Section 5 under two conditions: (a) ResNet18 trained and tested on denoised spectrograms after ORCA-CLEAN; (b) the same ResNet18 trained and tested on the original noisy spectrograms, using identical train/test splits, hyperparameters, and evaluation protocol. If condition (b) accuracy exceeds condition (a) by the reported 5.8-point margin, the abstract's claim that AquaSignal improves task performance is directly false. Also, recompute novelty detection accuracy from the reported precision, recall, and 1% contamination rate; if the result does not match 91.5%, the reported metrics are internally inconsistent.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract claims AquaSignal 'improves signal clarity and task performance.' Yet Section 8 explicitly states that when the classifier is trained on original noisy samples, accuracy is 77.1%, whereas on denoised data it drops to 71.3%. Since the pipeline places ORCA-CLEAN denoising before ResNet18 classification, the pipeline's reported classification performance is worse than the same classifier without denoising. This is not an external baseline dispute; it is an internal contradiction between the stated contribution and the reported evidence. The only escape would be to redefine 'task performance' to exclude classification, but the paper presents classification accuracy as a primary metric. Additionally, the 91% novelty detection accuracy is below the trivial majority-class baseline given a 1% tug contamination rate, and it is numerically inconsistent with the reported precision (87.8%) and recall (81.3%), which under standard definitions would imply accuracy near 99%. These quantitative problems compound the qualitative contradiction, making the central claim unsupported even before considering generalization issues.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces AquaSignal, a four-stage processing pipeline for underwater acoustic signals: preprocessing (2-second, 32 kHz segments from DeepShip and ONC background data), denoising with the ORCA-CLEAN U-Net, classification with a fine-tuned ResNet18, and unsupervised novelty detection with an autoencoder. The reported headline results are 71.3% classification accuracy and 91.5% novelty-detection accuracy on a 25,000-sample curated set with strict train/test separation at the recording level. The paper also claims, in the abstract and introduction, that AquaSignal improves signal clarity and task performance, and it presents ablation tables comparing its classifier and detector against in-house reimplemented baselines.","tokens_in":11704,"tokens_out":3500,"duration_ms":34177,"significance":"An integrated, modular, open-dataset pipeline for denoising, classification, and novelty detection would be useful for maritime monitoring, especially if it can be shown to generalize under a leakage-free partition. The paper also addresses a real gap in the DeepShip literature by considering novelty detection. However, the central performance claims are not supported by the manuscript's own evidence: the denoising stage lowers classification accuracy relative to the same classifier on noisy data, and the novelty-detection metrics are internally inconsistent and below the trivial always-background baseline under the stated setup. As a result, the contribution, as evaluated, does not yet establish the claimed benefit.","major_comments":[{"comment":"The abstract and introduction claim that AquaSignal 'improves signal clarity and task performance,' but Section 8 states that 'when trained on the original noisy samples, the classifier returned an accuracy of 77.1%, whereas when trained on the denoised data, the accuracy drops to 71.3%.' Since the pipeline described in Section 5 feeds denoised spectrograms into the ResNet18 classifier, the end-to-end classification accuracy is 71.3%, six points below the same classifier without denoising. This is an internal contradiction between the stated contribution and the reported evidence. The claim should be revised to state that denoising improves visual clarity but degrades classification accuracy in this setting.","section":"Section 8, Discussion and Concluding Remarks"},{"comment":"The novelty-detection results are not credible as reported. With 1% tug contamination in the background test set, an always-background classifier achieves 99% accuracy, so the reported 91.5% accuracy is below a trivial baseline. Moreover, the reported precision (87.8%) and recall (81.3%) are mutually inconsistent with 91.5% accuracy under standard definitions: with 1% positives, precision 87.8% and recall 81.3% imply an accuracy of roughly 99.7%. The authors should report the confusion matrix, the true positive/negative counts, and a majority-class baseline.","section":"Section 7, Table 2 and Section 5, Detection"},{"comment":"The reconstruction-error threshold is set at the 70th percentile of background-only training errors. If the test background distribution resembles the training background distribution, this threshold by construction classifies about 30% of test background samples as novel. With only 1% contamination, the maximum achievable accuracy under that assumption is approximately 70%, not 91.5%. The described methodology is therefore incompatible with the reported accuracy unless the threshold definition is misstated, the test background distribution is substantially different, or the threshold was re-calibrated on test data. This point is load-bearing for the novelty-detection claim and must be clarified and validated on held-out data.","section":"Section 5, Detection"},{"comment":"The ablation comparisons in Tables 1 and 2 are all against models reimplemented in-house under the same pipeline. No error bars, confidence intervals, repeated runs, or tests of statistical significance are provided, and no external published results are used as anchors. The conclusion that AquaSignal 'outperformed the state-of-the-art methods when trained using the same policy' is therefore not established. Reporting variance or a small number of seeds, and comparing against published results under a comparable leakage-free partition, would be necessary to support the comparison.","section":"Section 7, Tables 1 and 2"}],"minor_comments":[{"comment":"There are several typographical errors that should be corrected, including 'achitecture' in the opening of Section 8, 'datset' in the Introduction, and 'DesNet169' in Section 8.","section":"General"},{"comment":"References [9] and [10] are the same paper (He et al., deep residual learning) and should be merged or renumbered; similarly, [10] and [31] in the original numbering are listed separately but refer to the same work.","section":"References"},{"comment":"The sentence 'The results presented in the 3 section cannot be directly compared with those reported in the 7 section' should be rewritten to refer to Sections 3 and 7 by name rather than by bare numerals.","section":"Section 7"},{"comment":"In the Detection paragraph, 'Following an extensive hyperparameter optimisation and procedure' is ungrammatical and should be rephrased.","section":"Section 5"},{"comment":"The ResNet18 description calls it a '72-layer architecture with 18 deep layers,' which is confusing without a more precise architectural specification; consider stating the standard ResNet18 configuration explicitly.","section":"Section 4"}],"recommendation":"reject","confidential_remarks":"The manuscript's central claims are contradicted by its own results (denoising lowers classification accuracy) and by the internal inconsistency of the novelty-detection metrics (91.5% accuracy vs. 87.8% precision / 81.3% recall at 1% contamination). These are not presentation-level issues; they concern the validity of the headline numbers. A major revision would need to re-run the experiments, report error bars and baselines, and substantially soften the claims, at which point the contribution would be much narrower. Given the scope of the paper as written, reject is appropriate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The punchline: the reported results don't support the abstract's claim that AquaSignal \"improves signal clarity and task performance.\" Section 8 openly states that classification accuracy on noisy data is 77.1%, while on denoised data it drops to 71.3%. Since the pipeline denoises before classifying, the pipeline's classification performance is strictly worse than skipping the denoising block. That is an internal contradiction, not a debate about baselines.\n\nThe novelty detection result is also shaky. With 1% tug contamination, always predicting background gives 99% accuracy. Their autoencoder gets 91.5%, which is below that trivial baseline. Worse, the reported precision (87.8%) and recall (81.3%) are mathematically inconsistent with an accuracy of 91.5% at 1% prevalence—those numbers imply accuracy near 99%. Something is off in the metrics, and the paper doesn't acknowledge it.\n\nWhat's genuinely new: this is the first application of novelty detection to DeepShip, and the specific combination of ORCA-CLEAN + ResNet18 + AE is new, even if each component is off-the-shelf. The ablation study is a reasonable attempt to compare alternatives, and the authors are transparent about the denoising trade-off in the discussion—they don't hide the 6-point drop. That honesty counts for something.\n\nThe soft spots are serious but fixable. No error bars, no code release, and the novelty threshold is set at the 70th percentile of background training errors with no sensitivity analysis. The baseline comparisons are in-house reimplementations, and some results (DDAD at 1.7% accuracy) suggest those implementations may be flawed. The paper also lacks a majority-class baseline, which is exactly what would have exposed the novelty detection issue.\n\nWho this is for: anyone working on underwater acoustic pipelines or novelty detection in imbalanced acoustic settings could learn from the integration and the cautionary tale about denoising hurting classification. But as a claim of a robust, high-performing framework, it falls short.\n\nMy recommendation: send it to peer review. The flaws are substantial but addressable, and the first-claim on DeepShip novelty detection plus the honest discussion make it worthy of careful referee time. I would not accept it in its current form—the headline claim needs to be rewritten, the metrics need to be reconciled, and baselines need to be reported—but a revision could be salvageable.","headline":"The paper's own numbers contradict its central claim—denoising hurts classification and novelty detection is below a trivial baseline—but the integration and honest discussion of the trade-off give it enough substance to referee, not enough to accept as-is.","tokens_in":12190,"tokens_out":2295,"would_cite":false,"duration_ms":24265,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AquaSignal is a modular pipeline that denoises underwater recordings with a U-Net, classifies known ship types with a ResNet18, and flags never-seen acoustic events with an autoencoder, reporting 71.3% classification accuracy and 91.5%…","keywords":["underwater acoustics","signal denoising","ship classification","novelty detection","autoencoder","ResNet18","U-Net","DeepShip dataset"],"falsifier":"Sweep the detection threshold across percentiles and plot a full ROC curve on held-out background and tug samples, then re-run the fixed 70th-percentile threshold on background recordings from different seasons, depths, or sensor gains and count false positives. If accuracy drops sharply away from the chosen percentile or in new background conditions, the reported generalization is an artifact of calibration rather than a property of the learned representation.","tokens_in":11312,"feed_emoji":"🌊","tokens_out":11768,"duration_ms":104413,"temperature":0.7,"pith_summary":"AquaSignal asks whether underwater acoustic monitoring can be handled by one integrated system rather than by separate tools bolted together. The pipeline turns raw hydrophone recordings into two-second clips, turns each clip into a spectrogram (a time-frequency picture of the sound), cleans it with a U-Net denoiser, sorts known ship types with a ResNet18 classifier, and flags anything unfamiliar with an autoencoder that measures how well it can rebuild the clip. On a combined dataset of real ship recordings and public background-ocean noise, the paper reports 71.3% accuracy for classifying five classes (background, cargo, passenger ship, tanker, tug) and 91.5% accuracy for novelty detection of a withheld ship type. The authors also report a clear trade-off: classifying the original noisy clips gives 77.1% accuracy, while classifying the denoised clips gives 71.3%, so their case for the pipeline rests on the added novel-event detection, not on a classification record.","feed_headline":"AquaSignal pipeline: 71% ship accuracy, 91% novel sounds","feed_subtitle":"A single modular chain denoises, classifies, and flags unfamiliar underwater acoustic events for real-time monitoring.","key_machinery":"The load-bearing mechanism is the novelty detector: a feedforward autoencoder trained only on background-noise spectrograms, whose reconstruction error becomes an anomaly score, with a cutoff fixed at the 70th percentile of reconstruction errors computed from background-only training samples. Test inputs above that cutoff are flagged as novel. Around this detector sits a U-Net denoiser (the ORCA-CLEAN method) that works on spectrograms, and an ImageNet-pretrained ResNet18 fine-tuned for the five-class ship and background task. The whole chain is evaluated under a strict no-leakage protocol: two-second clips from the same original recording never appear in both training and test sets.","core_discovery":"The discovery AquaSignal tries to establish is that segmenting, denoising, classifying, and novelty-detecting underwater sound can be chained into one pipeline that works on real, noisy recordings, and that the autoencoder can recognize a withheld ship type (tug) as novel even though that class never appeared during training. On a combined dataset built from the DeepShip benchmark and public background-ocean recordings, the paper reports 71.3% classification accuracy and 91.5% novelty-detection accuracy, with precision, recall, and F1 in the same range. The paper also reports a cost: the classifier reaches 77.1% accuracy on the original noisy clips and 71.3% on denoised clips, so the denoiser removes information the classifier had been using. The authors' case is that the contribution is the combination—the first integrated preprocessing, denoising, classification, and novelty pipeline on this benchmark—rather than a new classification record.","pith_inferences":["Beyond the paper: the fixed 70th-percentile threshold is an arbitrary calibration point; a threshold chosen by cross-validation, or a full ROC-based decision rule, would probably transfer more reliably to new acoustic environments.","Beyond the paper: because real deployment noise changes with season, current, and sensor aging, an online recalibration of the autoencoder threshold would be a natural stress test of the 91.5% figure.","Beyond the paper: the denoising cost suggests an adaptive pipeline that classifies both raw and denoised clips and lets the novelty score arbitrate between them could recover some of the lost classification accuracy."],"forward_implications":["The 91.5% novelty-detection accuracy implies that a monitoring system can flag a ship type it never trained on by thresholding autoencoder reconstruction error, without any labeled anomalies.","The 71.3% classification accuracy on a strict no-leak split implies that realistic evaluations of vessel classifiers are harder than earlier reported numbers suggest, and that accuracy obtained without such splits can be inflated by information leakage.","The ablation study implies that under the data-scarce conditions of this benchmark, shallower models (ResNet18 over ResNet34 and DenseNet169, the autoencoder over EfficientNet and a diffusion-based detector) generalize better.","The measured drop from 77.1% to 71.3% when classifying denoised audio implies that a denoiser's value must be judged by its effect on downstream classification and detection, not by spectrogram clarity alone."],"supporting_citations":[{"why":"Supplies ORCA-CLEAN, the U-Net denoising method AquaSignal trains and applies in stage 2.","marker":"[1]"},{"why":"Supplies the DeepShip benchmark recordings and class labels used for the classification and novelty setup.","marker":"[14]"},{"why":"Supplies the background-ocean recordings from a public observatory network that create the background class and train the autoencoder.","marker":"[11]"},{"why":"Defines the leakage-free data-partitioning protocol and the evaluation metrics used throughout.","marker":"[3]"},{"why":"The in-text citation attached to the ResNet18 classifier used in the classification stage.","marker":"[7]"},{"why":"Provides the residual-learning architecture that underlies the ResNet family used in classification and in the ResNet34 ablation baseline.","marker":"[9]"},{"why":"Grounds the autoencoder reconstruction-error principle on which the novelty detector's threshold is based.","marker":"[20]"},{"why":"Serves as the diffusion-based anomaly-detection baseline in the ablation study.","marker":"[22]"}],"fun_headline_variants":["Underwater audio pipeline: 71% classification, 91% novelty detection","Denoise, classify, flag novel: AquaSignal unifies underwater sound","Odd trade-off: denoising drops AquaSignal classification from 77% to 71%","First integrated study on maritime vessel acoustic data: AquaSignal","AquaSignal: first integrated pipeline for underwater acoustic analysis"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The novelty-detection result assumes that a cutoff fixed at the 70th percentile of reconstruction errors on training background clips will also separate unseen tug sounds from background sounds in the test recordings; if the test background differs from the training background, the 91.5% figure overstates real-world performance.","fun_headline_variants_meta":{"raw":{"variants":["Underwater audio pipeline: 71% classification, 91% novelty detection","Denoise, classify, flag novel: AquaSignal unifies underwater sound","Odd trade-off: denoising drops AquaSignal classification from 77% to 71%","First integrated study on maritime vessel acoustic data: AquaSignal","AquaSignal: first integrated pipeline for underwater acoustic analysis"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000479,"raw_usage":{"total_tokens":2375,"prompt_tokens":951,"completion_tokens":1424,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":567,"completion_tokens_details":{"reasoning_tokens":1325}},"tokens_in":567,"tokens_out":1424,"duration_ms":11793,"temperature":1.0,"reasoning_tokens":1325,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:36:56.213109+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Sweep the detection threshold across percentiles and plot a full ROC curve on held-out background and tug samples, then re-run the fixed 70th-percentile threshold on background recordings from different seasons, depths, or sensor gains and count false positives. If accuracy drops sharply away from the chosen percentile or in new background conditions, the reported generalization is an artifact of calibration rather than a property of the learned representation.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the leakage-free data-partitioning protocol and the evaluation metrics used throughout."},{"cited_title":"Bergler et al","cited_arxiv_id":null,"evidence_quote":"Supplies ORCA-CLEAN, the U-Net denoising method AquaSignal trains and applies in stage 2."},{"cited_title":"Irfan et al","cited_arxiv_id":null,"evidence_quote":"Supplies the DeepShip benchmark recordings and class labels used for the classification and novelty setup."},{"cited_title":"Heesemann et al","cited_arxiv_id":null,"evidence_quote":"Supplies the background-ocean recordings from a public observatory network that create the background class and train the autoencoder."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The in-text citation attached to the ResNet18 classifier used in the classification stage."}],"review_version":1}