{"id":"b8f46f54-54c0-4959-9c93-938b7b5893d8","arxiv_id":"2411.14626","paper_version":3,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Underwater image enhancement hurts detection accuracy on average, yet a per-image oracle shows that selectively enhanced images can outperform original ones, motivating image-level selection.","lead":"This paper tests nine underwater image enhancement tools and finds that, on average, they do not help object detection across whole datasets. But some individual images do improve after enhancement, hinting that selective enhancement could boost detection if researchers can predict which images benefit.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The oracle mixed-set upper bound in Table 6 likely overstates per-image enhancement gains because selecting the best of ten noisy per-image mAP values can manufacture improvement; a permutation or random-baseline test is needed.","rationale":"The paper is a useful empirical study: the dataset-level finding that none of the nine enhancement models improves detection over the original detector is consistent with prior work and is supported by Tables 4 and 5. The genuinely novel claim is the image-level 'potential' of enhancement, and that claim rests on the oracle mixed set. The reader's CONDITIONAL verdict already identified per-image mAP stability as the weakest assumption; I agree. The concern can be sharpened: because the mixed set is the maximum of ten noisy per-image estimates, a positive gap over the original average is expected under a null hypothesis of no enhancement effect. The decisive check is a permutation test that preserves the max-selection procedure while breaking any real association between image content and enhancement benefit. If the observed mixed-set mAP is not extreme under that null, the quantitative evidence for potential disappears; only the outcome-selected qualitative examples remain, which are weak evidence because they are selected by outcome. If the observed values are extreme, the concern is resolved and the paper's conditional 'potential' claim stands. No change to the CONDITIONAL verdict is needed beyond requiring this analysis, which is why verdict_should_be is UNCHANGED. The public code and data make the proposed check straightforward.","tokens_in":20713,"tokens_out":6749,"duration_ms":70546,"concrete_test":"Run a permutation null for Table 6: for each test image, shuffle the per-image mAP values of the ten variants and record the maximum; average these maxima to form one null mixed-set mAP. Repeat 1,000 times to obtain a null distribution for CUPDD and RUOD. Also form a random-variant mixture with the per-algorithm proportions of Figs. 10/11. If the observed mixed-set values (0.64 and 0.77) are below the 95th percentile of the null, or if the random mixture is within one standard error of them, the per-image improvement in Section 6.2.2 is consistent with selection noise and should be reported only as an unvalidated upper bound. Reporting bootstrap confidence intervals for the per-image mAP differences would further settle the reader's stability concern.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The quantitative support for the paper's central claim is the oracle 'mixed set' in Section 6.2.2/Table 6, where each test image is assigned the variant (original or one of nine enhanced versions) with the highest per-image mAP. Per-image mAP on images with few labeled objects is highly discrete and noisy: for a single ground-truth box, AP at each IoU threshold is nearly a 0/1 outcome, so tiny detection changes can produce large mAP swings. Selecting the maximum over ten variants therefore creates an upward bias even if no enhancement algorithm has any systematic advantage, because the expected maximum of ten noisy estimates exceeds their average. The mixed set is also an oracle over detector identity, since each enhanced variant is scored by its own separately trained domain detector. No selector is proposed, and no permutation, bootstrap, or random-variant baseline is reported, so the claimed 0.64 vs 0.41 (CUPDD) and 0.77 vs 0.68 (RUOD) gains cannot yet be distinguished from selection noise. The qualitative cases in Section 6.2.1 are explicitly chosen by looking only at images where a domain detector did better, so they do not independently establish a systematic improvement.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper evaluates nine underwater image enhancement methods (non-physical, physical, and learning-based) on two underwater object detection datasets (CUPDD and RUOD). The authors train YOLO-NAS detectors on original and enhanced images, show that enhancement usually degrades detection performance at the dataset level (Section 5.1, Tables 4 and 5), propose a composite Q-index to study quality distributions, examine correlations between image quality metrics and mAP (Section 6.1), and then perform an image-level analysis that identifies individual images for which the domain detector outperforms the original detector (Section 6.2). The central claim is that enhancement has the \"potential\" to improve detection per image, supported by an oracle \"mixed set\" in Section 6.2.2/Table 6 that selects the best-performing variant for each image, achieving per-image mAP of 0.64 vs 0.41 on CUPDD and 0.77 vs 0.68 on RUOD.","tokens_in":20938,"tokens_out":3025,"duration_ms":30742,"significance":"If the per-image finding were statistically supported, the paper would make a useful contribution: it would show that dataset-level negative conclusions about underwater image enhancement are too coarse, and it would motivate selective enhancement pipelines. The study has concrete strengths: it covers nine enhancement methods and three detectors on two datasets, it releases data and code, it provides joint qualitative-quantitative visualizations, and it proposes a transparent Q-index for quality-distribution analysis. The dataset-level negative result (Section 5.1) is credible and consistent with prior work. However, the headline quantitative evidence for the paper's central claim is an oracle upper bound constructed using ground-truth labels, and the paper does not provide any statistical baseline to distinguish the reported per-image improvements from selection noise. This gap is load-bearing for the main thesis and must be addressed before the claims can be accepted.","major_comments":[{"comment":"The mixed-set mAP values of 0.64 vs 0.41 (CUPDD) and 0.77 vs 0.68 (RUOD) are produced by selecting, for each test image, the variant (original or one of nine enhanced versions) with the highest per-image mAP. Because the selection uses the ground-truth labels, this is an oracle upper bound: the expected maximum of ten noisy per-image mAP estimates exceeds their average, so the reported improvement is partly an artifact of selection and cannot be interpreted as the performance of any achievable enhancement-selection method. The paper acknowledges that the mixed set \"achieves the highest possible mAP\" (Section 6.2.2), but the abstract and conclusion present the 23% and 9% gains as evidence of enhancement's potential without this caveat. A permutation test or a random-variant baseline (e.g., randomly assigning each image to one of the ten variants, repeated many times) is needed to establish that the oracle gain exceeds what selection noise alone would produce.","section":"Section 6.2.2, Table 6"},{"comment":"The qualitative analysis is explicitly restricted to cases where a domain detector performed better than the original detector (\"This time, we only look at cases where the domain detector performed better than the Original detector\"). Therefore the selected examples cannot establish how frequently or systematically enhancement improves detection at the image level. The paper needs a systematic image-level comparison that reports the fraction of images improved, unchanged, and degraded by each enhancement method, together with a baseline or significance test, rather than only cherry-picked positive examples.","section":"Section 6.2.1"},{"comment":"All detection results are reported as point estimates from a single training run per condition, with no error bars, no repeated seeds, and no significance tests. Many reported differences are very small (e.g., Table 4: ACDC 0.34 vs TEBCF 0.33; Table 5: PCDE 0.61 vs TUDA 0.61), and the per-image mAP used in Table 6 is computed on single images often containing very few labeled objects, making it highly discrete and sensitive to small detection changes. Without variance estimates or multiple runs, the claimed per-image improvements cannot be distinguished from evaluation noise. This is especially important because the central claim depends on the stability of per-image mAP differences.","section":"Section 5.1, Tables 4 and 5"},{"comment":"The correlation study is based on scatter plots of only ten points (one per enhancement method, plus the original) and reports no correlation coefficients, confidence intervals, or significance tests. The conclusion that \"enhancement performance does not reliably predict detection performance\" is therefore underpowered and should be either quantified or softened. A simple Spearman or Pearson correlation with a confidence interval would make the claim testable.","section":"Section 6.1, Figure 7"}],"minor_comments":[{"comment":"The text says \"we selected four common object detection algorithms,\" but only three are listed (YOLO-NAS, RetinaNet, Faster R-CNN). Please correct the count.","section":"Section 3"},{"comment":"There is a typo: \"ON the contrary\" should be \"On the contrary.\"","section":"Section 4.2"},{"comment":"The first paragraph refers to \"Table 5\" when comparing YOLO-NAS, RetinaNet, and Faster R-CNN, but the actual comparison table appears to be Table 3. Please fix the cross-reference.","section":"Section 5"},{"comment":"The claim that \"the average increase in the Q-index value of low-quality images is much higher than the average increase in the Q-index values of high-quality images\" appears to be based on one randomly selected image per quality bin, not on a computed average across images. Please clarify whether this is an observation about the representative examples or a quantified average, and if the latter, report the supporting statistics.","section":"Section 4.3"},{"comment":"The Q-index construction does not specify how the four metrics are weighted in the averaging step, and the outlier threshold of three Median Absolute Deviations is an ad-hoc choice. Please state explicitly whether equal weights are used and discuss the sensitivity of the quality-distribution conclusions to the threshold choice, or at least note these as design decisions.","section":"Section 3, Q-index definition"}],"recommendation":"major_revision","confidential_remarks":"The paper's core new contribution is the image-level analysis, and it is currently undermined by the oracle construction of the mixed set and the absence of any statistical baseline. If the authors add a permutation/random-variant comparison and report variance or significance, the central claim could become defensible. The dataset-level negative result is not novel per se, but the per-image framing is a useful reframing; the paper should be clear about what is demonstrated versus what is only an upper bound."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is worth reading for the per-image result: dataset-level averages hide a real subgroup of images where enhancement helps detection. The authors reproduced the known adverse effect of enhancement on detection, then did something new by computing per-image mAP, showing a non-trivial minority of images improve. The oracle mixed set is honestly labeled an upper bound, and the qualitative examples are suggestive.\n\nWhat it does well: the study is broad and reproducible. Nine enhancement methods across categories, two datasets with different characteristics, public code/data. The quality distribution analysis is interesting, and the correlation study showing no link between quality metrics and mAP is a useful negative result. They do not oversell the dataset-level finding; they explicitly say no domain detector beat the original detector overall.\n\nWhere it gets soft: the headline numbers from the mixed set (0.64 vs 0.41, 0.77 vs 0.68) are maxima over ten noisy per-image estimates. With few boxes per image, per-image mAP is close to a binary outcome, so selecting the best variant per image manufactures gains even if enhancement is random noise. The paper does not report a permutation test, random-variant baseline, or any error bars. The qualitative cases are selected by looking only at images where a domain detector did better, so they do not independently confirm the effect. And the Q-index is built from ad hoc weights and outlier thresholds; fine for descriptive purposes, but not a validated metric.\n\nThe central descriptive claim—that enhancement helps some images and hurts others—holds up, and the paper is careful to frame the mixed set as an upper bound, not a method. But the magnitude of the claimed upper bound is likely inflated by selection noise, and the paper cannot currently distinguish that from a systematic per-image advantage.\n\nThis deserves a serious referee. The fixes are straightforward: add a random mixture baseline or permutation test, report variance across detector training runs, and maybe show how many images the mixed set actually draws from each source. That would turn a decent empirical study into a convincing one.","headline":"Useful per-image analysis, but the oracle mixed-set gains are probably selection noise until a permutation test is shown.","tokens_in":21507,"tokens_out":1874,"would_cite":true,"duration_ms":20241,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Underwater image enhancement can help detection when judged image by image, not just hurt it on average.","keywords":["Computer Vision","Underwater Object Detection (UOD)","Underwater Image Enhancement (UIE)","Underwater Datasets","Remote Sensing and Remotely Operated Vehicle (ROV)","Per-image mAP","Image quality metrics","Selective enhancement"],"falsifier":"Recompute the per-image mAP for the same images under several independent detector training runs (different random seeds or checkpoints); if the set of images scored as improved by enhancement changes substantially from run to run, the image-level gain is mostly evaluation noise rather than a real enhancement effect.","tokens_in":20552,"feed_emoji":"🌊","tokens_out":13012,"duration_ms":106614,"temperature":0.7,"pith_summary":"Underwater images are often degraded, and a common assumption is that running them through an enhancement model before object detection helps. This paper finds that at dataset level this assumption fails — detectors trained on enhanced images generally do worse than the original — but argues the assumption is too coarse. Measuring the same images one by one reveals that many low-quality images genuinely detect better after enhancement, and the images that improve are concentrated among the worst-quality ones. The final experiment builds a mixed set that chooses the best original or enhanced variant for each test image, and this set scores a per-image mAP of 0.64 versus 0.41 on CUPDD and 0.77 versus 0.68 on RUOD. The conclusion is that selective, image-level enhancement has real potential, so rejecting enhancement because of dataset averages would throw away a useful signal.","feed_headline":"Image by image, enhancement can boost underwater detection","feed_subtitle":"On CUPDD and RUOD, best-per-image enhancement reaches detection scores 0.64 and 0.77 vs 0.41 and 0.68.","key_machinery":"The carrying mechanism is the shift in the unit of evaluation from dataset to image, paired with a quality index. Rather than averaging detector performance over an entire test set, the paper computes mAP50-95 for each individual image, on the original and on each of the nine enhanced variants, and compares variants image by image; this per-image comparison is what exposes the split between images that benefit and images that are hurt. To sort images by quality and explain that split, the paper builds a Q-index, a bounded composite of four reference-free quality metrics (color, contrast/sharpness, chroma, entropy) that is rescaled and averaged so that original and enhanced images can be compared on one scale; the Q-index distributions show that low-quality images gain the most from enhancement while high-quality images tend to be over-enhanced. The demonstration that caps the argument is the mixed set, in which each test image is replaced by the variant with the highest per-image mAP, and the average of those per-image bests gives an upper bound of 0.64 on CUPDD and 0.77 on RUOD for what selective enhancement could achieve.","core_discovery":"On the paper's own terms, the central discovery is that the effect of underwater image enhancement on object detection is conditional on the image: it helps some images and hurts others, and dataset-level averages hide that split. The experimental route is systematic: nine enhancement models spanning physical, non-physical, and learning-based methods are applied to two datasets, one severely degraded and one large and varied; three detectors are compared on the originals, and the best one, YOLO-NAS, is retrained on every enhanced set to produce domain detectors. At the dataset level, no domain detector beats the original detector, which the paper sees as confirming earlier conclusions that enhancement degrades detection when used as a preprocessing step. At the image level, however, a substantial number of individual test images score higher mAP after enhancement, and the paper identifies a mechanism: enhancement reduces the disparity between low-quality, hard-to-detect scenes and the higher-quality scenes on which the detector was trained, effectively making some hard test images more like easy training images. The quantitative proof is the mixed image set — selecting the best-performing variant per image gives mAPs of 0.64 on CUPDD and 0.77 on RUOD, versus 0.41 and 0.68 for the originals — which the paper presents as an upper bound on what selective enhancement could deliver.","pith_inferences":["The mixed-set upper bound was built using ground-truth labels to pick the best variant for each image, so it is an oracle, not a deployable system; the paper's own framing implies a trained selector that predicts which variant will win, and a natural test is whether a lightweight classifier on image statistics plus current quality metrics can approximate half or more of that oracle gain.","The paper's proposed mechanism — that enhancement helps by making low-quality test images resemble high-quality training images — invites a direct test: measure how much the feature distribution of images shifts toward the training distribution after enhancement, and check whether that shift predicts the per-image mAP improvement.","The observation that visually pleasing deep-learning enhancers do not always beat simpler traditional enhancers on detection suggests that human aesthetics and machine-detector features are only partially aligned; a useful experiment would be to independently vary colorfulness, contrast, and edge sharpness and record which axis most affects per-image mAP."],"forward_implications":["Enhancement should not be dismissed as a preprocessing step solely because it fails on dataset averages; its effect is conditional on the image, with low-quality images benefiting most.","Because current image-quality metrics do not reliably predict detection performance, a practical selection policy needs metrics that encode machine perception as well as human-aesthetic quality.","If a selector could predict which images benefit from enhancement, the mixed-set result sets a concrete target: a per-image mAP gain of about 0.23 on CUPDD and 0.09 on RUOD over the original detector.","No domain detector beat the original at dataset level, so wholesale replacement of original training images with enhanced ones is not supported; the value of enhancement lies in per-image selection, not whole-dataset replacement."],"supporting_citations":[{"why":"provides the CUPDD dataset, the severely degraded testbed used for the image-level and mixed-set analyses","marker":"Saleem et al. (2023)"},{"why":"provides the RUOD dataset and the prior conclusion that enhancement hurts detection, which the paper's dataset-level results reproduce and its image-level results refine","marker":"Fu et al. (2023)"},{"why":"concluded via TIDE that enhancement increases false positives; the paper's dataset-level negative result aligns with this, while the per-image analysis extends it","marker":"Wang et al. (2023b)"},{"why":"concluded that restoration produces robustness rather than higher within-domain performance; the paper contrasts its own mixed-set result with this view","marker":"Chen et al. (2020b)"},{"why":"established that degradation removal does not restore CNN classification performance; the paper's detection-level negative result parallels this and is then qualified at image level","marker":"Pei et al. (2019)"},{"why":"supplies the YOLO-NAS detector implementation used for all detection models, the instrument through which per-image mAP is measured","marker":"Aharon et al. (2021)"}],"fun_headline_variants":["Enhancement's hidden win: it rescues hard underwater images","Dataset averages miss: underwater enhancement helps per image","Per-image enhanced detection beats originals underwater","Underwater detection: enhancement helps selectively, not broadly","Boost underwater detection with per-image enhancement"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that a per-image mAP computed on a single image, which often contains only one or a few labeled objects, is a stable measurement — stable enough that the images flagged as improved after enhancement reflect a systematic property of those images rather than random noise in evaluation.","fun_headline_variants_meta":{"raw":{"variants":["Enhancement's hidden win: it rescues hard underwater images","Dataset averages miss: underwater enhancement helps per image","Per-image enhanced detection beats originals underwater","Underwater detection: enhancement helps selectively, not broadly","Boost underwater detection with per-image enhancement"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000588,"raw_usage":{"total_tokens":2839,"prompt_tokens":1102,"completion_tokens":1737,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":718,"completion_tokens_details":{"reasoning_tokens":1663}},"tokens_in":718,"tokens_out":1737,"duration_ms":11550,"temperature":1.0,"reasoning_tokens":1663,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T15:04:49.885282+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the per-image mAP for the same images under several independent detector training runs (different random seeds or checkpoints); if the set of images scored as improved by enhancement changes substantially from run to run, the image-level gain is mostly evaluation noise rather than a real enhancement effect.","supporting_citations":[],"review_version":1}