{"id":"92a965db-c0e3-4d43-acfa-cc2f74aaa110","arxiv_id":"2412.13079","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A method that applies Fourier, wavelet, and median filter transforms to full images to detect whether a CNN's accuracy is driven by dataset background bias rather than object features.","lead":"This paper tests whether simple image transforms can reveal hidden dataset bias in image classification AI. The authors report that for images collected under controlled conditions, AI accuracy stays high after transforms, which can signal that the AI is using background artifacts instead of the objects.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed bias signal is confounded with dataset type; no control experiments separate imaging-process bias from color, background structure, or object variability.","rationale":"I read the paper in good faith. The core idea is simple and plausible: transforms that smooth or decorrelate spatial structure may remove contextual features while preserving per-class artifacts from the imaging process. The authors provide code, use datasets whose bias was established in prior work, and present a consistent qualitative pattern. However, the strongest claim in Section 5 is causal: the method 'can distinguish between contextual information and imaging process bias.' That causal claim requires that the differential response across natural versus synthetic/hybrid datasets is attributable to the presence of imaging-process bias, not to any other systematic difference between the dataset groups. The paper provides no control experiment that manipulates bias while holding dataset type fixed. The reader's weakest assumption identifies exactly this gap, and I agree. The specific confounds are concrete: color versus grayscale, backgrounds that are fixed versus heterogeneous, and object variability. The decision rule in Section 5 is post hoc and threshold-free; without error bars or multiple seeds, even the descriptive pattern is not firmly established. The proposed concrete test—injecting a controlled bias into a natural dataset and removing the known bias from a synthetic dataset—would settle the causal question. If the method distinguishes the manipulated versions correctly, the central claim is supported; if not, the pattern observed in the paper is an artifact of dataset grouping. I do not think this concern requires changing the reader's conditional verdict, because the paper can be accepted as a preliminary proof-of-concept provided the authors perform these controls in a revision. The recommendation is therefore UNCHANGED: keep the CONDITIONAL verdict, with the explicit condition being the control experiments described above.","tokens_in":14064,"tokens_out":4174,"duration_ms":41273,"concrete_test":"Use a single natural dataset (e.g., Imagenette) and create two versions: (A) unmodified images, (B) images with an artificial per-class imaging-process bias (e.g., a fixed color cast or fixed low-amplitude noise pattern added to every image of a class). Train and test VGG16 under the paper's exact protocol (70/15/15 split, 40 epochs, Adam) with at least 5 random seeds, and measure original, wavelet, median, and median+wavelet accuracy. If the method is truly sensitive to bias rather than dataset type, version B should show stable or increased accuracy after transforms relative to original, while version A should show a drop. As a second control, take COIL-20 and remove its known imaging-process bias by replacing the blank background with random natural textures (or cropping tightly to the object); the method should then show a drop in accuracy after transforms.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in Section 5 is that wavelet transforms, median filtering, and their combination can distinguish contextual information from imaging-process bias, with the operational decision rule that stable or higher post-transform accuracy indicates bias. This claim rests on the assumption that the differential response across datasets is caused by the presence or absence of imaging-process bias. However, the six datasets differ systematically in ways that are not controlled: the natural datasets (Imagenette, Natural Images) are color images with heterogeneous backgrounds and high object variability, while the synthetic datasets (COIL-20, Yale Faces, CT Scans) are grayscale, single-source, and have controlled or uniform backgrounds. The hybrid Caltech 20 sits between. The paper's own rationale for choosing the synthetic datasets is that they are known to contain bias from prior blank-background studies (Section 3.2), so the test is circular: the method is never challenged to detect bias in a natural dataset or to reject bias in a synthetic dataset from which the bias has been removed. Table 1 also shows no error bars or multiple seeds, and some decisive comparisons are close (e.g., CT Scans: 70.5% vs 70% for median+wavelet vs original), making it unclear whether the observed pattern is statistically meaningful. If the differential response is driven by low-level image statistics rather than imaging-process bias, the Section 5 decision rule would produce both false positives (flagging neutral datasets with robust features) and false negatives (missing real bias in natural-looking data). Thus the load-bearing weakness is the missing causal validation that the transform response tracks bias specifically, not dataset type.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a method to detect dataset bias in CNNs without cropping blank background regions, by applying image transforms (Fourier transform, Haar/Daubechies wavelet transforms, median filtering, and their combinations) to full images. Experiments are performed with VGG16 on six datasets, categorized as natural (Imagenette, Natural Images), synthetic (CT Scans, COIL-20, Yale Faces), and hybrid (Caltech 20). The main observation is that wavelet and median-based transforms decrease classification accuracy on natural datasets but keep it stable or increase it on synthetic and hybrid datasets; Fourier transform decreases accuracy everywhere. From this, the paper concludes in Section 5 that stable or higher post-transform accuracy indicates that classification is driven by imaging-process background bias, and positions the method as a practical bias-alert tool when blank backgrounds are unavailable. The code is publicly available.","tokens_in":14338,"tokens_out":2509,"duration_ms":25919,"significance":"If the proposed rule were validated, it would offer a simple and inexpensive proxy for detecting hidden imaging-process bias in datasets where blank-background controls are impossible, which is a real need for medical imaging and other controlled-capture domains. The paper has concrete strengths: it uses established benchmark datasets, the transforms are clearly described, the decision rule is explicitly stated, and the code is released. However, the current evidence is preliminary: there are no error bars or repeated-seed experiments, the six datasets differ systematically in color, grayscale, object variability, and background structure, and the biased/unbiased labels are taken from prior blank-background studies that also motivated the rule. Because the central claim is an empirical decision rule, the missing statistical support and the absence of controls against confounds are load-bearing, not presentation issues.","major_comments":[{"comment":"The decision rule in Section 5 rests on accuracy differences, but Table 1 reports only point estimates from a single run with no confidence intervals, error bars, or statistical tests. Several decisive comparisons are very close: CT Scans median+wavelet is 70.5% versus 70.0% original, and Caltech 20 median+wavelet is 36.0% versus 36.6% original, which is a decrease, contrary to the stated pattern. With a single 70/15/15 split and one training run, these differences are within plausible noise. The paper should report results over multiple random seeds and data splits, with mean, standard deviation, and paired significance tests (e.g., McNemar's test or a paired bootstrap) for each transform versus the original images.","section":"§5, Table 1"},{"comment":"The method is validated in a circular manner: the synthetic datasets are selected because they were already known from prior blank-background studies to contain bias, and the rule 'similar or higher accuracy after transforms implies bias' is then derived from the observed responses on these same datasets. No independent, out-of-sample validation is performed. The paper should either prospectively predict bias status for datasets not used to develop the rule, or at least apply the method to additional datasets whose bias status is not already documented, and report sensitivity and specificity under a pre-registered threshold.","section":"§3.2, §5"},{"comment":"The observed differential response is confounded with dataset type: natural datasets are color images with heterogeneous backgrounds and high object variability, while the synthetic datasets are grayscale, single-source, and have controlled or uniform backgrounds. The conclusion that the response is caused by the presence or absence of imaging-process bias requires control experiments that separate this factor from low-level image statistics. Suggested controls include: applying the same transforms to grayscale versions of Imagenette and Natural Images, adding natural-like background variability to synthetic datasets, or artificially implanting a known acquisition artifact into a natural dataset and checking that the method flags it.","section":"§3.1–§3.3, §4"},{"comment":"All experiments use a single architecture (VGG16) with one training protocol (40 epochs, Adam). The paper repeatedly speaks of 'CNNs' in general, but no evidence is given that the pattern holds across architectures, depths, or training settings. At minimum, the key comparisons in Table 1 should be repeated with one or two additional standard architectures (e.g., ResNet or a smaller CNN) and with changes in training budget, since the proposed method is meant to be a general diagnostic for CNN bias.","section":"§3.4, §4"}],"minor_comments":[{"comment":"The caption says 'VGG15 architecture' but the text describes VGG16; the caption should be corrected.","section":"Figure 7"},{"comment":"The sentence 'The table provides a higher resolution view of the results' appears twice in succession; remove the duplicate.","section":"§4.4"},{"comment":"The y-axis labels in Figures 14 and 16 say 'Wavelet Accuracy' where the bar labels are median-filtered and median+wavelet, respectively; the labels should match the transform being reported.","section":"§4.3 and Figures 14, 16"},{"comment":"The 'Natural Images' dataset from Roy et al. (2018) is used with the 'fruit' class discarded; a brief description of the class counts and image sizes would help reproducibility.","section":"§3.1"},{"comment":"There are several typos and grammatical errors throughout, including 'This transformations affect', 'the images are cannot be registered', and 'Wavelet transformation were generated'; a careful language edit is needed.","section":"§4"},{"comment":"Table 1 does not report the random-chance accuracy for each dataset, which is important for interpreting values like Caltech 20's 36.6% accuracy; adding a chance-level row would improve readability.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The core idea is potentially useful, but the manuscript currently conflates correlation with causation: the transform response pattern is shown only for datasets that differ in many ways beyond bias. The missing statistics and the absence of any independent validation mean that the Section 5 decision rule is not yet established. I would encourage the editor to treat the paper as a promising methods note that requires substantial additional experiments rather than as a finished validation study."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: this paper proposes using full-image transforms (wavelet, median filter, and their combination) to detect background bias in CNNs without cropping blank regions. That is a genuinely useful extension of the authors' earlier blank-background method, and it is new relative to the cited literature. The pattern they report is consistent: natural datasets drop in accuracy after wavelet/median transforms, while synthetic and hybrid datasets stay stable or rise. If that pattern holds, it would give practitioners a cheap test for hidden dataset bias, especially in medical imaging where blank backgrounds are unavailable.\n\nWhat the paper does well: it is clearly written, the code is public, and the authors are honest about limitations. They explicitly note that the method identifies bias but does not correct it, and that a negative result does not guarantee the absence of bias. The use of multiple transforms and the inclusion of F1/precision/recall in Table 1 are also reasonable.\n\nThe soft spots are real and load-bearing. First, the six datasets differ systematically beyond their bias status: natural datasets are color, heterogeneous, multi-source; synthetic datasets are grayscale, single-source, controlled. The differential response to transforms could be driven by low-level image statistics or color structure rather than imaging-process bias. The paper provides no control experiments to separate these. Second, the decision rule in Section 5 is post hoc: it was derived after observing the pattern on datasets whose bias status was already known from prior blank-background studies. There is no independent validation on a dataset with known bias that was not used to construct the rule. Third, the statistics are thin: no error bars, no multiple seeds, a single architecture (VGG16), and some comparisons are within a few percent (e.g., CT Scans 70.5% vs 70%). The claim of reliable distinction between contextual information and bias is therefore stronger than the evidence supports.\n\nThat said, the stress-test concern about circularity is only partially fair. The paper does not hide that it selected synthetic datasets because they are known to be biased; that is a deliberate testbed, not a hidden move. But the missing piece is the critical control: apply the method to a dataset where the background bias has been removed or artificially added, and show the transform response tracks that manipulation. Without that, the proposed test conflates bias with dataset type.\n\nWho this is for: researchers working on dataset bias, especially in medical imaging, who want a quick heuristic. It is a useful preliminary result, not a validated diagnostic.\n\nRecommendation: yes, send it to peer review, but with a request for additional control experiments and statistical rigor. The idea is worth airing, but the decision rule needs stronger empirical grounding before it is adopted.","headline":"A plausible but empirically thin extension of the authors' own blank-background bias work, worth a conditional read but not yet a reliable diagnostic.","tokens_in":14861,"tokens_out":668,"would_cite":false,"duration_ms":7832,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that applying wavelet transforms and median filtering to full images can reveal when a CNN's classification accuracy is driven by hidden background bias, eliminating the need to crop blank background regions for such a…","keywords":["dataset bias","convolutional neural networks","image transforms","wavelet transform","median filter","VGG16","benchmark datasets","medical imaging"],"falsifier":"Take a natural multi-source dataset and add a per-class background artifact of controlled strength, such as a faint fixed watermark or a constant gradient in every image of each class; then apply the wavelet-plus-median test. If the claim is correct, the transformed-versus-original accuracy pattern should shift from 'drops' to 'stable or rises' as the artifact is strengthened; if the pattern tracks dataset type instead, the diagnostic is not actually measuring imaging-process bias.","tokens_in":13868,"feed_emoji":"🔍","tokens_out":6655,"duration_ms":56382,"temperature":0.7,"pith_summary":"Deep neural networks can score high on benchmark image datasets by exploiting hidden background artifacts rather than the objects they are supposed to recognize. This paper claims to offer a simple screen for that kind of bias: apply a wavelet transform, a median filter, or both to the full images, retrain or test the same network, and compare accuracy. For natural datasets assembled from many independent sources, accuracy drops after the transforms. For controlled, single-source datasets previously shown to carry imaging-process bias, accuracy stays flat or rises. The stable-or-higher accuracy is the paper's indicator that classification is being driven by reproducible background signal, and the proposed reading is that the test can flag bias even when no blank background region is available to crop.","feed_headline":"Wavelet test reveals hidden bias in CNN benchmarks","feed_subtitle":"If accuracy survives wavelet and median filtering, a CNN is likely classifying background artifacts, not objects.","key_machinery":"The test is a comparison of classification accuracy across four versions of each dataset: the original images, images passed through a two-dimensional discrete wavelet transform (Haar or Daubechies), images smoothed by a 5×5 median filter, and images given the median filter followed by the wavelet transform. The wavelet–median combination is the carrier of the argument: median filtering removes salt-and-pepper noise while preserving edges, and the subsequent wavelet decomposition localizes the residual high-frequency structure, which the authors hold reveals the systematic signal left by the image-acquisition process while destroying the contextual content that supports genuine recognition. The decision rule is that if the transformed-image accuracy matches or exceeds the original accuracy, the CNN's predictions are likely being driven by background bias rather than by the objects of interest.","core_discovery":"On the paper's own terms, the discovery is that image transforms act asymmetrically on contextual object information and on the artifacts of the imaging process. Using the VGG16 architecture on six datasets, the authors show that Fourier transform reduces accuracy everywhere and is therefore unusable as a diagnostic. In contrast, Haar and Daubechies wavelet transforms, a 5×5 median filter, and the median-plus-wavelet combination lower accuracy on the natural datasets (Imagenette, Natural Images) while leaving accuracy unchanged or improved on the synthetic and hybrid datasets (CT Scans, COIL-20, Yale Faces, Caltech 20). The authors interpret the same experiments on 20×20 blank-background crops, where wavelet transforms raise the accuracy of crops from biased datasets, as direct evidence that the transform is exposing hidden background signal. The concluding claim is that accuracy on the transformed full images can distinguish contextual information from imaging-process bias and alert to bias without separating background from the original images.","pith_inferences":["A decisive control the paper does not run would be to inject a tunable per-class artifact, such as a faint watermark or a fixed illumination gradient, into natural images and verify that the accuracy pattern moves with artifact strength; such an experiment would isolate imaging-process bias from dataset provenance.","The six datasets differ in more than bias: the natural sets are color, multi-source photographs, while the biased positives are grayscale or single-setup captures, so the claimed diagnostic could in principle be tracking low-level statistical differences instead of bias. Converting the synthetic datasets to color or graying the natural datasets would test this.","The proposed rule is a warning flag, not a debiasing procedure; if adopted, it would motivate a follow-up stage that removes or neutralizes the flagged background signal rather than merely reporting it."],"forward_implications":["A researcher can screen a dataset for this class of bias without needing blank background crops: if accuracy after wavelet or median+wavelet transforms is comparable to or higher than original accuracy, the classifier is likely exploiting background signal.","The method is aimed at controlled-environment datasets such as medical images, where every pixel may carry diagnostic information and no clearly irrelevant background region exists to crop.","The test is a one-sided check: the paper states that a negative result does not guarantee the absence of bias, since other kinds of bias can still be present.","The Fourier transform is explicitly ruled out as a diagnostic, because it degrades accuracy across natural, synthetic, and hybrid datasets alike and cannot separate contextual from bias signal."],"supporting_citations":[{"why":"Established that object-recognition benchmarks including COIL-20 remain classifiable above chance from small blank background crops, defining the known-biased positive controls the transform test must flag.","marker":"(Model and Shamir, 2015)"},{"why":"Provided the blank-background classification experiments on biomedical, face, and object datasets that the proposed transform test is designed to replace when no blank background exists.","marker":"(Dhar and Shamir, 2021)"},{"why":"Showed face-recognition datasets can be classified from background-only sub-images, the earliest evidence cited for imaging-process bias in benchmark datasets.","marker":"(Shamir, 2008)"},{"why":"Contributed the class-activation-mapping evidence that COVID-19 X-ray CNNs rely on regions outside the ROI, motivating a no-crop bias detector.","marker":"(Majeed et al., 2020)"},{"why":"Supplied the VGG16 architecture used for all classification experiments in the paper.","marker":"(Simonyan and Zisserman, 2014)"},{"why":"Provides COIL-20, one of the synthetic controlled-environment datasets used as a known-biased positive control.","marker":"(Nene et al., 1996)"},{"why":"Provides the Yale Face Database, another known-biased positive control used to test the transform diagnostic.","marker":"(Georghiades et al., 2001)"}],"fun_headline_variants":["Wavelet test uncovers CNN bias without cropping images","No-crop wavelet trick reveals CNN background bias","Wavelet transforms spot CNN biases hidden in background","Detecting CNN bias via wavelet and median transforms","Wavelet diagnostic finds CNN bias without background crops"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that the different reaction of natural versus synthetic datasets to the transforms is caused by the presence or absence of imaging-process bias, and not by other systematic differences between the groups, such as color versus grayscale images, single-source versus multi-source collection, or the regularity of object poses.","fun_headline_variants_meta":{"raw":{"variants":["Wavelet test uncovers CNN bias without cropping images","No-crop wavelet trick reveals CNN background bias","Wavelet transforms spot CNN biases hidden in background","Detecting CNN bias via wavelet and median transforms","Wavelet diagnostic finds CNN bias without background crops"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000199,"raw_usage":{"total_tokens":1403,"prompt_tokens":1008,"completion_tokens":395,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":624,"completion_tokens_details":{"reasoning_tokens":323}},"tokens_in":624,"tokens_out":395,"duration_ms":5070,"temperature":1.0,"reasoning_tokens":323,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T13:26:19.767034+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a natural multi-source dataset and add a per-class background artifact of controlled strength, such as a faint fixed watermark or a constant gradient in every image of each class; then apply the wavelet-plus-median test. If the claim is correct, the transformed-versus-original accuracy pattern should shift from 'drops' to 'stable or rises' as the artifact is strengthened; if the pattern tracks dataset type instead, the diagnostic is not actually measuring imaging-process bias.","supporting_citations":[{"cited_title":"and Shamir, L","cited_arxiv_id":null,"evidence_quote":"Established that object-recognition benchmarks including COIL-20 remain classifiable above chance from small blank background crops, defining the known-biased positive controls the transform test must flag."},{"cited_title":"and Shamir, L","cited_arxiv_id":null,"evidence_quote":"Provided the blank-background classification experiments on biomedical, face, and object datasets that the proposed transform test is designed to replace when no blank background exists."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Showed face-recognition datasets can be classified from background-only sub-images, the earliest evidence cited for imaging-process bias in benchmark datasets."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Contributed the class-activation-mapping evidence that COVID-19 X-ray CNNs rely on regions outside the ROI, motivating a no-crop bias detector."},{"cited_title":"A., Nayar, S","cited_arxiv_id":null,"evidence_quote":"Provides COIL-20, one of the synthetic controlled-environment datasets used as a known-biased positive control."},{"cited_title":"S., Belhumeur, P","cited_arxiv_id":null,"evidence_quote":"Provides the Yale Face Database, another known-biased positive control used to test the transform diagnostic."}],"review_version":1}