{"id":"7d722132-b4e9-4a2a-9d4d-009c21ce4bf9","arxiv_id":"2505.11034","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"CleanPatrick is a new benchmark with expert-verified labels for off-topic images, near duplicates, and label errors in 16,577 dermatology photos, plus a ranking-based evaluation framework.","lead":"This paper introduces CleanPatrick, a large benchmark for cleaning image datasets, built from 496,377 crowd-worker annotations on dermatology images. It tests existing cleaning methods and finds that label-error detection remains an unsolved challenge.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Near-duplicate ground truth is generated by DINO nearest-neighbor search under Assumption 2 (Appendix G); without validation, the benchmark's duplicate positives are DINO-selected and SelfClean's near-duplicate advantage is partly an artifact of candidate generation.","rationale":"The reader's weakest assumption matches the most load-bearing concern: the near-duplicate ground truth is not independently established but is generated by DINO nearest-neighbor search under an unvalidated assumption. This is the single point on which the benchmark's headline near-duplicate result and its claim to be a neutral evaluation platform hinge. Appendix A(1) concedes the risk, which strengthens rather than weakens the concern. I also noted the abstract/body mismatch on label-error prevalence (32% vs 22%) and the presence of FINE/BHN in the header abstract but not in the results; these are real reporting inconsistencies and should be fixed, but they do not affect the structural validity of the near-duplicate evaluation. The proposed exhaustive-pair test is feasible at the scale already undertaken in this project and would settle whether Assumption 2 holds. Because the released data and code provide a path to validation, the concern does not warrant rejection; it supports keeping the reader's CONDITIONAL verdict, so no verdict adjustment is needed.","tokens_in":20873,"tokens_out":7419,"duration_ms":68326,"concrete_test":"Exhaustively annotate all pairs in a random subset of 100–200 images from the near-duplicate pool using the same protocol and expert annotators, then compare the resulting duplicate graph with the graph produced by the fast-duplicates procedure. If any pair is judged a near-duplicate in the exhaustive graph but absent from the benchmark's positive set, Assumption 2 is violated and the ground truth is DINO-dependent. Recompute SelfClean, pHash, and SSIM metrics restricted to this exhaustive subset; a shrinking SelfClean advantage relative to Table 2 would confirm that the headline near-duplicate result is an artifact of candidate selection.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that self-supervised representations 'excel at near-duplicate detection' (AP 0.88 vs pHash/SSIM near chance) depends on the completeness of the duplicate ground truth. The fast-duplicates procedure in Section 3.2 and Appendix G presents annotators only with DINO nearest-neighbor pairs, relying on Assumption 2: every near-duplicate of an image is closer in DINO embedding space than any non-duplicate. No validation of this assumption is reported, and the authors explicitly acknowledge the incompleteness risk in Appendix A (limitation 1). If Assumption 2 fails, any duplicate pair whose DINO distance is not minimal for at least one endpoint is never annotated and is absent from the positive set. SelfClean uses self-supervised embeddings closely related to DINO to score pairs, so it is evaluated on positives it can discover; pHash and SSIM are measured on a candidate set constructed by a representation their similarity measures do not share. Section 3.5 also states that unannotated pairs are not treated as negatives, so evaluation is confined to DINO-generated candidates plus human rejection labels. The headline near-duplicate comparison is therefore at least partly an artifact of how ground truth was generated, weakening the benchmark's claim to be a neutral testbed for data-cleaning methods.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces CleanPatrick, which it claims is the first large-scale benchmark for image data cleaning. The benchmark is built on the Fitzpatrick17k dermatology dataset and provides binary ground-truth labels for three data-quality issues: off-topic samples, near-duplicate images, and label errors. The labels are derived from 496,377 crowd annotations by 933 medical crowd workers, aggregated with a GLAD-based item-response-theory model and calibrated with expert review of a stratified subsample. The paper formalizes each issue as a ranking task and evaluates several existing cleaning methods, reporting that SelfClean excels at near-duplicate detection (AP 0.88), classical anomaly detectors are competitive for off-topic detection, and label-error detection remains close to chance. The dataset, annotations, and evaluation code are released publicly.","tokens_in":21162,"tokens_out":6938,"duration_ms":71327,"significance":"If the benchmark's ground truth is trustworthy and method-neutral, CleanPatrick would be a valuable community resource: it is built on real, naturally occurring contamination rather than synthetic noise; the annotation campaign is unusually large; the ranking-task formulation matches practical audit workflows; and the release of data and code supports reproducibility and future comparison. These strengths are real and should be credited. However, the benchmark's central claim to provide a neutral testbed is currently weakened by the way the near-duplicate ground truth was constructed and by several unresolved consistency issues in the reported statistics. The near-duplicate candidate set is generated with a DINO embedding, and the method that achieves the headline near-duplicate results, SelfClean, uses closely related self-supervised representations; without validation that the candidate-generation step does not bias the evaluation, the main comparative conclusion is not yet established.","major_comments":[{"comment":"The near-duplicate ground truth is not exhaustive: candidate pairs are generated by DINO nearest-neighbor search under Assumption 2 (Fast cleaning) in Appendix G, and only those candidates are annotated. The paper provides no validation of this assumption, and Appendix A explicitly acknowledges that it may lead to incomplete discovery of near-duplicate groups. Because §3.5 states that unannotated pairs are not treated as negatives, the near-duplicate evaluation is restricted to the DINO-generated candidate set. SelfClean uses self-supervised embeddings closely related to DINO, so its reported AP of 0.88 versus pHash/SSIM near chance in Table 2 and Figure 3 may reflect the candidate-generation process rather than a general superiority in duplicate detection. To support the benchmark's neutrality, the authors should validate Assumption 2, for example by comparing candidate recall against a second independent embedding or perceptual hash and by manually inspecting a random sample of non-candidate pairs, and they should report how the evaluation changes when additional duplicate pairs found by that validation are added to the ground truth.","section":"§3.2, §3.5, Appendix G, Table 2"},{"comment":"The abstract supplied with the submission states a 32% label-error rate, while the full-text abstract and Section 4.1 report 22% (3,666 images). The label-error prevalence is a headline quantity of the benchmark, and the inconsistency between the two versions of the abstract is a factual error that must be resolved in the final version.","section":"Abstract vs. §4.1"},{"comment":"The expert-calibrated thresholds t_OT=0.76, t_ND=0.70, and t_LE=0.77 are derived from only 400 expert-annotated samples per issue type, using a rule described only as 'the bin where the distribution of positive labels starts to increase.' The final ground-truth labels, the reported prevalences, and all evaluation numbers depend directly on these thresholds, yet no sensitivity analysis is provided. The paper should specify the threshold-selection rule precisely, report how the prevalence and the benchmark results change under reasonable alternative thresholds, and ideally provide uncertainty estimates for the threshold-dependent quantities.","section":"§3.4"},{"comment":"Section 2 states that the authors 'do not use any tools or methods for performing the data selection but only rely on exhaustive annotation,' but Section 3.2 and Appendix G describe a DINO-embedding-based selection of near-duplicate candidate pairs. This is an internal contradiction about a central design choice. The near-duplicate annotation is not exhaustive, and the claim of unbiased, tool-free annotation should be corrected and qualified accordingly.","section":"§2 vs. §3.2"},{"comment":"The paper describes the released ground truth as 'expert-verified,' but expert review was performed on only 400 samples per issue type for quality control and threshold calibration; the final labels for the remaining samples come from crowd annotations aggregated with GLAD and thresholded. This overstates the level of expert verification of the released labels and should be reworded to describe the actual procedure: expert review of a stratified subsample used to calibrate and validate crowd-derived labels.","section":"Abstract, §1, §3.4"}],"minor_comments":[{"comment":"Table 2 reports p+ = 3.7% for off-topic and 21.4% for near duplicates, while Section 4.1 reports 4% and 21%; please clarify whether p+ is computed on the evaluation subset or the full dataset and make the denominators consistent.","section":"Table 2 vs. §4.1"},{"comment":"The submission header abstract lists FINE and BHN among the benchmarked methods, but the full text, Appendix D, and Table 2 do not describe or report results for these methods; the abstract should be aligned with the actual experimental content.","section":"Abstract"},{"comment":"The positive criterion for the off-topic task says the image is off-topic 'as identified by the medical crowd workers,' but the final labels are obtained from GLAD aggregation plus expert-calibrated thresholds; the criterion should reference the actual label-generation process.","section":"§3.5"},{"comment":"The main text says the fast-duplicates procedure requires at most 2N pairwise judgments, while Lemma 1 in Appendix G states at most |D| sample pairs; the relationship between these bounds (for example, whether each pair is annotated once or twice) should be reconciled.","section":"§3.2 and Appendix G"},{"comment":"The statement that each sample received an average of 10 votes is only correct if the 496,377 annotations are divided across three issue-specific samples per image; the text should state this explicitly to avoid confusion.","section":"§3.2"}],"recommendation":"major_revision","confidential_remarks":"The near-duplicate candidate generation uses a DINO embedding, and SelfClean, the method that achieves the headline near-duplicate results, is a self-supervised representation method from the same research group. I have no reason to question the authors' integrity, but this design choice gives SelfClean an inbuilt advantage in the reported comparison, and the paper currently does not disclose or mitigate this sufficiently. The abstract-level inconsistencies (label-error rate, missing FINE/BHN) also suggest the manuscript needs a careful consistency pass before it is suitable for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper delivers what the image data-cleaning field has been missing: a large-scale benchmark with real, human-labeled contamination instead of synthetic noise. 496,377 binary votes from 933 medical crowd workers on Fitzpatrick17k, expert verification, and released data and code make this a solid, citable resource. The fast-duplicates procedure is a neat contribution with a clear lemma and proof, and the ranking-task formalization is sensible. Credit is due for the scale of the annotation campaign and for shipping the data and evaluation code.\n\nThe main soft spot is the near-duplicate ground truth. The positive set is not an exhaustive set of duplicate pairs; it is the set of pairs that DINO nearest-neighbor search surfaced, under Assumption 2 in Appendix G. That assumption is unvalidated, and the authors acknowledge the incompleteness risk in Appendix A. The consequence is exactly what the stress-test note says: SelfClean, which uses self-supervised embeddings closely related to DINO, is evaluated on positives it can discover, while pHash and SSIM are scored on a candidate set produced by a representation their similarity measures do not share. The AP 0.88 vs ~0.31 comparison is therefore partly an artifact of candidate generation. This does not sink the off-topic or label-error tasks, but the near-duplicate leaderboard should be read as \"performance on DINO-discoverable duplicates,\" not as a neutral comparison. The authors should either validate Assumption 2 on a random sample of pairs, or restrict their claims accordingly.\n\nOther soft spots are real but more minor. The abstract inconsistency on label-error prevalence (32% in one version, 22% in the full text) needs fixing. Expert thresholds are calibrated from only 400 samples per issue, and expert agreement on label errors is low (Krippendorff's alpha about 0.42), so that ground truth is noisy. The paper reports no error bars or significance tests for method comparisons; since the methods are deterministic on fixed features, this is less critical, but some bootstrap intervals would help. The decision not to treat unannotated pairs as negatives is defensible and stated, but it means the AP numbers are computed over a restricted, DINO-biased pair set.\n\nWho should read this: anyone building or evaluating data-cleaning methods for images, especially in medical imaging. The benchmark is a step forward even with its caveats. It deserves serious peer review; a good referee should push for validation of Assumption 2, a corrected abstract, and a more careful framing of the near-duplicate results. I would send it out.","headline":"A genuinely useful first large-scale image data-cleaning benchmark, but the DINO-generated near-duplicate ground truth biases the headline comparison and needs scrutiny before the benchmark is treated as neutral.","tokens_in":21720,"tokens_out":1668,"would_cite":true,"duration_ms":19407,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CleanPatrick is the first dedicated image-domain benchmark for data cleaning, built from 496,377 crowd-sourced judgments of off-topic images, near duplicates, and label errors in a real dermatology dataset.","keywords":["data cleaning","benchmark","near-duplicate detection","label error detection","off-topic detection","dermatology images","item response theory","crowdsourced annotations"],"falsifier":"Take a random subsample of several thousand Fitzpatrick17k images, annotate all image pairs within small candidate clusters using the same crowd protocol but without the nearest-neighbor pre-filter, and check whether any near-duplicate component exists in which an image's DINO nearest neighbor is not a duplicate of it; if such components appear at a non-negligible rate, the 21% duplicate figure and the duplicate rankings are biased toward DINO-like similarity. Re-running the same check with a different self-supervised embedding would show how large that bias is.","tokens_in":20718,"feed_emoji":"🧹","tokens_out":14598,"duration_ms":117862,"temperature":0.7,"pith_summary":"CleanPatrick is an attempt to give image-data cleaning the kind of standardized testbed that structured-data cleaning already has. The paper repurposes the 16,577-image Fitzpatrick17k dermatology collection, gathers 496,377 binary judgments from 933 medical crowd workers about three quality issues — off-topic images, near duplicates, and label errors — and converts the noisy votes into expert-verified ground truth with an item-response-theory aggregation model. The result is a benchmark that reports 4% off-topic samples, 21% near duplicates, and 22% label errors in a widely used real dataset, and that recasts each detection problem as a ranking task scored with AUROC, average precision, and top-k precision under review budgets. It matters because cleaning methods are currently compared on synthetic corruption that may not resemble real noise, whereas CleanPatrick measures them on contamination that actually occurs in an uncurated medical collection. Its main findings are that self-supervised representations nearly solve duplicate detection, classical anomaly detectors are competitive for off-topic detection under tight review budgets, and detecting clearly wrong labels in fine-grained medical classification remains unsolved.","feed_headline":"496,377 human labels reveal what AI image-cleaning actually catches","feed_subtitle":"A dermatology image audit shows what cleaning catches easily—and where label errors still stump machines.","key_machinery":"The benchmark runs on two mechanisms. The fast-duplicates procedure avoids the $O(N^2)$ cost of judging every image pair: each image is embedded with a DINO self-supervised encoder, each round sends only every image's nearest neighbor to annotators, positively confirmed pairs are merged into clusters, and the process iterates; under the assumption that an image's duplicates are always its closest neighbors, Lemma 1 guarantees that all duplicate groups are discovered in at most $\\lfloor \\log_2 K \\rfloor + 1$ rounds using at most $N$ pair judgments, where $K$ is the size of the largest group. The aggregation model is a modified GLAD item-response-theory model that treats annotator expertise and sample difficulty as latent variables and estimates, from the sparse binary votes, a posterior probability per image that the issue is present; the sign and magnitude of the difficulty parameter encode class and confidence, and expert review calibrates one threshold per issue type. These two pieces turn noisy crowd votes into the benchmark ground truth, and the ranking formulation (AUROC, AP, P@k, R@k) is what makes any two cleaning methods comparable on the same real data.","core_discovery":"The central claim is that a large, expert-verified contamination testbed for image cleaning is feasible, and that its construction exposes a sharp split in current methods. On CleanPatrick, the holistic method SelfClean reaches an AUROC of 0.92 and an average precision of 0.88 for near duplicates, with perfect precision among the top 1,000 ranked pairs, while perceptual hashing and SSIM stay near chance; classical anomaly detectors (Isolation Forest, HBOS, ECOD) reach AUROC 0.76–0.77 for off-topic detection and outperform richer methods on precision among the top 100 candidates; and for label errors every tested method performs at or barely above the 22% base rate. The authors conclude that self-supervised representations excel at duplicate detection, off-topic detection is serviceable when review budgets are small, and label-error detection in fine-grained medical classification remains an open problem that will need context-aware signals.","pith_inferences":["The near-duplicate ground truth is itself selected by a DINO embedding, so methods built on the same self-supervised representation family are being scored on positive examples tailored to that family's notion of similarity; re-deriving the duplicate labels with a second, independent embedding and re-running the benchmark would reveal how much of the 0.92 AUROC comes from sharing the selection rep","The instruction to mark only 'clearly wrong' labels deliberately excludes ambiguous and uncertain diagnoses, so the 22% figure is a conservative floor; a variant benchmark that scores label confidence rather than clear error would likely surface substantially more mislabels and might discriminate better among cleaning algorithms.","Nothing in the pipeline — task decomposition, fast-duplicates selection, GLAD aggregation, expert thresholding — is specific to dermatology, so the same procedure applied to other image collections would test whether the difficulty ordering observed here (duplicates easy, off-topic medium, labels hard) is a general property of real image contamination.","Because the final thresholds were calibrated by only three dermatologists, the published positive sets encode one narrow clinical judgment; re-calibrating thresholds with a larger, more diverse panel of experts is the most direct robustness check on every prevalence number in the paper."],"forward_implications":["Because 21% of the dataset consists of near duplicates, models trained on un-cleaned Fitzpatrick17k carry inflated class counts and a real risk of train/test leakage; the benchmark's duplicate ground truth makes both effects measurable and removable.","Self-supervised ranking alone places every true duplicate within the top 1,000 pairs (P@1000 = 1.00), so duplicate removal before training can be automated with very high precision.","Under a review budget of 100 images, SelfClean finds the most truly off-topic images (P@100 = 0.52) despite worse global ranking than the classical detectors, showing that the best method depends on whether the audit is budget-constrained.","Because label-error detection is at or near base rate, automated cleaning pipelines for fine-grained medical data cannot yet be trusted to find mislabels, and the paper leaves this as an open challenge.","Standardized ranking metrics on one real, expert-verified dataset allow cleaning methods to be compared directly against each other, replacing incompatible synthetic corruption protocols."],"supporting_citations":[{"why":"Fitzpatrick17k, the public dermatology dataset whose 16,577 images and original diagnoses CleanPatrick annotates and re-releases with quality labels.","marker":"[11]"},{"why":"DINO, the self-supervised encoder whose nearest-neighbor pairs define which image pairs the crowd annotators ever see in the fast-duplicates procedure.","marker":"[20]"},{"why":"GLAD, the item-response-theory model adapted to estimate annotator expertise and per-image difficulty from the sparse binary votes.","marker":"[21]"},{"why":"SelfClean, the holistic cleaning method benchmarked on all three issues and the source of the issue taxonomy and ranking formalization.","marker":"[4]"},{"why":"Confident Learning, one of the two label-error detection baselines that perform near the 22% base rate.","marker":"[5]"},{"why":"NoiseRank, the Markov-random-field label-noise detector used as the second label-error baseline.","marker":"[6]"},{"why":"Isolation Forest, one of the classical anomaly detectors shown to be competitive for off-topic detection.","marker":"[28]"},{"why":"The perceptual-hashing baseline that performs near chance on near-duplicate detection.","marker":"[31]"},{"why":"The SSIM baseline that performs near chance on near-duplicate detection.","marker":"[32]"},{"why":"Earlier work documenting leakage, duplicates, and mislabels in dermatology benchmarks, which motivates cleaning real contamination rather than synthetic noise.","marker":"[12]"}],"fun_headline_variants":["CleanPatrick: 496k labels show what image cleaning misses","Near-duplicate detection solved, label errors remain for AI cleaners","CleanPatrick benchmark: 32% label errors still stump AI cleaning","Self-supervised beats dupes, but label errors defeat all AI cleaners"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The near-duplicate ground truth rests on the assumption that for every image, all of its true duplicates are closer to it in the paper's self-supervised embedding space than any non-duplicate, which is what lets the annotators judge only nearest-neighbor pairs; if that assumption fails, whole duplicate groups are never discovered and every duplicate-detection result is scored against an incomplete target set.","fun_headline_variants_meta":{"raw":{"variants":["CleanPatrick: 496k labels show what image cleaning misses","Near-duplicate detection solved, label errors remain for AI cleaners","CleanPatrick benchmark: 32% label errors still stump AI cleaning","Self-supervised beats dupes, but label errors defeat all AI cleaners"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000575,"raw_usage":{"total_tokens":2712,"prompt_tokens":937,"completion_tokens":1775,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":553,"completion_tokens_details":{"reasoning_tokens":1701}},"tokens_in":553,"tokens_out":1775,"duration_ms":11606,"temperature":1.0,"reasoning_tokens":1701,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:59:01.666250+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random subsample of several thousand Fitzpatrick17k images, annotate all image pairs within small candidate clusters using the same crowd protocol but without the nearest-neighbor pre-filter, and check whether any near-duplicate component exists in which an image's DINO nearest neighbor is not a duplicate of it; if such components appear at a non-negligible rate, the 21% duplicate figure and the duplicate rankings are biased toward DINO-like similarity. Re-running the same check with a different self-supervised embedding would show how large that bias is.","supporting_citations":[{"cited_title":"Evaluating deep neural networks trained on clinical images in dermatology with the fitzpatrick 17k dataset","cited_arxiv_id":null,"evidence_quote":"Fitzpatrick17k, the public dermatology dataset whose 16,577 images and original diagnoses CleanPatrick annotates and re-releases with quality labels."},{"cited_title":"Whose V ote Should Count More: Optimal Integration of Labels from Labelers of Unknown Expertise","cited_arxiv_id":null,"evidence_quote":"GLAD, the item-response-theory model adapted to estimate annotator expertise and per-image difficulty from the sparse binary votes."},{"cited_title":"Navarini, and Marc Pouly","cited_arxiv_id":null,"evidence_quote":"SelfClean, the holistic cleaning method benchmarked on all three issues and the source of the issue taxonomy and ranking formalization."},{"cited_title":"Confident learning: Estimating uncertainty in dataset labels","cited_arxiv_id":null,"evidence_quote":"Confident Learning, one of the two label-error detection baselines that perform near the 22% base rate."},{"cited_title":"Noiserank: Unsupervised label noise reduction with dependence models","cited_arxiv_id":null,"evidence_quote":"NoiseRank, the Markov-random-field label-noise detector used as the second label-error baseline."},{"cited_title":"Isolation Forest","cited_arxiv_id":null,"evidence_quote":"Isolation Forest, one of the classical anomaly detectors shown to be competitive for off-topic detection."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The perceptual-hashing baseline that performs near chance on near-duplicate detection."},{"cited_title":"Bovik, H.R","cited_arxiv_id":null,"evidence_quote":"The SSIM baseline that performs near chance on near-duplicate detection."},{"cited_title":"Towards reliable dermatology evaluation benchmarks","cited_arxiv_id":null,"evidence_quote":"Earlier work documenting leakage, duplicates, and mislabels in dermatology benchmarks, which motivates cleaning real contamination rather than synthetic noise."}],"review_version":1}