{"id":"7d551126-47ec-471a-90df-5e8a51f98235","arxiv_id":"2506.21883","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A gradient-tracing data pruning method removes spurious images from a weakly supervised psoriasis severity classifier, raising test AUC by 5 points and catching most inter-rater label disagreements.","lead":"Gradient-based influence scores are used to identify and remove confusing training images from a psoriasis severity model trained on phone photos, improving held-out AUC from 85% to 90%. The same scores rank images where two dermatologists disagree, potentially replacing expensive manual annotation review.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported AUC gain lacks a random-removal control and is selected on the test set, so the causal role of TracIn-selected images is unestablished; a matched random-removal baseline with repeated seeds is needed.","rationale":"The reader's weakest-assumption analysis identifies exactly the load-bearing gap: the paper claims that TracIn-selected images are responsible for the AUC improvement, but the only ablation is the number of removed images. The improvement from 0.85 to 0.90 could plausibly arise from random dataset shrinkage, regularization, or retraining luck, especially because no random-removal baseline, multiple seeds, or error bars are reported. The paper has genuine strengths: the method is grounded in published influence-function work, the reported numbers are internally coherent, and the self-influence annotation-inconsistency result is a falsifiable empirical claim. However, those strengths do not control for the selection mechanism, which is the causal crux. The proposed test directly settles this: if random removal matches TracIn removal, the central claim falls; if TracIn clearly outperforms random removal with error bars and a validation-selected k, the claim is substantially supported. Because the reader already reached CONDITIONAL on this basis, the stress-test pass does not move the verdict; it sharpens the test needed before acceptance.","tokens_in":7836,"tokens_out":1896,"duration_ms":23806,"concrete_test":"Repeat the ConvNeXT pipeline with k=300 removed images (and ideally k=100-500) under two conditions: (1) TracIn-selected removal and (2) random removal of the same number of images, each repeated over at least 5 independent seeds. Report mean and standard deviation of test AUC-ROC for both conditions, plus a bootstrap 95% confidence interval on the difference. If random removal reproduces the 85-to-90 gain, the selection mechanism is not supported. Additionally, select k on the validation split only, and report the test AUC for that fixed k without re-selecting on the test set.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that gradient-traced images are the harmful ones, but Section 4 and Table 1 only vary k, the number of removed images, not the selection mechanism. Removing 2,292 of 28,060 images (8.2%) could improve test AUC from 85% to 90% simply because the training set is smaller, because training dynamics change with fewer examples, or because retraining with a different random seed produces a better model. The paper reports no repeated-seed runs, error bars, or confidence intervals, and the best k=300 is chosen by inspecting test-set AUC across k values, which selects an optimistic operating point. Without a random-removal control matched in size and repeated over seeds, the before/after comparison cannot distinguish TracIn's selection from a generic data-shrinkage or retraining-variance effect. A secondary ambiguity is that the displayed TracIn equation in Section 2.3 omits the dot product with the training gradient present in Algorithm 1, but the primary load-bearing gap is the absent random-removal baseline.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes GRASP-PsONet, a data-pruning framework for psoriasis severity classification from patient-captured mobile images. The method uses TracIn-style gradient tracing from misclassified validation visits to rank and remove influential training images, then retrains a ConvNeXT- or ViT-based multi-instance learning classifier. On a held-out test set, removing 8.2% of flagged training images improves ConvNeXT AUC-ROC from 0.85 to 0.90 (Reader 1) and from 0.86 to 0.89 (Reader 2); the ViT encoder also improves when 13.3% are removed. In a separate experiment, self-influence scores are reported to identify 90.3% of inter-rater label disagreements within the top 30% of a 100-visit doubly rated subset. The central claims are that gradient-traced images are the harmful ones and that self-influence can flag annotation inconsistencies.","tokens_in":8021,"tokens_out":5147,"duration_ms":54422,"significance":"If the causal claim is established, the work would provide a practical data-curation tool for medical imaging with noisy labels, and the annotation-consistency detection result could reduce manual review burden. The use of published TracIn and self-influence methods is appropriate, and the application to real-world patient photos with two independent dermatologist raters is clinically relevant. However, the experimental design does not yet support the causal attribution: the removal effect is not compared against random removal, the pruning amount is selected using the test set, and no repeated-seed variability is reported. The self-influence result is promising but is demonstrated on a small, stratified subset without a random-ordering baseline or confidence intervals. The strengths are the real-data setting, the two-reader evaluation, and the explicit connection to gradient-based influence estimation; the weaknesses are the missing controls and selection procedure.","major_comments":[{"comment":"The causal claim that TracIn-flagged images are the harmful ones is not supported because the only ablation is the number of removed images, not the selection mechanism. Removing 2,292 of 28,060 images (8.2%) could improve test AUC through generic data shrinkage, altered training dynamics, or retraining with a different seed. A matched random-removal control, with the same number of images removed at each k and repeated over multiple seeds, is essential to attribute the improvement to TracIn selection.","section":"Section 4, Table 1"},{"comment":"The hyperparameter k is selected using test-set AUC: the paper reports the best-k results for ConvNeXT (k=300) and ViT (k=500) after inspecting Table 1, which lists test-set performance for multiple k values. This is an optimistic selection procedure and invites overfitting to the test set. The authors should either fix k a priori on clinical or computational grounds, or select k on the validation set and report test performance only for the chosen k, ideally with the full k-sweep clearly labeled as exploratory.","section":"Section 3.3, Table 1"},{"comment":"All reported numbers come from single training runs without error bars, confidence intervals, or multiple seeds. Given that only 9 of 64 validation visits are misclassified and that training is stochastic, the observed AUC gains of 0.02–0.05 may be within run-to-run variability. Repeated-seed experiments with mean and standard deviation (or bootstrap confidence intervals) are needed to establish that the improvements are reliable.","section":"Section 3.3, Table 1"},{"comment":"The claim that self-influence identifies 90.3% of inter-rater disagreements within the top 30% of a 100-visit subset needs a random-ordering baseline and an uncertainty estimate. With only 100 visits and an enforced 84:16 same/different-label ratio, the result corresponds to roughly 14–15 of about 16 disagreements, so the precision of the 90.3% figure is low. The authors should also clarify whether disagreement is defined at the PASI severity-class level (mild/moderate/severe) or at the numeric PASI level, and report recall at other operating points with a random baseline.","section":"Section 3.4, Figure 3"}],"minor_comments":[{"comment":"The displayed TracIn formula omits the dot product with the training gradient and appears to define the score using only the validation gradient; this is inconsistent with Algorithm 1, which includes the dot product ∇L(wt,z')·∇L(wt,z). The text should be corrected to match the algorithm.","section":"Section 2.3, Eq. (TracIn)"},{"comment":"The text says the top-100 to top-500 removals represent 3–16% of the training dataset (900–4,500 images), but Table 1 shows actual unique removal counts of 794–3,734 images, corresponding to 2.8–13.3%. The percentages should be harmonized.","section":"Section 3.3"},{"comment":"The contribution statement says the method improves AUC by approximately 5% and 10% for ConvNeXT and ViT, respectively. For ViT, Table 1 shows AUC improvements of 17 points (0.72 to 0.89) for Reader 1 and 12 points (0.72 to 0.84) for Reader 2; the stated 10% figure is unclear and should be revised or defined as a relative improvement.","section":"Abstract and Contributions"},{"comment":"The sentence about reconstructing the dataset to include test data independently scored by two readers, yielding 780 patient visits, is confusing because the test set is already part of the original 844-visit split. The authors should clarify what dataset is used for the self-influence experiment and how the 780-visit number is derived.","section":"Section 3.4"},{"comment":"The ViT rows list only k=100, 300, and 500, while the ConvNeXT rows include k=100, 200, 300, 400, and 500. Please either report the full k-sweep for ViT or state why intermediate values are omitted.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The reader's stress-test concern is well placed: without a random-removal control, the central causal attribution is unproven. The missing control and the test-set-based selection of k are the two load-bearing issues; both are fixable with additional experiments. The paper is otherwise a reasonable application of TracIn and self-influence to a relevant medical-imaging problem. I would be willing to review a revised version that adds random-removal baselines, repeated seeds, and a clearer model-selection protocol."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a legitimate application of published influence-tracing methods to a real clinical problem, and the annotation-discrepancy result is genuinely useful if it holds. But the central performance claim is not yet proven because there is no random-removal control and k is chosen on the test set.\n\nWhat's actually new: using TracIn and self-influence on a weakly supervised MIL psoriasis classifier trained on patient-captured phone photos, and showing that self-influence rank-ordering surfaces inter-rater label disagreements. That second piece—90.3% recall in the top 30% of a 100-visit subset—is the most interesting result here because it directly attacks the expensive double-reading problem in derm trials. The paper is clearly written, the method description is faithful to TracIn/self-influence, and the numbers are internally consistent.\n\nThe soft spots are experimental design, not math. The headline AUC gain (85 to 90 for Reader 1) comes from removing 8.2% of training images, but the paper never retrains with the same number of randomly removed images. Without that matched removal baseline, the improvement could be generic data shrinkage or retraining variance—especially since the ViT results show the opposite trend (top-500, most removed, gives the best AUC). There are no error bars, no multiple seeds, and the best k=300 is selected by looking at test AUC in Table 1. That makes the reported peak optimistic. The annotation-recognition experiment is also on a hand-constructed 100-visit subset with the label ratio enforced to match the full double-rated set, so it tells you the method works on a curated sample, not necessarily in the wild. The proprietary dataset and absence of code prevent independent reproduction, which is a constraint typical of industry work rather than a scientific error.\n\nThis paper is for researchers working on data pruning for medical imaging and for clinical trial teams who want a cheaper path to label cleaning. It deserves a serious referee, but not in its current form—the random-removal baseline, repeated seeds, and a k-selection protocol that does not touch the test set are must-fix items. I would accept it for review with a strong request for revision.","headline":"Legitimate application of TracIn/self-influence to a real clinical dataset, but the AUC gain is unproven without a random-removal baseline; the annotation-discrepancy finding is the most valuable part.","tokens_in":8578,"tokens_out":2423,"would_cite":false,"duration_ms":25623,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Removing the 8.2% of training images that a gradient-tracing influence score flags as most responsible for misclassified psoriasis severity cases lifts held-out AUC from 85% to 90%.","keywords":["Psoriasis","Severity classification","Multi-instance learning","Explainability","Gradient tracing","Spurious correlations","Data attribution","Inter-rater variability"],"falsifier":"Retrain the same ConvNeXT MIL model after removing an equal number of training images chosen at random, matched to the top-300 removal count (2,292 images); if random removal reproduces the 85% to 90% AUC gain, then the TracIn-based selection is not the operative cause of the improvement.","tokens_in":7637,"feed_emoji":"🩺","tokens_out":4940,"duration_ms":45836,"temperature":0.7,"pith_summary":"This paper claims that gradient-based data attribution can find the training images that teach a psoriasis severity model spurious correlations, and that removing those images improves the model. The setting is psoriasis severity classification (mild, moderate, or severe by PASI score) from patient-captured phone photos, using a weakly supervised multi-instance framework where each patient visit contributes 46 images. Tracing TracIn influence scores from misclassified validation visits back to training images, the authors flag the most influential training images; deleting 8.2% of the training set (the top 300 per misclassified visit, with overlaps merged) raises held-out test AUC-ROC from 85% to 90% for Reader 1 and from 86% to 89% for Reader 2. The same family of scores, in self-influence form, ranks training images so that reviewing the top 30% catches 90.3% of the cases where two dermatologists disagreed on the severity label. If right, this gives clinical-trial workflows a cheap way to audit noisy labels and artifact-laden remote images without full manual re-review.","feed_headline":"Removing 8% of flagged images lifts psoriasis AUC from 85 to 90","feed_subtitle":"Gradient tracing finds training photos that push a psoriasis model into error, and flags rater disagreements too.","key_machinery":"The engine is the TracIn influence score, computed as a sum over training checkpoints of the Adam-normalized gradient of the loss on the training image dotted with the gradient of the loss on the misclassified validation image; high positive scores mark training images whose presence most raised validation loss. In the weakly supervised multi-instance framework the unit of scoring is a single image, not a patient visit, so flagged images can be dropped individually while keeping the visit's other 45 images. A second mechanism, self-influence (the TracIn score of a training image against itself), is used as a label-noise detector, with high self-influence assumed to flag mislabeled or atypical images.","core_discovery":"The central claim is that the training samples most responsible for validation misclassifications, identified by summing gradient-dot-product terms over training checkpoints (the TracIn score), are disproportionately the samples carrying annotation inconsistencies or non-clinical artifacts, and that pruning them is what unlocks generalization. The paper demonstrates this on the GRASP-PsONet pipeline: a ConvNeXT (or ViT) encoder feeding an attention-based MIL aggregator over 46 images per patient visit. On the held-out test set, removing the top-300 influential training images per misclassified validation visit improves Reader 1 AUC from 0.85 to 0.90 and Reader 2 AUC from 0.86 to 0.89, with a convex effect: removing too few (top-100) buys little and removing too many (top-500, 13.3%) hurts, dropping AUC to 0.83 and 0.80. The paper further claims that self-influence scores, a sample's influence on its own prediction, identify inter-rater label disagreement: over 90% of the disagreements in a 100-visit doubly rated subset sit in the top 30% of the self-influence ranking.","pith_inferences":["The paper's causal reading, that gradient-traced images are the harmful ones, is not yet separated from the alternative that any modest targeted data cut helps; a random-removal control at the same removal count would settle that.","The same TracIn sieve should transfer to other weakly supervised medical settings with per-subject labels and artifact-prone acquisition, such as whole-slide pathology or multi-field fundus photography, where per-image influence can be computed the same way.","A natural extension the authors gesture at is to re-score flagged images instead of deleting them; if re-scored images are re-introduced, the pipeline becomes a data-cleaning loop rather than a data-loss procedure.","Because the inter-rater experiment uses only 100 randomly chosen visits with a maintained 84:16 label-agreement split, the 90.3% recall figure is a small-sample estimate whose precision depends on that ratio and on the specific rater pair."],"forward_implications":["Removing the top-300 flagged images per misclassified validation visit (8.2% of training data) raises held-out test AUC by about 5 points for ConvNeXT, and the ViT encoder's best-row AUC rises from 0.72 to 0.89 under the same procedure, so the pruning effect is not tied to one architecture.","There is a sweet spot in how many images to remove: top-100 and top-200 buy little, top-300 is best for ConvNeXT, and top-500 (13.3%) overshoots and degrades performance, so the method needs a tuning knob per task.","The self-influence ranking concentrates label disagreements in a small, reviewable fraction of the data, so annotation audits can be targeted rather than exhaustive.","Because images are scored individually, a visit with ambiguous labels is not discarded wholesale; only the offending images go, preserving most of the training signal.","Cohen's Kappa improves alongside AUC for the best ConvNeXT setting (0.53 to 0.62 for Reader 1), suggesting the pruned model's severity calls align more closely with human raters."],"supporting_citations":[{"why":"Supplies the TracIn gradient-tracing method and the self-influence score that the paper's whole data-pruning and label-noise detection pipeline is built on.","marker":"[20]"},{"why":"Supplies the attention-based weakly supervised MIL framework that the paper adapts, including the per-visit multi-image aggregation the pruning is applied to.","marker":"[9]"},{"why":"Establishes the influence-function foundation in deep learning that TracIn approximates, grounding the claim that gradient similarity measures training-sample impact.","marker":"[11]"},{"why":"Provides the related data-cleansing approach for SGD-trained models that the paper contrasts with replay-based TracInCP.","marker":"[4]"},{"why":"Supplies the score-guided data-pruning baseline for medical imaging (Dynamic Average Dice) that the paper positions its own influence-based pruning against.","marker":"[5]"},{"why":"Establishes the PASI gold standard and its inter- and intra-rater reliability, motivating the annotation-consistency problem the paper addresses.","marker":"[2]"}],"fun_headline_variants":["Prune 8% of psoriasis images, gain 5 AUC pts","Gradient tracing removes spurious psoriasis images, boosts AUC","Prune 8% of flagged images, psoriasis AUC 85 to 90","Flag spurious psoriasis images, lift AUC from 85 to 90"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim that the removed images are the harmful ones rests on a before-and-after comparison only: the paper does not test whether deleting the same number of images at random produces the same AUC gain, so the improvement could come from shrinking the training set rather than from what the gradient tracing identified.","fun_headline_variants_meta":{"raw":{"variants":["Prune 8% of psoriasis images, gain 5 AUC pts","Gradient tracing removes spurious psoriasis images, boosts AUC","Prune 8% of flagged images, psoriasis AUC 85 to 90","Flag spurious psoriasis images, lift AUC from 85 to 90"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00055,"raw_usage":{"total_tokens":2684,"prompt_tokens":1065,"completion_tokens":1619,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":681,"completion_tokens_details":{"reasoning_tokens":1541}},"tokens_in":681,"tokens_out":1619,"duration_ms":11607,"temperature":1.0,"reasoning_tokens":1541,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T22:16:28.134419+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the same ConvNeXT MIL model after removing an equal number of training images chosen at random, matched to the top-300 removal count (2,292 images); if random removal reproduces the 85% to 90% AUC gain, then the TracIn-based selection is not the operative cause of the improvement.","supporting_citations":[{"cited_title":"Advances in Neural Information Processing Systems 33, 19920–19930 (2020)","cited_arxiv_id":null,"evidence_quote":"Supplies the TracIn gradient-tracing method and the self-influence score that the paper's whole data-pruning and label-noise detection pipeline is built on."},{"cited_title":"PSO-Net: Development of an automated psoriasis assessment system using attention-based interpretable deep neural networks","cited_arxiv_id":"2501.18782","evidence_quote":"Supplies the attention-based weakly supervised MIL framework that the paper adapts, including the per-visit multi-image aggregation the pruning is applied to."},{"cited_title":"Advances in Neural Information Processing Systems32 (2019)","cited_arxiv_id":null,"evidence_quote":"Provides the related data-cleansing approach for SGD-trained models that the paper contrasts with replay-based TracInCP."},{"cited_title":"Data-Centric Diet: Effective Multi-center Dataset Pruning for Medical Image Segmentation","cited_arxiv_id":"2308.01189","evidence_quote":"Supplies the score-guided data-pruning baseline for medical imaging (Dynamic Average Dice) that the paper positions its own influence-based pruning against."},{"cited_title":"British Journal of Dermatology155(4), 707–713 (2006)","cited_arxiv_id":null,"evidence_quote":"Establishes the PASI gold standard and its inter- and intra-rater reliability, motivating the annotation-consistency problem the paper addresses."}],"review_version":1}