{"id":"8958860e-73df-42e5-b930-9b498952251d","arxiv_id":"2511.12100","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"SS-CA masks the most decision-relevant regions identified by a counterfactual subset-selection attribution and trains on the relabeled original, improving ImageNet ID/OOD accuracy by up to ~2 points in most comparisons.","lead":"This paper proposes SS-CA, a training-time data augmentation that masks the image regions a model relies on most and replaces them with background, forcing the model to learn from other cues. On ImageNet variants it reports modest gains in accuracy and out-of-distribution robustness, but the headline numbers in the abstract do not match the tables.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Label-preservation of hard counterfactual samples is asserted but unvalidated; because hard mining selects exactly images where the model flipped away from y_gt, the original label may be wrong, undermining the causal claim.","rationale":"The reader identified the label-preservation assumption as the weakest point; I agree and consider it the most load-bearing issue. The method's causal interpretation depends entirely on the augmented image retaining the original class semantics. The harder the counterfactual sample (higher c_max after deletion), the more likely the mask removed the only discriminative evidence, making the original label invalid. The paper offers no empirical check of this assumption, and the qualitative figures are not a substitute. A human annotation study on the actual hard samples would settle whether the labels are trustworthy. If labels are unreliable, the method's success could stem from regularizing with noisy labels or exploiting donor background cues, not from learning more complete causal features. This does not require changing the reader's CONDITIONAL verdict; it sharpens the condition under which acceptance is warranted. The reader's other concerns—missing error bars and random-mask control—are secondary; if the label-validity test fails, the causal story collapses, and if it passes, the attribution-specific claim still needs a random-mask baseline to be fully established.","tokens_in":14540,"tokens_out":5758,"duration_ms":62065,"concrete_test":"Run Algorithm 1 on a fixed set of, say, 300 ImageNet-100 training images with the final trained model, producing the actual hard augmented samples that would enter B_aug. Show each augmented image to three independent human annotators without showing the original, and ask them to choose among y_gt, y_counter, and 'other/ambiguous'. Compute the proportion of samples for which the majority annotator selects y_gt. If this proportion is substantially below the model's original ID accuracy (e.g., <70%) or not clearly above chance, the label-preservation assumption fails. This single check directly tests whether the augmented samples used for training are correctly labeled.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central inductive bias is stated in Section 4.2/Algorithm 1 and Eq. (9): after replacing the counterfactual mask M_cf with donor background, the augmented image I'_j is assigned the original ground-truth label y_j. The method's causal justification—training enforces P(y_gt | I(V\\M_cf))—requires that the remaining visible content actually determines y_gt. This is not guaranteed and is actively threatened by the selection procedure. Counterfactual LIMA searches for a minimal region set whose removal flips the model's prediction to y_counter (Eq. 6 and Algorithm 1 lines 17–20), and hard mining keeps only samples with c_max > τ_aug (Eq. 9). Thus the augmented training set is concentrated on images where the model already predicts a different class after masking. If the mask covers the only discriminative evidence in that image, the 'hard sample' is semantically an image of a different class or is ambiguous, and training it under y_gt injects systematic label noise. The paper asserts humans can still recognize the object but provides no human study, no oracle probe, and no label-validity control. Without this validation, the reported ID/OOD gains could equally be explained by generic label-noise regularization or by learning donor-background artifacts, rather than by the proposed causal debiasing mechanism. This is load-bearing because every main result is interpreted through this causal intervention; if the labels are wrong, the mechanism is not what the paper claims.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SS-CA (Subset-Selected Counterfactual Augmentation), a training-time method that uses subset-selection-based counterfactual attribution (Counterfactual LIMA) to identify minimal image regions whose removal flips a model's prediction to a competing class. These regions are replaced with background patches from a donor image, and the resulting augmented sample is assigned the original ground-truth label and added to the training batch alongside the original samples. The authors claim that this closed-loop procedure reduces reliance on limited sufficient causes, improving in-distribution accuracy, out-of-distribution generalization, and robustness to corruptions across ImageNet-100, TinyImageNet-200, and ImageNet-1k, with several backbone architectures. The main results are reported in Tables 1--4, and ablations in Table 5 compare counterfactual LIMA with Grad-CAM and LIMA guidance.","tokens_in":14950,"tokens_out":4150,"duration_ms":40691,"significance":"If the claims are validated, the paper would make a useful contribution to explanation-guided training: it demonstrates a concrete mechanism for feeding faithful, subset-selected counterfactual attributions back into the training loop, and its ablation comparing factual versus counterfactual guidance (Table 5) is a sensible sanity check. The formulation in Eqs. (2)--(6) is explicit, and the use of background refilling rather than zero-masking is a reasonable design choice. However, the current evidence is not sufficient to support the central causal claim. The label-preservation assumption in Eq. (9) is load-bearing and unvalidated, key hyperparameters are omitted, and there are internal numerical inconsistencies, most notably between the abstract's claimed gains and the numbers in Table 3. The statistical reliability of the reported improvements is also unclear because no error bars or run counts are provided.","major_comments":[{"comment":"The label-preservation assumption is the core inductive bias but is never validated. Hard mining retains only samples with c_max > τ_aug, i.e., images for which the model flipped away from y_gt after masking. Assigning these augmented images the original label y_j presupposes that the remaining visible content is semantically sufficient for y_gt. The paper asserts that humans can still recognize the object, but provides no human study, no oracle probe, and no label-validity control. This is especially problematic because the main results are interpreted as a causal debiasing effect rather than label-noise regularization or background-artifact learning. Please add a direct validation of the labeling assumption (e.g., human annotation of augmented samples, an oracle-based check, or an ablation with flipped/donor labels).","section":"Section 4.2, Eq. (9), Algorithm 1"},{"comment":"The abstract claims 'gains of 5.70%/18.04% on ImageNet-1k/ImageNet-R with CLIP ViT/32b', but Table 3 reports 72.07% vs. 71.44% ID (+0.63%) and 31.89% vs. 31.63% OOD (+0.26%) for the same setting. This is a large discrepancy that cannot be attributed to rounding. Please reconcile the abstract with the actual table, and if the 5.70%/18.04% figures come from a different configuration or dataset, state that clearly. As written, this inconsistency undermines confidence in the reported results.","section":"Abstract vs. Table 3"},{"comment":"The text states that all scores are averaged over multiple runs 'to ensure statistical reliability', but no standard deviations, error bars, or number of runs are reported anywhere. Many gains are small (e.g., Table 3: +0.26% on ImageNet-R, +0.41% on ImageNet-S; Table 2 ViT: +0.52% ID). Without variance information it is impossible to determine whether these differences are significant. Please report means with standard deviations and, if feasible, significance tests or confidence intervals.","section":"Section 5.1, Tables 1--4"},{"comment":"The method depends on several hyperparameters that are never specified in the main text: λ1, λ2, the stopping threshold τ_cf, the hard-mining threshold τ_aug, the region budget k, the sub-region partition size m, and the donor-background patch selection procedure. The reader is referred to 'supplementary materials', but none are provided in the manuscript. These parameters directly control which samples are augmented and how aggressively, so the reported results cannot be reproduced or compared fairly without them. Please include the values and a sensitivity analysis.","section":"Section 4.1--4.2, Algorithm 1"},{"comment":"There are internal inconsistencies in the ablation results. The text says the full SS-CA framework 'achieves the peak performance of 91.18% (ID), 62.64% (ImageNet-R), and 59.11% (ImageNet-S)', but the 'w/ Additional Background' row in Table 5 reports 91.28, 62.68, 59.14, while the 'w/ Counterfactual LIMA' row reports 91.14, 62.59, 59.07. Section 5.2 also refers to 91.14/62.59/59.07 as the SS-CA result. Please correct the text or table so that the numbers are consistent.","section":"Table 5 and Section 5.3"},{"comment":"The experimental support for the central claim is thinner than the text suggests. On ImageNet-1k only one backbone (CLIP ViT-B/32, linear probing) is evaluated, while Tables 1--2 use three backbones. The comparison set is also narrow: only two recent debiasing baselines are included, and no standard augmentation methods (e.g., CutMix, RSC) or other explanation-guided augmentation baselines are compared, despite the related work surveying such methods. This limits the strength of the 'consistent improvement' claim and makes it hard to assess whether the gains are specific to counterfactual subset selection or shared by broader augmentation strategies. Please expand the baselines and, if feasible, add IN-1k results for at least one more backbone.","section":"Section 5.1, Tables 1--3"}],"minor_comments":[{"comment":"The title and introductory sentence contain a grammatical error: 'Did Models Sufficient Learn?' should be 'Did Models Learn Sufficiently?'.","section":"Global"},{"comment":"Figure 6 is referenced in Section 5.3, but the actual image content is not present in the manuscript; only the caption appears. Please include the figure or remove the reference.","section":"Figure 6"},{"comment":"The notation S_{·,j} for the cumulative subset is used without a formal definition. Please define it explicitly, e.g., S_{·,j} = {s_1, ..., s_j}.","section":"Notation, Eq. (1)"},{"comment":"Reference [37] for Tiny-ImageNet-200 uses an unusual URL format (cstr.cn). Please provide the standard citation or a stable DOI.","section":"References"},{"comment":"The contributions section claims 'Extensive experiments on three ImageNet variants', while the abstract says 'five ImageNet variants'. The experimental section actually evaluates five datasets (IN-100, TinyImageNet-200, IN-1k, ImageNet-R, ImageNet-S). Please make this consistent.","section":"Section 1"},{"comment":"The four-term utility function is clear, but the role of the insertion score as a 'fidelity regularizer' could be made more precise. In particular, the insertion term rewards f_y_gt(I(S)) and penalizes f_y_counter(I(S)), but the paper does not discuss what happens when the selected regions are not actually the most discriminative for y_gt. A brief discussion or a synthetic example would help.","section":"Section 4.1, Eq. (6)"}],"recommendation":"major_revision","confidential_remarks":"The discrepancy between the abstract's headline numbers (5.70%/18.04%) and the values in Table 3 (+0.63%/+0.26%) is particularly concerning and should be resolved before the paper is considered further. The self-citation of the authors' own LIMA work is heavy but not disqualifying; the more pressing issue is that the label-preservation assumption, which is central to the causal interpretation, is unvalidated and could be addressed with a relatively simple human-oracle or label-noise-control experiment. I recommend major revision with a request for these clarifications and validation experiments."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe core idea here is worth a look: feed counterfactual attributions back into training by masking the regions the model relies on and refilling with background. The execution has a problem though. The abstract claims 5.70% and 18.04% gains on ImageNet-1k and ImageNet-R with CLIP ViT-B/32, but Table 3 shows +0.63% and +0.26% for those same settings. That discrepancy is not explained. It makes the reporting look sloppy, and it is the first thing to fix.\n\nWhat is actually new is Counterfactual LIMA: a submodular search that finds a minimal region set to flip the model's prediction to a competing class, used during training. That is a real extension of LIMA, and the ablation gives it credit. Counterfactual Grad-CAM hurts performance while Counterfactual LIMA helps, and it beats factual LIMA too. So the selection mechanism is doing something, and the results are consistent across several backbones and datasets, though gains are modest.\n\nThe bigger concern is the label-preservation assumption. The method keeps the original ground-truth label on hard counterfactual samples, which are exactly the images where the model flipped its prediction after masking. If the mask covers the only discriminative evidence in that image, the augmented sample is effectively mislabeled. The paper asserts humans can still recognize the object, but there is no human study, oracle probe, or label-validity control. This is load-bearing: every result is interpreted through the causal mechanism of learning P(y_gt | I(V\\M_cf)). If the labels are wrong, the gains could just be generic label-noise regularization or learning background artifacts. The stress-test note is right about this.\n\nOther soft spots: no error bars anywhere despite the claim of averaging multiple runs, and gains as small as 0.26 points need variance to be meaningful. No random-mask control, so we can't attribute the effect specifically to counterfactual selection. Several hyperparameters (lambda_1, lambda_2, tau_cf, tau_aug, k, m, donor selection) are deferred to a supplementary that isn't included.\n\nWho gets value from this: people working on explanation-guided training and shortcut learning. The idea is promising and the method is clearly presented. But the evidence does not support the headline claims, and the label assumption needs validation. I'd send it to peer review with the expectation of heavy revision: fix the abstract, add variance and a random-mask baseline, disclose hyperparameters, and add a human/empirical check on label preservation. I wouldn't cite it in its current form.","headline":"A promising attribution-guided augmentation idea whose headline numbers don't match its own tables and whose label-preservation assumption is unvalidated; needs revision, not rejection.","tokens_in":15444,"tokens_out":3444,"would_cite":false,"duration_ms":29035,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Training with counterfactual masks lifts ImageNet-100 to 91.1% and ImageNet-R to 62.6%","keywords":["counterfactual augmentation","attribution-guided training","subset selection","shortcut learning","out-of-distribution generalization","model robustness","image classification"],"falsifier":"Measure human recognition accuracy on augmented samples (original image with the SS-CA mask replaced by background). If a sizable fraction are not recognized as the original class, the ground-truth labeling of these hard samples is unjustified, and the reported gains may partly reflect label noise rather than causal debiasing.","tokens_in":14444,"feed_emoji":"🎯","tokens_out":7201,"duration_ms":55629,"temperature":0.7,"pith_summary":"Many image classifiers rely on a single decisive visual cue, making them fail when that cue is missing or the distribution shifts. This paper proposes SS-CA, an attribution-guided training strategy that feeds a model's own counterfactual explanations back into the training loop. Using a subset-selection-based counterfactual attribution, it finds the minimal set of regions whose removal flips the prediction to a competing class, replaces those regions with background, and trains the model jointly on original and augmented samples. The authors report consistent gains in in-distribution accuracy and out-of-distribution robustness across multiple ImageNet variants and backbone architectures, with ImageNet-100 accuracy rising from 89.50% to 91.14% and ImageNet-R from 60.94% to 62.59% on a contrastively pretrained visual encoder.","feed_headline":"Counterfactual masks lift ImageNet-100 to 91.1%, ImageNet-R to 62.6%","feed_subtitle":"By replacing the exact regions a model relies on with background, SS-CA forces it to use the remaining evidence.","key_machinery":"The central object is a subset-selected counterfactual attribution computed by greedy submodular optimization. It seeks the minimal set of spatial regions whose removal pushes the model's confidence toward a counterfactual class while suppressing the ground-truth class, and whose insertion maintains fidelity to the original prediction. The augmentation then replaces these regions with background patches from a donor image, and a hard-mining filter retains only samples where the counterfactual flip confidence exceeds a threshold. Joint optimization on original and augmented batches with cross-entropy loss is what carries the training signal.","core_discovery":"The paper's central claim is that counterfactual attributions—explanations that identify which minimal region removals would change a model's prediction—can be used as training augmentations to correct incomplete causal learning. The proposed algorithm, SS-CA, extends a subset-selection attribution method into a counterfactual search that maximizes a utility function balancing the push toward a competing class and the fidelity to the ground-truth class. The resulting mask is applied to the image by replacing the selected regions with patches from a donor background image, and the augmented sample is given the original ground-truth label. Training on these hard samples together with the origi","pith_inferences":["The paper's labeling assumption—that an image with its most decision-relevant region replaced by background still belongs to the original class—could be tested directly by human recognition experiments; if a large fraction of augmented images are unrecognizable, the method may be inadvertently training on mislabeled examples, and the reported gains might partly reflect label noise rather than caus","Varying the source of donor background patches (e.g., same-class, other-class, or random natural images) could change the strength of the intervention and reveal with greater precision how the augmentation debiases the model.","The hard-mining threshold tau_aug controls a trade-off between the quantity and quality of augmented samples; a systematic sweep of this threshold could show whether the improvement is driven by the hardest samples or merely by the volume of extra data.","The counterfactual search cost scales with the number of region candidates per image; amortizing the search across epochs or using a faster approximation would make the method more practical for large-scale training."],"forward_implications":["Counterfactual attribution can be more than a diagnostic tool; feeding it back into training actively steers the model toward more complete feature reliance.","Improved performance on renditions and sketches suggests reduced dependence on texture and style shortcuts.","Robustness to Gaussian noise and other corruptions improves without needing corruption-specific augmentation at test time.","The method requires no extra human annotations—only the model's own logits and background image patches.","The gains appear across convolutional, transformer, and contrastively pretrained encoders, indicating architecture generality."],"fun_headline_variants":["Train models to see more by hiding what they over-rely on","SS-CA: counterfactual masks that boost OOD robustness","Hide the obvious, force the model to learn harder cues","Counterfactual augmentation lifts ImageNet-R by 18%","Attribution-guided training: mask the cue, keep the class"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The method assumes that an image with its most decision-relevant region erased and replaced by arbitrary background is still recognizably the original class, and this label-preservation assumption is not human-validated.","fun_headline_variants_meta":{"raw":{"variants":["Train models to see more by hiding what they over-rely on","SS-CA: counterfactual masks that boost OOD robustness","Hide the obvious, force the model to learn harder cues","Counterfactual augmentation lifts ImageNet-R by 18%","Attribution-guided training: mask the cue, keep the class"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001114,"raw_usage":{"total_tokens":4515,"prompt_tokens":825,"completion_tokens":3690,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":569,"completion_tokens_details":{"reasoning_tokens":3603}},"tokens_in":569,"tokens_out":3690,"duration_ms":24143,"temperature":1.0,"reasoning_tokens":3603,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T22:04:24.473843+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure human recognition accuracy on augmented samples (original image with the SS-CA mask replaced by background). If a sizable fraction are not recognized as the original class, the ground-truth labeling of these hard samples is unjustified, and the reported gains may partly reflect label noise rather than causal debiasing.","supporting_citations":[],"review_version":1}