{"id":"52f6a3fe-0952-41dd-9a19-4d57418bd3a6","arxiv_id":"2501.09086","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"SIP-AT preserves salient image regions during adversarial training, improving clean accuracy and low-epsilon robustness, but high-epsilon robustness is not maintained on all datasets.","lead":"A new adversarial training scheme, SIP-AT, masks out salient image regions so that perturbations are added only to the non-salient background, and the authors report higher clean accuracy and stronger low-epsilon robustness than four established baselines. The paper also includes a 330-person study showing humans rarely notice small perturbations, which motivates the focus on low-epsilon robustness.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"At ε=8/255, SIP-AT lags standard adversarial training on CIFAR-100 and CUB, contradicting the abstract's claim of maintaining robustness without sacrificing overall robustness.","rationale":"The reader's weakest_assumption focuses on the fidelity of the pixel-level salience mask, which is a plausible mechanism for the observed high-epsilon degradation. My concern is more direct: regardless of mechanism, the central claim of maintaining robustness across multiple epsilon levels is contradicted by the paper's own Tables 2 and 3 at ε=8/255. This is a load-bearing empirical inconsistency, not just an unvalidated assumption. The reader already flagged this degradation in the rationale and assigned CONDITIONAL, so my analysis reinforces the verdict rather than moving it. I credit the paper for consistent clean-accuracy gains and strong low-epsilon results, but the abstract's unqualified language is too strong. The proposed test would settle whether the high-epsilon drop is statistically meaningful across the five seeds; if it is, the paper's contribution should be presented as a low-epsilon robustness improvement with a known high-epsilon cost.","tokens_in":12645,"tokens_out":5404,"duration_ms":55418,"concrete_test":"Run a paired significance test (e.g., Wilcoxon signed-rank) over the five reported seeds comparing SIP-S/SIP-H against Madry at ε=8/255 for each dataset-architecture pair. If CIFAR-100 (ResNet18 and WRN-34) and CUB (DenseNet121) show a significant decrease (p < 0.05) relative to Madry, then the abstract's 'without sacrificing overall robustness' claim is falsified at high epsilon. Also compute the mean robust accuracy averaged over ε ∈ {1,2,4,8}; if SIP-AT's average falls below Madry's on CIFAR-100, the headline should be revised to claim only low-epsilon robustness.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's central claim is that SIP-AT 'maintain[s] a high degree of robustness against attacks at multiple epsilon levels' and relieves the robustness-accuracy trade-off 'without sacrificing overall robustness.' The paper's own results contradict this at the largest tested epsilon. In Table 2 (CIFAR-100), SIP-S at ε=8/255 reaches 11.79% (ResNet18) and 13.00% (WRN-34), versus Madry's 14.76% and 17.23%. In Table 3 (CUB), SIP-S at ε=8/255 reaches 1.64% (ResNet50) and 1.69% (DenseNet121), versus Madry's 4.59% and 5.68%. This is a substantial robustness sacrifice on two of three datasets. The theory in §3.1 actually anticipates this: by allowing models to learn non-robust salient features (Eq. 5), those features become an attack surface during evaluation, where §4.4 explicitly allows the adversary to perturb all pixels. The paper even notes in §5.1 that SIP-AT 'performance degrades more rapidly at higher epsilons.' Thus the unqualified claim that SIP-AT maintains robustness 'at multiple epsilon levels' and relieves the accuracy-robustness trade-off is not supported by the reported evidence; the trade-off has merely been shifted toward low-epsilon robustness.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Salient Information Preserving Adversarial Training (SIP-AT), a method that restricts adversarial perturbations during training to non-salient image regions, with the goal of retaining useful non-robust features and thereby improving clean accuracy without sacrificing robustness. The authors formalize a feature-level notion of salience following Ilyas et al., introduce a practical pixel-level mask based on top-k gradient magnitudes, and evaluate on CIFAR-10, CIFAR-100, and CUB-200-2011 with ResNet and WideResNet architectures. They report that SIP-AT improves clean accuracy relative to standard adversarial training and other baselines, while maintaining robustness at low perturbation budgets, and complement this with a human survey on detection of adversarial perturbations.","tokens_in":13004,"tokens_out":4520,"duration_ms":42826,"significance":"If the central claims held, SIP-AT would be a valuable, low-cost addition to adversarial training: it requires no additional training-time computation beyond a precomputed salience mask, and the reported clean-accuracy gains on multiple datasets are nontrivial. The paper is also careful in its evaluation: five repeats per condition, AutoAttack for robustness, and evaluation at four epsilon levels are all strengths. However, the significance is undercut by the fact that the high-epsilon robustness claims are not supported by the data on two of three datasets, and by the disconnect between the feature-level theory and the pixel-level implementation. The method still appears promising for low-epsilon robustness, but the paper needs substantial qualification and revision before it can be accepted.","major_comments":[{"comment":"The abstract claims SIP-AT maintains robustness 'at multiple epsilon levels' and relieves the robustness-accuracy trade-off 'without sacrificing overall robustness,' and §5.1 claims 'comparable or improved levels of robust accuracy at higher epsilons.' These claims are contradicted by Tables 2 and 3 at ε=8/255. On CIFAR-100 (ResNet18 and WRN-34), SIP-S achieves 11.79% and 13.00% versus Madry's 14.76% and 17.23%; on CUB (ResNet50 and DenseNet121), SIP-S achieves 1.64% and 1.69% versus Madry's 4.59% and 5.68%. This is a substantial robustness drop on two of three datasets. The abstract and discussion must be revised to state that the robustness benefit is concentrated at low epsilon values, and the 'without sacrificing overall robustness' phrasing should be removed or explicitly qualified.","section":"Abstract and §5.1"},{"comment":"The theoretical framework defines feature-level salience ST(x, f) and the restricted perturbation set Δ1 in Eq. (4), but the practical method in Eqs. (7)–(9) uses a pixel-level binary mask M(x) derived from the top-k gradient magnitude and restricts perturbations by zeroing masked elements. There is no formal or empirical bridge showing that zeroing perturbation on the pixel-level mask preserves the feature-level salience condition ST(x, y, f) = 1 → ST(x+δ, y, f) = 1. Consequently, the theoretical claim that models trained with the restricted set learn features in F_sal_robust \\ F_robust does not apply to the actual training procedure. The paper should either provide a proof that the pixel mask approximates the feature-level condition, or reframe the theory as a motivation and add experiments that directly test whether masked perturbations preserve the salience of the features identified by T.","section":"§3.1 and §3.2"},{"comment":"The discussion in §5.1 is internally inconsistent: it first states that SIP-AT models 'maintain comparable or improved levels of robust accuracy at higher epsilons' and then immediately notes that 'performance degrades more rapidly at higher epsilons.' The data show that, at ε=8/255, SIP-S is worse than Madry on CIFAR-100 and CUB, and even SIP-H is only marginally better on CUB (ResNet50: 5.04% vs 4.59%; DenseNet121: 6.70% vs 5.68%). The narrative that SIP-AT relieves the trade-off is not accurate across the tested epsilon range; the trade-off has been shifted toward low-epsilon robustness. The authors should report and interpret this trade-off honestly, and should avoid claiming that robustness is maintained 'at multiple epsilon levels' without specifying which epsilons and which datasets.","section":"§5.1, Tables 2 and 3"}],"minor_comments":[{"comment":"The notation ∇pT pxqq is ambiguous; it should be ∇ₓT(x) to indicate the gradient with respect to the input, and 'Top-k' should be formally defined (e.g., the set of coordinates with the k largest absolute gradient values, with ties broken deterministically).","section":"§3.2, Eq. (7)"},{"comment":"The human survey is interesting but its connection to the central claims of SIP-AT is not stated explicitly. The paper should clarify whether the survey is intended to motivate the focus on low-epsilon robustness, or to validate the threat model, and how it informs the method's design.","section":"§5.2"},{"comment":"The y-axis label 'Images Labeled Perturbed (%)' is unclear; it should be 'Percentage of images correctly labeled as perturbed' or similar, and the caption should state whether the plot shows mean detection rate across participants and images.","section":"Figure 5"},{"comment":"The paper does not report the computational overhead of SIP-AT relative to standard adversarial training; a brief statement that the only added cost is the precomputed salience mask and an element-wise multiplication would help readers assess practicality.","section":"§4.3"}],"recommendation":"major_revision","confidential_remarks":"The paper has solid experimental methodology (five repeats, AutoAttack, multiple epsilons) and a clear practical contribution for low-epsilon robustness, but the abstract and conclusions overstate the high-epsilon robustness results. The gap between the feature-level theory and the pixel-level implementation also needs to be addressed or explicitly labeled as heuristic. I believe these issues are fixable within the scope of a major revision; the core empirical observation that salience-preserving adversarial training boosts clean accuracy at the cost of high-epsilon robustness is valid and worth publishing once the claims are properly qualified."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: SIP-AT is a genuinely simple variant—zero the adversarial perturbation budget on pixels a salience map marks as important—and the experiments show it reliably lifts clean accuracy and low-epsilon robustness across CIFAR-10, CIFAR-100, and CUB, with two architectures each and five repeats. That is the part to keep. The central new result is the negative-space flip relative to PART: PART reweights perturbations outside important regions; SIP-AT forbids perturbations inside them, and the evaluation is against full-image AutoAttack, so the robust numbers are not circular. I agree with the reader's conditional verdict and the stress-test. The abstract's \"without sacrificing overall robustness\" and \"multiple epsilon levels\" are not supported. At ε=8/255, SIP-S trails Madry on CIFAR-100 (11.79 vs 14.76 for ResNet-18; 13.00 vs 17.23 for WRN-34) and on CUB (1.64 vs 4.59 for ResNet-50; 1.69 vs 5.68 for DenseNet-121). The paper's own Section 5.1 admits faster degradation at higher epsilons. So the honest claim is: SIP-AT buys clean accuracy and low-epsilon robustness partly by ceding high-epsilon robustness. The theory section is also not fully connected: the feature-level salience definition in Eqs. (3)-(5) does not justify the pixel-level top-k gradient mask in Eqs. (7)-(8), and the paper does not validate that the mask is actually preserving the non-robust useful features it claims. This is a real soft spot, but it is a framing gap rather than a fatal flaw; the empirical result stands on its own. Minor issues: the top-k threshold is a free parameter, no code is released, and the human survey is suggestive but not a formal perceptual study. None of that kills the paper. The benchmarks are solid: five repeats, AutoAttack evaluation, multiple datasets and architectures, and evaluation at several epsilons. I would send this to review. The authors need to reword the abstract and conclusions, acknowledge the high-epsilon trade-off directly, and ideally release code; the core method and evidence deserve referee time. I would cite it as related work in robustness-accuracy research and would bring it to reading group.","headline":"SIP-AT is a simple, well-tested trick that shifts the robustness-accuracy trade-off toward low-epsilon robustness, but the abstract oversells high-epsilon performance; worth refereeing with revisions.","tokens_in":13420,"tokens_out":1676,"would_cite":true,"duration_ms":17000,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Salient Information Preserving Adversarial Training lifts clean accuracy while keeping robustness.","keywords":["adversarial training","robustness-accuracy trade-off","salience maps","non-robust features","clean accuracy","AutoAttack","fine-grained classification","human attack detection"],"falsifier":"Train SIP-AT on CIFAR-10 with a mask that protects a randomly chosen set of pixels of the same size as the gradient mask; if clean accuracy rises as much as with the true salience mask, the claim that semantically meaningful features are being preserved is false, while if random masks do not reproduce the gain, the salience link is load-bearing.","tokens_in":12456,"feed_emoji":"🛡️","tokens_out":4827,"duration_ms":43302,"temperature":0.7,"pith_summary":"This paper claims that the robustness-accuracy trade-off of adversarial training can be reduced by refusing to perturb the image regions an annotator deems salient. Its method, SIP-AT, generates adversarial examples as usual but multiplies the perturbation by the complement of a salience mask, so meaningful pixels stay unchanged during training. The models then keep learning predictive non-robust features that ordinary adversarial training erases, gaining clean accuracy while retaining robustness against AutoAttack across several epsilon budgets. The claim is tested on CIFAR-10, CIFAR-100, and CUB-200-2011 with both human and automatic salience maps.","feed_headline":"Freezing salient pixels lifts clean accuracy without losing robustness","feed_subtitle":"SIP-AT spares meaningful image regions during attack generation, lifting clean accuracy while matching robust accuracy.","key_machinery":"The load-bearing object is the salience-preserving perturbation set $\\Delta'(x)$ and its practical proxy, a binary mask $M(x)$ built as the minimal top-$k$ set of pixels whose gradient magnitudes sum to at least half the total gradient magnitude. The adversarial perturbation $\\delta$ computed by PGD is masked to $\\delta' = \\delta \\cdot (1-M(x))$, so salient pixels are frozen during training. The theoretical framing imports the useful/robust feature taxonomy: a feature is useful if correlated with the label, robust if it stays useful under perturbations in $\\Delta$, and salient-or-robust if it stays useful under the restricted set $\\Delta'$. The key identity is $F_{\\mathrm{robust}}\\subseteq F_{\\mathrm{sal\\text{-}robust}}\\subseteq F_{\\mathrm{useful}}$, which is what enlarges the set of learnable features.","core_discovery":"On the paper's own terms, SIP-AT establishes that imposing the constraint $\\Delta'(x)=\\{\\delta \\in \\Delta(x) : M(x)_{c,h,w}=1 \\Rightarrow \\delta_{c,h,w}=0\\}$ on adversarial training lets a model learn from the enlarged feature set $F_{\\mathrm{sal\\text{-}robust}}\\setminus F_{\\mathrm{robust}}$: useful non-robust features that survive salient-region protection. In practice every PGD step perturbs only non-salient pixels during training, while evaluation still attacks all pixels. Across architectures and datasets, SIP-AT reports higher clean accuracy than standard adversarial training, TRADES, FAT, and PART baselines, and comparable or better low-epsilon robust accuracy; the clean gain is largest on CIFAR-10, for example 82.34 percent clean accuracy with a ResNet18 versus 74.39 for standard adversarial training, and fine-grained CUB-200-2011 benefits most when synthetic salience is used.","pith_inferences":["Editorial inference: a direct test of whether salience itself matters is to replace the salience mask with a random mask of equal coverage; if the clean-accuracy gain persists, the benefit would come from a reduced perturbation budget rather than from protecting semantically meaningful pixels.","Editorial inference: because SIP-AT enlarges the learnable feature set rather than reweighting the loss, it may combine additively with loss-based trade-off methods such as TRADES-style regularization, though this combination is not tested in the paper.","Editorial inference: the method transfers naturally to domains where local salience can be annotated, such as medical imaging, where protecting diagnostically meaningful regions during adversarial training could preserve clinically useful non-robust features.","Editorial inference: the human detection results suggest a testable design principle for deployment: optimize primarily for robustness at $\\epsilon\\le 2/255$ because higher-epsilon attacks are perceptible and therefore less dangerous in practice."],"forward_implications":["If SIP-AT holds, adversarially trained image classifiers can be noticeably more accurate on clean inputs without giving up robustness to small perturbations, which are exactly the attacks humans most often fail to notice.","The method reduces to a single masking step inside standard PGD adversarial training, so it can be added to existing pipelines with negligible extra cost once salience maps are available.","Because evaluation attacks all pixels, the reported robustness is not an artifact of test-time masking; the defense is in the training procedure itself.","Fine-grained classification responds differently to human versus synthetic masks: human masks protect object-only regions and yield more robust models, while synthetic masks that include correlated background yield higher clean accuracy.","The human survey implies that low-epsilon robustness matters most in human-in-the-loop deployment, since perturbations at $\\epsilon=4/255$ and $8/255$ are detected at semi-consistent or high rates while $\\epsilon=1/255$ and $2/255$ perturbations are almost always missed."],"supporting_citations":[{"why":"Supplies the useful/robust feature taxonomy and the theoretical setting from which SIP-AT's restricted perturbation set and enlarged feature set are derived.","marker":"[15]"},{"why":"Defines the minimax adversarial training formulation and PGD inner maximization that SIP-AT modifies with the masked perturbation set.","marker":"[18]"},{"why":"TRADES is a core accuracy-robustness benchmark method that SIP-AT is compared against on all datasets.","marker":"[37]"},{"why":"FAT is a benchmark method that SIP-AT must beat, and its minimally adversarial training is the closest alternative to SIP-AT's restricted perturbations.","marker":"[38]"},{"why":"PART is the closest prior method using pixel-wise importance maps for adversarial training, and the paper differentiates SIP-AT from it explicitly.","marker":"[39]"},{"why":"AutoAttack is the parameter-free evaluation ensemble used to report robust accuracy at all epsilon levels.","marker":"[6]"},{"why":"Supports the choice to generate salience masks from an adversarially trained model, since robust models produce more interpretable salience maps.","marker":"[31]"},{"why":"Also supports the connection between adversarial robustness and salience-map interpretability that motivates the mask-generation strategy.","marker":"[7]"}],"fun_headline_variants":["Freeze salient pixels, gain clean accuracy","Salient pixel freezing lifts accuracy, keeps robustness","SIP-AT: protect salient pixels, improve clean accuracy","Salience-aware adversarial training lifts clean accuracy","Salience-guided training preserves robustness, improves clean accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that a binary top-k gradient mask captures the same salient features a human annotator would protect, and that freezing exactly those pixels preserves the predictive non-robust features that drive the clean-accuracy gain.","fun_headline_variants_meta":{"raw":{"variants":["Freeze salient pixels, gain clean accuracy","Salient pixel freezing lifts accuracy, keeps robustness","SIP-AT: protect salient pixels, improve clean accuracy","Salience-aware adversarial training lifts clean accuracy","Salience-guided training preserves robustness, improves clean accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001393,"raw_usage":{"total_tokens":5640,"prompt_tokens":956,"completion_tokens":4684,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":572,"completion_tokens_details":{"reasoning_tokens":4611}},"tokens_in":572,"tokens_out":4684,"duration_ms":33127,"temperature":1.0,"reasoning_tokens":4611,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:11:25.752140+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train SIP-AT on CIFAR-10 with a mask that protects a randomly chosen set of pixels of the same size as the gradient mask; if clean accuracy rises as much as with the true salience mask, the claim that semantically meaningful features are being preserved is false, while if random masks do not reproduce the gain, the salience link is load-bearing.","supporting_citations":[{"cited_title":"Adversar- ial examples are not bugs, they are features","cited_arxiv_id":null,"evidence_quote":"Supplies the useful/robust feature taxonomy and the theoretical setting from which SIP-AT's restricted perturbation set and enlarged feature set are derived."},{"cited_title":"Towards deep learn- ing models resistant to adversarial attacks","cited_arxiv_id":null,"evidence_quote":"Defines the minimax adversarial training formulation and PGD inner maximization that SIP-AT modifies with the masked perturbation set."},{"cited_title":"Attacks which do not kill training make adversarial learning stronger","cited_arxiv_id":null,"evidence_quote":"FAT is a benchmark method that SIP-AT must beat, and its minimally adversarial training is the closest alternative to SIP-AT's restricted perturbations."},{"cited_title":"Improving accuracy-robustness trade- off via pixel reweighted adversarial training","cited_arxiv_id":null,"evidence_quote":"PART is the closest prior method using pixel-wise importance maps for adversarial training, and the paper differentiates SIP-AT from it explicitly."},{"cited_title":"Reliable evalua- tion of adversarial robustness with an ensemble of diverse parameter-free attacks","cited_arxiv_id":null,"evidence_quote":"AutoAttack is the parameter-free evaluation ensemble used to report robust accuracy at all epsilon levels."},{"cited_title":"Robustness may 9 be at odds with accuracy","cited_arxiv_id":null,"evidence_quote":"Supports the choice to generate salience masks from an adversarially trained model, since robust models produce more interpretable salience maps."},{"cited_title":"On the connection between adversarial ro- bustness and saliency map interpretability","cited_arxiv_id":null,"evidence_quote":"Also supports the connection between adversarial robustness and salience-map interpretability that motivates the mask-generation strategy."}],"review_version":1}