{"id":"945a34b9-98a6-4bf0-8a58-4ee7c106fa19","arxiv_id":"2506.05382","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"ECLIPSE, a black-box attack using Gaussian-blurred gradient estimates and surrogate-based masking, achieves a better trade-off among compression robustness, spectral detection evasion, and human invisibility than SimBA, SimBA-DCT, and Square Attack.","lead":"This paper tests whether black-box adversarial image attacks stay effective after JPEG compression, automatic detection, and human inspection, and proposes a new attack called ECLIPSE that uses blurred gradients and a local surrogate model. It introduces a three-part evaluation framework for real-world attack feasibility.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'perfect stealthiness' claim rests on internally inconsistent detection metrics in Table 3; the reported accuracy 0.87 with precision 0.03 and recall 0.01 cannot hold on a balanced 150/150 set.","rationale":"I focused on the internal inconsistency of the P2 detection metrics rather than the reader's stated weakest assumption (surrogate alignment) or the unfair comparison, because the P2 pillar is the one most directly tied to the strongest claim and is the least supported by the reported data. The surrogate alignment concern is plausible but partially mitigated by the ablation: removing the local surrogate changes AUC from 0.50 to 0.71, showing the surrogate contributes to P2, but even the no-surrogate variant still outperforms Square Attack (0.71 vs 0.96), so the trade-off claim does not collapse solely on that axis. The unfair comparison is also real, but the ablation again shows ECLIPSE without surrogate retains most of its P1 and P3 advantages. The detection table, however, is arithmetically impossible on a balanced set, which suggests a methodological error in how the SVM evaluation was performed or reported. If the AUC of 0.50 is an artifact of a degenerate classifier, the phrase 'perfect stealthiness' is unjustified, and the central claim loses one of its three co-equal pillars. The reader's rationale mentions a 'broken detector' as one of several issues, so we partially agree, but I elevate it to the primary load-bearing concern. A corrected evaluation could still yield a conditional accept, so the reader's CONDITIONAL verdict remains appropriate; the fix is to recompute and reinterpret the P2 results rather than to discard the paper.","tokens_in":11497,"tokens_out":9202,"duration_ms":88747,"concrete_test":"Recompute Table 3 from raw ROC scores with class-balanced cross-validation and report the full confusion matrix at a fixed decision threshold (e.g., Youden's J). If the confusion matrix for 'Normal vs ECLIPSE' does not match the reported accuracy/precision/recall, correct the table and re-evaluate whether AUC remains 0.50. Also report detection rate at a fixed false-positive rate (FPR = 1%) to separate 'detector is blind' from 'ECLIPSE is genuinely undetectable'.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The strongest conclusion (Section 6) claims 'perfect stealthiness (AUC 0.5)' for ECLIPSE against spectral detection. This rests entirely on Table 3 (Section 5, P2). For the 'Normal vs ECLIPSE' comparison, the table reports Accuracy 0.87 (±0.01), Precision 0.03 (±0.20), Recall 0.01 (±0.08), F1 0.02 (±0.11), and ROC AUC 0.50 (±0.03), with 150 adversarial and 150 benign samples. These numbers are mutually incompatible on a balanced set: precision 0.03 with recall 0.01 implies roughly 1.5 true positives and 48.5 false positives, yielding accuracy ≈ 0.34, not 0.87. The large standard deviations (precision 0.03 ± 0.20) indicate degenerate cross-validation folds in which the SVM almost never predicts the adversarial class. An AUC of 0.50 may therefore simply reflect a detector that has collapsed to predicting 'normal' for essentially every input, not a property of ECLIPSE. The paper's headline comparison (ECLIPSE AUC 0.5 vs Square Attack AUC 0.96) is meaningful only if the detector is functional; the reported metrics suggest it is not. This is load-bearing because the central claim bundles P2 as a co-equal pillar, and the 'perfect stealthiness' phrasing overstates what a broken detector can show.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ECLIPSE, a confidence-based black-box evasion attack that combines Hill Climbing with two novel components: Gaussian blurring of estimated gradients and a sampling mask derived from GradCAM heatmaps of a local surrogate model. The authors introduce three effectiveness properties for real-world attacks—robustness to JPEG compression (P1), stealthiness to automatic spectral detection (P2), and stealthiness to human inspection (P3)—and evaluate ECLIPSE against SimBA, SimBA-DCT, and Square Attack L∞ on the Animals-10 dataset using a ResNet152V2 oracle. The central claim is that ECLIPSE achieves the best published trade-off among the three properties, including 'perfect stealthiness' against spectral detection (AUC 0.5) and strong compression robustness.","tokens_in":11770,"tokens_out":4873,"duration_ms":43409,"significance":"If the results were fully supported, the proposed three-property evaluation framework would be a useful contribution to the adversarial machine learning literature, and ECLIPSE would be a noteworthy attack design. The paper also includes a human perception survey and an ablation study, which are valuable empirical components. However, the current evidence does not establish the central trade-off claim: the detection metrics in Table 3 are internally inconsistent, the ablation text contradicts the data in Tables 4 and 5, and the comparison to baselines is inequitable because ECLIPSE receives external surrogate knowledge. With these issues fixed, the framework and the attack could be of interest to the community, but as presented the main conclusions are not reliable.","major_comments":[{"comment":"The reported detection metrics for 'Normal vs ECLIPSE' are internally inconsistent on a balanced set of 150 normal and 150 adversarial examples. Precision 0.03 with recall 0.01 implies TP ≈ 1.5, FP ≈ 48.5, and TN ≈ 101.5, yielding an accuracy of approximately 0.34, not the reported 0.87. The large standard deviation around precision (±0.20) indicates degenerate cross-validation folds in which the SVM essentially never predicts the adversarial class. Consequently, the AUC of 0.50 cannot be interpreted as evidence of 'perfect stealthiness'; it may simply reflect a detector that has collapsed to predicting 'normal' for essentially every input. This directly undermines the P2 comparison in Figure 9 and the corresponding conclusion in Section 6.","section":"§5, Table 3 and Table 4"},{"comment":"The ablation text directly contradicts the data it cites. The paper states that removing Gaussian blur 'improves robustness to processing,' but Table 5 reports, for No Gaussian blur versus ECLIPSE, median loss 0.36 versus 0.15, low-loss 24.00% versus 89.33%, and surviving 0.67% versus 18.67%—every compression metric worsens when blur is removed. Similarly, the text says that removing the local surrogate 'benefits stealthiness, with a drop in ROC AUC from 0.71 to 0.5,' but Table 4 shows the opposite: ECLIPSE has AUC 0.50 and No Local Surrogate has AUC 0.71, meaning removal makes the attack more detectable. These contradictions must be resolved before the ablation can support the claimed contribution of either component.","section":"§5, 'ECLIPSE Ablation Study' and Table 5"},{"comment":"The experimental comparison is inequitable because ECLIPSE is given access to a local surrogate model (DenseNet201 trained on a 12-class ImageNet subset) while SimBA, SimBA-DCT, and Square Attack are strictly query-only. The GradCAM-derived mask is external knowledge about the oracle that the baselines do not receive, so the reported advantages in query counts, compression robustness, and detectability may reflect this additional information rather than the attack mechanism itself. The paper should either add an ECLIPSE variant without the surrogate to the main comparison, or explicitly frame ECLIPSE as a transfer-assisted attack and benchmark it against transfer-based baselines under the same assumptions.","section":"§4 (Local Surrogate) and §5 (Table 2)"},{"comment":"The phrase 'perfect stealthiness (AUC 0.5)' is an overstatement. A ROC AUC of 0.5 means no discriminability, which is not the same as evading a working detector; given the inconsistent metrics in Table 3, the paper has not excluded the possibility that the detector simply failed to train for this comparison. The claim should be softened to something like 'not detected by the tested spectral detector under the reported cross-validation conditions,' and the detector should be validated with positive and negative controls before any stealthiness conclusion is drawn.","section":"§6 Conclusions"}],"minor_comments":[{"comment":"Table 1 reports median loss, low-loss percentage, and surviving percentage without confidence intervals or the number of images per attack; given that the P2 experiments use 150 samples, it is unclear whether differences such as 18.67% versus 15.33% are statistically significant. Please report per-sample distributions and significance tests.","section":"§5, P1"},{"comment":"The description of the detection classifier omits the DCT feature extraction procedure, the dimensionality of the features, the SVM kernel and hyperparameters, and the exact cross-validation scheme (number of folds and stratification). Without these details the experiments cannot be replicated.","section":"§5, P2"},{"comment":"The notation f(C_{t-1}+1(i,j,c)) is ambiguous; please define the one-hot unit perturbation at coordinate (i,j,c) explicitly.","section":"Algorithm 1, line 9"},{"comment":"The caption contains a typo: 'attach' should be 'attack'.","section":"Figure 7"},{"comment":"The label 'P3 (% of 0-1 ans.)' is unclear; expand it to specify that it is the percentage of survey responses with scores 0 or 1, and consider adding error bars or confidence intervals.","section":"Figure 9"},{"comment":"Some references contain duplicated author names, for example reference [23] repeats 'Binghui Wang' and 'Michael I. Jordan' multiple times; these appear to be formatting artifacts and should be cleaned up.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper has a useful framework and an interesting attack design, but the central claims rest on detection metrics that are internally inconsistent and an ablation narrative that contradicts the reported tables. The authors should be asked to provide raw confusion matrices and detector training details, and to re-run or re-report the P2 experiments. The comparison fairness issue with the surrogate model also needs to be addressed, either by adding an ablation without the surrogate or by repositioning the paper. These are fixable within the manuscript's scope, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe thing to know: this paper has a genuinely new attack design and a sensible evaluation framework, but the experiments as reported don't support the headline claims. ECLIPSE combines Gaussian blurring of estimated gradients with a sampling mask derived from a local surrogate's GradCAM heatmaps, inside a hill-climbing loop. That combination is new to me, and the three-property formalization (compression robustness, automatic-detection evasion, human invisibility) is a useful way to organize the space. The human survey with 127 participants is real work, and the ablation study is the right instinct.\n\nThe soft spots are serious but fixable. First, the comparison is apples-to-oranges: ECLIPSE gets a surrogate model (DenseNet201) to guide its sampling, while SimBA, SimBA-DCT, and Square Attack are treated as query-only. That gives ECLIPSE information the baselines lack. Either give all attacks a surrogate or make ECLIPSE query-only. Second, the ablation narrative contradicts its own Table 5. The text says removing Gaussian blur improves robustness to processing and cuts queries; the table shows the opposite — blur removal worsens median loss (0.36 vs 0.15), low-loss rate (24% vs 89.33%), survival (0.67% vs 18.67%), and median queries (12870 vs 8125). That is not a minor typo; it inverts the conclusion for one of the two claimed contributions.\n\nThe bigger problem is the 'perfect stealthiness' claim. Table 3 reports, for Normal vs ECLIPSE, accuracy 0.87, precision 0.03, recall 0.01, AUC 0.50, on a balanced 150/150 set. Those numbers are mutually impossible: precision 0.03 and recall 0.01 imply about 1.5 true positives and 48.5 false positives, which gives accuracy around 0.34, not 0.87. The AUC of 0.50 with those precision/recall values means the SVM collapsed to predicting 'normal' for nearly everything. So 'perfect stealthiness' is not demonstrated; it's an artifact of a broken detector. Re-baseline the detector or soften the claim.\n\nIf these are fixed — equalize the threat model, correct the ablation discussion, and re-evaluate or re-report the detection results — the paper could be a reasonable contribution to the adversarial-ML literature. As it stands, I wouldn't rely on the reported numbers.\n\nRecommendation: send it to peer review, but expect heavy revision. The core idea is worth referee time, and the framework is useful for the community. I'd bring it to a reading group, mainly to discuss the evaluation pitfalls.","headline":"Novel attack design and a useful three-property framework, but the evaluation's unequal baselines, a self-contradictory ablation table, and a collapsed detector undermine the headline claims.","tokens_in":12323,"tokens_out":3217,"would_cite":false,"duration_ms":28083,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a new black-box attack, ECLIPSE, balances JPEG-compression robustness, automatic-detection stealth, and human invisibility at once, a trade-off no prior evaluated attack achieves.","keywords":["Adversarial examples","Black-box attacks","Computer vision","Evasion","Stealthiness","JPEG compression","Gaussian blur","Surrogate model"],"falsifier":"Take a black-box model with a markedly different inductive bias, such as a vision transformer or a CNN trained on a different dataset, and keep the DenseNet201 surrogate fixed; if ECLIPSE's JPEG-survival rate falls toward SimBA's level and the spectral detector's AUC rises well above 0.5, the transferability of the surrogate mask is refuted and the generality of the trade-off claim with it.","tokens_in":11274,"feed_emoji":"🕶️","tokens_out":9036,"duration_ms":81756,"temperature":0.7,"pith_summary":"The paper sets out to measure whether adversarial attacks on image classifiers are feasible outside the lab, where images get compressed, services watch for attack patterns, and humans might look at the picture. It defines three effectiveness properties — robustness to JPEG compression, stealthiness to spectral-based automatic detection, and stealthiness to human inspection — and evaluates three popular black-box confidence-based attacks (SimBA, SimBA-DCT, Square Attack) against them, finding that each attack satisfies some properties but fails others. The authors then propose ECLIPSE, a targeted attack that combines hill-climbing gradient estimation with Gaussian-blurred gradients and a gradient-sampling mask drawn from GradCAM heatmaps of a local surrogate model. In their experiments on the Animals-10 dataset (cats misclassified as dogs), ECLIPSE is the only method that combines a low-loss rate of 89.33% under JPEG compression, a spectral detector that performs at chance (AUC 0.50), and a 62.55% share of human ratings of 'not visible' or 'slightly visible'. The claim is that this closes the long-standing trade-off and shows that real-world black-box threats need to be taken at least as seriously as white-box ones.","feed_headline":"Black-box attack survives compression and evades detection at once","feed_subtitle":"ECLIPSE keeps 89% of attacks effective after JPEG while spectral detectors score no better than chance.","key_machinery":"The load-bearing mechanism is the ECLIPSE optimization loop, whose two novel steps interact. First, Gaussian blurring of the estimated gradient tensor ($k{=}3$, standard deviation $\\sigma$) spreads each sampled finite-difference gradient to neighbouring coordinates, producing smoother, coarser perturbations; the ablation study attributes JPEG-compression robustness and automatic-detection stealth mainly to this step. Second, a time-varying sampling mask derived from a GradCAM heatmap on a local surrogate DenseNet201 restricts gradient sampling to the most salient regions (mask threshold rising from 0.0 to 0.5), which mostly reduces query cost rather than visual stealth. The outer loop is standard hill climbing with a decaying step size, and the three effectiveness properties P1, P2, P3 form the evaluation scaffold that lets the paper compare ECLIPSE against SimBA, SimBA-DCT, and Square Attack $L_\\infty$.","core_discovery":"ECLIPSE (Evasion of Classifiers with Local Increase in Pixel Sparse Environment) is a targeted, confidence-based black-box evasion attack built on hill climbing. The attacker queries a remote oracle for the confidence score of the desired target class, samples a small batch of coordinates inside a mask, estimates their gradients by finite differences, blurs that gradient tensor with a Gaussian kernel, and updates the image within an $L_\\infty$ budget of 0.1 (0.05 for the human survey) only if the target confidence increases. The step size decays exponentially and the algorithm stops early once the target confidence exceeds 0.5. The mask is generated by GradCAM on a local DenseNet201 surrogate trained on an ImageNet subset, on the assumption that the surrogate's salient regions approximate those of the remote ResNet152V2 oracle. The paper claims these two ingredients let adversarial examples survive JPEG compression, remain statistically indistinguishable from benign images to a DCT-spectrum SVM detector, and are rated invisible or slightly visible by a majority of the 127 surveyed participants — the only evaluated method to score well on all three effectiveness properties simultaneously.","pith_inferences":["The evaluation uses one remote oracle (ResNet152V2) and one surrogate (DenseNet201), both CNNs trained on overlapping data; if the oracle were a vision transformer or trained on a different distribution, the GradCAM mask could lose alignment and the trade-off could narrow — the paper does not test this.","Median query cost is about 8,125 calls per example, which is high for real deployments; a rate-limited or billed API would force the attacker to spread queries over time or across accounts, an operational constraint the paper mentions but does not fold into the three-property framework.","Gradient blurring and attention masking are attack-agnostic ingredients that could in principle be grafted onto Square Attack or SimBA; the paper does not test that transfer, but it would be a natural way to see if the trade-off improvement is intrinsic to these steps.","The human survey finds that respondents who do not know what adversarial examples are score all images as more altered, including clean ones; this suggests absolute visibility numbers depend on the population, so cross-study comparisons should control for awareness or use paired clean-image baselines."],"forward_implications":["Confidence-scoring computer-vision APIs should be treated as practically attackable: an attacker can now craft examples that survive the compression and detection layers that such APIs typically apply.","JPEG compression alone is not a reliable defense against black-box attacks, and spectral-fingerprint detectors will miss ECLIPSE-style perturbations, so effective defenses need to combine multiple cues.","The three-property framework gives a common yardstick: future attacks can report P1, P2, P3 scores, and future defenses can measure how much they degrade each property, rather than only fooling rate.","Because ECLIPSE is a targeted attack demonstrated on a cat-to-dog task, the underlying machinery should transfer to other source-target pairs and to other confidence-scoring domains, widening the practical threat.","The ablation results indicate that the Gaussian blur is the main driver of compression robustness and detection stealth, while the surrogate mask mainly improves query efficiency — so even attackers with weak surrogates can keep part of the benefit."],"supporting_citations":[{"why":"Supplies SimBA and SimBA-DCT, the primary baselines and the confidence-score gradient-estimation paradigm ECLIPSE builds on.","marker":"[21]"},{"why":"Supplies the Square Attack L-infinity baseline that is ECLIPSE's closest competitor on JPEG-compression robustness.","marker":"[6]"},{"why":"Provides GradCAM, the saliency-mapping technique that generates the local-surrogate mask steering ECLIPSE's gradient sampling.","marker":"[36]"},{"why":"Establishes zeroth-order (hill-climbing) gradient estimation from confidence scores, the core optimization loop ECLIPSE inherits.","marker":"[11]"},{"why":"Supplies the Animals-10 dataset used for all experiments, including the cat-to-dog targeted attack task.","marker":"[14]"},{"why":"Introduces the notion of deployable adversarial examples that the three-effectiveness-properties framework extends.","marker":"[18]"},{"why":"Provides the hill-climbing metaheuristic that structures ECLIPSE's iterative optimization loop.","marker":"[30]"},{"why":"Supplies the Gaussian blur operation used to smooth the estimated gradients.","marker":"[19]"}],"fun_headline_variants":["ECLIPSE: robust black-box attack that survives JPEG and evades detection","Black-box attack stays stealthy under compression and detection","ECLIPSE attack: robust to JPEG, invisible to detectors and humans","Query-only attack passes JPEG compression and evades detection","ECLIPSE: JPEG-robust, detector-invisible, and human-invisible attack"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Everything hinges on the local DenseNet201 surrogate approximating the remote ResNet152V2 oracle closely enough that GradCAM heatmaps point to the same image regions; if the surrogate's attention diverges from the oracle's, the sampling mask guides queries to irrelevant areas and the attack's success, compression survival, and stealth all degrade together.","fun_headline_variants_meta":{"raw":{"variants":["ECLIPSE: robust black-box attack that survives JPEG and evades detection","Black-box attack stays stealthy under compression and detection","ECLIPSE attack: robust to JPEG, invisible to detectors and humans","Query-only attack passes JPEG compression and evades detection","ECLIPSE: JPEG-robust, detector-invisible, and human-invisible attack"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000685,"raw_usage":{"total_tokens":3083,"prompt_tokens":898,"completion_tokens":2185,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":514,"completion_tokens_details":{"reasoning_tokens":2089}},"tokens_in":514,"tokens_out":2185,"duration_ms":14896,"temperature":1.0,"reasoning_tokens":2089,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:19:11.146487+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a black-box model with a markedly different inductive bias, such as a vision transformer or a CNN trained on a different dataset, and keep the DenseNet201 surrogate fixed; if ECLIPSE's JPEG-survival rate falls toward SimBA's level and the spectral detector's AUC rises well above 0.5, the transferability of the surrogate mask is refuted and the generality of the trade-off claim with it.","supporting_citations":[{"cited_title":"International Conference on Machine Learning pp","cited_arxiv_id":null,"evidence_quote":"Supplies SimBA and SimBA-DCT, the primary baselines and the confidence-score gradient-estimation paradigm ECLIPSE builds on."},{"cited_title":"European Conference on Computer Vision (2020)","cited_arxiv_id":null,"evidence_quote":"Supplies the Square Attack L-infinity baseline that is ECLIPSE's closest competitor on JPEG-compression robustness."},{"cited_title":"In: Proceedings of the IEEE international conference on computer vision","cited_arxiv_id":null,"evidence_quote":"Provides GradCAM, the saliency-mapping technique that generates the local-surrogate mask steering ECLIPSE's gradient sampling."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Animals-10 dataset used for all experiments, including the cat-to-dog targeted attack task."},{"cited_title":"Pattern Recognition133, 108985 (2023)","cited_arxiv_id":null,"evidence_quote":"Introduces the notion of deployable adversarial examples that the three-effectiveness-properties framework extends."},{"cited_title":"Lulu, second edn","cited_arxiv_id":null,"evidence_quote":"Provides the hill-climbing metaheuristic that structures ECLIPSE's iterative optimization loop."},{"cited_title":"Prentice-Hall, Inc., USA (2006)","cited_arxiv_id":null,"evidence_quote":"Supplies the Gaussian blur operation used to smooth the estimated gradients."}],"review_version":1}