{"id":"c1fe4f6d-6965-4705-900d-ee0a3da28413","arxiv_id":"2608.11285","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"SegPAR, a class-centric decision-based sparse attack with a discrepancy reward, outperforms black-box sparse baselines in semantic segmentation MIoU reduction and sparsity efficiency.","lead":"SegPAR is a new black-box attack that breaks image segmentation models by changing very few pixels, using per-class exploration and a 'discrepancy reward' to guide queries. It reports larger accuracy drops than existing black-box attacks for the same or lower number of changed pixels, and close to gradient-based white-box attacks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. 1's proxy rewards any change from the initial prediction map, so correcting initially mis-segmented pixels earns positive reward while increasing MIoU; the reported MIoU reductions may not be driven by the optimized objective.","rationale":"The reader identified the proxy-to-MIoU alignment as the weakest assumption; I agree, but I sharpen it to a concrete failure mode: the reference prediction f(x) in Eq. 1 and in the discrepancy mask d(t) contains initial errors, so transitions that correct those errors receive positive reward while increasing MIoU. This is more specific than a general concern about non-smoothness and class imbalance, and it suggests a direct empirical test. I do not recommend changing the verdict because the paper's reported R.MIoU values are directly measured on ground truth, so the central empirical claim could still hold even if the proxy is imperfect. The proposed correlation/decomposition check would settle whether the optimization objective actually tracks the metric being reported. The lack of error bars and the single 100-image subset remain secondary concerns about statistical confidence, not about the internal logic of the method.","tokens_in":15016,"tokens_out":10114,"duration_ms":104399,"concrete_test":"Instrument the main DeepLabV3/Cityscapes run to log, for every candidate perturbation, the discrepancy reward R_d (Eq. 10) and the ground-truth MIoU difference between the candidate and the current image. Compute the Spearman correlation between R_d and ground-truth Delta MIoU across all logged steps; a well-aligned proxy should show a strong negative correlation. Additionally decompose positive-reward pixels into (i) initially correct predictions flipped to wrong classes, and (ii) initially wrong predictions flipped to the true class or another wrong class. If the correlation is weak or if a substantial fraction of positive reward comes from repairs, the proxy in Eq. 1 is misaligned with the reported MIoU reductions and the central claim requires qualification.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim depends on Eq. 1, which maximizes I[f(x_hat) != f(x)], i.e., the fraction of pixels whose prediction changes relative to the initial prediction, and the discrepancy reward in Eq. 10 inherits the same reference. Because no ground-truth label is used, an initially wrong pixel (true class A, initial prediction B) that the attack flips to A yields d_hat - d = +1 even though this corrects the model and increases per-class IoU. Likewise, flipping an initially wrong pixel from B to another wrong class C gives +1 with essentially no MIoU change. In semantic segmentation the initial prediction is not perfect: for example, DeepLabV3 on Cityscapes reaches MIoU 0.798, meaning a non-negligible fraction of pixels are initially wrong. Thus a meaningful share of positive rewards can come from 'repair' or 'wrong-to-wrong' transitions rather than from true-to-wrong damage. The paper's measured MIoU numbers are direct empirical evidence, so this concern does not disprove the claim, but it is the load-bearing assumption: if the proxy is frequently misaligned, the reward can be gamed and the reported advantage over baselines may reflect rewarding change rather than rewarding MIoU damage. The paper should demonstrate that positive rewards correlate with ground-truth MIoU decreases.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies decision-based black-box sparse attacks on semantic segmentation, a setting the authors argue is underexplored. The proposed method, SegPAR, adapts the authors' prior RL-based attack RFPAR to a class-centric paradigm: fixed per-class masks are extracted from the initial prediction, a masked softmax action sampler restricts perturbations to each predicted class region, and a new discrepancy reward is defined as the change in the number of pixels whose prediction differs from the initial prediction. The method is evaluated against Pointwise, SparseEvo, RFPAR, Pixle, and Sparse-RS on Cityscapes, ADE20K, and VOC2012 with four segmentation architectures, plus adversarially trained models, and is compared with the white-box attacks PGD0 and sPGD. The central claims are that SegPAR with the discrepancy reward achieves the strongest black-box performance in sparsity efficiency and MIoU reduction and is competitive with white-box sparse attacks.","tokens_in":15365,"tokens_out":4093,"duration_ms":75044,"significance":"If the results hold, the paper would provide a useful benchmark and a practical black-box attack for dense prediction, an area where sparse decision-based attacks have received little attention. The paper's strengths include a clear formulation of the class-centric exploration idea, a mathematically clean decomposition of the standard reward into four pixel-transition cases, a new reward that correctly assigns zero reward to fluctuations and negative reward to reversion, public code, and ablations that isolate the contributions of class-centric design and the discrepancy reward. The claimed gains are large and consistent across multiple architectures and datasets. However, the evaluation rests on a single random subset per dataset without variance estimates, and the optimized proxy is not shown to be aligned with the final MIoU objective on imperfectly segmented inputs. These issues are addressable and do not invalidate the approach, but they need to be resolved before the central claims can be accepted.","major_comments":[{"comment":"The optimized objective maximizes the number of pixels whose prediction changes relative to the initial prediction f(x), and the discrepancy reward uses the same initial prediction as the reference. Because the victim models are imperfect (e.g., DeepLabV3 on Cityscapes has initial MIoU 0.798 in Table 1, so over 20% of pixels are initially wrong), the proxy gives positive reward to transitions that can increase MIoU, such as correcting an initially wrong pixel to its true class, and to wrong-to-wrong transitions that do not change per-class IoU. The paper should demonstrate empirically that positive proxy reward correlates with ground-truth MIoU decrease, for example by reporting attack performance separately on initially-correct and initially-incorrect pixels, or by plotting proxy reward against MIoU reduction over training steps. Without such evidence, the reported MIoU gains may partly reflect reward gaming rather than genuine damage.","section":"Section 2, Eq. (1); Section 4.2, Eq. (10)"},{"comment":"All quantitative claims are based on a single fixed subset of 100 randomly sampled validation images per dataset, with no error bars, confidence intervals, or statistical significance tests. The abstract and Section 5 state that SegPAR 'significantly outperforms' baselines, but this is not supported by the reported numbers. The authors should provide means and standard deviations over multiple random subsets or training seeds, and ideally a paired significance test against the strongest baseline, to substantiate the superiority claim.","section":"Tables 1–3 and Section 5.1"},{"comment":"The white-box comparison equates one black-box forward query with one white-box gradient update, where a gradient update includes both forward and backward passes. This is an ad-hoc cost model and is load-bearing for the claim that SegPAR is 'competitive with white-box sparse attacks.' The paper should justify this equivalence, for example by reporting wall-clock time or FLOPs, or at least by acknowledging that the comparison is under a nonstandard budget and showing sensitivity to the chosen equivalence factor.","section":"Section 5.5"}],"minor_comments":[{"comment":"The caption contains garbled mathematical notation (e.g., 'd(0)', 'x(0)', and unreadable subscript expressions) that should be cleaned up for clarity.","section":"Figure 2 caption"},{"comment":"The notation 'MultinomialNR(pk; n)' and the description of sampling n locations without replacement are unclear; the exact sampling procedure should be stated explicitly, including how the number of locations n is chosen relative to the class mask size.","section":"Section 4.1, Eq. (3) and action sampling"},{"comment":"The layout of Table 3 is confusing: the 'Standard' and 'Discrepancy' subheaders alternate with model names, and the row 'DiscrepancyRFPAR' is not aligned with the model column. Please restructure the table so that each model, objective, and attack combination is unambiguous.","section":"Table 3"},{"comment":"The observation that applying the discrepancy reward to RFPAR on Cityscapes yields a smaller MIoU drop than the standard reward (Table 3) is attributed to sparse reward events, but the paper does not quantify how often informative reward events occur. A brief analysis of reward density would make the explanation more concrete.","section":"Section 5.4"},{"comment":"The paper relies heavily on the supplementary material for experimental details such as RL training hyperparameters, reward scaling, convergence thresholds, and baseline adaptation. At least the key hyperparameters and the exact adaptation of RFPAR to segmentation should be described in the main text to make the benchmark reproducible.","section":"General"},{"comment":"The phrase 'fatal semantic disruptions' is unnecessarily strong and unsupported by the reported metric; consider replacing it with a neutral description such as 'large MIoU reduction.'","section":"Figure 1 caption"}],"recommendation":"major_revision","confidential_remarks":"The paper adapts RFPAR, the authors' own prior work, as the base framework and as a baseline. This is a mild self-citation concern, but the inclusion of independent baselines (Pointwise, SparseEvo, Pixle, Sparse-RS) mitigates it. The main scientific risk is the unverified alignment between the decision-based proxy and MIoU; the authors should be pushed to provide evidence on this point. The evaluation's lack of variance is another reason to require a revision rather than acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"SegPAR is the first decision-based sparse attack designed for semantic segmentation, and the core ideas hold up. The authors shift from image-centric to class-centric exploration, sampling perturbations within each predicted class region, and introduce a discrepancy reward that only rewards pixels whose state changes relative to the initial prediction. The decomposition of pixel transitions into maintain/revert/new/fluctuate is correct, and the ablations show each component earns its keep. The empirical sweep is broad: four architectures, three datasets, two adversarially trained variants, plus comparisons against adapted classification sparse attacks and white-box sparse attacks. Code is provided. That is a solid package for an underexplored setting.\n\nThe soft spots are real but not disqualifying. First, the proxy in Eq. (1) maximizes pixel flips relative to the original prediction, not MIoU. A pixel that is initially wrong and gets corrected to its true class earns a positive reward even though it improves the model. The paper never quantifies how often such \"repair\" flips occur, nor shows that the optimized reward correlates with MIoU damage. Given the strong final numbers, the proxy is probably aligned on average, but the lack of analysis leaves the central mechanism under-justified. A referee should ask for a per-step or per-pixel correlation between Rd and ground-truth MIoU. Second, all results come from a fixed 100-image subset per dataset, with no error bars or multiple seeds. That makes the \"significantly outperforms\" claims less secure. Third, the white-box budget comparison equates one forward query with one gradient update, which is a useful first cut, but those are not equivalent computations; the paper should be clearer about what a fixed FLOP budget would say.\n\nThese are addressable. The class-centric insight is plausible and the empirical gains are large, so I'd send this to peer review rather than desk reject. If I were refereeing, I'd ask for the correlation analysis and repeated runs before accepting.","headline":"Solid first decision-based sparse attack for semantic segmentation; class-centric exploration plus discrepancy reward are effective, but the proxy-to-MIoU alignment and missing error bars need attention.","tokens_in":15815,"tokens_out":4966,"would_cite":true,"duration_ms":43195,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SegPAR, a decision-based black-box sparse attack that explores class by class and rewards only new misclassifications, achieves the strongest black-box sparse-attack results on segmentation and remains competitive with white-box sparse…","keywords":["black-box attack","semantic segmentation","sparse attack","decision-based attack","reinforcement learning","class-centric exploration","discrepancy reward","query efficiency"],"falsifier":"On an imbalanced segmentation benchmark, count how many perturbed pixels flip from the original prediction to a wrong class versus from one wrong class to another, and measure each subset's contribution to per-class IoU; if the discrepancy objective keeps rising while the second subset dominates or while MIoU stagnates, the proxy is not aligned with the claimed metric.","tokens_in":14857,"feed_emoji":"🎯","tokens_out":8305,"duration_ms":87605,"temperature":0.7,"pith_summary":"The paper sets out to show that sparse decision-based black-box attacks, where the attacker only sees the model's hard label output and may alter a small fraction of pixels, can be made practical for semantic segmentation. Its proposal, SegPAR, attacks one predicted class region at a time and rewards only the growth of pixels whose prediction differs from the original image. If the claims hold, segmentation models used in autonomous driving and dense perception are vulnerable to targeted sparse perturbations mounted with a few hundred label queries, and this vulnerability remains largely present at lower resolutions even after adversarial training.","feed_headline":"Fewer than 3% of pixels can gut a segmentation model","feed_subtitle":"Using only hard-label queries, a class-centric search with a discrepancy reward beats black-box sparse baselines.","key_machinery":"The two load-bearing mechanisms are class-centric exploration and the discrepancy reward. Class-centric exploration fixes binary masks $M^k_{i,j}=\\mathbb{I}[f(x)_{i,j}=k]$ from the initial prediction, builds a tight bounding box around each predicted class, and uses a masked softmax distribution over that box so sampled perturbations stay inside the class region. The discrepancy mask tracks attack success relative to the original prediction, $d^{(t)}_{i,j}=\\mathbb{I}[y^{(t)}_{i,j}\\neq y^{(0)}_{i,j}]$, and the class-wise reward $R_d^k=\\frac{1}{o^2 n}\\sum_{i,j} M^k_{i,j}\\,(\\hat{d}^{(t)}_{i,j}-d^{(t)}_{i,j})$ gives positive feedback only to newly misclassified pixels, zero to maintained or fluctuating ones, and negative feedback to reversions. This reward removes the misleading signal that standard transition-based rewards give to pixels that revert to the original class or oscillate between wrong classes.","core_discovery":"The central claim is that image-centric pixel accumulation is the wrong unit of search for segmentation, because an image contains many heterogeneous, class-specific decision boundaries. SegPAR therefore runs the search per predicted class: it derives fixed per-class masks from the victim's initial prediction, crops the agent state to each class's bounding box, and samples perturbed pixel locations inside the class mask from a learned masked-softmax probability map. On top of this, SegPAR changes what is rewarded: instead of rewarding any prediction change between consecutive steps, a discrepancy reward scores only the marginal increase in pixels that differ from the initial prediction. The paper reports that with these two changes it achieves the best black-box sparse-attack results on Cityscapes, ADE20K, and Pascal VOC2012 across CNN and Transformer segmentation models, while remaining competitive with gradient-based white-box sparse attacks under matched budgets.","pith_inferences":["The discrepancy reward is defined against the original prediction rather than against per-class ground truth, so the same cumulative-reward logic could transfer to other dense prediction tasks, including instance-level object detection, where reversion and fluctuation also waste queries during accumulation.","The class-centric decomposition suggests a natural prioritization scheme: allocate more queries to safety-critical or small-area classes instead of treating all predicted classes equally, which could produce even larger practical disruption per query.","One testable extension is to make class masks adaptive during an episode rather than fixed from the initial prediction; because the discrepancy reward is anchored to the initial prediction, refreshing masks after stable flips might accelerate convergence.","Because the attack needs only hard labels and no ground truth, it could serve as a low-cost auditing tool for deployed segmentation systems, exposing a failure profile different from that of dense perturbation attacks."],"forward_implications":["Decision-only attackers can push segmentation models to low MIoU values at roughly 2.4 to 4.0 percent pixel sparsity within a 1,000-query budget, so realistic black-box robustness evaluation should include sparse attacks.","Attacking class regions separately lets the attacker degrade safety-critical classes such as traffic signs, traffic lights, persons, and riders much earlier than image-centric exploration does.","Replacing the standard transition reward with the discrepancy reward suppresses pixel reversion and wasted queries, and the same reward can be plugged into other accumulation-based sparse attackers, improving their MIoU reduction at equal or lower sparsity.","A black-box sparse attack built this way is competitive with white-box sparse baselines, outperforming PGD0 and narrowing the gap to sPGD under equal query-versus-update budgets.","Adversarially trained segmentation models offer meaningful protection against sparse attacks on high-resolution images, but that protection appears substantially weaker at lower resolutions."],"supporting_citations":[{"why":"Supplies the reinforcement-learning pixel-accumulation framework that SegPAR reformulates into a class-centric paradigm and serves as a primary black-box baseline.","marker":"[29]"},{"why":"Decision-based sparse attack baseline adapted to segmentation using a success-ratio criterion for attack success.","marker":"[27]"},{"why":"Evolutionary decision-based sparse attack baseline adapted to semantic segmentation.","marker":"[34]"},{"why":"Score-based sparse attack baseline used to test whether the discrepancy objective improves pixel accumulation.","marker":"[12]"},{"why":"Score-based sparse attack baseline used in the same discrepancy-reward isolation experiments.","marker":"[26]"},{"why":"White-box sparse attack baseline, sPGD, whose equal-budget MIoU-versus-sparsity curve is the competitive target.","marker":"[43]"},{"why":"White-box sparse attack baseline, PGD0, that SegPAR is reported to outperform under matched budgets.","marker":"[13]"}],"fun_headline_variants":["Class-centric sparse attack beats pixel-based methods","SegPAR uses class masks and discrepancy reward for sparse attacks","Image-centric search wastes queries; SegPAR goes class-centric","Discrepancy reward fixes misleading feedback in sparse attacks","Segment-wise search cuts pixels more efficiently than image-wise"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Everything rests on the proxy that maximizing the fraction of pixels whose predicted label changes from the original prediction faithfully tracks reducing mean IoU; if most flipped pixels lie within already-wrong regions or are label swaps that do not touch per-class IoU, the claimed gains would not follow.","fun_headline_variants_meta":{"raw":{"variants":["Class-centric sparse attack beats pixel-based methods","SegPAR uses class masks and discrepancy reward for sparse attacks","Image-centric search wastes queries; SegPAR goes class-centric","Discrepancy reward fixes misleading feedback in sparse attacks","Segment-wise search cuts pixels more efficiently than image-wise"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000189,"raw_usage":{"total_tokens":1314,"prompt_tokens":901,"completion_tokens":413,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":517,"completion_tokens_details":{"reasoning_tokens":337}},"tokens_in":517,"tokens_out":413,"duration_ms":4342,"temperature":1.0,"reasoning_tokens":337,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:12:46.785497+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On an imbalanced segmentation benchmark, count how many perturbed pixels flip from the original prediction to a wrong class versus from one wrong class to another, and measure each subset's contribution to per-class IoU; if the discrepancy objective keeps rising while the second subset dominates or while MIoU stagnates, the proxy is not aligned with the claimed metric.","supporting_citations":[{"cited_title":"In: NeurIPS","cited_arxiv_id":null,"evidence_quote":"Supplies the reinforcement-learning pixel-accumulation framework that SegPAR reformulates into a class-centric paradigm and serves as a primary black-box baseline."},{"cited_title":"Query Efficient Decision Based Sparse Attacks Against Black-Box Deep Learning Models","cited_arxiv_id":"2202.00091","evidence_quote":"Evolutionary decision-based sparse attack baseline adapted to semantic segmentation."},{"cited_title":"In: AAAI","cited_arxiv_id":null,"evidence_quote":"Score-based sparse attack baseline used to test whether the discrepancy objective improves pixel accumulation."},{"cited_title":"Adversarial Training via Adaptive Knowledge Amalgamation of an Ensemble of Teachers","cited_arxiv_id":"2405.13324","evidence_quote":"White-box sparse attack baseline, sPGD, whose equal-budget MIoU-versus-sparsity curve is the competitive target."}],"review_version":1}