{"id":"ae2450c9-b254-4a8a-ba08-1bf3ff539a6e","arxiv_id":"2506.13024","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A certified robustness radius says nothing about whether a sample is clean or correctly predicted, so certification does not yet imply model security.","lead":"Certified robustness is often presented as a guarantee of security against adversarial attacks, but this paper argues the guarantee only shows how far an input is from the nearest class-flipping example, not that the input is clean or correctly labeled. The authors outline gaps in current certification research and propose open questions to make certificates useful in real deployments.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Certificates carry a theoretically grounded signal: any ε-bounded class-flipping input has certified radius ≤ ε, so the paper's dismissal of radii as 'only a heuristic' and its claim that only absence is actionable are too strong.","rationale":"The stress-test pass found one genuinely load-bearing weakness, and it coincides with the reader's weakest_assumption. The paper's core logical point is sound: a certified radius is relative to the predicted class and does not certify correctness. However, the paper goes beyond this to claim that certificates provide no information for distinguishing clean and attacked samples and that the only actionable information is absence of a certificate. That stronger claim is not supported. In fact, for any sound certificate, the radius at an adversarial input is bounded above by the perturbation norm used to create it, because the original clean input is itself a class-flipping example. This is a simple and rigorous argument, not a heuristic. It means adversarial examples produced by bounded attacks necessarily have small certificates, so the radius distribution carries statistical signal. The paper's own example of a misclassified clean sample far from the boundary does not apply to genuine bounded adversarial perturbations. This does not invalidate the title or the call for more rigorous certification practice, but it does mean the 'security theater' and 'only absence is actionable' conclusions are overstated. The reader's CONDITIONAL verdict is appropriate: the paper should either soften these claims or provide empirical evidence that the radius signal is unusable. No change to the verdict is needed beyond what the reader already recommended.","tokens_in":17483,"tokens_out":10517,"duration_ms":116778,"concrete_test":"On CIFAR-10 with randomized smoothing (Cohen et al. 2019), certify radii for (a) clean test images and (b) AutoAttack/PGD adversarial images constructed with a standard budget ε=8/255 and verified to change the predicted class. Compare the empirical distributions and compute the AUC of a one-dimensional radius-based clean-vs-attacked classifier. The theoretical bound predicts r(x_adv) ≤ ε for essentially all attacked samples; if the clean radii are not similarly concentrated at ε, the AUC will be well above 0.5, directly demonstrating that certificates carry actionable signal and refuting the 'only absence is actionable' claim. Repeating with IBP certificates on a small CNN would test generality.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing weakness is in Section 3. The paper asserts that 'the only reliable, actionable information that a certification technique may currently provide is the absence of a certification' (Section 3, paragraph 5), and justifies this by calling the observation that certifiable adversarial examples may have smaller radii 'only a heuristic, with no theoretical backing' (same paragraph). This dismissal is incorrect, and it is load-bearing because it converts the uncontroversial logical point (a certificate is label-relative) into the unsupported practical conclusion that certificate radii carry no useful signal.\n\nFor any two inputs x and y with F(x) ≠ F(y), the certification radius at y is a lower bound on inf{||y-z||_p : F(z) ≠ F(y)}. Since x itself is a class-flipping example at distance ||y-x||_p, every sound certificate satisfies r(y) ≤ ||y-x||_p. Consequently, any adversarial example produced by an ε-bounded attack from a correctly classified clean input has certified radius at most ε. This is a theorem, not a heuristic: under the standard threat model, attacked inputs have certificates stochastically bounded by the attack budget, while clean inputs can have large radii. Thus the distribution of radii does carry decision-relevant information about whether a sample was attacked, even though a large radius does not logically prove cleanliness.\n\nThe paper conflates 'no logical implication' with 'no information'. The central logical observation—that a certificate bounds distance to a class change rather than to a true-label adversarial example—remains correct and valuable. But the stronger claims that certifications provide 'no information for distinguishing between clean and attacked samples' (Section 3) and that they may be 'security theater' (Section 1.1) depend on the radii having no discriminative power.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This position paper argues that certified robustness certificates, as currently formulated, do not constitute a model-security guarantee. The central logical observation is that a certification bounds the distance to the nearest class-flipping example relative to the predicted class, not to the nearest adversarial example relative to the true label. Hence a misclassified (adversarial) input can itself be certified, and a certificate does not by itself distinguish clean from attacked samples. The paper discusses the three main certification families (randomized smoothing, interval bound propagation, global Lipschitz), notes the risk of norm-mismatch attacks (Theorem 2), warns that certificates can guide attackers, and argues that presenting certifications as guarantees can lead to 'security theater.' It closes with open questions about practical deployment, evaluation, coverage, and secure implementation.","tokens_in":17694,"tokens_out":11466,"duration_ms":130517,"significance":"The paper's core point is correct and important: the field often conflates 'certified robustness' with 'security,' and the distinction between a certificate of prediction invariance and a certificate of correctness is real. If the community heeds the call, this paper could help change how certified robustness results are framed. The paper is honest about relying on earlier work for the key empirical claims (e.g., certifiable adversarial examples and certificate-guided attacks) and includes a formal norm-containment theorem. Its main weakness is that Section 3 overreaches: it dismisses as 'only a heuristic' a theoretically grounded relationship between attacked inputs and small certified radii, and it states Theorem 1 without proof. These issues do not undermine the title claim, but they need to be fixed before the position is publishable in its current form.","major_comments":[{"comment":"The statement that 'the only reliable, actionable information that a certification technique may currently provide is the absence of a certification' is too strong and is contradicted by the paper's own Eq. (2). For any inputs x and y with F(x) != F(y), the certified radius at y satisfies r(y) <= ||y-x||_p, because x is an admissible witness in the infimum defining r(y). Consequently, any adversarial example produced by an epsilon-bounded attack from a correctly classified clean input has certified radius at most epsilon, while a clean input can in principle have a much larger radius. This is a mathematically grounded signal, not 'only a heuristic, with no theoretical backing' as claimed in the same paragraph. The paper conflates the correct logical claim that a certificate does not prove the input is clean with the unsupported claim that certificate radii convey no decision-relevant information. I recommend rewriting this passage to acknowledge that radii can serve as a risk signal while making clear that they are not a certificate of correctness.","section":"Section 3, fifth paragraph"},{"comment":"Theorem 1 is stated as a theorem but is not proved, and its key terms are undefined: there is no formal model of the attack process, no sample space over clean and adversarial inputs, and no definition of 'information.' As written, the theorem is effectively an observation that a certified radius is defined relative to the predicted class. That observation is correct and is sufficient for the paper's core thesis. However, the theorem's conclusion ('the existence of a certification does not intrinsically provide any information regarding if the sample x has been attacked') is a stronger statistical claim that is not established by the preceding argument and that repeats the conflation described above. Please either prove a formal version of Theorem 1 or recast it as a definitional observation, and replace 'no information' with 'no logical guarantee of cleanliness.'","section":"Section 3, Theorem 1"}],"minor_comments":[{"comment":"The phrase 'Potential adversarial attacks ... exist for rq > ...' overstates what the proof shows. The proof establishes that the certified ℓp ball does not cover the whole ℓq ball, so attacks in that region are not ruled out; it does not establish the existence of an actual class-flipping input. Rephrase as 'the certificate does not rule out attacks for rq > ...'.","section":"Section 4, Theorem 2"},{"comment":"The displayed equations for randomized smoothing (Eq. (3) and Eq. (4)) contain typesetting artifacts that make them difficult to read; in particular, the expectation expression and the certified-radius formula are not fully legible as rendered. Please provide clean, self-contained statements with proper citations, including one for the Cohen et al. certified radius.","section":"Section 2.2.1"},{"comment":"The abstract introduces 'the paradox of detection without distinction,' but this phrase does not appear in the body of the paper. Please define it in the Introduction or remove it from the abstract.","section":"Abstract and Introduction"},{"comment":"The text describes a certification as 'a heuristic measure of how likely it is that a sample may have been manipulated,' which is in tension with the same section's earlier dismissal of the smaller-radius pattern as 'only a heuristic.' These two statements should be aligned after the Section 3 revision.","section":"Section 3, Open Question 1"},{"comment":"The notation for F is introduced as a one-hot encoding of the predicted class, but later uses (e.g., Eq. (6) and the surrounding text) treat F(x) as a scalar label or as a value comparable to the abstention marker. Please standardize the notation for F throughout.","section":"Section 2.2"}],"recommendation":"major_revision","confidential_remarks":"This is a position paper, not an experimental contribution, but it is appropriate for ICML and the core logical point is timely and valid. The paper relies on the authors' own prior work for the two key empirical ingredients (certifiable adversarial examples and certificate-guided attacks); this reliance is transparent and the claims are independently checkable, so I do not see a circularity problem. The main risk is the Section 3 overstatement: if left as is, the paper could be cited as showing that certificate radii carry no useful information, which is stronger than the evidence warrants. If the authors revise Section 3 along the lines suggested and either prove or downgrade Theorem 1, I would support publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: this is a position paper worth reading, but not a paper that earns every claim it makes. The core observation is correct and important: a certified robustness radius bounds the distance to the nearest class-flipping example, not the distance to the nearest \"true\" adversarial example. Because the model's predicted label may itself be wrong, a certificate can exist for an attacked input, and by itself proves nothing about label correctness or input cleanliness. That point is made clearly in Section 3 and it is the right corrective to how certification results are often marketed.\n\nThe paper does well in two other ways. First, it turns the observation into a concrete set of open questions about deployment context, evaluation criteria, threat-model coverage, and secure implementation of certification code. Second, Theorem 2, while simple, correctly quantifies how an ℓp certificate leaves room for smaller ℓq attacks when q > p, and the norm-containment arithmetic checks out.\n\nNow the soft spots. The big one is in Section 3, where the authors claim that the only reliable, actionable information from a certificate is its absence, and dismiss the observation that certified adversarial examples may have smaller radii as \"only a heuristic, with no theoretical backing.\" That is not right. For any clean input x and any ε-bounded attack y with F(x)≠F(y), a sound certificate at y satisfies r(y) ≤ ||y−x||_p ≤ ε, because y itself is a class-flipping example. So attacked inputs have certificates that are stochastically bounded by the attack budget, while clean inputs can have large radii. That is a theorem, not a heuristic, and it means the radius distribution does carry decision-relevant information about whether a sample was attacked. The authors conflate \"no logical implication\" with \"no information.\" This matters because the stronger claims about \"no information for distinguishing between clean and attacked samples\" and \"security theater\" rest on that conflation.\n\nOther concerns are minor. Theorem 1 is more a definitional restatement than a theorem; it would help to say that directly rather than invite a proof where none is needed. The paper leans heavily on the authors' own prior work (Cullen et al. 2024b, 2024c) for the key premises, but those premises are independently checkable and the self-citation is transparent, so I do not treat that as a fatal issue.\n\nWho is this for? Anyone working on certified robustness who wants a candid critique of the gap between the guarantee and the marketing. The open questions are a useful agenda. It deserves a serious referee, though a referee should push back on Section 3's \"only absence\" claim and ask the authors to either soften it or support it.\n\nRecommendation: send it out, but expect revisions.","headline":"A useful cautionary position paper whose central logical point is sound, but whose claim that only certificate absence is actionable overstates the case.","tokens_in":18348,"tokens_out":2177,"would_cite":true,"duration_ms":21154,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Certified robustness certificates bound the distance to the nearest class-flipping example, not the distance to the nearest adversarial example, so a certificate provides no information about whether a sample has been attacked.","keywords":["certified robustness","adversarial examples","model security","security theater","certification radius","randomized smoothing","threat models","position paper"],"falsifier":"Run a standard attack such as projected gradient descent on a classifier that outputs certified guarantees, and record the guarantee size for each successfully misclassified sample; if many attacked samples receive nonzero guarantees, the paper's Theorem 1 is confirmed. The stronger claim that only the absence of a guarantee is reliable would be falsified by showing that the full distribution of guarantee sizes separates clean from attacked samples with high accuracy that transfers across datasets and models.","tokens_in":17239,"feed_emoji":"🛡️","tokens_out":8992,"duration_ms":95142,"temperature":0.7,"pith_summary":"This position paper argues that certified robustness certificates, which are widely presented as guaranteed protection against adversarial examples, should not yet be treated as model security. The key distinction is that a certificate bounds the distance to the nearest example that changes the model's predicted class, not the distance to the nearest example that changes the true class; a misclassified or attacked input can therefore carry a certificate just as a clean input can. Because certificates exist for both clean and adversarial samples, their existence gives no information about whether a sample has been attacked, and the authors conclude that the only reliable, actionable information a certificate currently provides is its absence. The paper also warns that marketing certificates as guarantees can create security theater: overconfidence that diverts attention from other defenses, while certificates themselves can reveal to attackers where adversarial examples cannot exist. If the argument is right, certified robustness should be reframed as one input to risk management and evaluated against realistic threat models, not presented as a standalone security guarantee.","feed_headline":"Certified robustness is not yet real security","feed_subtitle":"A certificate bounds the distance to the nearest class-flipping example, so attacked inputs can be certified too.","key_machinery":"The load-bearing object is the certification radius $r^{\\star}=\\inf\\{\\|x-x'\\|_p : x'\\in S,\\; F(x)\\neq F(x')\\}$, where $F$ is the one-hot encoded prediction of the model. The paper's move is to read this definition literally: it is a distance to a class change in the model's output, and nothing in the definition mentions whether the predicted class is correct. Randomized smoothing, interval bound propagation, and global Lipschitz margins all produce this same type of quantity, so the ambiguity is not an artifact of one technique. Theorem 1 formalizes the clean/adversarial ambiguity, and Theorem 2 shows how an $\\ell_p$ certificate translates to an $\\ell_q$ threat model: for $q>p$, $\\ell_q$-norm adversarial examples smaller than the certified region can exist, so certificates are tied to the specific norm being certified.","core_discovery":"The central claim is that current certified defenses produce a certification radius that measures the distance to the nearest class-flipping example relative to the model's own prediction, with no reference to ground truth. In deployed settings this distance can be measured from a clean sample to an adversarial one, from an adversarial sample to a different adversarial class, or from an adversarial sample to the true class; the certificate itself cannot distinguish these cases. Theorem 1 states that a certification of size $\\epsilon$ could correspond either to a certification of the correct class or to a certification of an incorrect class, so the existence of a certificate provides no intrinsic information about whether the input has been attacked. A corollary is that a certificate does not measure the effort needed to attack a sample, because certificates can be exploited to guide adversarial search. The paper's practical conclusion is that the only reliable, actionable information a certification technique currently provides is the absence of a certification.","pith_inferences":["The paper's logic generalizes beyond $\\ell_p$ classifiers: any certificate defined relative to the model's own prediction, such as edit-distance certificates for sequence models or policy-smoothing certificates for reinforcement learning, inherits the same clean/adversarial ambiguity because it never references ground truth.","If the conceded empirical correlation between small radii and adversarial inputs is stable, certificates could still be repurposed as risk scores in a statistical sense even though they are not logical proofs; the paper does not rule this out, it only notes the lack of theoretical backing.","Whether certificate absence is reliable is an empirical question the paper leaves open: a detector that flags low-radius inputs could make certificates useful for attack detection if its false-positive rate on clean data is acceptable, which would soften the paper's 'only absence is reliable' claim.","The security-theater argument implies a testable human-factors prediction: users told a model is 'certified robust' will choose to deploy it with fewer additional controls than users told the same model has a bounded-risk indicator; a simple behavioural experiment could test that."],"forward_implications":["A certificate is not evidence that a sample is benign; deployers should treat it as a bound on where the model's prediction changes, and combine it with other monitoring rather than relying on it as a guarantee.","Certificates should be kept secret from untrusted parties, because they encode information about where adversarial examples can and cannot exist and can be used to construct stronger attacks.","Security claims must be tied to the exact threat model certified: an $\\ell_p$ certificate does not cover an attacker who can use a different $\\ell_q$ norm, and geometric or semantic perturbations may be outside the certified region entirely.","Certification research needs standardized, application-driven evaluation, covering resource use, parallelism, dataset coverage, and suitability for the deployed task, rather than only average radii on reference datasets.","Presenting certificates as guarantees rather than as constrained, heuristic risk indicators risks security theater: users may drop other defenses precisely because they believe the guarantee is absolute."],"supporting_citations":[{"why":"Supplies the prior observation that certificates can be constructed for adversarial examples and that certificates can be exploited to guide stronger attacks, which is the paper's core premise.","marker":"(Cullen et al., 2024b)"},{"why":"Provides the randomized smoothing certificate formula the paper reinterprets, and is the leading example of a practical certified defense.","marker":"(Cohen et al., 2019)"},{"why":"Establishes certified robustness via differential privacy and randomized smoothing, a main mechanism family the paper's argument applies to.","marker":"(Lecuyer et al., 2019)"},{"why":"Introduces evasion attacks at test time, defining the class of manipulations certificates are meant to address.","marker":"(Biggio et al., 2013)"},{"why":"Demonstrates that neural networks are vulnerable to imperceptible perturbations, motivating the need for certified defenses.","marker":"(Szegedy et al., 2014)"},{"why":"Supplies the framing of norm distance as a proxy for detectability and attack cost, which the paper argues does not transfer to certificates.","marker":"(Gilmer et al., 2018)"}],"fun_headline_variants":["Certified defenses can certify attacks too","A certificate of robustness won't save you","Robustness certificates don't measure attack cost","Certified robustness may be a false promise","The certificate gap: why robustness isn't security"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument's load-bearing premise is that the sizes of the certified regions around samples carry no usable information about whether a sample was attacked; the paper grants that attacked samples may produce smaller guarantees but treats that as an unsupported heuristic, so the conclusion that only the absence of a guarantee is reliable rests on that correlation being useless in practice.","fun_headline_variants_meta":{"raw":{"variants":["Certified defenses can certify attacks too","A certificate of robustness won't save you","Robustness certificates don't measure attack cost","Certified robustness may be a false promise","The certificate gap: why robustness isn't security"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000236,"raw_usage":{"total_tokens":1442,"prompt_tokens":821,"completion_tokens":621,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":437,"completion_tokens_details":{"reasoning_tokens":554}},"tokens_in":437,"tokens_out":621,"duration_ms":7013,"temperature":1.0,"reasoning_tokens":554,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T00:36:50.953565+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a standard attack such as projected gradient descent on a classifier that outputs certified guarantees, and record the guarantee size for each successfully misclassified sample; if many attacked samples receive nonzero guarantees, the paper's Theorem 1 is confirmed. The stronger claim that only the absence of a guarantee is reliable would be falsified by showing that the full distribution of guarantee sizes separates clean from attacked samples with high accuracy that transfers across datasets and models.","supporting_citations":[{"cited_title":"Certified R obustness to A dversarial E xamples with D ifferential P rivacy","cited_arxiv_id":null,"evidence_quote":"Establishes certified robustness via differential privacy and randomized smoothing, a main mechanism family the paper's argument applies to."},{"cited_title":"Intriguing P roperties of N eural N etworks","cited_arxiv_id":null,"evidence_quote":"Demonstrates that neural networks are vulnerable to imperceptible perturbations, motivating the need for certified defenses."}],"review_version":1}