{"id":"eb148f0f-0dc9-44a6-8dbd-54065954917c","arxiv_id":"2501.13094","paper_version":2,"verdict":"ACCEPT","confidence":"MODERATE","novelty_score":8.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"rRCM, a contrastive denoising pre-training and fine-tuning scheme, gives a single-pass robust classifier that beats diffusion-based defenses on ImageNet and CIFAR-10 while reducing inference cost by up to 85x.","lead":"This paper trains an image classifier to give consistent answers for clean and noise-corrupted versions of the same image, using a training trick borrowed from diffusion models. The result is a certified-robust classifier that is much faster than prior diffusion-based defenses, with better accuracy at large perturbation radii.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline accuracy and cost claims compare 100k-sample rRCM against 10k-sample diffusion baselines; the reported 5.3% gain and 85× speedup are not reproducible from Tables 1–2.","rationale":"The reader identified Eq. (11)'s same-noise proxy as the weakest assumption, and the paper does acknowledge that approximation in Appendix E. However, the most load-bearing concern for the central claim is not the proxy itself: the method's own 10k row still shows positive gains, and the proxy is an internal training choice that could be validated or replaced. The larger threat is that the headline accuracy and efficiency numbers are computed under mismatched certification budgets and are not reproducible from the paper's own tables. Randomized smoothing results are sensitive to the number of smoothing noises through the Clopper-Pearson bound in Eq. (5), so comparing 100k-sample rRCM rows to 10k-sample diffusion baselines conflates sampling effort with algorithmic improvement. The 85× latency figure is likewise unsupported by any explicit protocol in the text. This is a concrete, checkable discrepancy rather than a speculation about future behavior. The method may still be a genuine advance, and the 10k-vs-10k comparison suggests it is, but the central quantitative claims as stated should be corrected or justified before the paper is accepted as-is. Recommend conditional acceptance pending rerunning the baselines at equal N and recomputing the headline numbers.","tokens_in":18507,"tokens_out":12156,"duration_ms":127500,"concrete_test":"Recompute Table 1 with DDS, DensePure, and DiffSmooth evaluated at 100,000 smoothing noises using the released code, and also evaluate rRCM-B-Deep at 10,000 smoothing noises. If the equal-N accuracy gap over DDS at large radii is ≤2 percentage points, or if the average latency reduction across the three diffusion baselines is not approximately 85×, the abstract's quantitative claims should be revised to reflect the fair comparison.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim's quantitative support is weakened by a certification-budget mismatch. In Table 1, DDS, DensePure, and DiffSmooth rows are marked with ‡ (10,000 smoothing noises), while the headline rRCM-B (53s) and rRCM-B-Deep (1min41s) rows use 100,000 smoothing noises. Randomized smoothing's certified radius depends on a finite-sample lower confidence bound (Eq. 5), so increasing N from 10k to 100k tightens the bound and can certify larger radii for the same base classifier. Part of the claimed improvement over diffusion-based methods may therefore reflect more smoothing samples rather than the method itself. The paper's own 10k row, rRCM-B‡, is the fair comparison: against DDS‡ it gains only 0.4–5.8 percentage points, far below the abstract's 5.3% average and 11.6% maximum. In addition, the abstract's 85× average inference-cost reduction is not recoverable from Tables 1–2: the latency ratios are roughly 39× vs DDS (3min52s/6s), 523× vs DensePure K=5, and 56× vs DiffSmooth m=15; no stated averaging protocol yields 85×. These issues do not refute the core method—the 10k row still beats DDS‡—but they mean the headline claims, as written, are not supported by the reported evidence.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes rRCM, a training framework that turns the generative denoising objective of diffusion models into a discriminative representation-alignment task. During pre-training, temporally adjacent points on approximate PF ODE trajectories are treated as positive pairs in an InfoNCE-style loss; after fine-tuning with class labels, the model performs implicit denoising-then-classification in a single forward pass. The authors report state-of-the-art certified accuracy under randomized smoothing on ImageNet and CIFAR-10, with large reductions in inference latency compared with diffusion-based purification methods such as DDS, DensePure, and DiffSmooth.","tokens_in":18818,"tokens_out":5828,"duration_ms":59419,"significance":"If the central claims hold, the paper makes a substantial practical contribution: it replaces the two-stage denoising-plus-classification pipeline of diffusion-based randomized smoothing with a single classifier, reducing certification cost by orders of magnitude while matching or exceeding the certified accuracy of prior diffusion-based methods at large radii. The work is also notable for its extensive experiments, including ablations on noise-level ranges, EMA schedules, and comparisons with MoCo-v3 with noise augmentation, as well as the appendix's quantitative analysis of semantic similarity along stochastic forward trajectories. These strengths are real and are part of my positive assessment. The main weakness is that the headline quantitative claims (5.3% average accuracy gain and 85× inference-cost reduction) are not supported by the tables as reported, because of a certification-budget mismatch and an unspecified latency-averaging protocol.","major_comments":[{"comment":"The headline claim of a 5.3% average certified-accuracy improvement over diffusion-based methods (Abstract and Section 1) compares rRCM-B/rRCM-B-Deep evaluated with N=100,000 smoothing noises against DDS, DensePure, and DiffSmooth rows marked with ‡, which use N=10,000 smoothing noises. Because the certified radius in Eq. (5) is a finite-sample lower confidence bound, increasing N can itself raise certified accuracy even for a fixed base classifier. The paper's own matched-budget row, rRCM-B‡, shows gains of only 0.4–5.8 percentage points over DDS‡ (e.g., 45.2 vs 41.4 at r=1.0), far below the abstract's averages. Please report the primary comparison at matched N, or explicitly state that the 5.3% figure is an unmatched-budget comparison and revise the claims accordingly.","section":"Section 4.1, Table 1, and Abstract"},{"comment":"The claimed average inference-cost reduction of 85× is not recoverable from the latencies in Tables 1 and 2. For example, DDS‡ is reported at 3min52s (232s) and rRCM-B‡ at 6s, giving a ratio of about 39×; DensePure K=5 is 52min20s (3140s), which is about 59× relative to rRCM-B at 53s and about 523× relative to rRCM-B‡ at 6s; DiffSmooth m=15 is 5min35s (335s), about 6.3× relative to rRCM-B at 53s. No averaging protocol stated in the paper yields 85× across these baselines. Please specify the exact set of baseline latencies and the averaging procedure used to obtain the 85× figure, or correct the claim.","section":"Abstract and Section 4.1"},{"comment":"The construction of positive pairs uses x_{t_{n-1}} = x_{t_n} + (t_{n-1} - t_n) ε, which is a same-noise forward-path step rather than a PF ODE step. The paper acknowledges in Appendix E that points on the stochastic forward trajectory \"share similar, rather than identical, semantics\" to the initial point. Because the pre-training objective's ability to transfer denoising ability into the classifier rests on this approximation, its validity is load-bearing. I ask for a quantitative evaluation of the approximation (e.g., measuring the actual PF ODE step error at the noise levels used) or a control experiment using a pretrained score model to generate the positive pairs, to establish that the approximation is not the source of the method's gains.","section":"Section 3.3, Eq. (11), and Appendix E"},{"comment":"The pre-training hyper-parameters in Table 4 list time steps in the range 20 to 80, while fine-tuning and certification use noise levels σ ∈ {0.25, 0.5, 1.0} (Section 3.4). If \"time steps\" here refers to the noise magnitudes t in Eq. (3), then the pre-training never aligns points at the small noise levels used during certification. The paper should clarify the meaning of this hyper-parameter and provide evidence or an explicit argument for why consistency learned at t ≥ 20 transfers to t ≤ 1. Without such clarification, the connection between the pre-training objective and the certified robustness at the evaluated radii is not fully established.","section":"Table 4 and Section 3.3"}],"minor_comments":[{"comment":"The phrase \"unwilling computation trade-off\" appears to be a typo; it should likely be \"unwieldy computation trade-off\" or \"undesirable computation trade-off.\"","section":"Section 4.2"},{"comment":"The comment \"Gassian noise sampled\" contains a typo: it should be \"Gaussian noise sampled.\"","section":"Algorithm 1"},{"comment":"The notation pν for the projector head conflicts with the use of p for probability distributions (e.g., p(x0), pϕ(xσ)). Consider using a different symbol such as hν or qν to avoid confusion.","section":"Section 3.3, Eq. (9)"},{"comment":"The sentence \"Consequently, theses points share similar...\" contains a typo: \"theses\" should be \"these.\"","section":"Appendix E"},{"comment":"The phrase \"consistency model Song et al. (2023)\" should be \"consistency models (Song et al., 2023)\" for grammatical correctness.","section":"Section 5"}],"recommendation":"major_revision","confidential_remarks":"The paper is technically sound in its core method and the experimental campaign is broad. The main issue is that the abstract's quantitative claims go beyond what the tables support under a matched certification budget; this is fixable by re-reporting with matched N and clarifying the latency computation. The Eq. (11) approximation and the pre-training time-step range also deserve explicit attention in revision. I do not see grounds for rejection, but the claims as written are currently overstated."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know up front. First, the core idea is real: aligning representations along diffusion noise trajectories via contrastive discrimination, then fine-tuning into a single forward-pass classifier, is a genuine departure from DDS/DensePure/DiffSmooth and from consistency models. Second, the abstract's numbers don't survive contact with Tables 1-2. The claimed 5.3% average and 11.6% max improvement over diffusion-based methods, and the 85x average speedup, are not reproducible from the tables as reported.\n\nThe paper does a lot well. The pre-training objective (consistency loss plus a MoCo-style contrastive loss) is well-motivated, the ablation against MoCo-v3 with Gaussian noise augmentation is exactly the right control, and the ImageNet and CIFAR-10 experiments are extensive with model-size and batch-size scaling. The code and hyperparameters are given. Appendix E is honest about the Eq. (11) proxy: points on the stochastic forward path have similar but not identical semantics to the PF ODE trajectory, and the paper quantifies the semantic drift with linear probing and Frechet distances. That is real evidence, not hand-waving.\n\nThe soft spots are in the headline reporting. In Table 1, DDS, DensePure, and DiffSmooth are all certified with 10k smoothing noises, while rRCM-B and rRCM-B-Deep use 100k. Randomized smoothing's certified radius is a lower confidence bound, so a 10x larger N tightens the bound; comparing 100k rRCM rows against 10k baselines is apples to oranges. The paper's own 10k row, rRCM-B-dagger, is the fair baseline comparison, and there the gains over DDS-dagger are 0.4 to 5.8 percentage points, not 5.3% average with 11.6% max. The 85x speedup also doesn't come out of the latency column--ratios are roughly 4-6x versus DDS at 100k, and 39x/523x/56x at 10k. The abstract and intro should be rewritten to state the comparison protocol.\n\nThe theoretical gap in Eq. (11) is real but acknowledged, and the ablation in Appendix G partially addresses it. The paper is not a takedown; the method's 10k results still beat DDS at large radii, and the 100k results suggest scaling helps. The presentation, not the method, is the problem.\n\nWho should read it: anyone working on certified robustness via randomized smoothing, especially the diffusion-purification crowd. It will get them thinking about whether single-pass training can replace two-stage pipelines. If I were the editor, I would send it to reviewers--ideally reviewers who check arithmetic--with instructions that the abstract's quantitative claims must match the evidence. As it stands, the work belongs in the literature, but the headline needs correction.","headline":"A genuinely new training framework for certified robustness, but the abstract's headline numbers don't survive contact with the tables; the method itself still deserves review.","tokens_in":19326,"tokens_out":4670,"would_cite":true,"duration_ms":41128,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A single forward-pass classifier trained with contrastive denoising can replace the two-stage purification-and-classification pipeline of diffusion-based randomized smoothing, giving higher certified accuracy at large radii at roughly 85×…","keywords":["randomized smoothing","certified robustness","adversarial robustness","diffusion models","contrastive learning","representation consistency","Vision Transformer","ImageNet"],"falsifier":"Train the same Vision Transformer with only the fine-tuning loss in Eq. (12), omitting the contrastive consistency pre-training, and compare certified accuracy at large radii; if it matches rRCM, the trajectory-alignment pre-training is not what carries the gain. Alternatively, construct positive pairs with a pretrained score model instead of the same-noise approximation in Eq. (11) and check whether certified accuracy at high noise levels changes.","tokens_in":18324,"feed_emoji":"🛡️","tokens_out":11342,"duration_ms":101388,"temperature":0.7,"pith_summary":"Certified adversarial robustness normally comes from randomized smoothing: add Gaussian noise many times, classify each noisy copy, and aggregate. Diffusion-based variants purify each copy with a separate denoiser before classifying, which is slow and weak at large radii. This paper claims that both steps can be collapsed into one forward pass of a single Vision Transformer that has been pre-trained to give consistent representations to points on the same diffusion trajectory. After supervised fine-tuning, the network implicitly denoises and classifies in one shot. The reported result is a 5.3% average gain in certified accuracy over diffusion-based methods on ImageNet (up to 11.6% at large radii) with roughly 85× lower inference cost.","feed_headline":"One forward pass beats diffusion certified smoothing, 85x faster","feed_subtitle":"Folding denoising into one ViT forward pass lifts certified accuracy at large radii and cuts latency to seconds.","key_machinery":"The central object is a two-stage training objective. Pre-training combines an infoNCE-style consistency loss over temporally adjacent points on the same probability-flow trajectory with a contrastive loss over augmented views of the clean image (Eq. 9); fine-tuning applies the consistency-regularized cross-entropy loss of Eq. (12) at each smoothing noise level. Positive pairs are cheap because they are built from one Gaussian sample: $x_{t_n}=x_0+t_n\\epsilon$ and $x_{t_{n-1}}=x_{t_n}+(t_{n-1}-t_n)\\epsilon$, so the two points differ only by a known rescaling of the same noise. The network is a Vision Transformer with a time embedding and a learnable class token, and the class-token output is the single prediction used at certification time. An EMA-updated target model provides the contrastive targets, and a projector head is used only for the contrastive loss.","core_discovery":"The central claim is that a discriminative objective over diffusion trajectories transfers denoising ability into a classifier, so the two-stage denoise-then-classify pipeline can be replaced by a single prediction. Positive pairs are noisy versions of the same clean image at adjacent time steps, generated with the same Gaussian noise via the approximate probability-flow step $x_{t_{n-1}} = x_{t_n} + (t_{n-1} - t_n)\\epsilon$; the model pulls these pairs together in representation space and pushes apart points from other trajectories. After this pre-training, a supervised fine-tuning stage with a consistency-regularized cross-entropy loss aligns the representations with class labels. In the paper's experiments this yields better certified accuracy than diffusion-based baselines on ImageNet and CIFAR-10, with the ImageNet gains largest at large perturbation radii.","pith_inferences":["An extension the paper leaves open is to build positive pairs with a pretrained score model instead of the same-noise approximation in Eq. (11); if certified accuracy at high noise levels improves, the approximation is the limiting factor.","Because the consistency pre-training is independent of the classification head, the same encoder could be fine-tuned for noise-robust downstream tasks beyond classification, such as segmentation or detection under Gaussian noise.","Other two-stage defenses that currently purify with a diffusion model and then classify could in principle be re-engineered as single rRCM-style models, inheriting a large latency reduction without the exact training recipe."],"forward_implications":["Certification becomes one forward pass per smoothing sample, cutting ImageNet certification latency from minutes to seconds without majority voting or a separate purification network.","Because the model is trained across the diffusion noise schedule, certified accuracy is maintained at radii where two-stage purification degrades, the regime where the reported gains are largest.","The same pre-trained encoder is fine-tuned separately for each smoothing noise level, so one training pipeline serves all certified radii.","The ImageNet results continue to improve with model depth and batch size, indicating that the reported certified accuracy is not a performance plateau."],"supporting_citations":[{"why":"Establishes randomized smoothing and the certification bound in Eq. (5) that rRCM and all compared methods build on.","marker":"Cohen et al. (2019)"},{"why":"The DDS diffusion-based purification baseline that rRCM claims to outperform in certified accuracy and inference cost.","marker":"Carlini et al. (2022)"},{"why":"Provides the consistency-model formulation and the same-noise approximation in Eq. (11) used to construct positive pairs.","marker":"Song et al. (2023)"},{"why":"Defines the EDM forward SDE, noise schedule, and time discretization used in pre-training.","marker":"Karras et al. (2022)"},{"why":"Supplies the contrastive learning framework and projector-head design adapted for the contrastive loss.","marker":"Chen et al. (2020)"},{"why":"Introduces the infoNCE loss on which both pre-training terms in Eq. (9) are based.","marker":"Oord et al. (2018)"},{"why":"Provides the consistency-regularized fine-tuning objective in Eq. (12) and the 'Consistency' baseline.","marker":"Jeong & Shin (2020)"},{"why":"DensePure, the diffusion purification method whose multi-step majority voting drives the paper's efficiency comparison.","marker":"Xiao et al. (2022)"}],"fun_headline_variants":["Single-pass ViT beats diffusion certified smoothing, 85x faster","Implicit denoising via contrastive learning: 85x faster, better accuracy","Folding denoising into classifier: 85x faster, +5.3% certified acc","One forward pass: better certified robustness, 85x less inference cost","Contrastive denoising replaces diffusion: single pass, SOTA certified accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that two noisy versions of the same image made with the same Gaussian noise at adjacent noise levels behave like two points on the same denoising trajectory, so aligning their representations transfers denoising into the classifier; the paper concedes in Appendix E that such points share similar, not identical, semantics, and the gap grows with noise.","fun_headline_variants_meta":{"raw":{"variants":["Single-pass ViT beats diffusion certified smoothing, 85x faster","Implicit denoising via contrastive learning: 85x faster, better accuracy","Folding denoising into classifier: 85x faster, +5.3% certified acc","One forward pass: better certified robustness, 85x less inference cost","Contrastive denoising replaces diffusion: single pass, SOTA certified accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001015,"raw_usage":{"total_tokens":4286,"prompt_tokens":944,"completion_tokens":3342,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":560,"completion_tokens_details":{"reasoning_tokens":3236}},"tokens_in":560,"tokens_out":3342,"duration_ms":23618,"temperature":1.0,"reasoning_tokens":3236,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T16:25:57.478100+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same Vision Transformer with only the fine-tuning loss in Eq. (12), omitting the contrastive consistency pre-training, and compare certified accuracy at large radii; if it matches rRCM, the trajectory-alignment pre-training is not what carries the gain. Alternatively, construct positive pairs with a pretrained score model instead of the same-noise approximation in Eq. (11) and check whether certified accuracy at high noise levels changes.","supporting_citations":[{"cited_title":"Consistency regularization for certified robustness of smoothed classifiers","cited_arxiv_id":null,"evidence_quote":"Provides the consistency-regularized fine-tuning objective in Eq. (12) and the 'Consistency' baseline."}],"review_version":1}