{"id":"39fc4278-7f79-4591-9ed9-a405e671f886","arxiv_id":"2411.12575","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Median smoothing plus a trained denoiser with ranking loss yields certified l2 robustness for no-reference image quality metrics while preserving correlation with subjective scores better than prior smoothing baselines.","lead":"This paper wraps no-reference image quality metrics in a median smoothing shield that gives certified score bounds against small l2 perturbations, and it adds a trained denoiser and ranking loss to keep the defended scores close to human judgments. The authors test the defense on five quality metrics and three datasets, reporting better correlation preservation than two prior smoothing baselines.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported certificates are empirical quantiles from N=2,000 noise samples with no finite-sample confidence interval, so the 'provable' robustness guarantee is not actually demonstrated as stated.","rationale":"The paper's underlying construction is plausible and the core median-smoothing theorem is correctly imported from Chiang et al. The auxiliary denoiser and the three-part loss are reasonable design choices, and the idea of applying median smoothing to no-reference IQA is novel. There is no internal inconsistency in the main theorem itself; the concern is about the gap between the population-level guarantee and the finite-sample quantities actually reported. The reader's weakest assumption identifies exactly this gap: N=2,000 empirical quantiles without confidence intervals cannot support a 'guaranteed' interval. My proposed test would settle whether the missing confidence bounds materially change the reported certified deltas. Since this is a standard fix and does not necessarily invalidate the approach, the appropriate verdict remains CONDITIONAL, matching the reader's verdict. I therefore do not recommend changing the verdict, but the paper should be revised to include finite-sample confidence statements and ideally release the code so the order-statistic computations can be checked.","tokens_in":13672,"tokens_out":5991,"duration_ms":69627,"concrete_test":"Recompute S_l and S_u for Tables 3 and 4 using distribution-free order-statistic confidence intervals. For each image, with N=2,000 i.i.d. draws of M(D(x+r)) and p=Phi(eps/sigma), choose lower index l and upper index u such that P(H_p <= X_(l)) >= 1-alpha and P(H_p >= X_(u)) >= 1-alpha, using binomial quantiles at alpha=10^-3; then set CD = X_(u) - X_(l). If the corrected CD values are substantially wider than the reported ones, or if the ordering 'DMS-IQA maintains comparable certified guarantees' is reversed, the reported certificates do not support the central claim. If they remain comparable, the concern reduces to a missing confidence statement rather than a substantive failure.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that DMS-IQA provides a certified, provably robust no-reference IQA metric. What the theory (Theorem 1, from Chiang et al. [6]) actually certifies is a population-level statement: for every ||u||_2 <= epsilon, the true median of M(D(x+r)) is bracketed by the Phi(-eps/sigma)- and Phi(eps/sigma)-quantiles of M(D(x+r)). That theorem is sound, and it transfers correctly to the composed metric M(D(.)). The problem is the operational step. Section 4 fixes N=2,000 noise samples, and all reported S, S_l, S_u, CD, CD_l, and CD_u are empirical estimates of these population quantiles, as described in Section 4.2. No confidence level, order-statistic correction, or finite-sample concentration bound is given. For the strong use case eps/sigma=2, the relevant upper quantile is around the 97.7th percentile; with 2,000 samples, the empirical order statistic is biased and can fall below the true quantile with probability far larger than any claimed guarantee. Thus Tables 3 and 4 do not actually report certificates. A violation does not require the attack to break the median-smoothing theorem; it only requires the Monte Carlo estimate to land on the wrong side of the true quantile. Because the paper's novelty is precisely the provable-robustness claim, this missing statistical calibration is load-bearing.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DMS-IQA, a defense for no-reference image quality assessment (NR-IQA) metrics based on median randomized smoothing with an auxiliary denoiser. The method is model-agnostic and does not require retraining the base metric. The authors report experiments on five NR-IQA metrics across three datasets, comparing with plain median smoothing (MS) and denoised median smoothing (DMS). They claim that DMS-IQA achieves better correlation with subjective scores (in terms of τSROCC and τPLCC) while maintaining comparable certified l2 robustness bounds (called certified deltas) for perturbation radii 0.06 and 0.36. They also show a use case where the smoothed metric is used as a loss for image denoising.","tokens_in":13946,"tokens_out":6028,"duration_ms":60456,"significance":"If the certified robustness claims were properly substantiated, this would be a valuable contribution: it is the first certified defense for no-reference IQA metrics, it is architecture-agnostic, and it does not require retraining the base metric. The theoretical core (Theorem 1, imported from Chiang et al.) is sound and transfers correctly to the composed metric M(D(.)). The denoiser training scheme with a ranking loss is a reasonable way to mitigate the accuracy loss of median smoothing, and the empirical results suggest that the proposed method preserves correlation better than the baselines. However, the paper's central 'guarantee' statement is not yet supported, because the reported bounds are empirical quantiles from only 2,000 noise samples with no finite-sample confidence level. This gap is fixable by adding standard concentration bounds or order-statistic confidence intervals, but as written it undermines the 'provable' claim.","major_comments":[{"comment":"The certified bounds Sl, Su, and CD reported in Tables 3 and 4 are empirical quantiles of M(D(x+r)) estimated from N=2,000 Gaussian samples, as stated in Section 4. Theorem 1 provides a population-level guarantee for the true quantiles of the smoothed function; it does not certify the finite-sample estimates. No confidence level, Clopper-Pearson interval, or any concentration bound is given. For the strong use case (ε/σ=2), the upper quantile corresponds to p=Φ(2)≈0.977, and with 2,000 samples the empirical order statistic can deviate substantially from the true quantile. Consequently, the sentence in Section 1 that the method 'guarantees ... the IQA score is bounded' is not established by the reported experiments. Please provide finite-sample confidence guarantees (e.g., binomial or order-statistic intervals) and report the bounds as confidence intervals, or soften the 'guarantee' language accordingly.","section":"Section 4, 'Use cases' and Tables 3/4"},{"comment":"Lemma 1 as stated is not correct. If |a_i - b_i| ≤ Δ(X) for all i, then a pair (x_i, x_j) can change order only if |b_i - b_j| ≤ 2Δ(X), not ≤ Δ(X). The lemma's definition of t as 'arg min_{δ̂∈B} : δ̂ > Δ(X)' also has issues: it defines t as a value, not an index or a count, so the phrase 'at most t out of m(m−1)/2 possible ranking errors' is not well-formed. The correct statement would involve pairs with δ̂ ≤ 2Δ(X) and t as the number of such pairs. Since this lemma is presented as a theoretical basis for the rank-correlation behavior, it should be fixed or removed.","section":"Section 3.3, Lemma 1"}],"minor_comments":[{"comment":"The definition of τ-closeness uses Pearson correlation, while Remark 1 extends it to Spearman correlation. The tables report 'τSROCC' and 'τPLCC' with arrows indicating lower is better; please clarify how these numbers relate to the threshold τ in Definition 3.1 and state whether a specific τ value is considered acceptable.","section":"Section 3.1, Definition 3.1"},{"comment":"Eq. (17) defines the attack as maximizing M(x+Δx)−M(x), but Eq. (18) defines a loss that includes M(x)−M(x+Δx) and the text says the loss is minimized. Please reconcile the sign convention and specify whether the optimizer maximizes or minimizes the objective.","section":"Section 4.2, Eqs. (17)–(18)"},{"comment":"The caption says 'Green dots represent the certified guarantees' but does not explain what the orange dots, blue dots, or error bars represent, nor how the averaging across images is performed. Please expand the caption for interpretability.","section":"Figure 2"},{"comment":"References [13] and [14] are the same paper (LipSim) listed twice; please consolidate them into a single entry and correct the citation numbering.","section":"References"},{"comment":"The text states that 'In the Supplementary materials, we provide the results of experiments on finding the optimal values of Cr and Ct from Eq. (11) and the batch size N.' No supplementary material is included with this submission; please either provide it or remove the sentence.","section":"Section 3.2"},{"comment":"Table 3 reports results averaged over five NR-IQA metrics, but individual-metric results are not shown anywhere. Including per-metric tables in the supplementary material would strengthen the evaluation and allow readers to assess variance across architectures.","section":"Table 3"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely and relevant problem, and the proposed combination of median smoothing with a denoiser trained via ranking loss is a natural and potentially useful extension of prior work. The main obstacle to acceptance is the unsupported finite-sample certificate: the reported bounds are empirical quantiles without confidence levels, which contradicts the 'guarantee' wording. This is fixable by adding standard statistical calibration (e.g., binomial confidence intervals) and adjusting the claims. The error in Lemma 1 should also be corrected. If the authors address these points, the paper could be a solid contribution to the certified robustness literature for IQA."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper does something genuinely new: it applies median randomized smoothing to no-reference IQA, and the core theorem from Chiang et al. transfers cleanly to the composed metric M(D(·)) with a denoiser. The ranking-loss denoiser training is a sensible fix for the accuracy drop that vanilla median smoothing causes, and the experiments cover five metrics on three datasets. The use of the smoothed metric as a loss function for denoising is a nice practical touch—Table 5 shows a real RMSE improvement. The theoretical core is cited, not re-derived, and that is fine; it does apply here.\n\nThe soft spot is load-bearing: the reported bounds are empirical quantiles from N=2,000 noise samples, with no confidence interval or finite-sample concentration bound. Section 4 fixes N=2,000 and all reported S, Sl, Su, CD are empirical estimates of the population quantiles in Theorem 1. For ε/σ=2, the upper quantile is around the 97.7th percentile; with 2,000 samples, the empirical order statistic can fall below the true quantile with non-negligible probability. So Tables 3 and 4 do not actually report certificates. The 'provably robust' claim in the abstract is not demonstrated as stated. This is fixable—use conservative order statistics or add Clopper-Pearson intervals—but it is not a minor issue for a paper whose contribution is precisely the certified guarantee.\n\nAlso, Lemma 1 has an off-by-one error: if t is the index of the first δ greater than Δ, then the number of ranking errors is at most t−1, not t. Minor but worth correcting. The experimental tables lack error bars, and the code is hidden (typical for submission, but still a limitation). The τ-closeness measure is aligned with the training objective, which is a mild circularity, not a flaw.\n\nOverall, this is a serious paper with a sound core idea and useful experiments. The central issue is the missing statistical calibration, not the underlying math. I would send it to peer review and expect the authors to add confidence bounds and fix the lemma. For a reading group, it is a maybe—useful for anyone working on IQA robustness, but the point is fairly niche.","headline":"Genuinely new application of median smoothing to NR-IQA with a sound core theorem, but the 'certified' bounds are empirical quantiles from 2,000 samples with no confidence interval, so the provable claim overreaches as stated.","tokens_in":14515,"tokens_out":2093,"would_cite":true,"duration_ms":21283,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Median smoothing gives blind IQA its first certified defense","keywords":["blind image quality assessment","certified robustness","median randomized smoothing","denoised smoothing","no-reference IQA","adversarial defense","l2 perturbation","ranking loss"],"falsifier":"Recompute the certified interval on the same test images with 100,000 noise samples instead of 2,000; if the 2,000-sample interval fails to contain the larger-sample estimate, or the bounds shift by more than the reported certified delta, the guarantee is a small-sample artifact rather than a mathematical bound.","tokens_in":13459,"feed_emoji":"🛡️","tokens_out":4232,"duration_ms":37812,"temperature":0.7,"pith_summary":"The paper tries to make no-reference image-quality metrics provably robust to small adversarial perturbations without giving up their agreement with human opinions. It proposes DMS-IQA, which replaces a metric's score on an image by the median of the metric's scores on many noisy copies of that image, after a small trained denoiser cleans the copies. The paper claims that for any perturbation $\\Delta x$ with $\\ell_2$ norm at most $\\epsilon$, the defended score stays inside a certified interval, and that the defended metric's correlation with subjective scores beats prior median-smoothing defenses. If the claim holds, this is the first certified defense for blind IQA that preserves ranking performance.","feed_headline":"Median smoothing gives blind IQA its first certified defense","feed_subtitle":"A trained denoiser keeps the smoothed metric correlated with human scores while bounding attack effects.","key_machinery":"The load-bearing identity is the median-smoothing bound: with $p = \\Phi(-\\epsilon/\\sigma)$ and $\\bar p = \\Phi(\\epsilon/\\sigma)$, the smoothed score of any perturbed image lies between the $p$ and $\\bar p$ percentiles of the smoothed-score distribution on the clean image. The second piece is the denoiser, a lightweight seven-layer U-Net trained with three losses: MSE for pixel reconstruction, a ranking loss that penalizes pairs where the smoothed metric and the subjective scores disagree in order, and a target loss that keeps smoothed scores close to source-metric scores. The denoiser keeps the smoothed metric accurate; the median makes the bounds hold.","core_discovery":"The central claim is that median randomized smoothing, paired with a denoiser trained to preserve both pixel fidelity and the ranking behavior of the source metric, produces a certified no-reference IQA metric. For image $x$ and Gaussian noise $r$, the method outputs $G(M(D(x))) = \\mathrm{med}[M(D(x+r))]$, and the median-smoothing theorem bounds $G(M(D(x+u)))$ between two percentile limits $H_p$ and $H_{\\bar p}$ whenever $\\lVert u\\rVert_2 \\le \\epsilon$. Experiments across five NR metrics and three datasets report better SROCC and PLCC than plain MS and DMS, comparable certified deltas, and a smaller adversarial gain on attacked images.","pith_inferences":["The 2,000-sample quantile estimates could be replaced by finite-sample confidence intervals, which would turn the empirical guarantee into a rigorous probabilistic one.","The method's practical robustness will depend on the base metric being locally smooth under Gaussian noise; metrics with sharp score discontinuities may need larger denoisers or more samples.","The same wrapper could be applied to video-quality metrics or full-reference similarity metrics by changing the target loss to compare similarity scores."],"forward_implications":["Any existing no-reference IQA metric can be protected by DMS-IQA without retraining the metric itself.","For perturbations within the certified radius $\\epsilon$, an adversary cannot push the metric's score outside the announced interval $[S_l, S_u]$.","The defended metric can serve as a loss function for image-processing optimizers; the paper reports lower RMSE with DMS-IQA than with the undefended DBCNN metric in a denoising task.","Training the denoiser on multiple datasets improves transfer to an unseen dataset such as KADID."],"supporting_citations":[{"why":"Supplies the median-smoothing theorem (Theorem 1) that gives the certified bounds for the smoothed score.","marker":"[6]"},{"why":"Introduces denoised smoothing, the template for training a denoiser to align smoothed predictions with the source model.","marker":"[32]"},{"why":"KonIQ dataset, one of the three testbeds for evaluating correlation and certified deltas.","marker":"[20]"},{"why":"CLIVE dataset, used to evaluate the defended metric's correlation with subjective scores.","marker":"[12]"},{"why":"SPAQ dataset, the third evaluation set for the method.","marker":"[11]"}],"fun_headline_variants":["Certified defense for blind IQA via median smoothing","Median smoothing gives blind IQA provable robustness","Provably robust no-reference IQA with median smoothing","Certified robust blind IQA: median smoothing plus denoiser","Blind IQA hardened: certified median-smoothing defense"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The certified bounds $S_l$ and $S_u$ are treated as exact guarantees, but they are empirical quantiles estimated from only 2,000 noise samples, with no stated confidence level.","fun_headline_variants_meta":{"raw":{"variants":["Certified defense for blind IQA via median smoothing","Median smoothing gives blind IQA provable robustness","Provably robust no-reference IQA with median smoothing","Certified robust blind IQA: median smoothing plus denoiser","Blind IQA hardened: certified median-smoothing defense"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000932,"raw_usage":{"total_tokens":3937,"prompt_tokens":841,"completion_tokens":3096,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":457,"completion_tokens_details":{"reasoning_tokens":3016}},"tokens_in":457,"tokens_out":3096,"duration_ms":20250,"temperature":1.0,"reasoning_tokens":3016,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T17:22:02.848123+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the certified interval on the same test images with 100,000 noise samples instead of 2,000; if the 2,000-sample interval fails to contain the larger-sample estimate, or the bounds shift by more than the reported certified delta, the guarantee is a small-sample artifact rather than a mathematical bound.","supporting_citations":[{"cited_title":"De- tection as regression: Certified object detection with median smoothing","cited_arxiv_id":null,"evidence_quote":"Supplies the median-smoothing theorem (Theorem 1) that gives the certified bounds for the smoothed score."},{"cited_title":"Denoised smoothing: A provable defense for pretrained classifiers","cited_arxiv_id":null,"evidence_quote":"Introduces denoised smoothing, the template for training a denoiser to align smoothed predictions with the source model."},{"cited_title":"Koniq-10k: An ecologically valid database for deep learning of blind image quality assessment","cited_arxiv_id":null,"evidence_quote":"KonIQ dataset, one of the three testbeds for evaluating correlation and certified deltas."},{"cited_title":"Massive online crowdsourced study of subjective and objective picture qual- ity","cited_arxiv_id":null,"evidence_quote":"CLIVE dataset, used to evaluate the defended metric's correlation with subjective scores."},{"cited_title":"Perceptual quality assessment of smartphone photog- raphy","cited_arxiv_id":null,"evidence_quote":"SPAQ dataset, the third evaluation set for the method."}],"review_version":1}