{"id":"30a5bc6a-1409-4a01-91f7-dee4a17e5734","arxiv_id":"2601.21738","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"GMC maps an IQA model's agreement with human scores across the quality-level and quality-difference landscape, exposing local strengths that global PLCC/SRCC hide.","lead":"This paper proposes GMC, a new evaluation metric that measures an IQA model's correlation with human quality scores locally as a 3D surface over quality level and quality difference, rather than collapsing everything into a single global number. It also derives a global score designed to stay stable when test-set quality distributions shift.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"GMCg's 'distribution-agnostic' claim is tested only for variance; low variance across shifted subsets does not establish unbiasedness, so the 'true reflection' claim is unsubstantiated.","rationale":"The reader's Gaussian-weight concern is real, but it mainly threatens local interpretation; the more load-bearing issue is the global claim. The paper's own motivation (Fig. 2) is distributional instability, and its main proposed remedy, GMCg, is asserted to be distribution-agnostic. The evidence is variance-only. A variance-only robustness experiment is insufficient: a constant score has zero variance but no discriminative validity. A bias check against a full-dataset reference is inexpensive and decisive. I still see enough value in the local correlation surface and the applications to keep the paper under CONDITIONAL review; the missing analysis is fixable. Hence no change to the reader's verdict, but the condition should explicitly require the bias test.","tokens_in":21212,"tokens_out":5119,"duration_ms":60593,"concrete_test":"Using the same nine sampling distributions as Sec. 4.2(5), compute GMCg on the full PIPAL and SPAQ datasets as reference values. For each subset, record bias = mean over repeated LHS runs of GMCg_subset − GMCg_full, with bootstrapped CIs. Also compute the between-model GMCg gap for the closest pair (e.g., MS-SSIM vs LPIPS on PIPAL). If |bias| exceeds roughly half that gap, or if the ranking relative to the full-dataset ranking changes, the distribution-agnostic claim fails. Report the same bias analysis for SRCC as a control.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that GMCg is 'inherently distribution-agnostic' and gives a 'true' reflection of model capability is not established by the reported experiments. GMCg (Eq. 22) integrates the fitted surface over the observed [Q_s_min, Q_s_max] x [Q_d_min, Q_d_max] rectangle. The surface itself is estimated from a single sample: the Distribution Regulator (Eqs. 16-18) reweights observed pairs but cannot create information in unsupported MOS/|ΔMOS| regions, where local linear regression must interpolate or extrapolate. Hence GMCg remains a functional of the sample-dependent fitted surface, not a distribution-free quantity. Sec. 4.2(5) compares only the standard deviation of GMCg across nine Gaussian-shaped subsets; low variance can coexist with large systematic bias. No comparison to the full-dataset GMCg, no bias/coverage analysis, and no distribution-robust baseline (e.g., PWRC) is provided. The 'stability' could simply reflect that inverse-density weighting attenuates the signal in sparse regions, compressing all GMCg values. Therefore the paper's broadest claim needs a bias check before GMCg can be recommended as a standardized metric.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Granularity-Modulated Correlation (GMC), a generalization of Kendall's Generalized Correlation Coefficient (GCC) in which pair weights depend on absolute MOS and pairwise MOS difference through Gaussian kernels (Granularity Modulator), and are inversely weighted by a kernel-smoothed quality density (Distribution Regulator). Local weighted correlations are sampled via Latin hypercube sampling and fit by local linear regression to produce a 3D correlation surface over (MOS, |ΔMOS|); integrating this surface yields a global score GMCg. The paper reports experiments on KADID-10k, PIPAL, LIVEC, and SPAQ with both FR and NR IQA models, and uses GMC for local diagnostics, scenario-specific model selection, fine-grained optimization, model integration, and robustness to sampling distributions. The mathematical extension of GCC with nonnegative symmetric weights is sound, and the local surface construction is clearly described, but the strongest claim—that GMCg is an \"inherently distribution-agnostic\" and \"true\" reflection of model capability—is not supported by the current evidence.","tokens_in":21510,"tokens_out":6977,"duration_ms":66971,"significance":"If validated, GMC would be a useful diagnostic tool for the IQA community: it localizes correlation performance in a way global PLCC/SRCC cannot, and the surface visualization plus GMCg could support model selection and integration. The authors provide clean derivations linking PLCC, SRCC, and KRCC to GCC, and they release code, which aids reproducibility. The core novelty is moderate but potentially valuable, especially for applications requiring fine-grained discrimination. However, the most ambitious claim—distribution-agnostic and unbiased global scoring—needs substantially more evidence, and some validation experiments are qualitative or statistically thin. With additional bias analysis, confidence intervals, and a corrected interpretation of the Gaussian weights, the contribution could be solid.","major_comments":[{"comment":"The Gaussian weighting is presented as a probability model, but Eqs. (13)–(15) conflate the MOS estimate q_i with the random subjective rating R_i. If R_i ~ N(q_i, σ_i^2), then σ_i is the spread of individual ratings, not the uncertainty of the mean q_i; Eq. (13) is not the probability that q_i equals Q_s. Moreover, if R_i and R_j are independent Gaussians, |R_i − R_j| follows a folded normal distribution, not N(|q_i − q_j|, σ_i^2 + σ_j^2), so Eq. (15) is not a valid probability for |q_i − q_j| = Q_d. Since these weights define every localized correlation and all GMC_s/GMC_d conclusions, the manuscript must either provide a correct probabilistic derivation or explicitly re-frame Eqs. (13)–(15) as heuristic kernel weighting and analyze sensitivity to the choice of σ_i (and to Beta-regression estimates when σ_i is unavailable).","section":"§3.3.1, Eqs. (13)–(15)"},{"comment":"The claim that GMCg is \"inherently distribution-agnostic\" and yields a \"true\" reflection of model capability is not established. GMCg integrates the fitted surface over the observed rectangle [Q_s_min, Q_s_max] × [Q_d_min, Q_d_max]. The surface is estimated from a single sample: inverse-density weighting in Eqs. (16)–(18) cannot create information in unsupported MOS/|ΔMOS| regions, where local linear regression must interpolate or extrapolate. Section 4.2(5) reports only the standard deviation of GMCg across nine Gaussian-shaped subsets; low variance can coexist with large systematic bias. No comparison to the full-dataset GMCg, no bias/coverage analysis, and no distribution-robust baseline (e.g., PWRC) are provided. The authors should add a bias analysis and confidence intervals, and temper the \"true reflection\" claim accordingly.","section":"§3.3.3, Eq. (22); §4.2(5)"},{"comment":"The ablation of kernel-smoothed density estimation does not support the stated conclusion. Across five models on two datasets, the reduction in standard deviation from raw bin-frequency density is on the order of 1e-4 to 1e-5, and on SPAQ CLIP-IQA the kernel variant has higher std (0.007805 vs 0.007800). No significance test or effect on model ranking is reported. Since the Distribution Regulator is a core module, this experiment needs to be redesigned—for example, with more distribution shifts, repeated sampling, and reporting bias as well as variance—or the conclusion must be weakened.","section":"Table 3, §4.3"},{"comment":"The adversarial optimization experiment is presented as confirmation that MS-SSIM is superior in the low-difference regime, but the evidence is qualitative: four example images per setting, no quantitative metrics for constraint satisfaction (e.g., the fraction of runs where the adversarial metric remains within ε), no average improvements in the target metric, and no human evaluation of the optimized images. Please provide quantitative summaries and statistical comparisons; otherwise the link between the GMC surface and optimization utility remains anecdotal.","section":"§4.2(3), Fig. 7"}],"minor_comments":[{"comment":"Both sampling coordinates use the same u_k ~ U(0,1). Standard LHS requires independent perturbations u_k^x and u_k^y; using the same draw induces a correlation between Q_s and Q_d sample coordinates.","section":"Eqs. (19)–(20)"},{"comment":"The bandwidths for the kernel density estimator in Eqs. (17)–(18) and for the local linear regression in Eq. (21) are not specified, and the Beta-regression procedure for estimating σ_i is not described. These are important free parameters for reproducibility.","section":"§4.1"},{"comment":"The model names QALIGN and TOPIQ-NR appear in Table 3 but are not among the NR models introduced in Section 4.1; please define them and align the table with the model list.","section":"Table 3"},{"comment":"DISTS is cited as [31] in Table 1, but the correct reference is [5]. Check also whether references [60] and [22] are duplicated.","section":"Table 1"},{"comment":"The caption contains a typo: \"PIAPL\" should be \"PIPAL\".","section":"Fig. 7"},{"comment":"The density estimator is written as an integral over Y, but Y is elsewhere described as a set of bins; please clarify the notation and the kernel bandwidth.","section":"Eq. (18)"},{"comment":"The axes are not labeled, making it hard to verify the convergence claim. Add axis labels and a legend.","section":"Figs. 10–11"}],"recommendation":"major_revision","confidential_remarks":"This is a methodologically interesting paper that could fit TPAMI if the central claims are tightened. The distribution-agnostic claim is currently overreaching and needs a bias/coverage analysis, not just low variance. The Gaussian-weight issue is fixable by reframing the weights as kernels rather than probabilities. The ablation in Table 3 is too weak to support the Distribution Regulator's importance. I would not reject; the core surface idea is sound and the code release is a plus, but the experiments need more rigor before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this paper deserves a serious referee. GMC is not just another weighted correlation; the joint MOS-|ΔMOS| surface with pair-level Gaussian weights gives a structured view of where models succeed and fail, and the two downstream experiments (adversarial optimization and model integration) are genuinely prospective: GMC generates a hypothesis that is then tested against independent objectives. That is not circular, and it shows the surface has real content.\n\nWhat's new: the 3D correlation surface as a function of both absolute quality and pairwise difference, with a Distribution Regulator for density compensation and a surface-integrated global score. The math is a clean extension of GCC to nonnegative pair weights; correlation values stay in [-1,1]. The qualitative demonstrations are the strongest part — CLIP-IQA's high-quality niche and NIQE's low-quality niche would be invisible in a scalar. Code is stated to be available.\n\nSoft spots, in order. First, the 'inherently distribution-agnostic' claim for GMCg. The paper shows lower variance across nine shifted subsets, but variance is not bias. GMCg is an integral of a fitted surface over the observed rectangle; inverse-density weighting can compress values in sparse regions, and nothing in the experiments rules out systematic shifts. The paper needs a bias/coverage analysis against the full-dataset value, or a formal statement of what invariance is actually guaranteed. This is the load-bearing overclaim, but it doesn't sink the diagnostic use of the surface.\n\nSecond, the Gaussian weights in Eqs. (13)-(15) assume individual raw ratings are Gaussian around MOS, and then treat sigma as known or Beta-regressed. The paper conflates rating spread with MOS uncertainty. For the surface to be trustworthy as a local probe, this assumption needs an ablation or at least a sensitivity check using a homogenized sigma.\n\nThird, missing comparisons: PWRC is cited as the closest weighted global metric but never used as a baseline. Confidence intervals are absent, which matters because the K=100 surfaces are random via LHS. Code link has no commit hash or license — minor.\n\nBottom line: a useful methodology paper with one overclaimed global property. For the IQA community, the diagnostic surface is worth adopting; GMCg needs more work. I'd send it to review.","headline":"GMC is a genuinely useful diagnostic surface for IQA evaluation — worth engaging seriously — but the 'distribution-agnostic' claim is not yet supported; it needs a bias analysis before GMCg can be treated as a standard metric.","tokens_in":22015,"tokens_out":1628,"would_cite":true,"duration_ms":16870,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that single-number correlation metrics mislead IQA benchmarking, and introduces a correlation surface and a distribution-robust global score.","keywords":["image quality assessment","correlation coefficient","fine-grained evaluation","distribution robustness","model benchmarking","rank correlation","perceptual metrics"],"falsifier":"Measure GMC on a dataset whose rating noise is deliberately non-Gaussian (e.g., heavy-tailed or bimodal within-image ratings) while keeping the same MOS values; if the surface and GMC_g change materially, the Gaussian weighting is the load-bearing assumption. A simpler calculation: replace σ_i in Eqs. (13)–(15) with a constant global σ across images and check whether conclusions about regime-specific strengths (e.g., CLIP-IQA vs NIQE) invert—if they do, the per-image σ estimates are driving the result, not the data.","tokens_in":21092,"feed_emoji":"📊","tokens_out":4206,"duration_ms":43644,"temperature":0.7,"pith_summary":"Image quality assessment (IQA) models are usually compared by a single number — PLCC, SRCC, or KRCC — computed over a whole benchmark. The paper argues that this scalar hides which quality regimes a model actually handles: two models can tie on SRCC while one is accurate on high-quality images and the other is better at telling apart nearly identical distortions. It proposes Granularity-Modulated Correlation (GMC), a weighted correlation that can be queried at any combination of absolute quality (MOS) and pairwise quality difference (|ΔMOS|), then assembled into a 3D correlation surface. Averaging that surface gives a global score, GMC_g, which the paper claims is stable when the test set's quality distribution changes, unlike SRCC or PLCC. If true, GMC would give researchers a way to rank models fairly and to choose models for specific applications, such as evaluating generative outputs versus guiding compression.","feed_headline":"3D correlation surface replaces single-score IQA ranking","feed_subtitle":"Plots accuracy by quality level and difference, yields a stable global score, and shows all models struggle on tiny quality gaps.","key_machinery":"The machinery is the Generalized Correlation Coefficient (GCC) extended with a per-pair weight w^k_ij = P^s_k(i,j) * P^d_k(i,j) * P^t_k(i,j). P^s and P^d form the Granularity Modulator: Gaussian probabilities that images i and j have MOS about Q^s_k and pairwise difference about Q^d_k, using each image's rating standard deviation σ_i. P^t is the Distribution Regulator: the inverse of kernel-smoothed density at each image's MOS. Replacing GCC's uniform pairing with these weights turns PLCC, SRCC, and KRCC into local probes; a Latin Hypercube design samples the (Q^s, Q^d) plane, local linear kernel regression fits the surface, and surface integration produces the global GMC_g.","core_discovery":"The central claim is that IQA performance is a two-dimensional function — prediction accuracy along absolute MOS and discrimination capability along pairwise MOS difference — and that both dimensions can be measured with a generalized correlation coefficient whose per-pair weights come from two modulators: a Granularity Modulator that down-weights pairs far from a queried (MOS, |ΔMOS|) via Gaussians with per-image rating variance, and a Distribution Regulator that reweights pairs by the inverse of a kernel-smoothed MOS density. Sampling these weighted correlations at Latin-Hypercube points and fitting a local-linear kernel regression yields a continuous correlation surface; integrating the s","pith_inferences":["The same weighted-correlation surface could be ported to other perceptual or non-perceptual evaluation settings where a scalar metric hides local behavior, such as video quality, text-to-image alignment, or ranking models in information retrieval.","GMC_g's distribution-agnostic claim depends on the Gaussian weighting surviving non-Gaussian rating noise; if σ_i is badly estimated (e.g., via Beta regression on sparse ratings), a non-parametric or heteroscedastic weighting would be a natural robustness check.","The surface could be inverted into a training signal: penalizing a model's low-difference-regime GMC during training might directly improve fine-grained discrimination rather than global SRCC.","Because GMC_g integrates over the whole MOS/ΔMOS domain, it effectively requires a test set spanning the full quality range; applying it to truncated or narrow-range datasets would need explicit domain extension, which the paper does not address."],"forward_implications":["Two IQA models with identical global SRCC can be separated by their correlation surfaces, enabling benchmarks to report regime-specific strengths instead of one ambiguous number.","Model selection becomes scenario-aware: the surface identifies models suited to high-MOS/fine-difference tasks (e.g., generative image evaluation) versus low-MOS/coarse-difference tasks (e.g., heavy compression).","GMC_g provides a global ranking that is more stable than SRCC when the test set's quality distribution shifts, making cross-dataset and re-sampled comparisons fairer.","The finding that all tested models degrade sharply at small |ΔMOS| indicates a concrete open problem: current IQA models lack fine-grained discrimination, and GMC makes that gap visible and measurable.","GMC-guided integration of complementary models (e.g., MS-SSIM with LPIPS) outperforms integration chosen by SRCC, suggesting the surface can drive ensembles."],"fun_headline_variants":["3D correlation surface reveals IQA models' true strengths and gaps","Beyond SRCC: a surface that maps IQA performance","IQA scorecards get a 3D makeover, not just a number","From single score to full map: IQA performance in 3D","Granular IQA: see where models fail across quality gaps"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The entire surface and the stability of GMC_g rest on treating each image's MOS as a Gaussian random variable with known variance σ_i^2, so that the granularity weights in Eqs. (13)–(15) faithfully represent perceptual relevance; if σ_i is missing or the Gaussian model is wrong, every local score inherits that error.","fun_headline_variants_meta":{"raw":{"variants":["3D correlation surface reveals IQA models' true strengths and gaps","Beyond SRCC: a surface that maps IQA performance","IQA scorecards get a 3D makeover, not just a number","From single score to full map: IQA performance in 3D","Granular IQA: see where models fail across quality gaps"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000767,"raw_usage":{"total_tokens":3282,"prompt_tokens":838,"completion_tokens":2444,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":582,"completion_tokens_details":{"reasoning_tokens":2352}},"tokens_in":582,"tokens_out":2444,"duration_ms":20885,"temperature":1.0,"reasoning_tokens":2352,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T06:50:48.100544+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure GMC on a dataset whose rating noise is deliberately non-Gaussian (e.g., heavy-tailed or bimodal within-image ratings) while keeping the same MOS values; if the surface and GMC_g change materially, the Gaussian weighting is the load-bearing assumption. A simpler calculation: replace σ_i in Eqs. (13)–(15) with a constant global σ across images and check whether conclusions about regime-specific strengths (e.g., CLIP-IQA vs NIQE) invert—if they do, the per-image σ estimates are driving the result, not the data.","supporting_citations":[],"review_version":1}