{"id":"a2c2daa0-73e3-4546-a55f-07332737629d","arxiv_id":"2608.12088","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"RA-CLIPScore extends CLIP-based generative model evaluation to attribute-specific and spatial distribution scoring via dual prompts and intermediate patch tokens.","lead":"RA-CLIPScore is a new metric that scores generated images attribute by attribute and region by region, using paired text prompts and local image patches from CLIP. The authors report that its spatial divergence score matches human diversity judgments perfectly and reveals where models like StyleGAN-XL and BigGAN place objects.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (6) never projects local patch tokens into CLIP's joint vision-language space; in released CLIP models their dimension differs from text embeddings, so Eq. (4) is undefined without an omitted projection, and every SaD/PaD/R-SaD result, including r=1.0, lacks semantic grounding.","rationale":"The reader's CONDITIONAL verdict already rests partly on this weak assumption, and my independent reading confirms it is not just an unstated detail: in released CLIP checkpoints, patch tokens at layer L-1 have a different dimensionality from the text embeddings, and no projection appears in Eq. (6). Since Eqs. (12)-(14) average these patch-level cosine similarities, the headline R-SaD r=1.0 result and the spatial-bias claims are not reproducible from the submitted description. The user-study design (20 binary choices, no confidence intervals, classes selected for spatial bias, r=1.0 fragile) and the ablation table (e.g., Male w/o LT=3.35 vs Full=3.16; Eyeglasses w/o GE=4.41 vs Full=3.03) are additional concerns, but the projection issue is more fundamental. A single shape check on an official checkpoint settles it; if the dimensions match in some unreleased variant, the authors need to name that variant and validate the semantic alignment. This is why I would move from CONDITIONAL to REJECT for the current manuscript: the core equations cannot be executed as written, not merely because evidence is incomplete.","tokens_in":20195,"tokens_out":8800,"duration_ms":80656,"concrete_test":"Load a released CLIP ViT-B/32 checkpoint, extract H^{L-1} for a batch of images, compute the adapted features per Eq. (6), and print their shape alongside Et('This is a photo of zebra.') from the same checkpoint. If the patch-token dimension is 768 and the text-embedding dimension is 512, Eq. (4) cannot be evaluated as written, confirming the missing projection. Then apply the CLIP visual projection to each patch token, rerun the Sec. 4.3 and Sec. 4.5 pipelines, and check whether Table 4 and Table 6 values change; any change shows the submitted results depend on the unspecified operation.","verdict_should_be":"REJECT","load_bearing_attack":"Eq. (4) defines the coarse attribute score as a softmax over cosine similarities between adapted patch-token features E_v(x_n[j]) and text-prompt embeddings E_t(Prompt). The paper claims (Sec. 3.1) that 'local patch tokens are projected directly into CLIP's unified vision-language embedding space,' but Eq. (6) defines the adapted features as H^L_dense = H^{L-1} + H^{L-1} W_V^L + MLP^L(H^{L-1} W_V^L) -- i.e., the standard transformer update, with no projection to the text-embedding dimension. In released CLIP ViT models (e.g., ViT-B/32), the visual hidden state is 768-d while the text-embedding output is 512-d, and only the final [CLS] token is passed through the visual projection head into the shared space. Consequently, as written, the cosine similarity in Eq. (4) cannot be computed; if a projection is silently assumed, it is not validated and the resulting scores are not necessarily aligned with CLIP's semantic space. Since SaD (Eq. 12), PaD (Eq. 13), and R-SaD (Eq. 14) all aggregate these patch-level scores, the reported r=1.0 human correlation and the spatial-bias analyses rest on an unspecified and, on released checkpoints, undefined operation. This is an internal consistency problem, not a disagreement with community consensus.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes RA-CLIPScore, a CLIP-based evaluation metric for generative image models that combines dual prompts (positive/negative attribute prompts) with local patch-token features to support attribute-wise and spatial distribution evaluation. The metric feeds refined patch-level scores into Single-Attribute Divergence (SaD), Pair-Attribute Divergence (PaD), and a new Regional Single-Attribute Divergence (R-SaD). The authors present experiments on CelebA, FFHQ, and ImageNet, an ablation study, injection controls, and a user study in which R-SaD reportedly achieves perfect correlation (r=1.0) with human judgments of sample diversity. The paper claims that RA-CLIPScore is more robust and interpretable than HCS and CLIPScore and reveals previously undocumented spatial biases in generative models without additional training or significant computational overhead.","tokens_in":2084,"tokens_out":2317,"duration_ms":66674,"significance":"If the load-bearing technical gaps are closed, RA-CLIPScore would be a genuinely useful contribution: it extends CLIP-based evaluation from global semantics to spatially localized attribute analysis, an underexplored direction, and it does so with a single forward pass and no learned parameters. The manuscript has clear strengths: a thoughtfully designed ablation (Table 3), well-controlled injection experiments (Fig. 3), an unusually detailed supplementary, and an attempted external check through human preferences. The claim of a new spatial-diversity dimension is interesting and, if substantiated, would be of value to the generative-model evaluation community. However, at present the method is not fully defined: Eq. (6) omits the projection that would place patch tokens in CLIP's joint embedding space, Eq. (8) contains a type error that makes the refinement step ambiguous, and the r=1.0 human-correlation figure is asserted on a small, possibly unrepresentative sample. These issues are correctable, but they currently block confident acceptance.","major_comments":[{"comment":"The central mechanism is incomplete: Eq. (6) defines the dense features as the output of the final transformer block with attention bypassed for local tokens, leaving the patch features in the visual transformer's hidden space. No projection to CLIP's text-embedding dimension or to the joint vision-language space is specified. In released CLIP ViT models the visual hidden dimension and the text-embedding dimension differ, and only the final CLS token is projected into the shared space. Consequently, the cosine similarity in Eq. (4) between the adapted visual encoder output for patch j and the text prompt embedding is undefined as written, and all quantities derived from it (SaD, PaD, R-SaD, and the r=1.0 result) inherit this gap. The authors must either specify the exact projection used (learned, fixed, or the existing CLIP visual projection head) and validate that the projected patch features are semantically aligned with text embeddings, or reformulate the scoring so that it operates in a well-defined space.","section":"Sec. 3.1, Eqs. (4) and (6)"},{"comment":"Equation (8) as printed is not implementable. The left side is the refined score for token j, but the right side multiplies the attention vector for token j (a row over all tokens) by the scalar coarse score for token j, and no summation over neighboring tokens is shown. The intended operation appears to be a weighted average of the coarse scores across tokens. As written, the definition of the refined scores that feed into RA-CLIPScore (Eq. 9) and R-SaD (Eq. 14) is ambiguous, and the reader cannot reproduce the experiments.","section":"Sec. 3.1, Eq. (8)"},{"comment":"The claim that R-SaD achieves perfect correlation (r=1.0) with human-perceived diversity is disproportionate to the evidence. The user study uses only 20 attributes, selects classes with clear spatial biases (Tab. 5), and reports an inter-rater agreement of only 72.9%. A perfect Pearson correlation under such conditions strongly suggests a small or non-representative sample, a selection artifact, or an effective tie in the majority votes. The authors should report the exact number of paired comparisons, a confidence interval or significance test for r, a per-attribute breakdown, and an analysis of agreement (e.g., Cohen's kappa) rather than only the correlation coefficient. The wording 'perfect correlation (r=1)' should be revised unless the analysis is substantially expanded.","section":"Sec. 4.5, Table 6"},{"comment":"The replacement of Gaussian KDE by a single Gaussian fit is asserted to be 'sufficient to capture distributional shifts' without evidence. The score distributions visualized in Fig. 4 appear asymmetric and in some cases bimodal, and a misspecified Gaussian model directly changes the KL-divergence values in Eqs. (12) and (13). The authors should justify the single-Gaussian assumption with a goodness-of-fit analysis or demonstrate that model rankings are unchanged when a more flexible estimator (e.g., KDE or a mixture model) is used. Without such validation, all SaD/PaD numbers in Tables 3, 4, and 8 are conditional on an unverified distributional assumption.","section":"Sec. 3.2 and Fig. 4"},{"comment":"The spatial-bias 'discoveries' (e.g., BigGAN's 45-degree guitar orientation, LDM's upward burrito shift) are outputs of the proposed metric alone; there is no external ground truth or independent localization check confirming that these heatmaps correspond to actual object-position differences between real and generated images. Since the spatial extension is a principal novelty of the paper, the authors should validate at least a few of the heatmaps against a detection-based or human-annotation-based position distribution, or explicitly frame Section 4.4 as a hypothesis-generating case study rather than a validated measurement.","section":"Sec. 4.4 and Fig. 5"}],"minor_comments":[{"comment":"There are several typos and inconsistencies: 'metri' in Section 4, 'explainations' in Section 4.1, and the terms 'R-SAD' and 'R-SaD' are used interchangeably; the abbreviations should be unified.","section":"General"},{"comment":"Reference [44] and [45] both point to the same paper (Rombach et al., CVPR 2022); one duplicate should be removed and the citation indices adjusted accordingly.","section":"References"},{"comment":"The description of the attention mask is not precise: the text says the first row contains all 0s and that the local submatrix becomes an identity, but the displayed matrix has -inf values on the diagonal of the local block, which would prevent tokens from attending to themselves and contradict the 'identity' statement. The mask should be specified explicitly.","section":"Eq. (7)"},{"comment":"In Eq. (14), the notation for the refined regional scores on the generated set is used, but Eq. (8) was introduced only for the training set; the authors should define the analogous refinement for generated samples consistently.","section":"Eq. (14)"},{"comment":"The text 'dual prompts: first refine with attention, then softmax' appears in the heatmap figures and seems to be a leftover debugging annotation rather than a figure caption; it should be removed.","section":"Fig. 5"}],"recommendation":"major_revision","confidential_remarks":"The paper has a promising core idea and a careful experimental apparatus, but the omitted projection in Eq. (6) is an internal-consistency issue that must be fixed before the results can be trusted. The r=1.0 user-study correlation is also a red flag that deserves closer editorial scrutiny; I would ask the authors to make the study data and per-class results available in the revision. The paper is within the scope of the journal and, after addressing these points, could become a publishable contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper has a genuinely useful idea: use dual prompts and intermediate patch tokens to turn CLIP into an attribute-wise and spatially aware evaluator for generative models. The R-SaD divergence is simple and the spatial heatmaps are a nice diagnostic. The injection experiments and the robustness check against irrelevant attributes are well done. I think the core concept is worth pursuing.\n\nThe problem is that the stress-test note is right. Eq. (6) claims to project local patch tokens into CLIP's unified vision-language space, but it only applies a residual value projection and an MLP. In released ViT-B/32, the visual hidden state is 768-d and the text embedding is 512-d; only the final CLS token goes through the projection head. So the cosine similarity in Eq. (4) cannot be computed as written. This is not a style complaint; it means every SaD, PaD, and R-SaD number, including the r=1.0 human correlation, is semantically ungrounded until the authors specify and validate an actual projection. The Limitation section is honest about CLIP's weaknesses but silent on this gap.\n\nOther soft spots are less severe but still real. The user study reports perfect correlation from 29 participants answering 20 binary forced choices, with no error bars. r=1.0 over 20 data points, where many metrics tie or random choices would be common, is not credible as reported. The ablation table also contradicts the \"all components are necessary\" summary: removing local tokens improves Male separation, and removing global embeddings improves Eyeglasses. That is a minor explanatory flaw, but it should be fixed. And the paper gives no code or data, which matters for a metric paper.\n\nOn the positive side, the paper is clearly written, the related work is relevant, and the proposal is a genuine extension of HCS/CLIPScore. The spatial-bias direction is important and underexplored. The circularity burden is mild: the design choices are selected on the same benchmarks, but the benchmarks include external labels and human judgments.\n\nMy recommendation: send it to peer review, but as major revision. The idea deserves referee time; the projection issue must be addressed, the user study needs proper uncertainty quantification, and the ablation interpretation should match the table. If those are fixed, this could be a solid contribution to the generative evaluation toolbox.","headline":"Strong idea for attribute- and spatial-level generative evaluation, but the missing projection in Eq. (6) is a load-bearing gap that undermines the quantitative claims until fixed.","tokens_in":21048,"tokens_out":2403,"would_cite":false,"duration_ms":22220,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"RA-CLIPScore claims to make generative-model evaluation interpretable by scoring each attribute independently through dual prompts and pooling local patch tokens; its spatial component, R-SaD, is reported to match human diversity…","keywords":["generative model evaluation","CLIP","attribute-wise metrics","spatial diversity","interpretability","dual prompts","R-SaD","explainable evaluation metrics"],"falsifier":"Implement the paper's Eq.~(6) on a released CLIP ViT and test whether patch-token--text-prompt cosine similarities can localize attributes where ground truth is known, such as object bounding boxes in COCO or attribute regions in CelebA; near-chance retrieval would falsify the joint-space assumption. A second check is to rerun the diversity user study with a larger participant pool and more attributes, since $r=1.0$ on 29 raters and 20 attributes is fragile.","tokens_in":19900,"feed_emoji":"📊","tokens_out":9188,"duration_ms":79209,"temperature":0.7,"pith_summary":"This paper tries to make generative-model evaluation explain why a model fails, not just how far its outputs sit from real images. It proposes RA-CLIPScore, a CLIP-based metric that scores each attribute independently through positive/negative prompt pairs and pools local patch tokens so the score is grounded in image regions. The authors claim this yields attribute-wise divergences, SaD and PaD, that stay stable when unrelated attributes are removed, and it extends CLIP-based evaluation to spatial distribution alignment through Regional Single-Attribute Divergence (R-SaD). If the claims hold, practitioners could diagnose biases such as objects always generated at fixed positions, which scalar scores like FID hide. The paper reports that R-SaD matched human judgments of sample diversity perfectly ($r=1$) in its user study, while standard diversity metrics scored between $0.29$ and $0.7$.","feed_headline":"Spatial diversity metric matches human raters perfectly","feed_subtitle":"RA-CLIPScore scores images attribute-by-attribute and region-by-region, exposing biases like fixed-angle objects.","key_machinery":"The mechanism is a modified final block of CLIP's visual encoder combined with dual prompts. In the paper's Eq.~(6)-(7), the attention mask nulls attention among local patch tokens so they pass through the last residual block without the global mixing of the final attention layer, while the [CLS] token still aggregates global context; the output is the global token concatenated with the dense local tokens, $\\tilde{E}_v(X)=[H^L[0],\\,H^L_{\\text{dense}}[1:]]$. Each attribute is turned into two fixed prompts, a positive and a negative, and the softmax of their cosine similarities with each patch token gives a binary presence score per region. Attention maps from earlier layers refine those patch scores, a softmax-weighted sum over patches pools them into one per-attribute score, and KL divergences (SaD, PaD) or per-patch mean differences (R-SaD) turn the scores into dataset-level diagnostics that can be drawn as heatmaps.","core_discovery":"RA-CLIPScore replaces the single global embedding used by CLIPScore with a representation formed jointly by the final [CLS] token and dense local patch tokens taken from layer $L-1$ with the last attention operation bypassed. Each attribute $t_i$ is scored by a softmax over a positive prompt (``This is a photo of $t_i$'') and a negative prompt (``This is a photo without $t_i$''), and the regional scores are refined by attention and aggregated by patch-to-prompt similarity. The paper's central claim is that this decouples attributes that softmax contrastive training entangles, stays stable under attribute-set perturbation where HCS's mean-centering provably collapses into a two-attribute zero-sum relation, and reveals spatial biases such as BigGAN's tendency to draw electric guitars at a fixed $45^\\circ$ diagonal. In the paper's user study, R-SaD achieves a Pearson correlation of $1.0$ with human preferences for sample diversity, against $0.29$-$0.7$ for FID, KID, LPIPS, precision/recall, density, and coverage.","pith_inferences":["The $r=1.0$ result rests on 29 participants and 20 attributes; if it replicates with larger pools, spatial diversity should become a standard axis in generative evaluation, but the effect size needs re-measurement.","The paper's zero-sum argument against HCS applies generally: any attribute metric built on mean-centering CLIP embeddings in a two-attribute setting forces the scores to be exact opposites, so the fragility is not specific to HCS's particular implementation.","If the layer $L-1$ joint-space assumption fails for released CLIP models, the regional scores are cosine similarities in a space not semantically aligned with text; a projection head or a learned linear probe on patch tokens would be the minimal fix.","A cheap test of the spatial claim: retrain or fine-tune a generator on data with deliberately perturbed object positions and check whether R-SaD heatmaps track the perturbation magnitude per patch."],"forward_implications":["Attribute-wise evaluation becomes stable under attribute-set changes: RA-CLIPScore scores resist removal of a single attribute, where HCS shifts noticeably.","Spatial placement can be audited per class with no extra training or second forward pass; the paper reports distinct regional biases for StyleGAN-XL, BigGAN, and LDM.","Diversity judgments that ignore layout are incomplete: R-SaD's reported perfect human correlation implies where an object appears is part of perceived diversity.","SaD and PaD rise monotonically when biased images are injected into a dataset and stay flat for neutral injections, so they can serve as controlled sanity checks.","Because the aggregation runs in one forward pass and uses only CLIP features, the metric adds negligible cost over a plain CLIPScore evaluation."],"supporting_citations":[{"why":"CLIP's joint vision-language embedding space and encoders are the backbone of every score in RA-CLIPScore.","marker":"[41]"},{"why":"CLIPScore is the global semantic baseline that RA-CLIPScore extends and compares against.","marker":"[16]"},{"why":"HCS is the attribute-wise baseline; the paper proves its mean-centering collapses for two attributes and beats it on stability.","marker":"[24]"},{"why":"Dual prompts originate from DualCoop; the paper adapts them with fixed text to define presence and absence per attribute.","marker":"[51]"},{"why":"TagCLIP motivates local-to-global patch usage; the paper contrasts its separate-pass cost with the single-pass trick.","marker":"[31]"},{"why":"StyleGAN-XL provides ImageNet samples whose spatial bias R-SaD exposes, such as rotary dial phones.","marker":"[49]"},{"why":"BigGAN's generated electric guitars at a fixed 45-degree angle serve as a spatial-bias case study.","marker":"[4]"},{"why":"LDM supplies both FFHQ and ImageNet generations used in attribute, spatial, and user-study evaluations.","marker":"[45]"},{"why":"FID is the standard scalar metric that RA-CLIPScore is designed to supplement and that the user study benchmarks against.","marker":"[17]"},{"why":"LPIPS is a perceptual diversity baseline that the user study compares with R-SaD.","marker":"[58]"}],"fun_headline_variants":["Spatial scoring metric reveals hidden biases in generative models","Patch-wise metric matches human perception of image diversity","RA-CLIPScore decouples attributes and exposes spatial bias","New metric matches human raters on diversity","Interpretable evaluation reveals fixed-angle object bias"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that CLIP's layer $L-1$ patch tokens, after passing through the modified residual block (value projection plus MLP, no attention), lie in CLIP's shared vision-language embedding space, so cosine similarity with text prompts is semantically meaningful; in released CLIP models only the final [CLS] token is projected into that joint space.","fun_headline_variants_meta":{"raw":{"variants":["Spatial scoring metric reveals hidden biases in generative models","Patch-wise metric matches human perception of image diversity","RA-CLIPScore decouples attributes and exposes spatial bias","New metric matches human raters on diversity","Interpretable evaluation reveals fixed-angle object bias"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000795,"raw_usage":{"total_tokens":3517,"prompt_tokens":981,"completion_tokens":2536,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":597,"completion_tokens_details":{"reasoning_tokens":2461}},"tokens_in":597,"tokens_out":2536,"duration_ms":19209,"temperature":1.0,"reasoning_tokens":2461,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:16:42.245627+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Implement the paper's Eq.~(6) on a released CLIP ViT and test whether patch-token--text-prompt cosine similarities can localize attributes where ground truth is known, such as object bounding boxes in COCO or attribute regions in CelebA; near-chance retrieval would falsify the joint-space assumption. A second check is to rerun the diversity user study with a larger participant pool and more attributes, since $r=1.0$ on 29 raters and 20 attributes is fragile.","supporting_citations":[{"cited_title":"In:Proceedingsofthe 41stInternational ConferenceonMachine Learning","cited_arxiv_id":null,"evidence_quote":"HCS is the attribute-wise baseline; the paper proves its mean-centering collapses for two attributes and beats it on stability."},{"cited_title":"In: Advances in Neural Information Processing Systems (2022)","cited_arxiv_id":null,"evidence_quote":"Dual prompts originate from DualCoop; the paper adapts them with fixed text to define presence and absence per attribute."},{"cited_title":"In: Proceedings of the AAAI Conference on Artificial Intelligence","cited_arxiv_id":null,"evidence_quote":"TagCLIP motivates local-to-global patch usage; the paper contrasts its separate-pass cost with the single-pass trick."},{"cited_title":"In: IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR)","cited_arxiv_id":null,"evidence_quote":"LDM supplies both FFHQ and ImageNet generations used in attribute, spatial, and user-study evaluations."},{"cited_title":"In: Advances in Neural Information Processing Systems (2017)","cited_arxiv_id":null,"evidence_quote":"FID is the standard scalar metric that RA-CLIPScore is designed to supplement and that the user study benchmarks against."}],"review_version":1}