{"id":"e9310d86-8ebf-4e70-81b3-8c2a628b9755","arxiv_id":"2412.14580","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"A diffusion U-Net's attention features, aligned with a bidirectional attention score, can rank visual similarity competitively with CLIP, DINO, and LPIPS.","lead":"DiffSim uses Stable Diffusion's attention-layer features to score how similar two images are, aligning the images through attention before comparing them. The paper reports that this training-free metric matches or beats CLIP, DINO, and LPIPS on human, style, and instance similarity benchmarks, and introduces two new benchmarks for style and instance consistency.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Per-benchmark grid search on test sets inflates DiffSim's SOTA margins; a fixed-configuration evaluation is needed to validate the central claim.","rationale":"The reader's weakest assumption identifies the most load-bearing issue in the paper: the per-benchmark grid search over layer and timestep on the test set. This is not a minor methodological quibble; it directly undermines the state-of-the-art claim because the metric's hyperparameters are fitted to the evaluation data. The paper discloses the practice, but disclosure does not mitigate the overfitting. The narrow margins on key benchmarks make the results fragile, and the supplementary analysis shows extreme sensitivity to the chosen configuration. The self-built Sref and IP benchmarks are also a concern, but they are secondary because the method also claims superiority on established benchmarks like NIGHTS, where the test-set selection issue is sufficient to invalidate the claim. I considered whether other flaws, such as the missing AAS wrapper in Equation 11 or the stochastic noise injection, are more damaging, but the test-set selection is the clear primary threat: if a fixed configuration performs comparably to baselines, the central contribution of a training-free SOTA metric collapses. The proposed concrete test—evaluating with a fixed configuration or with held-out hyperparameter selection—would settle the question directly. Therefore, the reader's REJECT verdict remains justified, and no change to the verdict is needed.","tokens_in":15916,"tokens_out":4256,"duration_ms":31157,"concrete_test":"Evaluate DiffSim with a single fixed configuration (e.g., self-attention layer U_0, timestep t=600, resolution 512) on all seven benchmarks, without per-benchmark tuning, and compare against the Table 1 baselines. If the fixed configuration does not beat DINO v2 and CLIP on NIGHTS, IP, Sref, and InstantStyle by at least the reported margins, then the SOTA claim depends on test-set grid search. As a robustness check, also split each test set in half, select layer/timestep on the first half, and report accuracy on the second half; if the held-out accuracy drops below the baselines, the central claim is refuted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim that DiffSim achieves state-of-the-art visual similarity rests on per-benchmark selection of U-Net layer and denoising timestep on the test set. Section 4.1 explicitly says, \"We also conducted grid searches across different layers and denoising time steps for each task, reporting the best results among these choices.\" Because the layer and timestep are hyperparameters of the metric, choosing them per benchmark from the same test set used for evaluation constitutes test-set overfitting. The margins in Table 1 are small where DiffSim wins: NIGHTS 86.52% vs. DINO v2 85.24% (1.28 points), IP 91.84% vs. CLIP 91.70% (0.14 points), Sref 97.40% vs. DINO v2 AAS 95.90% (1.50 points), InstantStyle 99.05% vs. CLIP AAS 97.80% (1.25 points). Moreover, DiffSim is not uniformly SOTA: LPIPS beats it on TID2013 (94.50% vs. 94.17%), DINO v2 beats it on CUTE (77.27% vs. 76.17%) and Dreambench++ (72.25% vs. 71.50%). The supplementary figures show that accuracy varies by tens of points across timesteps and layers (e.g., NIGHTS ranges from roughly 65% to 86% in Figure 7), so the chosen configuration substantially determines the result. Without a held-out or fixed configuration, the claimed superiority is not established. Additionally, the method injects noise before feature extraction, yet no averaging over noise samples or error bars are reported, which could further affect the narrow margins.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"DiffSim proposes a training-free visual similarity metric that extracts features from the attention layers of a pretrained Stable Diffusion U-Net and computes an Aligned Attention Score (AAS) to handle spatial misalignment. The paper claims that this is the first use of pretrained diffusion models for visual similarity assessment, reports state-of-the-art accuracy on human-align, style, and instance benchmarks, and introduces two new benchmarks, Sref and IP bench, for style and instance consistency evaluation. The method is evaluated against CLIP, DINO v2, LPIPS, FFA, and Gram on seven benchmarks, with additional ablations, video consistency experiments, and image retrieval results.","tokens_in":16315,"tokens_out":4930,"duration_ms":66807,"significance":"If the empirical claims are correct, DiffSim would be a valuable addition to the toolkit for evaluating generated images: it is training-free, requires no fine-tuning, and leverages the rich appearance information in diffusion U-Net features. The AAS alignment mechanism is simple and is shown to also improve CLIP and DINO v2 when applied to their attention layers. The two new benchmarks, Sref and IP bench, address an important gap in style and instance consistency evaluation. However, the reported state-of-the-art status rests on a protocol that selects hyperparameters on the test set, and the margins over strong baselines are often very small. Without a corrected evaluation protocol and uncertainty quantification, the central contribution is not yet established.","major_comments":[{"comment":"The authors state that they \"conducted grid searches across different layers and denoising time steps for each task, reporting the best results among these choices.\" This is test-set hyperparameter selection: the U-Net layer and denoising timestep are chosen on the same benchmark used for evaluation. Supplementary Figures 7-13 show that accuracy varies by tens of percentage points across timesteps and layers (for NIGHTS, the range is roughly 65% to 86%). The winning margins over the next-best baseline are small (e.g., 1.28 points on NIGHTS, 0.14 points on IP). A state-of-the-art claim requires a fixed configuration chosen without access to test labels, or a nested validation protocol with the selected configuration reported for each benchmark. As written, the comparison is optimistically biased and the central claim is not established.","section":"Section 4.1, Table 1, Supplementary Figures 7-13"},{"comment":"No error bars, significance tests, or repeated-noise averaging are reported. DiffSim injects noise into the input images before feature extraction, yet each accuracy appears to come from a single run without multiple noise realizations or seeds. Several decisive margins are very narrow (IP: 91.84% vs 91.70% for CLIP; NIGHTS: 86.52% vs 85.24% for DINO v2). Without bootstrap confidence intervals or paired significance tests, it is unclear whether these differences are statistically meaningful. The authors should report repeated runs with different noise samples and provide uncertainty estimates.","section":"Section 4.4, Table 1"},{"comment":"The two newly introduced benchmarks, Sref bench and IP bench, are constructed with diffusion-based generators (Midjourney, Flux, and IP-Adapter). Because DiffSim extracts features from the denoising U-Net of Stable Diffusion, there is a risk that the benchmark favors the proposed method by construction, since the reference and candidate images share the same generative prior. The paper does not analyze this potential bias. A concrete test would be to evaluate on non-diffusion style and instance benchmarks, or to show that the performance advantage persists when candidate images are produced by GANs or other non-diffusion generators.","section":"Section 3.4, Section 4.4"},{"comment":"The abstract claims \"state-of-the-art performance\" across benchmarks, but Table 1 shows that DiffSim is not the best method on several benchmarks: LPIPS outperforms on TID2013 (94.50% vs 94.17%), and DINO v2 outperforms on CUTE (77.27% vs 76.17%) and Dreambench++ (72.25% vs 71.50%). The text describes the Dreambench++ result as \"comparable to\" DINO v2, which is misleading given the numerical gap. The claims should be scoped to the benchmarks where DiffSim actually leads, or the method should be improved to be competitive across all evaluated settings.","section":"Abstract, Section 4.4, Table 1"}],"minor_comments":[{"comment":"There is a typo in the Introduction: \"video gneration\" should be \"video generation.\"","section":"Section 1"},{"comment":"The word \"caculate\" should be \"calculate.\"","section":"Section 3.3.1"},{"comment":"Equation (11) appears to be missing the AAS operator in the second term: it reads \"+zB_t,cross,n, IPA)\" but should be \"+AAS(zB_t,cross,n, IPA)).\"","section":"Equation (11)"},{"comment":"There are typos in the experimental section: \"orginal\" should be \"original,\" and \"stylish similarity\" should be \"style similarity.\"","section":"Section 4.4"},{"comment":"The caption contains a typo: \"middel\" should be \"middle.\"","section":"Figure 4 caption"},{"comment":"References [53] and [54] are duplicate entries for the same IP-Adapter paper; they should be consolidated into a single reference.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper's central empirical claim is undermined by the per-benchmark test-set grid search, which likely inflates DiffSim's accuracy. The authors should be asked to re-run all experiments with a fixed configuration chosen without test labels, or to use a proper validation split, and to report uncertainty. The novelty claim of being \"the first\" to use pretrained diffusion models for visual similarity should also be carefully checked against existing work such as CSD and other diffusion-feature-based metrics. In addition, the new Sref and IP benchmarks are generated by diffusion models, which creates a potential circularity that needs explicit analysis. These issues are substantial but addressable through additional experiments and revisions; if the corrected numbers no longer support state-of-the-art claims, the paper may need to be reframed as a method presentation rather than a SOTA claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Core idea is worth a serious look. AAS — aligning U-Net features via attention before cosine similarity — is a simple, training-free trick that the ablation shows is doing real work: Table 3 shows drops of roughly ten points without it. The generalization to CLIP and DINO also shows gains on several tasks. Code is released, and the Sref and IP benchmarks are plausible additions to the evaluation toolbox, even though they need more scrutiny.\n\nThe soft spot is the evaluation protocol, and it is load-bearing. Section 4.1 admits they grid-searched layers and timesteps per task and report the best results. That is test-set selection. The margins where DiffSim wins are thin — IP 91.84 vs 91.70, NIGHTS 86.52 vs 85.24 — and on CUTE and TID2013 it loses to baselines. The supplementary figures show accuracy swinging tens of points across timesteps and layers, so the configuration largely decides whether DiffSim looks SOTA. No error bars either, and the method injects noise, so a few seeds could flip narrow wins. The SOTA claim is not supported as stated.\n\nThe 'first to discover' framing is also inconsistent: they cite CSD for diffusion-based style similarity but the abstract still claims to be the first to use pretrained diffusion models for visual similarity. That is sloppy. The circularity worry about the self-built diffusion-generated benchmarks is real but not fatal; they could still be useful benchmarks if the construction and labels are sound.\n\nThe fix is straightforward: fix one layer and timestep, or choose on a validation split, and report all benchmarks with that same configuration, with repeated noise draws for error bars. If the lead holds, great; if not, the paper becomes an honest 'configuration-sensitive metric' paper, which would still be a contribution.\n\nFor whom: people building evaluation metrics for generative models, especially style and identity consistency. It deserves a serious referee, not a desk reject — the evaluation problem is real but fixable. Make the referee check the fixed-configuration numbers.","headline":"The attention-aligned diffusion feature trick is new and useful, but the per-benchmark test-set grid search makes the SOTA claims untrustworthy.","tokens_in":16768,"tokens_out":3121,"would_cite":true,"duration_ms":21481,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Diffusion U-Net attention can judge image similarity.","keywords":["visual similarity","diffusion models","perceptual metrics","Aligned Attention Score","style similarity","instance similarity","generative image evaluation"],"falsifier":"Re-run the NIGHTS, Sref, and IP comparisons with DiffSim configured by a single layer and timestep chosen on a validation split before seeing test labels; if NIGHTS accuracy drops from 86.52% to at or below DINO v2's 85.24%, the headline state-of-the-art claim depends on test-set selection.","tokens_in":15726,"feed_emoji":"🖼️","tokens_out":5388,"duration_ms":44405,"temperature":0.7,"pith_summary":"This paper claims that the features inside a pretrained text-to-image diffusion model, specifically the attention layers of its denoising U-Net, can be repurposed as a perceptual similarity metric without any fine-tuning. The proposed DiffSim score aligns two images through attention before comparing them, which the authors argue captures both appearance and style while avoiding the pixel-alignment problem of low-level metrics and the information compression of CLIP and DINO. Across human-judgment, style, instance, and low-level benchmarks, DiffSim is reported to match or beat CLIP, DINO v2, and LPIPS, and an ensemble with CLIP and DINO improves further. The paper also introduces two new benchmarks, Sref for style and IP for instance consistency, to test similarity in custom generation settings. If correct, this gives generative-model practitioners a training-free way to check that generated images stay visually consistent with references.","feed_headline":"Diffusion models can now grade image similarity","feed_subtitle":"DiffSim's attention-aligned score beats CLIP, DINO, and LPIPS on human-aligned benchmarks.","key_machinery":"The load-bearing object is the Aligned Attention Score (AAS), computed inside an attention layer of a pretrained diffusion U-Net. For two images A and B, AAS takes A's query against B's keys and values and B's query against A's keys and values, then averages the cosine similarity of the two attended outputs; this symmetric cross-attention alignment is what replaces pixel-aligned feature comparison. DiffSim-S applies AAS in self-attention layers of Stable Diffusion 1.5, while DiffSim-C applies it in cross-attention layers using IP-Adapter Plus image tokens. The layer index and denoising timestep act as a control dial: shallow blocks and high timesteps correlate with style and low-level appearance, and deep blocks and low timesteps correlate with semantic and instance content.","core_discovery":"DiffSim's central claim is that the self-attention and cross-attention layers of a Stable Diffusion U-Net carry features that can measure human-perceived visual similarity. The paper introduces the Aligned Attention Score (AAS), which feeds image A's query features to image B's key/value features in an attention layer and symmetrically does the reverse, then averages the cosine similarities of the two attention outputs. This implicit alignment lets the metric compare images that differ in layout, pose, or style without first registering pixels. By varying which U-Net block and which denoising timestep are used, the same method can emphasize low-level/style similarity (shallow layers, high timesteps) or semantic/instance similarity (deeper layers, lower timesteps). The paper reports that DiffSim outperforms CLIP, DINO v2, and LPIPS on the NIGHTS human-alignment benchmark and on its new Sref and IP style/instance benchmarks, and that applying AAS to CLIP and DINO v2 also improves them on some tasks.","pith_inferences":["A practical version of DiffSim would need a fixed default layer and timestep chosen on validation data; the paper's reported numbers select the best setting on each test set, so a frozen configuration would likely change the exact margins.","Because AAS also helps CLIP and DINO, the active ingredient may be attention-based feature alignment rather than diffusion-specific knowledge; a control experiment using random or weakly trained attention features would isolate this.","The observed layer and timestep gradient suggests a multi-scale score that pools features across several blocks and timesteps could be more robust than any single setting, though the paper does not test this.","The approach could extend to video diffusion models directly, evaluating frame-to-frame consistency with the same attention alignment rather than a separate video metric."],"forward_implications":["A training-free perceptual metric for generated images becomes available directly from the U-Net already used to create those images, so no extra data labeling or model training is needed.","The same metric can be tuned by layer and timestep to measure style, instance identity, or low-level distortion, replacing several specialized metrics with one configurable score.","AAS is architecture-agnostic enough to improve CLIP and DINO v2 on style and low-level tasks, suggesting attention alignment is a generally useful add-on for similarity scoring.","An ensemble of CLIP, DINO v2, and DiffSim votes to higher accuracy on several benchmarks, meaning DiffSim adds signal not already captured by existing metrics.","For video, AAS-based scores have lower variance across frames of the same subject, which the paper interprets as better robustness to motion and layout change."],"supporting_citations":[{"why":"Supplies the pretrained Stable Diffusion U-Net whose attention features DiffSim repurposes as a similarity metric.","marker":"[38]"},{"why":"Provides IP-Adapter Plus image tokens used by DiffSim-C in cross-attention layers.","marker":"[53]"},{"why":"Provides the NIGHTS human-similarity benchmark and the evaluation protocol DiffSim is measured against.","marker":"[12]"},{"why":"DINO v2 is a principal baseline whose self-supervised features DiffSim must beat and which AAS is also applied to.","marker":"[32]"},{"why":"CLIP is a principal baseline for semantic similarity and one component of the ensemble.","marker":"[37]"},{"why":"LPIPS is the standard low-level perceptual baseline that DiffSim compares against on TID2013 and other tasks.","marker":"[57]"},{"why":"CUTE supplies the object-instance similarity benchmark and the FFA baseline.","marker":"[26]"},{"why":"InstantStyle supplies the style benchmark and its generated style-consistency test set.","marker":"[48]"},{"why":"Provides the Dreambench++ human-aligned benchmark for personalized image generation.","marker":"[34]"}],"fun_headline_variants":["Diffusion models judge image similarity via aligned attention","DiffSim: align attention layers to measure visual similarity","Beyond CLIP and DINO: diffusion attention scores human similarity","DiffSim uses Stable Diffusion to score style and instance similarity","Attention alignment turns diffusion models into similarity judges"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported state-of-the-art results assume that picking the best U-Net layer and denoising timestep by grid search on each benchmark's own test set is a fair way to estimate DiffSim's performance.","fun_headline_variants_meta":{"raw":{"variants":["Diffusion models judge image similarity via aligned attention","DiffSim: align attention layers to measure visual similarity","Beyond CLIP and DINO: diffusion attention scores human similarity","DiffSim uses Stable Diffusion to score style and instance similarity","Attention alignment turns diffusion models into similarity judges"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000201,"raw_usage":{"total_tokens":1381,"prompt_tokens":952,"completion_tokens":429,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":568,"completion_tokens_details":{"reasoning_tokens":353}},"tokens_in":568,"tokens_out":429,"duration_ms":4643,"temperature":1.0,"reasoning_tokens":353,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T12:05:41.492794+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the NIGHTS, Sref, and IP comparisons with DiffSim configured by a single layer and timestep chosen on a validation split before seeing test labels; if NIGHTS accuracy drops from 86.52% to at or below DINO v2's 85.24%, the headline state-of-the-art claim depends on test-set selection.","supporting_citations":[{"cited_title":"Learning transferable visual models from natural language supervision","cited_arxiv_id":null,"evidence_quote":"CLIP is a principal baseline for semantic similarity and one component of the ensemble."},{"cited_title":"The unreasonable effectiveness of deep features as a perceptual metric","cited_arxiv_id":null,"evidence_quote":"LPIPS is the standard low-level perceptual baseline that DiffSim compares against on TID2013 and other tasks."},{"cited_title":"Are these the same apple? comparing images based on object intrinsics","cited_arxiv_id":null,"evidence_quote":"CUTE supplies the object-instance similarity benchmark and the FFA baseline."}],"review_version":1}