{"id":"0b736a9e-d8db-4414-9528-9ed0da1992fc","arxiv_id":"2504.17234","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"SPIPS fuses traditional and deep features, separated into semantic and perceptual streams, to predict human-perceived image quality and reports small gains over LPIPS on BAPPS.","lead":"SPIPS is a new image quality metric that combines per-pixel traditional scores (PSNR, SSIM, MS-SSIM) with deep features split into perceptual and semantic layers, then fuses them with learned weights. Tests on the BAPPS benchmark suggest slightly better agreement with human judgments than LPIPS and DISTS, though the gains are small and training details are missing.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Main tables may not measure what they claim: 2AFC ground truth is binary preference, yet PLCC/SRCC/KRCC are reported without defining the scalar labels or computation; central 'highest correlation' claim is unverifiable.","rationale":"I read the paper as a hybrid FR-IQA method whose main contribution is a fusion of traditional metrics, low-level deep features, and high-level semantic features, evaluated on BAPPS. The central claim is improved human consistency. For that claim, Tables V-VII are the key evidence. The most load-bearing issue is that the evaluation protocol for these tables is not specified: BAPPS 2AFC labels are binary preferences, and computing Pearson/Spearman/Kendall correlations requires a scalar ground truth that is never defined. A reader cannot determine whether the reported numbers are meaningful or comparable to standard IQA correlations. If the standard 2AFC accuracy were computed, it might or might not support the claim. This is more fundamental than the layer-split heuristic flagged by the reader, although the split is also a valid concern: without an ablation over split points, the semantic/perceptual story is not established. Additionally, the text explicitly asserts 'data-free' while also describing training on BAPPS, a contradiction that further erodes confidence in the reported comparison. I do not accuse the authors of wrongdoing; the paper simply lacks the definitions and code needed to verify the central claim. The proposed check - re-evaluating with standard preference accuracy and publishing the exact correlation definitions - would settle the issue. If the standard metric reproduces the claim, the verdict should be CONDITIONAL or ACCEPT; if not, it should be rejected or revised.","tokens_in":13000,"tokens_out":7455,"duration_ms":76266,"concrete_test":"Re-run the BAPPS 2AFC evaluation using the standard protocol: for every trial in the test split, compute SPIPS's predicted distance d(ref, image0) and d(ref, image1); mark the trial as correct when the image preferred by human raters has the smaller predicted distance. Report per-category accuracy for SPIPS, LPIPS, DISTS, PSNR, SSIM, VIF, and the two ablations, and also report the exact definitions of the scalar 'ground-truth scores' behind Tables V-VII. If SPIPS is not first in per-category accuracy, or if the PLCC/SRCC/KRCC values cannot be reproduced from the stated definitions, the central claim is not supported.","verdict_should_be":"UNVERDICTED","load_bearing_attack":"The central claim is the BAPPS 2AFC result in Section IV-D: 'our model consistently achieves the highest correlation across all categories.' The load-bearing evidence is Tables V-VII, which report PLCC, SRCC, and KRCC on 2AFC trials. However, BAPPS 2AFC provides binary human preferences (which of two images is closer to the reference), not scalar human quality scores. The paper never defines what ground-truth vector is correlated with the model scores, nor how PLCC/SRCC/KRCC are computed for paired-preference data. The standard evaluation for 2AFC is per-trial preference accuracy (the fraction of trials where the model's distance ranks the human-preferred image closer to the reference), sometimes reported as a rank correlation over trials; PLCC against a binary label is point-biserial and cannot be compared to the usual IQA PLCC values. Without the exact protocol, 'consistently highest correlation' is not a well-defined statement. This concern is compounded by the paper's own contradiction: Section II.A calls the method 'data-free' and Section II.B says 'our method requires no training at all,' while Section IV.A states BAPPS 'is used to train and validate SPIPS model' and the Figure 2 caption says parameters are 'iteratively optimized via backpropagation.' Thus the reader cannot tell whether the reported gains come from a clearly specified evaluation or from a trained model compared against fixed baselines under an ad hoc metric.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SPIPS, a full-reference image quality assessment (IQA) metric that fuses three streams: per-pixel traditional IQA maps (PSNR, SSIM, MS-SSIM), low-level perceptual deep features, and high-level semantic deep features extracted from a pretrained CNN. The deep features are split by layer index into perceptual and semantic groups, each passed through its own convolution block, and the final score is a weighted combination of the three streams. The authors train the model on the BAPPS dataset and report PLCC, SRCC, and KRCC on the BAPPS 2AFC and JND subsets, claiming that SPIPS consistently achieves the highest correlation with human judgments among compared methods (PSNR, SSIM, VIF, DISTS, LPIPS). The paper presents qualitative examples, two ablation studies, and a conclusion outlining future work.","tokens_in":13313,"tokens_out":3856,"duration_ms":38726,"significance":"If the reported results are reproducible and the evaluation protocol is well defined, the central idea of explicitly separating semantic from perceptual deep features and combining them with traditional metrics is a plausible and potentially useful contribution to full-reference IQA, especially for DNN-generated and DNN-processed images. The qualitative examples in Figures 3 and 4 are illustrative, and the ablation study is the right kind of evidence for the fusion design. However, the manuscript currently contains a direct contradiction between the claimed 'data-free' / 'no training' status of the method and the explicit description of training on BAPPS, and it leaves the evaluation protocol for 2AFC data unspecified. These issues prevent verification of the central claim and must be resolved before the contribution can be assessed.","major_comments":[{"comment":"The evaluation protocol for the 2AFC subset is not defined. BAPPS 2AFC provides binary human preferences (which of two images is closer to the reference), not scalar quality labels. PLCC, SRCC, and KRCC require scalar ground-truth values, and the paper never states how the binary preferences are converted into the ground-truth vectors used in Tables V-VII, nor how the model's scores for image0 and image1 are aggregated into a per-trial or per-image prediction. Without this description, 'our model consistently achieves the highest correlation across all categories' is not a well-defined, reproducible claim. The authors should specify the exact aggregation and correlation computation, and note whether the values are comparable to standard IQA PLCC/SRCC numbers.","section":"Section IV.A, Section IV.B, Tables V-VII"},{"comment":"The claim that the method is 'data-free' and 'requires no training at all' is contradicted by Section IV.A ('BAPPS ... is used to train and validate SPIPS model') and by the Figure 2 caption ('The SPIPS model parameters are iteratively optimized via backpropagation'). This is not a wording issue: it changes the interpretation of the experimental comparison, since SPIPS is a trained model while several baselines are fixed metrics. The authors must remove the data-free claim or clearly specify what is trained and what is fixed, including the training set split.","section":"Section II.A, Section II.B, Section IV.A, Figure 2 caption"},{"comment":"Per-pixel PSNR and per-pixel MS-SSIM maps are not standard definitions and are not defined in the paper. PSNR is normally a global scalar computed from MSE, and MS-SSIM is a scalar or, in some multi-scale formulations, a map of per-pixel local scores, but the paper does not give the formula used, and Eq. (3) states Q_msssim has C channels (the number of scales) while each Q_p value is claimed to be in R^{3xHxW}. The normalization N(·) is also unspecified. Without these definitions, the traditional IQA module cannot be reproduced, and the reported maps in Figures 3 and 4 are not interpretable.","section":"Section III.A, Equations (1)-(3)"},{"comment":"The split of CNN layers into 'perceptual' (first L-2 layers) and 'semantic' (last two layers) is load-bearing for the entire fusion design, but the paper offers no justification or ablation for this choice. It is stated as an example for AlexNet with no evidence that the split is optimal or even robust across backbones. An ablation over alternative splits (e.g., different cut points, or learned layer weights) is needed to support the claim that this specific distinction is what drives the reported gains.","section":"Section III.B, Equations (7)-(8)"},{"comment":"The ablation study reports only point estimates (e.g., SPIPS-abla1 vs. SPIPS on CNN: 0.78 vs. 0.81 in PLCC) with no error bars, number of trials, statistical significance tests, or details about training seeds. Many differences are small (0.01-0.05) and could be within run-to-run variation. The authors should report multiple runs or at least a significance analysis to support the statement that 'both ablation models performed worse than the full model across all metrics.'","section":"Section IV.C, Tables II-IV"},{"comment":"The claim of 'consistently highest correlation' is not consistent with the JND results on the CNN category: SPIPS (PLCC 0.60) is lower than both LPIPS (0.63) and DISTS (0.63). The text acknowledges this in passing, but the abstract and Section IV.D's opening claim of consistent superiority should be reconciled with these numbers, and the conditions under which the method is superior should be stated precisely.","section":"Section IV.D, Table VIII"}],"minor_comments":[{"comment":"Reference [6] is truncated: 'Transactions on Machine Le' should be completed with the journal name and volume/page details.","section":"References"},{"comment":"The text says 'height and weight' where it should say 'height and width'; also the feature map dimensions for different layers are never stated, so it is unclear how the error maps of different resolutions are aligned before convolution.","section":"Section III.B, text after Eq. (5)"},{"comment":"The constraint lambda_1 + lambda_2 + lambda_3 = 1 is stated, but no non-negativity constraint is imposed; the meaning of negative weights in the final score should be discussed or the weights should be constrained.","section":"Section III.D, Eq. (12)"},{"comment":"The captions do not explicitly state that lower SPIPS scores indicate better quality, which is essential for interpreting the numerical values in the figures; this should be stated in each caption.","section":"Figures 3 and 4"},{"comment":"The paper says BAPPS 'is used to train and validate SPIPS model' but never specifies the train/validation/test split, whether the 'test split' mentioned in Section IV.B refers to the official BAPPS test split, and whether any baselines were retrained or used with their published settings.","section":"Section IV.A"}],"recommendation":"major_revision","confidential_remarks":"The manuscript appears to be an early draft: there are internal inconsistencies (data-free vs. trained), missing definitions for core components, and incomplete references. The central claim is not verifiable as written because the 2AFC evaluation protocol is undefined. I believe the paper has a plausible core idea but needs substantial additional specification and experimentation before it can be accepted; I recommend major revision rather than rejection, provided the authors can resolve the evaluation-protocol and training-contradiction issues."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know about this one. The core idea is a sensible hybrid: take LPIPS-style deep features from a pre-trained net, split them into low-level perceptual and high-level semantic groups, combine them with per-pixel versions of PSNR/SSIM/MS-SSIM, and learn the fusion weights on BAPPS. That is a reasonable incremental step over LPIPS, and the ablations do show that both the semantic stream and the traditional metrics contribute. The qualitative examples are illustrative. The writing is clear.\n\nThe soft spots are real, and the stress-test flag is correct. First, the paper contradicts itself on training. Section II.A calls the method data-free and Section II.B says it 'requires no training at all,' but Section IV.A says BAPPS is used to train the model and the Figure 2 caption says parameters are optimized via backpropagation. That is not a wording quibble; it changes what the method is. Second, Tables V-VII report PLCC, SRCC, and KRCC on the 2AFC subset, but the paper never defines the ground-truth scalar or the computation. BAPPS 2AFC gives binary human preferences, so PLCC against a binary label is point-biserial and the values are not comparable to standard IQA PLCC. The 'consistently highest' claim is unverifiable until the protocol is specified. Third, implementation details are missing: the backbone is not named, the per-pixel PSNR map is undefined (PSNR is normally a single global number), and there are no training hyperparameters or code. Minor point: 'consistently highest' in Section IV-D is an overstatement, since Table VII shows SPIPS ties with LPIPS on the SR subcategory for KRCC.\n\nThe good news is these are fixable. The architecture is described well enough that a motivated reader could reimplement it once the per-pixel maps and the training setup are clarified. The idea is not earth-shattering, but it is honest incremental work, and the BAPPS results, if confirmed under the correct protocol, would be a small but useful improvement over LPIPS. For now the numbers should be treated as unverified.\n\nThis paper deserves peer review rather than a desk reject. A good referee will ask for a corrected 'data-free' claim, a precise evaluation protocol for 2AFC correlations, and at least a named backbone with training details. I'd bring it to a reading group only as a case study in how easily evaluation protocols can be underspecified. I would not cite it in its current form.","headline":"A sensible hybrid IQA metric whose main experimental claim is currently undercut by an undefined 2AFC correlation protocol and a contradictory 'data-free' claim; worth a serious round of revision, not a rejection.","tokens_in":13824,"tokens_out":5004,"would_cite":false,"duration_ms":46786,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SPIPS, a hybrid IQA metric that separates semantic and perceptual deep features and adds traditional scores, reports the highest agreement with human judgment among compared methods on every BAPPS 2AFC category.","keywords":["image quality assessment","full-reference IQA","perceptual metric","deep features","semantic features","BAPPS dataset","LPIPS","SSIM"],"falsifier":"Train SPIPS on BAPPS with the perceptual/semantic boundary moved to $L-1$, $L-3$, or a learned per-layer weighting, and compare SRCC on the 2AFC test set; if any alternative split matches or exceeds the reported numbers, the specific claim that the last two layers are uniquely semantic is falsified.","tokens_in":12766,"feed_emoji":"🖼️","tokens_out":9186,"duration_ms":74123,"temperature":0.7,"pith_summary":"SPIPS is a full-reference image quality metric that fuses three evidence streams: classic pixel and structural scores (PSNR, SSIM, MS-SSIM), low-level perceptual deep features, and high-level semantic deep features from a pretrained CNN. The paper's thesis is that distinguishing semantic content from perceptual detail, and adding traditional metrics, yields judgments closer to human perception than any single-stream metric. On the BAPPS two-alternative forced-choice benchmark, SPIPS reports the highest correlation with human ratings in every distortion category among the tested methods, and it generally leads on the just-noticeable-difference subset as well. If these results hold, SPIPS offers a straightforward recipe for scoring AI-generated or heavily processed images in line with what people actually see.","feed_headline":"Hybrid quality metric beats LPIPS on every BAPPS category","feed_subtitle":"SPIPS splits deep features into semantic and perceptual streams and adds classic scores to match human judgment.","key_machinery":"The load-bearing object is the layer partition in Eqs. (7)-(8): for a backbone with $L$ feature extraction layers, the first $L-2$ layers are the 'perceptual' group and the last two are the 'semantic' group, and the squared difference between reference and evaluated features at each layer produces a quality map that enters one of those two streams. The second mechanism is the fusion rule in Eq. (12), where the mean activations of three independently convolved streams—traditional, perceptual, and semantic—are combined with learned weights summing to one. The partition operationalizes the paper's top-down account of human vision, in which overall structure is grasped before fine details, and the weighted fusion lets the network balance global content against local artifacts when producing a single score.","core_discovery":"SPIPS's central claim is that image quality is best measured as a blend of separable streams rather than one global difference. From a pretrained CNN, the squared element-wise differences between the evaluated and reference feature maps are partitioned by layer: the first $L-2$ layers produce 'perceptual' quality maps and the last two layers produce 'semantic' quality maps. In parallel, per-pixel maps from PSNR, SSIM, and MS-SSIM are inverted so that lower values mean better quality. Each of the three map groups passes through its own small convolutional feature extractor with ReLU, and the mean of each extracted stream is combined in a weighted sum, with weights $\\lambda_1,\\lambda_2,\\lambda_3$ and the small convolutions trained on the BAPPS dataset following the LPIPS training protocol. The paper reports that the resulting score achieves the highest PLCC, SRCC, and KRCC across all six BAPPS 2AFC categories when compared with PSNR, SSIM, VIF, DISTS, and LPIPS, and the best SRCC and KRCC on the JND split.","pith_inferences":["A learnable layer split could reveal whether the semantic/perceptual boundary actually sits at different depths for different distortion types, and might improve the scores further; the fixed $L-2$ split in the paper is only one point in that design space.","The largest reported gains over the next best metric appear on colorization and traditional distortions, which suggests the traditional-metric stream carries the most weight where deep features are weakest; a per-category ablation of that stream would test this directly.","The same three-stream disentanglement could transfer to no-reference quality assessment or video quality, and the paper's stated use of vision transformers would let the semantic stream operate on token relationships rather than layer groups.","The per-category reporting leaves the practitioner to choose how to aggregate; a natural next step is a single pooled correlation over all BAPPS splits, which the paper does not provide."],"forward_implications":["On BAPPS 2AFC, SPIPS reports the highest PLCC, SRCC, and KRCC in all six distortion categories, beating PSNR, SSIM, VIF, DISTS, and LPIPS.","On the BAPPS JND subset, SPIPS achieves the best SRCC and KRCC on the CNN and Trad splits, while its PLCC on the CNN split is slightly below LPIPS and DISTS.","Ablations that remove either the semantic stream or the traditional metrics reduce correlation on every reported metric, indicating that both streams contribute to the final score.","Because the backbone stays pretrained and only the fusion weights and small per-stream convolutions are learned, adapting SPIPS to a new distortion family requires comparatively little training."],"supporting_citations":[{"why":"Provides the BAPPS dataset and the training protocol for deep perceptual metrics, and is the primary learned baseline SPIPS must beat.","marker":"[5]"},{"why":"DISTS, the structure-texture similarity baseline that defines, along with LPIPS, the learned full-reference comparison class.","marker":"[19]"},{"why":"SSIM, a classic structural metric used both as an input stream to SPIPS and as a baseline in the experiments.","marker":"[32]"},{"why":"MS-SSIM, the multi-scale structural metric used as another input stream and baseline.","marker":"[54]"},{"why":"VIF, the information-theoretic full-reference metric included as a comparison baseline.","marker":"[60]"},{"why":"Provides the PSNR formulation used both as an input stream and as a baseline metric.","marker":"[30]"},{"why":"Supplies the evidence that better ImageNet classifiers do not necessarily yield better perceptual similarity, motivating the separate treatment of semantic features in SPIPS.","marker":"[6]"}],"fun_headline_variants":["SPIPS beats LPIPS on every BAPPS category","New IQA score disentangles semantic and perceptual details","Hybrid IQA metric: semantic + perceptual streams beat LPIPS","SPIPS scores image quality the way humans do","Best IQA on BAPPS: SPIPS blends deep features and classic scores"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the fixed division of network layers into the first $L-2$ (perceptual) and last two (semantic) mirrors the human visual system's global-before-local judgment; the paper offers no ablation or evidence for alternatives to this split.","fun_headline_variants_meta":{"raw":{"variants":["SPIPS beats LPIPS on every BAPPS category","New IQA score disentangles semantic and perceptual details","Hybrid IQA metric: semantic + perceptual streams beat LPIPS","SPIPS scores image quality the way humans do","Best IQA on BAPPS: SPIPS blends deep features and classic scores"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000582,"raw_usage":{"total_tokens":2786,"prompt_tokens":1040,"completion_tokens":1746,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":656,"completion_tokens_details":{"reasoning_tokens":1661}},"tokens_in":656,"tokens_out":1746,"duration_ms":11338,"temperature":1.0,"reasoning_tokens":1661,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:44:39.556444+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train SPIPS on BAPPS with the perceptual/semantic boundary moved to $L-1$, $L-3$, or a learned per-layer weighting, and compare SRCC on the 2AFC test set; if any alternative split matches or exceeds the reported numbers, the specific claim that the last two layers are uniquely semantic is falsified.","supporting_citations":[{"cited_title":"Efros, Eli Shechtman, and Oliver Wang","cited_arxiv_id":null,"evidence_quote":"Provides the BAPPS dataset and the training protocol for deep perceptual metrics, and is the primary learned baseline SPIPS must beat."},{"cited_title":"Simoncelli","cited_arxiv_id":null,"evidence_quote":"DISTS, the structure-texture similarity baseline that defines, along with LPIPS, the learned full-reference comparison class."},{"cited_title":"Bovik, Hamid R","cited_arxiv_id":null,"evidence_quote":"SSIM, a classic structural metric used both as an input stream to SPIPS and as a baseline in the experiments."},{"cited_title":"Simoncelli, and Alan C","cited_arxiv_id":null,"evidence_quote":"MS-SSIM, the multi-scale structural metric used as another input stream and baseline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"VIF, the information-theoretic full-reference metric included as a comparison baseline."},{"cited_title":"Digital image processing","cited_arxiv_id":null,"evidence_quote":"Provides the PSNR formulation used both as an input stream and as a baseline metric."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the evidence that better ImageNet classifiers do not necessarily yield better perceptual similarity, motivating the separate treatment of semantic features in SPIPS."}],"review_version":1}