{"id":"b102610a-d7c7-4a52-9082-28cd04fb8da1","arxiv_id":"2411.17489","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Puzzle Similarity detects artifacts in novel views of 3D scenes by max-pooling feature similarity against training views, and it outperforms prior quality metrics in correlating with human artifact segmentations.","lead":"This paper introduces Puzzle Similarity, a metric that finds broken areas in new 3D-rendered views by matching small image pieces against a set of reference photos of the same scene. It reports that the metric agrees with human judgments better than existing no-reference, cross-reference, and even full-reference metrics, and it releases a new human-labeled dataset for evaluation.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Evaluation-set tuning of the backbone and layer weights undermines the reported state-of-the-art claim.","rationale":"The reader's weakest_assumption focuses on the spatial-consistency limitation of max-patch matching in Eq. (1). That is a genuine theoretical limitation, but the central claim is an empirical one on the collected benchmark. The most direct threat to that empirical claim is the explicit test-set tuning: the authors chose both the feature backbone and the layer-combination weights based on the same human-labeled images used for evaluation. This is a concrete, identifiable flaw that could fully explain the reported margin, and it is stated in the manuscript itself. The spatial-consistency issue, by contrast, is a plausible failure mode but not directly evidenced in the current experiments; it would limit generalization but does not necessarily invalidate the comparison on the tested artifact types. Both concerns support a conditional acceptance, so the verdict should remain CONDITIONAL. I agree with the reader's overall recommendation but identify a different primary vulnerability, hence 'partial' agreement.","tokens_in":14435,"tokens_out":4040,"duration_ms":39059,"concrete_test":"Hold out a random subset of scenes (e.g., 4 of 12) from all design decisions. On the remaining scenes, select the backbone and the three layer weights via cross-validation, then freeze them and evaluate on the held-out scenes. Compute average Pearson correlation (with the same 5-parameter logistic mapping) for PuzzleSim against CrossScore and PaQ-2-PiQ on the held-out scenes. If PuzzleSim no longer ranks first on average, the reported superiority is not established. Additionally, perturb the frozen weights by ±0.1 and report the resulting spread of correlations to assess robustness.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim depends on the empirical comparison in Tab. 2, but the method's design choices were made using the same human-labeled examples that are later used for evaluation. In Sec. 3 the authors state: 'We opted for SqueezeNet as it aligned best with our test examples, specifically using layers ℓ∈{2,3,4} with the weights w2=0.67, w3=0.2, and w4=0.13, which we found heuristically.' These 36 renderings (Sec. 4.1) are exactly the images on which Tab. 1 and Tab. 2 report correlations. No validation split or sensitivity analysis is described. The competing baselines (CrossScore, PaQ-2-PiQ) were applied as off-the-shelf without any dataset-specific tuning. Consequently, the reported 0.615 vs 0.510 advantage may reflect selection on the test set rather than a genuine superiority in artifact localization, and the conclusion 'outperforms all tested full-reference, cross-reference and no-reference metrics' is not yet supported by the experiments as reported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Puzzle Similarity, a cross-reference metric for detecting and localizing artifacts in novel views of 3D scene reconstructions. The metric embeds image patches using a pretrained CNN (SqueezeNet), then for each patch in the test view computes the maximum cosine similarity against all patches in a set of unaligned reference views, across all spatial positions. The similarity maps from layers 2, 3, and 4 are weighted (0.67, 0.20, 0.13) and combined into a final artifact map. The authors collect a new human-labeled dataset of artifact masks for 36 renderings from 12 scenes reconstructed with 3D Gaussian Splatting, and evaluate the metric's correlation with averaged human masks, comparing against no-reference, cross-reference, and full-reference baselines. They report that Puzzle Similarity achieves the highest average Pearson and Spearman correlations (Tab. 2), and demonstrate an iterative inpainting application in Sec. 5.","tokens_in":14674,"tokens_out":3522,"duration_ms":32623,"significance":"If the reported results hold under a properly controlled evaluation, Puzzle Similarity would be a practical and cheap artifact-localization tool for 3D reconstruction, with the advantage of not requiring aligned references. The paper also contributes a public human-labeled dataset for cross-reference metric evaluation, which is a useful resource for the community. The method's strengths are its clarity, simplicity, differentiability, and the efficient blockwise implementation described in Sec. 3. However, the central state-of-the-art claim is currently weakened by the evaluation protocol, in which the method's design choices were selected on the same test examples used for the final comparison, and by the small hand-picked evaluation set. The significance of the claimed advance is therefore conditional on additional validation.","major_comments":[{"comment":"The backbone, layer set, and layer weights are selected on the same data used for the final evaluation. The paper states in Sec. 3 that SqueezeNet 'aligned best with our test examples, specifically using layers ℓ∈{2,3,4} with the weights w2=0.67, w3=0.2, and w4=0.13, which we found heuristically.' Since these are the same 36 renderings (Sec. 4.1) used to compute the headline correlations in Tabs. 1 and 2, the reported 0.615 vs. 0.510 average Pearson advantage over CrossScore may reflect test-set tuning rather than a genuine property of the metric. The baselines are applied off-the-shelf without dataset-specific tuning, so the comparison is asymmetric. The authors should either introduce a separate validation split for choosing hyperparameters, or report a sensitivity analysis showing that the advantage persists across a range of weights, layer sets, and backbones. This is load-bearing for the conclusion that Puzzle Similarity 'outperforms all tested' metrics.","section":"Sec. 3 (Pre-trained Model Choice) and Sec. 4.2, Tab. 2"},{"comment":"The metric's core assumption is that every well-reconstructed region has a near-perfect patch match somewhere in the reference views and that every artifact region lacks such a match. However, Eq. (1) takes a spatial max over all reference patches, so artifacts composed of valid patches in invalid spatial arrangements (e.g., ghosting, duplicated texture, swapped scene parts) can match a reference patch very well and be incorrectly labeled as clean. The paper even notes in Sec. 3 that the max search 'relinquishes spatial relations.' The current dataset, which the authors state is dominated by blur, holes, and texture mismatches, cannot exercise this failure mode. The authors should test the metric on examples with spatial rearrangement artifacts, or explicitly acknowledge and bound this limitation. This is a principal risk for the claimed perceptual alignment.","section":"Sec. 3, Eq. (1); Sec. 4.1 dataset composition"},{"comment":"The conclusion claims that Puzzle Similarity outperforms 'all tested full-reference, cross-reference and no-reference metrics,' but the full-reference comparison is not reported in the main text. Sec. 4.5 only states that 'we also provide an extensive comparison' and mentions results in qualitative terms; the actual FR numbers are deferred to the Supplementary. Since the central claim explicitly includes FR metrics, the main text should display the FR results (at least a summary table comparable to Tab. 2). Without this, the stated scope of the claim cannot be verified from the paper as presented. Additionally, the evaluation is based on only 36 hand-picked images (3 per scene, 12 scenes), and the per-scene correlations in Tab. 1 are computed from just three images each, which limits the statistical strength of the average comparison.","section":"Sec. 4.1 and Sec. 4.5; Conclusion"}],"minor_comments":[{"comment":"The outer-product formulation in Eq. (4) is a useful implementation detail, but the notation is slightly inconsistent: the text defines Sℓ(I) as a map, while the equation appears to express the row-max of a matrix; please clarify the dimensions and the flattening order (N, Hℓ, Wℓ, Cℓ) to make the memory-efficient tiling easier to follow.","section":"Sec. 3, Eq. (4)"},{"comment":"The 5-parameter logistic fit is applied per scene and per metric; please state explicitly whether the logistic parameters are fit on the same human masks used for the reported correlation values. If so, this is a standard but potentially optimistic procedure, and it should be described in sufficient detail to assess whether all metrics benefit equally.","section":"Sec. 4.2, Eq. (5)"},{"comment":"The Limitations section already acknowledges that the layer weights are empirically calibrated, but it does not mention that the calibration was performed on the test data. Please add an explicit statement about the validation protocol and, ideally, point to a sensitivity analysis that reassures readers the result is not an artifact of overfitting to the 36 images.","section":"Sec. 6 (Limitations)"},{"comment":"The paper reports 22 participants but only states that gender and age distributions are in the Supplementary. Please move at least the participant count and any exclusion criteria into the main text, as the quality of the ground-truth labels is central to the evaluation.","section":"Sec. 4.1, participant details"}],"recommendation":"major_revision","confidential_remarks":"The core metric idea is simple and potentially useful, and the paper is generally well written. However, the evaluation protocol has a clear selection-on-test-set issue that the authors themselves inadvertently expose in Sec. 3 and Sec. 6. I would ask the authors to provide a validation-based tuning procedure or a sensitivity analysis, and to report the full-reference comparison in the main text. If the advantage over CrossScore and PaQ-2-PiQ persists under a properly held-out hyperparameter choice, this could become a solid contribution. The small sample size (36 images, 3 per scene) is concerning but possibly acceptable for a first benchmark if the limitations are clearly stated; the main blocking issue is the tuning protocol."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper contributes a genuinely useful dataset and a simple, cheap cross-reference metric for artifact localization in 3D reconstructions. The metric itself is not radically new—max-pooling cosine similarity in a pretrained CNN feature space is the same operation used in contextual loss—but the adaptation to cross-reference artifact detection and the human-labeled benchmark are real contributions.\n\nWhat it does well: the method is clearly described, efficient to run, and the authors release the dataset. The inpainting showcase is a nice demonstration that the artifact maps can drive a practical downstream task. The evaluation protocol (5-parameter logistic per scene, Pearson/Spearman on human maps) is standard, even if the per-scene fit with only three images is a bit loose.\n\nThe soft spot is exactly the one the stress-test flags. In Section 3 the authors state they chose SqueezeNet and the layer weights because they 'aligned best with our test examples' and found them 'heuristically.' Those test examples are the same 36 images used in Table 2 to report the state-of-the-art correlation. That makes the reported 0.615 vs 0.510 advantage uninterpretable as a fair comparison—the baselines were not given the same tuning opportunity. It doesn't mean the metric is bad, but it means the central conclusion is not supported by the experiment as run.\n\nThere's also a structural limitation that the evaluation doesn't stress-test. Equation (1) takes the maximum similarity against all reference patches, so any artifact region that happens to contain a valid-looking patch (duplicated texture, ghosting, swapped scene parts) will score high even though it looks wrong to a human. The dataset is dominated by blur, holes, and texture mismatches, so it can't resolve this.\n\nThe dataset itself is small—36 images across 12 scenes—and there's no inter-observer agreement reported, so we don't know how reliable the ground truth is.\n\nWho should read it: anyone working on quality metrics for novel view synthesis, and anyone who wants a benchmark for cross-reference artifact detection. The dataset alone is worth having.\n\nMy recommendation: send it to peer review, but require the authors to (1) report validation with a held-out set or a sensitivity analysis for the backbone and weights, (2) report inter-observer agreement, and (3) ideally add cases with spatial-rearrangement artifacts. With those changes, the paper would be solid.","headline":"Useful dataset and a sensible metric, but the reported SOTA comes from tuning on the test set.","tokens_in":15184,"tokens_out":3485,"would_cite":true,"duration_ms":36270,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Puzzle Similarity claims to outperform every tested full-reference, cross-reference, and no-reference metric at localizing artifacts in novel views of 3D reconstructions, using only training-view patches as references.","keywords":["cross-reference metric","artifact detection","3D reconstruction","novel view synthesis","image quality assessment","human-labeled dataset","patch similarity","perceptual metric"],"falsifier":"Take a well-reconstructed novel view, duplicate a small texture patch and paste it at a different location, or swap two similarly textured regions, then have human observers mark artifacts while Puzzle Similarity scores the same image. If humans flag the duplication but the metric's similarity stays near its maximum because each pasted patch exactly matches a reference patch, the assumption that artifact regions never match reference patches fails.","tokens_in":20,"feed_emoji":"🧩","tokens_out":8503,"duration_ms":182361,"temperature":0.7,"pith_summary":"The paper proposes Puzzle Similarity, a cross-reference metric that localizes artifacts in novel views of 3D reconstructions without any aligned reference image. It treats each training view as a pile of puzzle pieces: every patch of the test image is embedded by a pretrained CNN, and its score is the maximum cosine similarity to any patch of any training view, across several network layers. Regions whose best match is low are flagged as artifacts. On a new dataset of 36 human-labeled renderings from 12 scenes, the metric's artifact maps reach an average Pearson correlation of 0.615 with human segmentations, ahead of the cross-reference baseline at 0.510 and the best no-reference method at 0.402. The authors argue this makes the metric useful for automatic inpainting, guided acquisition, and few-shot reconstruction.","feed_headline":"Patch-matching metric beats all tested artifact detectors","feed_subtitle":"Puzzle Similarity finds 3D reconstruction errors from training views alone, scoring 0.615 against human labels.","key_machinery":"The load-bearing object is the similarity map $S(I)$ defined by Eq. (1) through Eq. (3): for each CNN layer $\\ell$, each feature vector of the test embedding is matched to its nearest neighbor, meaning the maximum cosine similarity, among all spatially flattened feature vectors of all $N$ training views, and the per-layer maps are upsampled and affinely combined with weights $w_2=0.67$, $w_3=0.2$, $w_4=0.13$ in the SqueezeNet backbone. This single maximum-match-to-any-reference-patch operation carries the entire argument, because it defines an artifact as a region with no close puzzle piece in the training views. The layered CNN embedding provides perceptual alignment and lets the metric localize artifacts at multiple scales.","core_discovery":"The central claim is that Puzzle Similarity outperforms all tested full-reference, cross-reference and no-reference metrics in capturing artifacts aligned with human perception, as measured on the authors' human-labeled dataset. The metric computes, for each pixel of the embedded test view, the maximum cosine similarity against all feature vectors extracted from all training views across several CNN layers, then combines the per-layer maps with fixed weights. Because the search ignores spatial position, it is robust to camera shifts; because it never needs a reference aligned to the test view, it applies in the setting where reconstruction-quality assessment is hardest. The claim is supported by an average Pearson correlation of 0.615 (std 0.120), ahead of the cross-reference baseline at 0.510 (std 0.204) and the best no-reference method at 0.402 (std 0.178), with lower variance indicating more consistent performance across artifact types.","pith_inferences":["The metric's blind spot is spatial rearrangement: because Eq. (1) maximizes similarity over all reference patches regardless of location, a view with duplicated or swapped textures can receive high scores even where humans see artifacts; adding a spatial-consistency or epipolar check would close this gap.","One can extend the evaluation to synthetic perturbations: replaying the same scene with known pasted or ghosted artifacts would quantify exactly how much of Puzzle Similarity's human correlation comes from local texture mismatch rather than geometric correctness.","A natural relaxation of the hard max operation is a softmax over reference patches, which could make the metric more suitable for gradient-based optimization; the paper notes that the current max operation is unlikely to produce useful gradients.","The reported standard deviations (0.120 vs 0.204 for the cross-reference baseline) suggest the advantage is as much about consistency across artifact types as about average accuracy, and scenes with blurry or unnatural textures are where the gap appears largest."],"forward_implications":["On the new human-labeled dataset, Puzzle Similarity's artifact maps correlate with human segmentations better on average than every no-reference, cross-reference, and full-reference metric tested, with smaller variance across the 12 scenes.","Because the metric needs only the training views of the scene and a pretrained CNN, it can flag artifacts in any new view without a ground-truth image, which is the setting where reconstruction quality is otherwise hardest to assess.","The resulting artifact masks can drive automatic inpainting: the paper's iterative thresholding framework uses Puzzle Similarity to select masks and reports monotone improvement in similarity.","Swapping the CNN backbone adapts the metric to a new domain with no retraining, so the same cross-reference principle transfers beyond 3D reconstruction to any image set that defines a distribution.","The released human-labeled artifact dataset gives other cross-reference metrics a benchmark with ground-truth maps, which previously did not exist."],"supporting_citations":[{"why":"Provides the main cross-reference baseline whose quality maps are compared with Puzzle Similarity on human-labeled data.","marker":"[47]"},{"why":"The reconstruction technique used to render the artifact-ridden novel views that participants labeled.","marker":"[15]"},{"why":"Supplies the segmentation tool used to collect the human artifact masks in the user study.","marker":"[49]"},{"why":"The inpainting model that consumes Puzzle Similarity masks in the automatic restoration application.","marker":"[37]"},{"why":"A no-reference metric producing quality maps, used as a baseline for artifact localization.","marker":"[53]"},{"why":"A patch-based no-reference baseline whose maps are correlated with human segmentations.","marker":"[13]"},{"why":"An artifact-localization baseline from image synthesis tasks that is compared in the evaluation.","marker":"[61]"},{"why":"A full-reference visible-difference predictor used to benchmark the claim of outperforming reference-based metrics.","marker":"[21]"}],"fun_headline_variants":["Puzzle Similarity: top artifact locator for 3D reconstructions","3D artifact detector wins with human-aligned accuracy","Patch-based metric outshines all for 3D error spotting","No-reference metric best at finding 3D reconstruction flaws","Puzzle Similarity leads in human-aligned 3D artifact mapping"],"cache_read_input_tokens":17408,"weakest_assumption_plain":"The metric assumes that every correctly reconstructed region has at least one closely matching patch somewhere in the training views and that every artifact region has none, with no check on whether the matched patches are spatially consistent; artifacts made of valid patches in wrong places (duplicated texture, ghosting, swapped parts) can therefore score as high as clean regions.","fun_headline_variants_meta":{"raw":{"variants":["Puzzle Similarity: top artifact locator for 3D reconstructions","3D artifact detector wins with human-aligned accuracy","Patch-based metric outshines all for 3D error spotting","No-reference metric best at finding 3D reconstruction flaws","Puzzle Similarity leads in human-aligned 3D artifact mapping"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000202,"raw_usage":{"total_tokens":1409,"prompt_tokens":1002,"completion_tokens":407,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":618,"completion_tokens_details":{"reasoning_tokens":318}},"tokens_in":618,"tokens_out":407,"duration_ms":4181,"temperature":1.0,"reasoning_tokens":318,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T12:03:34.374790+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a well-reconstructed novel view, duplicate a small texture patch and paste it at a different location, or swap two similarly textured regions, then have human observers mark artifacts while Puzzle Similarity scores the same image. If humans flag the duplication but the metric's similarity stays near its maximum because each pasted patch exactly matches a reference patch, the assumption that artifact regions never match reference patches fails.","supporting_citations":[{"cited_title":"CrossScore: Towards Multi-View Image Evaluation and Scoring","cited_arxiv_id":null,"evidence_quote":"Provides the main cross-reference baseline whose quality maps are compared with Puzzle Similarity on human-labeled data."},{"cited_title":"3D Gaussian Splatting for Real-Time Radiance Field Rendering.ACM Transactions on Graphics, 42(4):1–14, 2023","cited_arxiv_id":null,"evidence_quote":"The reconstruction technique used to render the artifact-ridden novel views that participants labeled."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the segmentation tool used to collect the human artifact masks in the user study."},{"cited_title":"Resolution-robust Large Mask Inpainting with Fourier Convolutions","cited_arxiv_id":null,"evidence_quote":"The inpainting model that consumes Puzzle Similarity masks in the automatic restoration application."},{"cited_title":"From Patches to Pic- tures (PaQ-2-PiQ): Mapping the Perceptual Space of Picture Quality","cited_arxiv_id":null,"evidence_quote":"A no-reference metric producing quality maps, used as a baseline for artifact localization."},{"cited_title":"Convo- lutional Neural Networks for No-Reference Image Quality Assessment","cited_arxiv_id":null,"evidence_quote":"A patch-based no-reference baseline whose maps are correlated with human segmentations."},{"cited_title":"Perceptual Artifacts Local- ization for Image Synthesis Tasks","cited_arxiv_id":null,"evidence_quote":"An artifact-localization baseline from image synthesis tasks that is compared in the evaluation."},{"cited_title":"Mantiuk, Gyorgy Denes, Alexandre Chapiro, An- ton Kaplanyan, Gizem Rufo, Romain Bachy, Trisha Lian, and Anjul Patney","cited_arxiv_id":null,"evidence_quote":"A full-reference visible-difference predictor used to benchmark the claim of outperforming reference-based metrics."}],"review_version":1}