Pith. sign in

REVIEW 4 major objections 6 minor 2 cited by

Benchmarking Image Similarity Metrics for Novel View Synthesis Applications

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Pith's one-line read The paper argues that DreamSim, a learned perceptual similarity metric, should replace SSIM, PSNR, and LPIPS for scoring novel-view-synthesis renders because it degrades gradually with corruption severity and ignores pixel changes humans…

desk verdict Real benchmark results with a load-bearing external-validity gap; the DreamSim-for-NVS recommendation outruns the evidence. read the letter →

arxiv 2506.12563 v1 pith:562MMD2I submitted 2025-06-14 cs.CV

classification cs.CV
keywords DreamSimnovelviewsynthesisimagesimilaritymetricsSSIMPSNRLPIPSrenderqualityassessmentcorruptionrobustness
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Novel view synthesis (NVS) models produce new camera angles of a scene, but the standard scores used to judge their output — SSIM, PSNR, and LPIPS — were built for image compression and react strongly to pixel-level changes that human viewers neither notice nor care about. The paper benchmarks these metrics against DreamSim, a learned perceptual similarity metric, using a purpose-built corpus of corrupted images that mimic the artifacts NVS renders show: blur, floaters, splats, shadows, color shifts, and others. Across 12 corruption types at 20 severity levels, plus ImageNet-C and localized foreground/background corruption experiments, DreamSim is the only metric whose scores decline gradually and discriminatively as corruption worsens. The paper concludes that DreamSim, or a future NVS-specific version of it, should be used for general evaluation of NVS models. The stakes are practical: evaluation choices steer model development, and a metric that matches human perception could change which NVS models are judged successful.

What carries the argument

The load-bearing mechanism is a controlled corruption benchmark, called NVS Corruptions, that applies twelve artifact types (blur, brightness, color shift, contrast, floaters, grayscale, pixelation, rotation, saturation, shadows, splats, warp) at twenty severity levels to building-scene images, together with a 750k-image ImageNet-C subset and localized foreground/background and pixel-crop corruptions. All metric outputs are normalized to a common 0-to-1 scale, and the paper reads the resulting median score-versus-severity curves for monotonic decline and discriminative power. DreamSim, a learned perceptual similarity model trained on human-judged similarity pairs, is the object whose behavior the curves are meant to validate.

What would settle it

Run a study in which humans rate real NVS renders from a NeRF or 3D Gaussian Splatting model for utility, then compare how well DreamSim, SSIM, PSNR, and LPIPS correlate with those ratings; the paper's recommendation fails if a classical metric correlates as well as or better than DreamSim, or if DreamSim's score-severity curves on the synthetic corruption suite diverge from its curves on the real artifact set.

Watch

Extended reading notes

Core claim

The paper claims that DreamSim is robust in scoring NVS renders in a predictable and differentiable manner and is less sensitive to corruptions that are imperceptible to humans. The experimental evidence is a set of normalized score-severity curves: on the NVS Scenes building imagery, SSIM, PSNR, and LPIPS collapse after minimal blur or other corruption and then stay flat, while DreamSim's median score descends steadily across severity levels and separates major intensity increments. On 750k ImageNet-C images, DreamSim keeps this monotone response on ten of twelve corruption types, with the two exceptions (fog and elastic transform) explained by the corruptions themselves having little perceptual change or introducing rotation. In the local-corruption experiments, DreamSim alone is nearly unaffected by cropping of 1 to 10 pixels and shows stronger penalties for foreground than background corruption, which the paper interprets as agreement with human visual priorities. The stated conclusion is that DreamSim or a future NVS-trained version should be used for general NVS evaluation because it quantifies render alignment with human judgment.

Load-bearing premise

The load-bearing premise is that the twelve artificial corruptions faithfully stand in for artifacts real NVS models produce and that score changes under those corruptions represent how useful humans would find the rendered image.

Editorial extensions

If this is right

  • NVS model rankings produced with SSIM, PSNR, or LPIPS could change substantially when scored with DreamSim, since the classical metrics conflate imperceptible artifacts with real quality loss.
  • Evaluation of real-world NVS deployments, such as site modeling or first-responder imagery, should weight scene-level content preservation rather than per-pixel fidelity.
  • Small camera pose errors that shift edges by a few pixels will no longer dominate quality scores, so pose-estimation failures will be judged by their actual impact on perceived content.
  • A DreamSim variant fine-tuned on NVS-specific render artifacts would be the natural next evaluation tool and would likely sharpen the already monotone response to corruption.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper treats its synthetic corruptions as stand-ins for real NVS artifacts, but it does not measure metric behavior on actual NeRF or 3D Gaussian Splatting renders; that comparison is the direct test the study leaves open.
  • Because the foreground/background asymmetry reflects DreamSim's training data, applications where background content matters (e.g., site modeling) may need a differently calibrated metric rather than a universally preferred one.
  • The human-utility interpretation is inferred from perceptual plausibility rather than directly measured; a human-rating study on the same corrupted corpus would settle whether DreamSim's monotonic curves correspond to task usefulness.
  • If DreamSim responds monotonically to synthetic severity while classical metrics do not, then using DreamSim could also serve as a cheap automated corruption-robustness monitor for NVS systems.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper benchmarks four image similarity metrics (SSIM, PSNR, LPIPS, and DreamSim) for use in novel view synthesis (NVS) evaluation. The authors construct two corrupted-image corpora: a new 'NVS Scenes' set with 12 custom corruption types at 20 severity levels, and a 750k-image subset of ImageNet-C with its standard 15 corruptions and 5 severities. They also apply localized foreground/background corruptions and small pixel-level cropping to ImageNet images, then plot median normalized metric scores across severity levels. Based on the resulting score curves, the paper concludes that DreamSim is more robust to minor corruptions, more discriminative across severity levels, and better aligned with human judgment than SSIM, PSNR, and LPIPS, and recommends DreamSim for general NVS evaluation.

Significance. If fully validated, the paper would provide a large-scale empirical characterization of how four image similarity metrics respond to synthetic corruptions, and would support a specific recommendation for NVS evaluation. The scale of the ImageNet-C experiment (750k variants) and the use of segmentation to separate foreground and background corruptions are notable strengths. However, the paper's central recommendation rests on two externally oriented claims that are not directly tested: that the synthetic corruptions faithfully represent artifacts produced by real NVS models, and that the observed score trends correspond to human perceptual judgment. As presented, the work is a useful internal comparison of metric behaviors under controlled distortions, but the human-alignment and real-NVS applicability conclusions are unsubstantiated.

major comments (4)
  1. [Section 3.1 and Section 5] The paper's central recommendation—that 'DreamSim or a future NVS-specifically trained version of DreamSim should be used for general evaluation of NVS models'—rests on the assertion (Section 3.1) that the 12 NVS Corruptions 'simulate common corruptions typically seen in NVS model outputs.' The text provides example real artifacts in Figure 2 but never quantitatively compares the distribution of those artifacts with the synthetic corruption suite (e.g., type frequencies, severity ranges, spatial extent). Without such validation, the benchmark measures metric behavior under hand-chosen distortions, not specifically under NVS render artifacts. Please either add a validation experiment on real NeRF/3DGS renders (e.g., apply the metrics to paired real renders and categorize their artifacts) or considerably temper the claim that the results transfer to NVS outputs.
  2. [Sections 4.1, 4.3, 4.4] The paper repeatedly claims that DreamSim's score curves 'align with human judgement' (Section 4.1), that its foreground/background sensitivity 'align[s] more closely with human judgement' (Section 4.3), and that small pixel crops are 'imperceptible' and do not affect 'scene understanding' (Sections 3.4 and 4.4). No human opinion scores are collected or cited for this specific benchmark, and DreamSim's training on human similarity judgments (reference [2]) does not by itself establish that its monotonic curves track human utility for NVS renders; this is a partial circularity. Please add a small human-rating study on a subset of corrupted and cropped images, or compare against an established subjective NVS quality dataset such as references [6] or [9], to support the human-alignment claims.
  3. [Section 3.3, Figures 9, 12, 13] The paper presents median score curves without any error bars, confidence intervals, or per-condition sample sizes. The NVS Scenes dataset has only 24 images, and the foreground/background experiment is limited to approximately 500 images due to manual mask validation. Because the central claims about discriminative power and relative robustness rely on the shapes of these median curves, the absence of variance measures makes it impossible to assess whether observed differences (e.g., DreamSim versus LPIPS plateau behavior, or the foreground/background gap) are statistically meaningful. Please report per-condition quantiles or bootstrap confidence intervals, and state the number of images per condition for each figure.
  4. [Section 3.2 and Section 4.4] The phrase 'imperceptible to the human eye' is used to describe pixel crops of 5–10 pixels (Section 3.2) and the paper later refers to 'small, imperceptible artifacts' (Section 4.4). Whether a given crop is perceptually invisible depends on image resolution, viewing distance, and content; this is asserted rather than measured. Without a perceptual verification (e.g., two-alternative forced choice with human observers, or a published visibility threshold), the claim that DreamSim is 'less sensitive to corruptions imperceptible by humans' conflates small pixel shifts with imperceptibility. Please either verify imperceptibility empirically or rephrase the claim to refer to 'small pixel crops' rather than 'imperceptible corruptions.'
minor comments (6)
  1. [Section 4.1] The text states that fog 'has little perceptual change across severity levels, depicted in Figure 10,' but Figure 10 shows pixelation and Figure 11 shows fog. Please correct the cross-reference.
  2. [Figure 9 caption] The caption says 'our NVS rendered dataset' while the text and Table 1 refer to the 'NVS Scenes' dataset. Use consistent terminology throughout.
  3. [Table 1] The row for ImageNet-C lists '10k' images but does not explain that this is 200 classes × 50 images per class. Please include this derivation in the text or table note.
  4. [Figure 13(b) caption] The caption states that 'DreamSim and SSIM are normalized to the same 0 to 1 scale' but does not explain how PSNR and LPIPS are normalized in the same plot. Please clarify the normalization applied to each metric.
  5. [References] Reference [12] has garbled author names ('Benand Mildenhall,' 'Matthewand Tancik,' 'Raviand Ramamoorthi,' 'Andreaand Vedaldi,' etc.). Please correct these entries.
  6. [Abstract] The sentence 'traditional metrics are unable to effectively differentiate between images with minor pixel-level changes and those with substantial corruption' is too strong given that the benchmark uses specific synthetic corruptions; it would be more accurate to say 'in the tested corruption scenarios.'

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the benchmark evaluates metric behavior on synthetic corruptions without fitting any parameter to the conclusions; the human-alignment premise is imported from external prior work, not re-derived from the present paper's outputs.

full rationale

The paper makes no fitted-parameter predictions and contains no equations whose outputs are definitionally equal to their inputs. Its experimental contribution is a set of sensitivity and discriminability measurements (median metric scores vs. corruption severity) on its own NVS Corruptions suite and on ImageNet-C. Those measurements are self-contained: the claim that DreamSim decreases gradually and differentiates severity levels is directly read from Figure 9 and Figure 13 and does not presuppose the conclusion. The recommendation to use DreamSim for NVS evaluation does rely on the premise that DreamSim aligns with human judgment, but that premise is cited to the external DreamSim training work [2] (which collected human similarity ratings) and to human-vision literature [8]. No author of the present paper is an author of those sources, so this is external evidentiary support rather than a self-citation loop. The chief weakness is that the paper never validates that its synthetic corruptions match real NVS artifact distributions or collects human ratings on this benchmark; however, those are unverified assumptions about external validity, not circular derivations. Accordingly, under the specified criteria no step reduces to its own inputs, and the appropriate circularity score is 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No free parameters are fitted in the paper; the corruption suite and severity levels are experimental design choices. The key axioms are that artificial corruptions represent real NVS artifacts, that DreamSim's pretrained behavior can stand in for human judgement, and that normalized median scores permit cross-metric comparison. No invented entities are introduced.

assumptions (4)
  • domain assumption Artificial corruptions emulate NVS render artifacts
    Section 3 asserts the 12 'NVS Corruptions' simulate common corruptions typically seen in NVS model outputs; this proxy is never validated against real NVS renders in the paper.
  • domain assumption DreamSim's pretrained similarity scores reflect human judgement
    Section 2.2 cites DreamSim's training on human judgments, and Section 5 uses this to conclude human alignment without collecting human ratings on the benchmark images.
  • domain assumption Foreground clarity bias is a human-like trait
    Section 4.3 cites [8] to interpret DreamSim's stronger foreground sensitivity as consistent with human vision, but no human experiment is run here.
  • domain assumption Normalizing each metric to 0-1 makes scores comparable across metrics
    Section 3 states all scores are normalized to a common scale, but the original scales and distributions of SSIM, PSNR, LPIPS, and DreamSim differ, so median curve shapes may not be directly comparable.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Benchmarking Image Similarity Metrics for Novel View Synthesis Applications." pith.science (2026). https://pith.science/paper/562MMD2I

@misc{pith2026250612563,
  author       = {Pith},
  title        = {Pith review of: Benchmarking Image Similarity Metrics for Novel View Synthesis Applications},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/562MMD2I}},
  note         = {Machine review of arXiv:2506.12563}
}
read the original abstract

Traditional image similarity metrics are ineffective at evaluating the similarity between a real image of a scene and an artificially generated version of that viewpoint [6, 9, 13, 14]. Our research evaluates the effectiveness of a new, perceptual-based similarity metric, DreamSim [2], and three popular image similarity metrics: Structural Similarity (SSIM), Peak Signal-to-Noise Ratio (PSNR), and Learned Perceptual Image Patch Similarity (LPIPS) [18, 19] in novel view synthesis (NVS) applications. We create a corpus of artificially corrupted images to quantify the sensitivity and discriminative power of each of the image similarity metrics. These tests reveal that traditional metrics are unable to effectively differentiate between images with minor pixel-level changes and those with substantial corruption, whereas DreamSim is more robust to minor defects and can effectively evaluate the high-level similarity of the image. Additionally, our results demonstrate that DreamSim provides a more effective and useful evaluation of render quality, especially for evaluating NVS renders in real-world use cases where slight rendering corruptions are common, but do not affect image utility for human tasks.

Figures

Figures reproduced from arXiv: 2506.12563 by the authors.

Figure 1
Figure 1. Which metric most effectively scores the similarity of these images to the reference? Our paper assesses commonly used image similarity metrics on how well they evaluate images with corruptions commonly found in NVS renders. Each row represents a sequence of images scored by a different metric, where the scores are normalized from 0 (lowest similarity) to 1 (highest similarity). Abstract Traditional image similarity… view at source ↗
Figure 2
Figure 2. Examples of real NVS artifacts [9]. Notably, NVS ar￾tifacts can affect the entire image as well as local patches of an image. rithm, commonly a structure-from-motion based approach [15], to estimate camera positioning from input images. Second, a training step leverages the same images and their previously estimated camera positions as input and opti￾mizes a model to minimize the per-pixel difference between generat… view at source ↗
Figure 3
Figure 3. The 12 unique corruption types applied to the NVS [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Example of the splat corruption at intensity levels 2, 5, [PITH_FULL_IMAGE:figures/full_fig_p003_4.png]
Figure 7
Figure 7. Figure 7: Foreground/Background Corruptions overview: original [PITH_FULL_IMAGE:figures/full_fig_p004_7.png]
Figure 5
Figure 5. Figure 5: Examples of the corruptions included in the ImageNet-C [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]
Figure 6
Figure 6. Figure 6: Illustration of each severity level of the Gaussian noise [PITH_FULL_IMAGE:figures/full_fig_p004_6.png]
Figure 8
Figure 8. Figure 8: A zoomed-in illustration of a corner of an image [PITH_FULL_IMAGE:figures/full_fig_p004_8.png]
Figure 9
Figure 9. Figure 9: Median scores graphed across each severity level in our NVS rendered dataset. All metric scores are normalized to the same [PITH_FULL_IMAGE:figures/full_fig_p005_9.png]
Figure 11
Figure 11. Figure 11: An example of the fog corruption, which had little [PITH_FULL_IMAGE:figures/full_fig_p005_11.png]
Figure 12
Figure 12. Figure 12: Median normalized scores of each metric as pixels are [PITH_FULL_IMAGE:figures/full_fig_p006_12.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Walk through Paintings: Egocentric World Models from Internet Priors

    cs.CV 2026-01 conditional novelty 7.0 of 10

    A lightweight action-conditioning layer inserted into pre-trained video diffusion models turns them into egocentric world models that follow 3-DoF and 25-DoF action commands and generalize to unseen environments such ...

  2. Heterogeneous Element-Aware Cross-Version Differencing of Scientific Documents via Layout-Aware Alignment and Structure-Aware Reasoning

    cs.CL 2026-05 conditional novelty 5.0 of 10

    A layout-aware, alignment-first framework decomposes two PDF versions into typed elements, aligns them, and reports detection, localization, and structure-aware changes, outperforming element-specific baselines on a p...

Reference graph

Works this paper leans on

19 extracted references · 15 canonical work pages · cited by 2 Pith papers

  1. [2]

    Dream- sim: Learning new dimensions of human visual similarity using synthetic data

    Stephanie Fu, Netanel Tamir, Shobhita Sundaram, Lucy Chai, Richard Zhang, Tali Dekel, and Phillip Isola. Dream- sim: Learning new dimensions of human visual similarity using synthetic data. InAdvances in Neural Information Pro- cessing Systems, volume 36. Curran Associates, Inc., 2023. 1, 3

  2. [6]

    Perceptual quality assessment of nerf and neural view syn- thesis methods for front-facing views, 2023

    Hanxue Liang, Tianhao Wu, Param Hanji, Francesco Ban- terle, Hongyun Gao, Rafal Mantiuk, and Cengiz Oztireli. Perceptual quality assessment of nerf and neural view syn- thesis methods for front-facing views, 2023. 1, 2

  3. [9]

    Nerf view synthesis: Subjective quality assessment and objective metrics evaluation, 2024

    Pedro Martin, Antonio Rodrigues, Joao Ascenso, and Maria Paula Queluz. Nerf view synthesis: Subjective quality assessment and objective metrics evaluation, 2024. 1, 2

  4. [1]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition. Ieee, 2009. 3

  5. [3]

    Benchmarking neu- ral network robustness to common corruptions and perturba- tions

    Dan Hendrycks and Thomas Dietterich. Benchmarking neu- ral network robustness to common corruptions and perturba- tions. arXiv preprint arXiv:1903.12261, 2019. 3

  6. [4]

    3d gaussian splatting for real-time radiance field rendering

    Bernhard Kerbl, Georgios Kopanas, Thomas Leimk ¨uhler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Trans. Graph., 42(4), 2023. 2

  7. [5]

    Berg, Wan-Yen Lo, Piotr Doll ´ar, and Ross Girshick

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C. Berg, Wan-Yen Lo, Piotr Doll ´ar, and Ross Girshick. Segment anything. In 2023 IEEE/CVF In- ternational Conference on Computer Vision (ICCV) , 2023. 4

  8. [7]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499, 2023. 4

Show all 19 references
  1. [8]

    Across the planes: Dif- fering impacts of foreground and background information on visual search in scenes

    Louisa Man and Monica Castelhano. Across the planes: Dif- fering impacts of foreground and background information on visual search in scenes. Journal of Vision, 18, 2018. 6

  2. [10]

    Ricardo Martin-Brualla, Noha Radwan, Mehdi S. M. Sajjadi, Jonathan T. Barron, Alexey Dosovitskiy, and Daniel Duck- worth. Nerf in the wild: Neural radiance fields for uncon- strained photo collections. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Re...

  3. [11]

    lang-segment-anything

    Luca Mederios. lang-segment-anything. https : / / github . com / luca - medeiros / lang - segment - anything, 2024. 4

  4. [12]

    Srinivasan, Matthewand Tan- cik, Jonathan T

    Benand Mildenhall, Pratul P. Srinivasan, Matthewand Tan- cik, Jonathan T. Barron, Raviand Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In Andreaand Vedaldi, Horstand Bischof, Thomasand Brox, and Jan-Michael Frahm, editors, Co...

  5. [13]

    An evalu- ation of quality metrics for neural radiance field

    Chibuike Onuoha, Jean Atsumi Flaherty, Shihao Luo, Truong Thu Huong, and Truong Cong Thang. An evalu- ation of quality metrics for neural radiance field. In 2023 IEEE 15th International Conference on Computational In- telligence and Communication Networks (CICN) , 2023. 1, 2

  6. [14]

    Nerf-nqa: No-reference quality assess- ment for scenes generated by nerf and neural view synthesis methods

    Qiang Qu, Hanxue Liang, Xiaoming Chen, Yuk Ying Chung, and Yiran Shen. Nerf-nqa: No-reference quality assess- ment for scenes generated by nerf and neural view synthesis methods. IEEE Transactions on Visualization and Computer Graphics, 30(5):2129–2139, 2024. 1, 2

  7. [15]

    Schonberger and Jan-Michael Frahm

    Johannes L. Schonberger and Jan-Michael Frahm. Structure- from-motion revisited. In Proceedings of the IEEE Confer- ence on Computer Vision and Pattern Recognition (CVPR) ,

  8. [16]

    Doi- baa-wriva-fy22-02, 2022

    Acquisition Services Directorate (AQD) The Department of the Interior (DOI), Interior Business Center (IBC). Doi- baa-wriva-fy22-02, 2022. 2

  9. [17]

    Benchmarking robustness in neural radiance fields

    Chen Wang, Angtian Wang, Junbo Li, Alan Yuille, and Cihang Xie. Benchmarking robustness in neural radiance fields. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR) Workshops ,

  10. [18]

    Image quality assessment: from error visibility to structural similarity

    Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Si- moncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):600–612, 2004. 1, 2

  11. [19]

    The unreasonable effectiveness of deep features as a perceptual metric

    Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shecht- man, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 586–595, 2018. 1, 2

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.