Pith. sign in

REVIEW 3 major objections 4 minor 2 references

CGVQM+D: Computer Graphics Video Quality Metric and Dataset

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read CGVQM, a full-reference video quality metric based on pre-trained 3D CNN features, significantly outperforms existing metrics on modern rendering artifacts and yields both per-pixel error maps and global quality scores.

desk verdict CG-VQD is a valuable new dataset and the 3D-CNN finding is credible, but CGVQM's reported advantage is inflated by test-loss selection; worth a serious referee. read the letter →

arxiv 2506.11546 v1 pith:DA53GV2H submitted 2025-06-13 cs.GR cs.CV

Akshay Jindal , Nabil Sadaka , Manu Mathew Thomas , Anton Sochenov , Anton Kaplanyan This is my paper · ORCID
classification cs.GRcs.CV
keywords videoqualityassessmentfull-referencemetriccomputergraphicsrenderingartifacts3DconvolutionalneuralnetworksubjectivedatasetDMOSsupersampling
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Modern rendering techniques—neural supersampling, path tracing, neural denoising, Gaussian splatting, frame interpolation, and variable rate shading—produce artifacts that standard video quality metrics misjudge. The paper builds a new subjective dataset, CG-VQD, of 80 short clips rated by 20 participants, and shows that existing full-reference metrics top out at 0.78 Pearson correlation with human opinion. It then shows that distances measured in the feature space of a pre-trained 3D ResNet-18 align far better with human perception, and calibrates a small set of channel weights to turn those feature distances into a metric, CGVQM, that reaches about 0.87 correlation and also produces per-pixel error maps. If correct, CGVQM gives graphics engineers an automatic way to detect and localize the specific artifacts viewers will notice.

What carries the argument

The load-bearing object is the feature-distance computation from a frozen 3D ResNet-18, a spatiotemporal convolutional network that processes short video clips. Reference and distorted videos are passed through the network; at each of five selected convolutional layers the feature maps are unit-normalized along the channel dimension, each channel is scaled by a learned weight $\omega_l$, and the squared $\ell^2$ distance between the normalized feature tensors is computed. Summing this distance over space, time, and layers gives the global quality score $q(x,x_0)$; interpolating the per-layer distances to full resolution gives the per-pixel error map $e(x,x_0)$. Only about 1000 channel weights are learned by maximizing Pearson correlation with human ratings on three calibration datasets—GamingVideoSET, LIVE Livestream, and CG-VQD itself—while all convolutional kernels stay frozen, which is what keeps the small quality datasets from causing overfitting.

What would settle it

A controlled-lab subjective study with a larger, screened, more diverse participant pool rating a fresh set of rendering-distorted videos (new scenes and artifact levels not drawn from CG-VQD) would settle the claim: if CGVQM's Pearson correlation with the resulting DMOS no longer significantly exceeds that of the best existing metric (e.g., HaarPSI or ColorVideoVDP), the central claim fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that the feature space of a pre-trained 3D convolutional neural network—trained for action recognition, not for quality—is a better perceptual space for judging rendering artifacts than any existing full-reference quality metric. On the new CG-VQD dataset of 80 videos with distortions from six modern rendering techniques, existing metrics reach a maximum Pearson correlation of 0.78 with human difference-mean-opinion scores; CGVQM, which compares channel-normalized deep features of the reference and distorted videos through a frozen 3D ResNet-18 and calibrates only per-channel weights, reaches about 0.87 on the same test split and also produces per-pixel error maps that localize artifacts. The authors report that this advantage holds across six additional video quality datasets covering compression, livestreaming, flicker, and neural view synthesis, that 3D temporal modeling is essential (2D-based LPIPS trails), and that both the network's pre-training and the calibration step contribute significantly to the gain.

Load-bearing premise

The central claim rests on the subjective ratings from 20 remote volunteers—collected without controlled viewing conditions or vision screening—being reliable and representative enough to serve as the ground truth that decides which metric is better.

Editorial extensions

If this is right

  • Rendering engineers can use CGVQM to rank and compare rendering settings without running user studies, since it correlates at about 0.87 with human DMOS on modern rendering artifacts.
  • CGVQM's per-pixel error maps localize where viewers will notice artifacts, such as ghosting around bright lights or noise in shadow areas, a capability that global-only metrics like VMAF and DISTS lack.
  • Strong cross-dataset performance makes CGVQM a reliable full-reference metric for synthetic content beyond CG-VQD, including cloud gaming, livestreaming, flicker, and neural view synthesis.
  • The lighter CGVQM-2 variant is 27% faster and nearly as accurate on some benchmarks, so artifact assessment with error localization is feasible in time-constrained rendering pipelines.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because CGVQM-2 needs only the first two ResNet blocks, a GPU-specialized implementation might run in real time inside a game engine or streaming encoder, enabling online artifact monitoring—an extension the paper does not claim.
  • The acknowledged weakness of mean pooling on localized errors suggests that replacing it with saliency-weighted or max pooling would improve CGVQM on scenes where a small artifact dominates perception; that fix is directly testable on CG-VQD.
  • If 3D CNN features generalize this well across distortion types, a no-reference variant that scores a single video against a learned clean-feature distribution is a natural next step for settings where no reference exists, such as live game capture.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The manuscript introduces CG-VQD, a subjective video quality dataset of 80 clips containing distortions from modern rendering techniques (neural supersampling, path tracing, neural denoising, Gaussian splatting, frame interpolation, and variable rate shading), collected remotely from 20 participants. It proposes CGVQM, a full-reference metric that extracts features from a frozen pretrained 3D ResNet-18, weights channels by a vector omega optimized to maximize Pearson correlation with human ratings, pools patch-level qualities, and can produce per-pixel error maps. The authors calibrate omega on GamingVideoSET, LIVE Livestream, and CG-VQD, and report results on those datasets plus six independent datasets, claiming significant improvement over existing full-reference metrics and strong error-localization performance on LocVis.

Significance. If the reported performance survives a fair re-evaluation, this is a useful contribution: the dataset addresses an underexplored set of rendering artifacts, the finding that pretrained 3D CNN features align with human quality judgments is of independent interest, and the metric provides both global scores and localized error maps. Concrete strengths include the public release of dataset and code, comparison with a large set of baseline metrics, generalization tests on six external datasets, and a detailed inter-rater reliability analysis in Appendix A1.1. The main reservation is that the central claim of 'significantly outperforms existing metrics' is supported primarily by the three calibration datasets, where the model selection protocol uses test labels; the external datasets provide mixed evidence. These issues are addressable and do not negate the dataset contribution.

major comments (3)
  1. [Section 4.1, 'Feature calibration'] The paper states that 'the system was trained for 100,000 epochs and the ω corresponding to the lowest test loss was selected' and that 'no separate validation set was used.' Because the same test splits (60, 252, and 65 videos for GamingVideoSET, LIVE Livestream, and CG-VQD) are subsequently used to report PLCC/SRCC in Figure 4c and Table 1, early stopping is performed on the evaluation labels. With 1027 free parameters and a small training set, this can bias the reported correlations upward; the 10 random splits and bootstrap error bars do not correct selection bias, since selection on test labels is repeated in every split. This is load-bearing for the abstract's claim of significant outperformance on these datasets. Please re-evaluate with a validation-based early-stopping rule, a nested cross-validation, or a held-out set that never influences ω selection, and report both the selected-model performance and the performance under the fair protocol.
  2. [Section 4.2, 'Comparison with existing metrics', Eq. (4), and Table 1] The claim that CGVQM-5 'significantly outperforms existing metrics' is not consistently supported by the independent datasets. On CGVDS, NVS, and BVI-HD, CGVQM-5 obtains PLCC of 0.67, 0.697, and 0.69, respectively, while DISTS achieves 0.85 and 0.852 on CGVDS and NVS, and VMAF and FoVVideoVDP achieve 0.772 and 0.796 on BVI-HD. Strong advantages over all baselines appear mainly on the three calibration datasets, which are exactly the ones affected by test-loss-based model selection. The external results support a claim of competitiveness and robustness, not uniform significant superiority. Please either temper the claim or provide independent evidence, e.g., a validation-based re-evaluation showing significant gains on held-out data.
  3. [Section 4.1, 'Feature calibration' (patch pooling) and Section 5] There is an inconsistency in the pooling description. Section 4.1 defines the overall video quality as the minimum quality over patches, while Section 5 states 'we use mean pooling over space and time to compute the overall quality of a video' without mentioning the minimum over patches. The choice of min pooling for patch aggregation is nontrivial and can strongly affect the metric's behavior for localized distortions, which the paper itself discusses. Please clarify the exact aggregation used in the experiments and consider reporting sensitivity to the min-pooling rule.
minor comments (4)
  1. [Equation 1] Equation (1) is difficult to read because the tensor subscripts and superscripts are garbled; please rewrite it with clear indices for the feature maps and pooling operations.
  2. [Section 3.3 and Appendix A1.1] The remote-viewing and vision-screening caveat is an important limitation of the subjective ground truth; consider moving a summary of it, along with the participant demographics, into the main text rather than leaving it only in the appendix.
  3. [Section 4.2, Eq. (4)] Please state explicitly whether the five-parameter nonlinear mapping in Eq. (4) is fitted on the same test set used for the reported correlations. If so, note that this is a standard but slightly optimistic protocol, and describe how the fitting is done for CGVQM versus the baseline metrics.
  4. [Appendix A1.3, Table A3] The significance-testing description would benefit from specifying how bootstrap samples are generated for the paired t-test and whether any multiple-comparison correction is applied across the many metric pairs.

Circularity Check

1 steps flagged · score 5.0 of 10

CGVQM's headline gain on its three calibration datasets is partly an artifact of selecting omega by test loss on those same datasets; only the six external datasets provide fully independent evidence.

  1. fitted input called prediction [Section 4.1, 'Feature calibration' (after Eq. 3); reported against test splits in Section 4.2 / Table 1]
    "The system was trained for 100,000 epochs and the ω corresponding to the lowest test loss was selected. Both training and test loss curves were observed to plateau asymptotically. No separate validation set was used due to the limited size of the available data."

    The only learned component of CGVQM is the 1027-element channel weight vector ω, obtained by maximizing PLCC with human DMOS on training subsets (Eq. 3), while the checkpoint is chosen by minimizing loss on the test subsets of the same three datasets (GamingVideoSET, LIVE Livestream, CG-VQD). The paper then reports PLCC/SRCC on those test subsets as evidence that CGVQM 'significantly outperforms existing metrics' (abstract and Fig. 4c/Table 1). Because test labels participate in model selection, the reported test correlations are selected rather than independently predicted, and no baseline metric receives a comparable selection step. Eq. 1 itself is not definitionally equal to DMOS and the six external datasets were not used for selection, so the circularity is partial rather than total.

full rationale

The core metric definition is not circular: q(x,x0) in Eq. 1 is a frozen-feature, weighted l2 distance, and optimizing ω to maximize PLCC with human ratings in Eq. 3 is standard calibration rather than a tautology. The paper also reports valuable independent evidence: CGVQM's weights are frozen and evaluated on LIVE Meta, CGVDS, LIVE Flicker, NVS, AVT-VQDB-UHD-1, and BVI-HD, none of which were used for calibration or selection; Table 1 and the LocVis error-localization experiment are therefore genuinely external checks. However, the central claim of 'significantly outperforming existing metrics' leans heavily on the three calibration datasets, and the paper explicitly states that no validation set was used and that the ω corresponding to the lowest test loss was selected. This means the reported test performance on those datasets is optimistically biased by checkpoint/early-stopping selection on the same labels used for evaluation. The 10 random splits and bootstrapping reduce variance but not this selection bias. This is a statistical leakage in the benchmark protocol rather than a definitional equivalence between inputs and outputs, so the appropriate score is moderate: 5.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central metric depends on 1027 fitted channel weights, a small ground-truth dataset from 20 participants, and the assumption that frozen Kinetics-400 features are perceptually relevant for synthetic content. The model-selection procedure introduces an additional implicit assumption that the test loss used for early stopping does not bias results.

free parameters (4)
  • Channel weight vector omega for ResNet = Not disclosed (learned vector)
    1027 weights in Eq. 1 and 3, optimized by maximizing PLCC on training splits of GamingVideoSET, LIVE Livestream, and CG-VQD.
  • Channel weight vector omega for C3D = Not disclosed (learned vector)
    1475 weights for the C3D variant used in comparisons.
  • Five-parameter nonlinear mapping eta (Eq. 4) = Fit per dataset
    Fit to each benchmark dataset before computing PLCC/RMSE, applied to all metrics.
  • Patch size and minimum pooling = 30x512x512, min pooling
    Design choices for handling high resolution and emphasizing the worst patch.
assumptions (4)
  • domain assumption Pre-trained 3D ResNet-18 features on Kinetics-400 provide perceptually relevant representations for synthetic content and rendering artifacts.
    Core premise of the metric; supported by the paper's calibration experiments but not independently established.
  • domain assumption The DMOS derived from 20 remote participants under uncontrolled viewing conditions accurately reflects perceived quality.
    Section 3.3 discloses lack of control over viewing conditions and no vision screening; reliability analysis in Appendix A1.1 mitigates but does not eliminate this risk.
  • ad hoc to paper A single global channel-weight vector omega can capture the perceptual importance of features across diverse distortion classes.
    The metric uses one omega for all videos and distortion types; the paper does not test distortion-specific weights.
  • ad hoc to paper Selecting the epoch with the lowest test loss does not materially bias the reported test correlations.
    Section 4.1 states no validation set was used and test loss was used for model selection, which is an implicit (and likely violated) assumption.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CGVQM+D: Computer Graphics Video Quality Metric and Dataset." pith.science (2026). https://pith.science/paper/DA53GV2H

@misc{pith2026250611546,
  author       = {Pith},
  title        = {Pith review of: CGVQM+D: Computer Graphics Video Quality Metric and Dataset},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DA53GV2H}},
  note         = {Machine review of arXiv:2506.11546}
}
read the original abstract

While existing video and image quality datasets have extensively studied natural videos and traditional distortions, the perception of synthetic content and modern rendering artifacts remains underexplored. We present a novel video quality dataset focused on distortions introduced by advanced rendering techniques, including neural supersampling, novel-view synthesis, path tracing, neural denoising, frame interpolation, and variable rate shading. Our evaluations show that existing full-reference quality metrics perform sub-optimally on these distortions, with a maximum Pearson correlation of 0.78. Additionally, we find that the feature space of pre-trained 3D CNNs aligns strongly with human perception of visual quality. We propose CGVQM, a full-reference video quality metric that significantly outperforms existing metrics while generating both per-pixel error maps and global quality scores. Our dataset and metric implementation is available at https://github.com/IntelLabs/CGVQM.

Figures

Figures reproduced from arXiv: 2506.11546 by the authors.

Figure 1
Figure 1. We collected a video quality dataset containing distortions generated by modern rendering algorithms. Using this dataset, we demonstrate that the feature space of pre-trained 3D-CNNs exhibits a high correlation with human ratings and propose a new video quality metric, CGVQM, which outperforms existing quality metrics. Abstract While existing video and image quality datasets have extensively studied natural videos a… view at source ↗
Figure 2
Figure 2. User interface used in the experiment. periment was authorized by an internal institutional review board and participants received token compensation. It was conducted remotely on the participants’ device. Each participant reviewed a briefing and signed a written consent form before starting the ex￾periment. All participants reported prior experience in video quality evaluation but were naïve to the content of this … view at source ↗
Figure 3
Figure 3. Experiment stimuli and results from our computer graphics video quality dataset. Examples of distortions can be found in [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Details on proposed metric architecture and quantitative comparison with other quality metrics. Error bars were generated via bootstrapping and denote 95% confidence interval. submitted to High-Performance Graphics (2025) [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Metric performance for different rendering methods. available in Appendix Table A1. While some existing metrics per￾form well on specific datasets, such as absolute error (ABS) on GamingVideoDataset or VMAF on video compression datasets, CGVQM-5 consistently achieves h…
Figure 7
Figure 7. Figure 7: Reference/distorted images, observers’ markings and metric predictions for selected examples from the LocVis dataset. Metric predictions must be viewed in color [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]
Figure 9
Figure 9. Figure 9 [PITH_FULL_IMAGE:figures/full_fig_p011_9.png]
Figure 8
Figure 8. Figure 8: Example error maps generated by CGVQM-2 for the Bridge (3×) and Jungle2 (4spp) scenes. CGVQM-2 effectively high￾lights ghosting and noise artifacts as perceptually significant in bright and diffuse regions (car light and tree shadows), respec￾tively, while appropriatel…
Figure 10
Figure 10. Figure 10: Examples of distortions arising from different rendering methods used in CG-VQD. Only highest levels of distortions are visual￾ized in this figure. submitted to High-Performance Graphics (2025) [PITH_FULL_IMAGE:figures/full_fig_p016_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

2 extracted references · 2 linked inside Pith

  1. [12]

    Study of spatio-temporal model- ing in video quality assessment

    2024. URL: https : / / www . fab . com / listings / d813ecef - 8346 - 4d8c - 9484 - 169da72a80aa. [EW21] Epic Games and Weta Digital. Meerkat Demo . Ac- cessed: 2025-04-12. May 2021. URL: https : / / submitted to High-Performance Graphics (2025) Jindal et al. / CGVQM+D 13 of 27 www . unrealengine . com / marketplace / en - US / product / meerkat - demo - ...

    arXiv 2025

  2. [2019]

    A subjective and objective study of space-time subsampled video quality

    URL: https : / / github . com / photosynthesis-team/piq. [Lee+21] Dae Yeol Lee et al. “A subjective and objective study of space-time subsampled video quality”. In: IEEE Transactions on Image Processing 31 (2021), pp. 934–948. [Li+20] Zhi Li et al. “A simple model for subject behav- ior in subjective experiments”. In: arXiv preprint arXiv:2004.02067 (2020...

    arXiv 2021

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.