{"id":"cee32971-0919-4bfa-b368-d9a88db7d7c2","arxiv_id":"2506.11546","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A new 80-video dataset and a 3D CNN-based full-reference metric, CGVQM, aim to assess artifacts from modern rendering techniques and outperform existing metrics.","lead":"The authors built a video quality dataset (CG-VQD) with 80 videos showing artifacts from modern rendering techniques such as neural supersampling, path tracing, and frame interpolation, rated by 20 human viewers. They then created CGVQM, a full-reference metric using pre-trained 3D CNN features, which they report outperforms existing quality metrics on this new data.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"CGVQM's reported advantage is inflated because its 1027 feature weights are selected on the same test splits used for reporting; a validation-based re-evaluation is required to support the claim of significant improvement.","rationale":"In good faith, the paper makes two credible contributions: a new graphics-specific video quality dataset and a finding that pre-trained 3D CNN features correlate with human ratings of rendering artifacts. The cross-dataset results on datasets never used for calibration provide independent evidence that the approach generalizes somewhat, and the released code makes the work reproducible. However, the strongest claim in the abstract explicitly says CGVQM \"significantly outperforms existing metrics,\" and the evidence for that claim is weakened by a methodological flaw: the feature weights ω are selected by minimizing loss on the very test splits whose correlation values are then reported. The manuscript itself flags this when it says \"No separate validation set was used due to the limited size of the available data.\" This is an internal inconsistency with standard evaluation practice, not merely a disagreement with community consensus, so it is a correctness risk rather than a novelty dispute. The reader's stated weakest assumption was DMOS reliability, but the appendix's inter-rater analyses and outlier rejection partly address that concern, whereas the calibration leak is not addressed by any reported experiment. A validation-based re-run is concrete, low-cost, and would settle whether the reported margins over HaarPSI and ColorVideoVDP are real. Therefore the conditional verdict should stand until that check is performed.","tokens_in":27347,"tokens_out":3303,"duration_ms":34110,"concrete_test":"Re-run the ω calibration with a clean train/validation/test split: hold out scene-disjoint validation sets, including 15 CG-VQD videos, optimize ω on the training split only, select the epoch and ω with best validation PLCC, then lock ω and evaluate once on the held-out test split. Compare the resulting CGVQM-5 PLCC and SRCC on CG-VQD against the reported 0.871/0.877 and against HaarPSI's 0.777/0.794 and ColorVideoVDP's 0.744/0.775. If the validation-based PLCC falls to within error bars of the second-best metric, the abstract's claim of significant improvement is unsupported; if it remains above with non-overlapping confidence intervals, the central claim survives.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that CGVQM-5 \"significantly outperforms existing metrics\" rests on the calibration protocol in Section 4.1. The paper states: \"The system was trained for 100,000 epochs and the ω corresponding to the lowest test loss was selected... No separate validation set was used due to the limited size of the available data.\" Because the same test splits (60/252/65 videos for GamingVideoSET, LIVE Livestream, and CG-VQD) are used both to select ω (1027 free parameters for ResNet-18) and to report PLCC/SRCC, the reported scores are optimistically biased estimates of performance on unseen data. This is not a minor implementation detail: the only learned component of CGVQM is ω, and it is fit by maximizing Pearson correlation (Eq. 3) on a portion of each dataset and then chosen by maximizing correlation on the remaining portion. Existing metrics are not retrained and receive no such selection, so the comparison is asymmetric. The 10 random splits and bootstrapped error bars do not remove the bias; repeated selection on test labels can only increase overfitting. External datasets (LIVE Meta, CGVDS, LIVE Flicker, NVS, AVT, BVI-HD) were not used for selection and therefore provide genuinely independent evidence, but CGVQM-5's headline advantage on the three calibration datasets is not trustworthy as reported. A related secondary issue is that early stopping is also performed on test loss, adding another selection step on the same labels. The subjective DMOS reliability concern raised by the reader is partially mitigated by the reported ICC(2,k)=0.97 and one participant excluded as an outlier, so the calibration leak is the more load-bearing concern.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript introduces CG-VQD, a subjective video quality dataset of 80 clips containing distortions from modern rendering techniques (neural supersampling, path tracing, neural denoising, Gaussian splatting, frame interpolation, and variable rate shading), collected remotely from 20 participants. It proposes CGVQM, a full-reference metric that extracts features from a frozen pretrained 3D ResNet-18, weights channels by a vector omega optimized to maximize Pearson correlation with human ratings, pools patch-level qualities, and can produce per-pixel error maps. The authors calibrate omega on GamingVideoSET, LIVE Livestream, and CG-VQD, and report results on those datasets plus six independent datasets, claiming significant improvement over existing full-reference metrics and strong error-localization performance on LocVis.","tokens_in":27730,"tokens_out":5526,"duration_ms":58973,"significance":"If the reported performance survives a fair re-evaluation, this is a useful contribution: the dataset addresses an underexplored set of rendering artifacts, the finding that pretrained 3D CNN features align with human quality judgments is of independent interest, and the metric provides both global scores and localized error maps. Concrete strengths include the public release of dataset and code, comparison with a large set of baseline metrics, generalization tests on six external datasets, and a detailed inter-rater reliability analysis in Appendix A1.1. The main reservation is that the central claim of 'significantly outperforms existing metrics' is supported primarily by the three calibration datasets, where the model selection protocol uses test labels; the external datasets provide mixed evidence. These issues are addressable and do not negate the dataset contribution.","major_comments":[{"comment":"The paper states that 'the system was trained for 100,000 epochs and the ω corresponding to the lowest test loss was selected' and that 'no separate validation set was used.' Because the same test splits (60, 252, and 65 videos for GamingVideoSET, LIVE Livestream, and CG-VQD) are subsequently used to report PLCC/SRCC in Figure 4c and Table 1, early stopping is performed on the evaluation labels. With 1027 free parameters and a small training set, this can bias the reported correlations upward; the 10 random splits and bootstrap error bars do not correct selection bias, since selection on test labels is repeated in every split. This is load-bearing for the abstract's claim of significant outperformance on these datasets. Please re-evaluate with a validation-based early-stopping rule, a nested cross-validation, or a held-out set that never influences ω selection, and report both the selected-model performance and the performance under the fair protocol.","section":"Section 4.1, 'Feature calibration'"},{"comment":"The claim that CGVQM-5 'significantly outperforms existing metrics' is not consistently supported by the independent datasets. On CGVDS, NVS, and BVI-HD, CGVQM-5 obtains PLCC of 0.67, 0.697, and 0.69, respectively, while DISTS achieves 0.85 and 0.852 on CGVDS and NVS, and VMAF and FoVVideoVDP achieve 0.772 and 0.796 on BVI-HD. Strong advantages over all baselines appear mainly on the three calibration datasets, which are exactly the ones affected by test-loss-based model selection. The external results support a claim of competitiveness and robustness, not uniform significant superiority. Please either temper the claim or provide independent evidence, e.g., a validation-based re-evaluation showing significant gains on held-out data.","section":"Section 4.2, 'Comparison with existing metrics', Eq. (4), and Table 1"},{"comment":"There is an inconsistency in the pooling description. Section 4.1 defines the overall video quality as the minimum quality over patches, while Section 5 states 'we use mean pooling over space and time to compute the overall quality of a video' without mentioning the minimum over patches. The choice of min pooling for patch aggregation is nontrivial and can strongly affect the metric's behavior for localized distortions, which the paper itself discusses. Please clarify the exact aggregation used in the experiments and consider reporting sensitivity to the min-pooling rule.","section":"Section 4.1, 'Feature calibration' (patch pooling) and Section 5"}],"minor_comments":[{"comment":"Equation (1) is difficult to read because the tensor subscripts and superscripts are garbled; please rewrite it with clear indices for the feature maps and pooling operations.","section":"Equation 1"},{"comment":"The remote-viewing and vision-screening caveat is an important limitation of the subjective ground truth; consider moving a summary of it, along with the participant demographics, into the main text rather than leaving it only in the appendix.","section":"Section 3.3 and Appendix A1.1"},{"comment":"Please state explicitly whether the five-parameter nonlinear mapping in Eq. (4) is fitted on the same test set used for the reported correlations. If so, note that this is a standard but slightly optimistic protocol, and describe how the fitting is done for CGVQM versus the baseline metrics.","section":"Section 4.2, Eq. (4)"},{"comment":"The significance-testing description would benefit from specifying how bootstrap samples are generated for the paired t-test and whether any multiple-comparison correction is applied across the many metric pairs.","section":"Appendix A1.3, Table A3"}],"recommendation":"major_revision","confidential_remarks":"The paper fits the scope of High-Performance Graphics and has a worthwhile dataset contribution. The main issue is methodological: the headline performance on the three calibration datasets is selected on the test labels, and the independent datasets tell a more mixed story. I would require a validation-based re-evaluation or a clearly softened claim before acceptance; the dataset and error-map analysis can likely be salvaged."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the dataset is the real contribution: CG-VQD covers distortions that no public VQA dataset touches—neural supersampling, Gaussian splatting, path tracing, neural denoising, frame interpolation, VRS—and the collection is more careful than most: 20 participants, three repetitions, MLE scoring, outlier exclusion, and an ICC(2,k) of 0.97. That is a benchmark people will use. Second, the metric's headline advantage is overstated. The authors select the 1027 channel weights by taking the epoch with the lowest test loss on the same splits they report, with no validation set. That biases the reported PLCC/SRCC upward and makes the comparison with untrained existing metrics asymmetric. I don't think this is fatal—the paper discloses the protocol—but the abstract's 'significantly outperforms' should not be taken at face value.\n\nWhat is genuinely new: the finding that pre-trained 3D ResNet features, after linear channel-weight calibration, align with human ratings of rendering artifacts better than 2D LPIPS and most hand-crafted metrics. That is a useful empirical result. The per-pixel error map evaluation on LocVis is a nice addition, and the lighter CGVQM-2 variant is sensible.\n\nThe soft spots, in proportion. The test-loss selection is the main one. Repeating random splits and bootstrapping does not remove it; each split still selects on test labels. The six external datasets are not used for calibration and therefore give a fairer read, but there the picture is mixed: CGVQM-5 wins on LIVE Meta, ties on Flicker, and loses to DISTS on NVS and CGVDS, and to VMAF on AVT. So 'consistently high correlation' is too strong. The small remote study with 20 participants is a minor concern; the reliability analysis largely addresses it.\n\nWho should read this: anyone building or benchmarking VQA for game engines, streaming, or neural rendering. The dataset will be a reference point regardless of the metric's fate.\n\nRecommendation: send it to peer review. Require a re-calibration with a proper validation split (or report only the external datasets as the unbiased test), and ask the authors to qualify the performance claims. The dataset and the empirical finding are worth the referee time.","headline":"CG-VQD is a valuable new dataset and the 3D-CNN finding is credible, but CGVQM's reported advantage is inflated by test-loss selection; worth a serious referee.","tokens_in":28247,"tokens_out":2853,"would_cite":true,"duration_ms":27196,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CGVQM, a full-reference video quality metric based on pre-trained 3D CNN features, significantly outperforms existing metrics on modern rendering artifacts and yields both per-pixel error maps and global quality scores.","keywords":["video quality assessment","full-reference metric","computer graphics","rendering artifacts","3D convolutional neural network","subjective dataset","DMOS","neural supersampling"],"falsifier":"A controlled-lab subjective study with a larger, screened, more diverse participant pool rating a fresh set of rendering-distorted videos (new scenes and artifact levels not drawn from CG-VQD) would settle the claim: if CGVQM's Pearson correlation with the resulting DMOS no longer significantly exceeds that of the best existing metric (e.g., HaarPSI or ColorVideoVDP), the central claim fails.","tokens_in":27178,"feed_emoji":"🎮","tokens_out":13482,"duration_ms":109184,"temperature":0.7,"pith_summary":"Modern rendering techniques—neural supersampling, path tracing, neural denoising, Gaussian splatting, frame interpolation, and variable rate shading—produce artifacts that standard video quality metrics misjudge. The paper builds a new subjective dataset, CG-VQD, of 80 short clips rated by 20 participants, and shows that existing full-reference metrics top out at 0.78 Pearson correlation with human opinion. It then shows that distances measured in the feature space of a pre-trained 3D ResNet-18 align far better with human perception, and calibrates a small set of channel weights to turn those feature distances into a metric, CGVQM, that reaches about 0.87 correlation and also produces per-pixel error maps. If correct, CGVQM gives graphics engineers an automatic way to detect and localize the specific artifacts viewers will notice.","feed_headline":"New 3D-CNN metric tops 0.87 on rendering-artifact videos","feed_subtitle":"Existing video metrics stall near 0.78 on rendering artifacts; CGVQM hits 0.87 and maps where errors show.","key_machinery":"The load-bearing object is the feature-distance computation from a frozen 3D ResNet-18, a spatiotemporal convolutional network that processes short video clips. Reference and distorted videos are passed through the network; at each of five selected convolutional layers the feature maps are unit-normalized along the channel dimension, each channel is scaled by a learned weight $\\omega_l$, and the squared $\\ell^2$ distance between the normalized feature tensors is computed. Summing this distance over space, time, and layers gives the global quality score $q(x,x_0)$; interpolating the per-layer distances to full resolution gives the per-pixel error map $e(x,x_0)$. Only about 1000 channel weights are learned by maximizing Pearson correlation with human ratings on three calibration datasets—GamingVideoSET, LIVE Livestream, and CG-VQD itself—while all convolutional kernels stay frozen, which is what keeps the small quality datasets from causing overfitting.","core_discovery":"The paper's central claim is that the feature space of a pre-trained 3D convolutional neural network—trained for action recognition, not for quality—is a better perceptual space for judging rendering artifacts than any existing full-reference quality metric. On the new CG-VQD dataset of 80 videos with distortions from six modern rendering techniques, existing metrics reach a maximum Pearson correlation of 0.78 with human difference-mean-opinion scores; CGVQM, which compares channel-normalized deep features of the reference and distorted videos through a frozen 3D ResNet-18 and calibrates only per-channel weights, reaches about 0.87 on the same test split and also produces per-pixel error maps that localize artifacts. The authors report that this advantage holds across six additional video quality datasets covering compression, livestreaming, flicker, and neural view synthesis, that 3D temporal modeling is essential (2D-based LPIPS trails), and that both the network's pre-training and the calibration step contribute significantly to the gain.","pith_inferences":["Because CGVQM-2 needs only the first two ResNet blocks, a GPU-specialized implementation might run in real time inside a game engine or streaming encoder, enabling online artifact monitoring—an extension the paper does not claim.","The acknowledged weakness of mean pooling on localized errors suggests that replacing it with saliency-weighted or max pooling would improve CGVQM on scenes where a small artifact dominates perception; that fix is directly testable on CG-VQD.","If 3D CNN features generalize this well across distortion types, a no-reference variant that scores a single video against a learned clean-feature distribution is a natural next step for settings where no reference exists, such as live game capture."],"forward_implications":["Rendering engineers can use CGVQM to rank and compare rendering settings without running user studies, since it correlates at about 0.87 with human DMOS on modern rendering artifacts.","CGVQM's per-pixel error maps localize where viewers will notice artifacts, such as ghosting around bright lights or noise in shadow areas, a capability that global-only metrics like VMAF and DISTS lack.","Strong cross-dataset performance makes CGVQM a reliable full-reference metric for synthetic content beyond CG-VQD, including cloud gaming, livestreaming, flicker, and neural view synthesis.","The lighter CGVQM-2 variant is 27% faster and nearly as accurate on some benchmarks, so artifact assessment with error localization is feasible in time-constrained rendering pipelines."],"supporting_citations":[{"why":"Supplies the deep-features-as-perceptual-metric paradigm that CGVQM extends from images to video.","marker":"[Zha+18b]"},{"why":"Provides the 3D ResNet-18 architecture with R3D, MC3, and R(2+1)D convolution variants used as the frozen feature extractor.","marker":"[Tra+18]"},{"why":"Gives the C3D architecture used as the main alternative 3D feature space and as the basis of prior DeepVQUE work.","marker":"[Tra+15]"},{"why":"Defines the classic SSIM baseline and the full-reference framing that CGVQM is measured against.","marker":"[Wan+04]"},{"why":"The VMAF ensemble baseline, one of the strongest traditional video metrics that CGVQM must beat.","marker":"[Ras17]"},{"why":"ColorVideoVDP, the leading psychophysical video difference predictor used as a key comparison in all benchmarks.","marker":"[Man+24]"},{"why":"Maximum-likelihood subject model used to convert raw ratings into the DMOS ground truth for CG-VQD.","marker":"[Li+20]"},{"why":"LocVis dataset and its probability-of-detection maps used to validate CGVQM's per-pixel error localization.","marker":"[Wol+18]"},{"why":"GamingVideoSET, a calibration dataset of compressed game footage used to learn the channel weights $\\omega$.","marker":"[Bar+18]"},{"why":"LIVE Livestream, a calibration dataset of natural streaming distortions also used to learn $\\omega$.","marker":"[Sha+21]"}],"fun_headline_variants":["3D CNN features beat existing metrics on rendering artifacts","CGVQM: 3D-CNN metric hits 0.87 for rendering artifact quality","Pre-trained 3D CNN wins on synthetic video quality assessment","From 0.78 to 0.87: deep 3D features judge rendering artifacts"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central claim rests on the subjective ratings from 20 remote volunteers—collected without controlled viewing conditions or vision screening—being reliable and representative enough to serve as the ground truth that decides which metric is better.","fun_headline_variants_meta":{"raw":{"variants":["3D CNN features beat existing metrics on rendering artifacts","CGVQM: 3D-CNN metric hits 0.87 for rendering artifact quality","Pre-trained 3D CNN wins on synthetic video quality assessment","From 0.78 to 0.87: deep 3D features judge rendering artifacts"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000206,"raw_usage":{"total_tokens":1377,"prompt_tokens":909,"completion_tokens":468,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":525,"completion_tokens_details":{"reasoning_tokens":385}},"tokens_in":525,"tokens_out":468,"duration_ms":4571,"temperature":1.0,"reasoning_tokens":385,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T04:04:17.709689+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A controlled-lab subjective study with a larger, screened, more diverse participant pool rating a fresh set of rendering-distorted videos (new scenes and artifact levels not drawn from CG-VQD) would settle the claim: if CGVQM's Pearson correlation with the resulting DMOS no longer significantly exceeds that of the best existing metric (e.g., HaarPSI or ColorVideoVDP), the central claim fails.","supporting_citations":[],"review_version":1}