{"id":"117be049-5fe1-4d05-b4a2-8fd9f2e3be43","arxiv_id":"2506.14642","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A human-rated dataset of 15,200 compressed 3D Gaussian Splatting images from 10 scenes and 6 algorithms, benchmarked against 30 quality metrics.","lead":"Researchers built a 15,200-image, human-rated benchmark for judging how compressed 3D Gaussian Splatting renderings look, covering 6 compression methods and 10 real scenes. It is the largest such dataset to date and could help train better perceptual quality metrics for 3D graphics.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"MOS reliability is unestablished; without per-image confidence or inter-subject agreement, the 15,200 labels cannot support the benchmark and view-dependence claims.","rationale":"The reader's weakest assumption identifies exactly the load-bearing issue: the reliability of the MOS ground truth. The paper provides no quantitative evidence that 15 ratings per image from 60 students yield precise scores, and the unusually low 2% rejection rate raises concern about the subject-screening procedure. This is not an ad hominem or a dispute with consensus; it is a request for standard psychometric reporting in subjective IQA. The concrete test is feasible because the authors can release the raw ratings or run a small re-collection study. The paper's other issues (the 720/760 model count typo and the confounding of the view-dependent conclusion by the deliberate selection of challenging novel viewpoints) are real but secondary; the former is an arithmetic slip that does not affect the dataset count if 760 is correct, and the latter can be rephrased as a property of the chosen viewpoint sampling rather than a universal 3DGS phenomenon. Since the reader already conditioned acceptance on reporting MOS reliability, and our analysis does not require moving to a different verdict, UNCHANGED is appropriate.","tokens_in":14119,"tokens_out":6394,"duration_ms":68225,"concrete_test":"If raw per-subject ratings are released, compute for each image the standard deviation and 95% bootstrap confidence interval of the MOS, and report ICC(2,k) or Krippendorff's alpha across the 15 raters. Then perform a split-half reliability test: randomly partition the raters into two groups, recompute MOS for each half, and re-rank the top metrics in Table 4; if the average per-image CI width exceeds 2 on the 0-10 scale, or if the two halves disagree on the top-3 SRCC metric ranking, the MOS labels are too noisy to support the benchmark and view-dependence conclusions.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central value of 3DGS-IEval-15K is its human MOS labels, which underpin the benchmark (Table 4) and the view-dependent quality analysis (Figure 4b). Section 3.4 reports only that 60 students each rated 15 images per sample, an 11-level impairment scale, and a 2% ITU-R BT.500 rejection rate. No inter-subject agreement (e.g., ICC, Krippendorff's alpha), per-image standard deviation, or confidence intervals are given. With 15 ratings per image and 15,200 images, the experiment involves 228,000 annotations, or about 3,800 annotations per subject; the paper does not describe session breaks or fatigue control. A 2% rejection rate is unusually low for naive subjects and may indicate that the BT.500 screening was applied too leniently to remove unreliable raters. If the MOS values are noisy, the benchmark correlations in Table 4 have unknown error bars, the ranking of 30 metrics may not be stable, and the train/test MOS gap in Figure 4b could be partly an artifact of rater noise rather than a genuine 3DGS view-dependent quality effect. Because all downstream uses of the dataset inherit the label quality, this is the most load-bearing weak point in the paper.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces 3DGS-IEval-15K, a large-scale image quality assessment (IQA) dataset for compressed 3D Gaussian Splatting. It describes the selection of 10 real scenes from Mip-NeRF 360, Tanks & Temples, and Deep Blending; a viewpoint selection strategy using k-means clustering for training viewpoints and a four-criterion composite score for testing viewpoints; the generation of distortions by training six 3DGS algorithms at multiple compression levels; a subjective experiment with 60 viewers and 15 ratings per image using an 11-level impairment scale; and a benchmark of 30 IQA metrics. The authors claim this is the first large-scale 3DGS-specific IQA dataset, containing 15,200 images, and use it to analyze view-dependent quality differences between training and novel viewpoints.","tokens_in":14380,"tokens_out":5313,"duration_ms":48654,"significance":"If the dataset is valid, it is a substantial community resource. It is considerably larger than prior NVS-QA datasets (e.g., GSC-QA with 120 samples and ENeRF-QA with 440), covers six compression algorithms with separate geometry and color distortion levels, and the authors publicly release the data. The 30-metric benchmark, including deep-learning and LLM-based methods, provides a useful baseline for future 3DGS quality assessment research, and the view-dependent MOS analysis, if confirmed with proper statistical controls, could inform 3DGS optimization. However, the benchmark rankings and the view-dependence conclusions inherit the quality of the subjective labels, and the current manuscript does not yet establish that quality. The reproducible public release is a clear strength, but the arithmetic error in the model count and the missing MOS reliability analysis need to be addressed before the resource can serve as a trusted benchmark.","major_comments":[{"comment":"The model-count arithmetic is inconsistent. The paper states 'we train 10 scenes×(4×16+2×6) DLs = 720 3DGS models', but 4×16+2×6 = 76, and 10×76 = 760, not 720. Since 760×20 = 15,200, the correct model count is 760; the printed '720' would imply 14,400 images. This error affects the headline dataset size and must be corrected wherever the count appears.","section":"3.3.2"},{"comment":"MOS reliability is not established. The manuscript reports 60 student viewers, 15 ratings per image, an 11-level impairment scale, and a 2% ITU-R BT.500 rejection rate, but provides no inter-subject agreement (e.g., ICC, Krippendorff's alpha), per-image standard deviations, confidence intervals, or validation against a second lab. With only 15 naive ratings per image, label noise could change the Table 4 metric rankings and the Figure 4(b) view-dependence comparison. Please report agreement statistics and the exact screening procedure, and justify the unusually low 2% rejection rate.","section":"3.4"},{"comment":"The four dataset configurations are not defined precisely enough to rule out leakage. The 'All' configuration is formed by concatenating the train and test sets of Geometry-Only, Color-Only, and Geometry & Color Mix, but the paper does not state whether these subsets are disjoint image sets, nor whether the 4:1 split is performed at the scene level or the image level. If the same source scene appears in both a model's training and test sets, the reported SRCC/PLCC values can be inflated. Please specify the split granularity and explicitly confirm that the three distortion subsets are disjoint.","section":"4.3.1"},{"comment":"For InternVL2.5 (8B) and Gemini1.5-pro, the reported SRCC values (≈0.67–0.71) and PLCC values (≈0.08–0.10) are implausibly far apart. Such a large discrepancy is unexpected for a quality prediction task and suggests a nonlinearity or an evaluation artifact in the reference-score alignment. The paper should explain the alignment/calibration procedure for LLM outputs, or these entries should be re-evaluated and corrected.","section":"Table 4"},{"comment":"The claim that test viewpoints yield lower MOS than training viewpoints is based on two overlaid histograms without a significance test or per-scene control. Since viewpoint selection and scene content are confounded, a paired or mixed-effects analysis stratified by scene is needed before describing this as 'the first systematic investigation' of view-dependent quality in 3DGS. Please provide per-scene statistics and a formal test (e.g., a scene-blocked comparison).","section":"4.2 / Figure 4(b)"}],"minor_comments":[{"comment":"The standards are mislabeled: 'ITUR BT.500' should be 'ITU-R BT.500' and 'ITU-TP.910' should be 'ITU-T P.910'.","section":"3.4"},{"comment":"The symbol S_j is used both for the composite score in Eq. (1) and for the sparsity criterion introduced immediately after; using distinct symbols for the composite score and the sparsity term would remove ambiguity.","section":"3.2"},{"comment":"Figure 1(d) states '15 Annotators per image', while the text says 15,200 images are randomly divided into 8 groups and rated by 60 students; please clarify how the 228,000 annotations are distributed across viewers and sessions, and whether each image is rated exactly 15 times.","section":"Figure 1(d)"},{"comment":"The definitions of 'Geometry-Only', 'Color-Only', and 'Geometry & Color Mix' should state explicitly which compression levels are held fixed for the other distortion type, since the multi-parameter algorithms combine 16 distortion levels through pairwise combinations.","section":"4.3.1"},{"comment":"No confidence intervals or significance tests are reported for the SRCC/PLCC/KRCC values; at least bootstrap intervals for the top-performing metrics would strengthen the benchmark comparisons.","section":"Table 4"},{"comment":"The paper alternates between 'Scaffold' and 'Scaffold-GS', and between 'LightGS' and 'LightGaussian'; please unify the naming throughout the text and tables.","section":"3.3.1"}],"recommendation":"major_revision","confidential_remarks":"The dataset is potentially valuable and the construction pipeline is mostly plausible, but the missing MOS reliability analysis and the arithmetic error in the model count need to be fixed before the resource can be fully trusted as a benchmark. I am not recommending rejection; the issues are addressable within the manuscript's scope. Please also ask the authors to double-check the Table 4 PLCC/SRCC discrepancy for InternVL2.5 and Gemini1.5-pro, which may indicate a bug in the evaluation pipeline rather than a meaningful finding."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe headline: this is the largest human-rated corpus for compressed 3DGS quality by a wide margin, and it is a genuinely useful resource if the labels hold up. The paper reports 15,200 images from 10 scenes, 6 compression algorithms, 20 viewpoints per scene, 60 viewers, plus a 30-metric benchmark including LLM scorers. That scale is the contribution; nothing in the cited prior work comes close (largest previous NVS-QA corpus is under 500 samples). The compression-level design, separating geometry and color parameters into a 4x4 grid for four methods and 6 levels for two others, is thoughtful and gives the field a systematic distortion space to train on. The benchmark results, with deep IQA models (MANIQA, HYPERIQA) near 0.94 SRCC and LLMs ranging from near-zero to Q-Align's 0.77, are useful reference points even if not surprising. The repo is public, the methodology follows ITU-style protocols, and the paper is clearly written.\n\nNow the soft spots, in order of severity.\n\nFirst, the MOS ground truth is under-documented. Section 3.4 gives 60 students, 15 ratings per image, an 11-level scale, and a 2% BT.500 rejection rate, but no inter-subject agreement, no per-image standard deviation, no confidence intervals. With 3,800 annotations per subject, fatigue and rater consistency matter; a 2% rejection rate for naive subjects is low enough to warrant justification. If the MOS values are noisy, every benchmark ranking in Table 4 inherits that noise, and the train/test MOS gap in Figure 4b could be partly rater noise. This is the most load-bearing weakness because the dataset's value is the labels. It is fixable: report ICC or alpha, per-image std, and ideally show a reproducibility check on a subset.\n\nSecond, the view-dependent quality conclusion (Section 4.2) is largely a selection artifact. Test viewpoints were chosen specifically to maximize distance, sparsity, extrapolation, and directional novelty from training viewpoints. Of course they score lower. Calling this 'the first systematic investigation of quality differences between training and novel viewpoints' overstates what is essentially a manipulation check. The data can support a weaker claim—that the chosen challenging viewpoints are indeed harder—but not a general statement about view-dependent 3DGS quality.\n\nThird, a concrete arithmetic slip: the text says '10 scenes ×(4×16+2×6) DLs = 720 3DGS models', but 10×(64+12) is 760, and 760×20 = 15,200. The 720 number is internally inconsistent with the dataset size. Minor, but it needs a correction because the model count appears in the abstract-level claims.\n\nOverall: the dataset is a real contribution and the construction is plausible. The missing reliability statistics and the over-interpreted view-dependence claim are addressable. I'd send this to a serious reviewer, and I'd want to see the reliability numbers before treating the MOS as ground truth. If those check out, this becomes the default benchmark for 3DGS compression IQA.","headline":"The largest 3DGS IQA dataset to date, useful if the MOS labels hold up; three fixable issues (arithmetic slip, missing reliability stats, over-read view-dependence) stand between this and a default benchmark.","tokens_in":14938,"tokens_out":3600,"would_cite":true,"duration_ms":32420,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"3DGS-IEval-15K provides 15,200 human-scored images for compressed 3D Gaussian Splatting quality.","keywords":["Image quality assessment","3D Gaussian Splatting","3DGS compression","Subjective quality evaluation","Mean opinion score","View-dependent quality","IQA benchmark","Novel view synthesis"],"falsifier":"Re-run the subjective experiment on a random subset of about 500 images with a different group of at least 30 viewers; if the two labs' MOS rankings agree with SRCC below 0.9, the single-lab labels are too unstable to support the benchmark's metric rankings.","tokens_in":13924,"feed_emoji":"🖼️","tokens_out":9663,"duration_ms":89805,"temperature":0.7,"pith_summary":"This paper introduces 3DGS-IEval-15K, a dataset of 15,200 rendered images that pairs compressed 3D Gaussian Splatting scenes with human quality scores. It is the first image-quality-assessment resource built specifically for compressed 3DGS representations, and it is large enough to train specialized quality metrics rather than reusing generic ones. The authors also benchmark 30 existing quality metrics on the dataset, showing that learned metrics track human scores better than handcrafted ones and that all models lose accuracy when distortions are isolated by type. The dataset matters because 3DGS storage is a bottleneck for deployment, and without a perceptual ground truth, compression choices cannot be tuned to what viewers actually see.","feed_headline":"15,200 human-scored images for compressed 3D scene quality","feed_subtitle":"The database and its 30-metric benchmark show how compression degrades perceived quality and why novel views look worse.","key_machinery":"The central object is the dataset itself, built on a three-part construction pipeline. Distortion design is the load-bearing mechanism: for the four multi-parameter algorithms, geometry and color compression levels are varied on four-point scales and combined pairwise into 16 distortion levels, while the two single-parameter algorithms receive 6 levels each, producing systematic coverage of 3DGS-specific artifacts. Viewpoint selection uses k-means clustering over position and direction for training views and a four-criterion composite score for test views, isolating view-dependent quality. Human scores are collected with a double-stimulus impairment scale and processed with ITU-R BT.500 outlier rejection to yield MOS values; the benchmark then measures 30 metrics with SRCC, PLCC, and KRCC. The construction is what allows the paper to attribute quality differences to distortion type and viewpoint rather than to scene content.","core_discovery":"On its own terms, the paper's central claim is that human perception of compressed 3DGS can be measured systematically, and that doing so reveals patterns generic IQA metrics miss. The authors construct 720 trained 3DGS models from 10 real scenes using 6 compression algorithms, render each from 20 viewpoints chosen to include both representative and challenging novel views, and collect 15 ratings per image from 60 viewers to produce MOS scores. They find that novel viewpoints receive systematically lower quality scores than training viewpoints, and that while the best deep-learning IQA metrics reach SRCC above 0.93 on the full dataset, every method degrades on geometry-only or color-only distortion subsets. From this they argue that the dataset provides the scale and distortion coverage needed to develop 3DGS-specific perceptual metrics.","pith_inferences":["A natural next step the authors do not take is training a quality regressor on the Gaussian attributes (positions, opacities, SH coefficients) rather than on rendered pixels; the dataset's compression-level design makes such attribute-level supervision possible.","The paired geometry and color distortion levels could support ablation studies separating which compression parameter most affects perceived quality, and possibly guide bit allocation between geometry and color during compression.","If the view-dependent MOS disparity replicates across more scenes, viewpoint selection in 3DGS training could be adapted to spend more capacity on directions that viewers find perceptually vulnerable."],"forward_implications":["Specialized 3DGS quality metrics can now be trained on 15,200 labeled images instead of relying on generic IQA models.","View-dependent quality is a measurable property of 3DGS: novel viewpoints score lower than training viewpoints, so viewpoint sampling is itself a quality factor.","Existing metrics are not interchangeable on 3DGS content: deep-learning models lead the benchmark while BRISQUE and several LLM zero-shot models correlate weakly with human scores.","Geometry-only and color-only distortion subsets expose generalization gaps, meaning metric evaluations on mixed distortions can hide weaknesses."],"supporting_citations":[{"why":"Supplies six of the ten source scenes (bicycle, flowers, garden, counter, kitchen, room).","marker":"[2]"},{"why":"Supplies the outdoor train and truck scenes used as source content.","marker":"[16]"},{"why":"Supplies the indoor playroom and drjohnson scenes.","marker":"[11]"},{"why":"One of the six compression algorithms; its prune and vector-quantization parameters generate geometry and color distortion levels.","marker":"[7]"},{"why":"One of the six compression algorithms; its codebook and importance parameters generate distortion levels.","marker":"[31]"},{"why":"One of the six compression algorithms; its hashmap and VQ parameters generate distortion levels.","marker":"[17]"},{"why":"One of the six compression algorithms; its codebook size generates color and geometry distortion levels.","marker":"[30]"},{"why":"One of the two single-parameter algorithms; its lambda parameter defines six distortion levels.","marker":"[4]"},{"why":"The other single-parameter algorithm; its vsize parameter defines six distortion levels.","marker":"[23]"},{"why":"The subjective testing standard used for outlier detection and subject rejection before averaging MOS.","marker":"[3]"}],"fun_headline_variants":["First large-scale IQA dataset for compressed 3DGS","15K human-scored 3DGS images reveal compression's perceptual toll","New benchmark: 15,200 images show why compressed 3D scenes look worse","Largest 3DGS quality dataset: 15K images, 30 metrics, human scores","Novel views score lower: 15K-image 3DGS quality benchmark"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The dataset's value rests on the 60 student viewers' ratings being trustworthy ground truth, but the ratings come from one lab with no reported inter-subject agreement, confidence intervals, or second-lab validation.","fun_headline_variants_meta":{"raw":{"variants":["First large-scale IQA dataset for compressed 3DGS","15K human-scored 3DGS images reveal compression's perceptual toll","New benchmark: 15,200 images show why compressed 3D scenes look worse","Largest 3DGS quality dataset: 15K images, 30 metrics, human scores","Novel views score lower: 15K-image 3DGS quality benchmark"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000666,"raw_usage":{"total_tokens":3047,"prompt_tokens":963,"completion_tokens":2084,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":579,"completion_tokens_details":{"reasoning_tokens":1978}},"tokens_in":579,"tokens_out":2084,"duration_ms":14509,"temperature":1.0,"reasoning_tokens":1978,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:50:14.804099+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the subjective experiment on a random subset of about 500 images with a different group of at least 30 viewers; if the two labs' MOS rankings agree with SRCC below 0.9, the single-lab labels are too unstable to support the benchmark's metric rankings.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies six of the ten source scenes (bicycle, flowers, garden, counter, kitchen, room)."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"One of the six compression algorithms; its prune and vector-quantization parameters generate geometry and color distortion levels."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"One of the six compression algorithms; its codebook and importance parameters generate distortion levels."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"One of the six compression algorithms; its codebook size generates color and geometry distortion levels."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"One of the two single-parameter algorithms; its lambda parameter defines six distortion levels."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The other single-parameter algorithm; its vsize parameter defines six distortion levels."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The subjective testing standard used for outlier detection and subject rejection before averaging MOS."}],"review_version":2}