{"id":"3c6f0a92-20c9-46bd-9d85-9c5486b6fca9","arxiv_id":"2608.03279","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A new 15,200-image human-annotated dataset and an LMM-based metric that jointly predicts overall, geometry, and color quality of compressed 3D Gaussian Splatting images.","lead":"This paper builds a 15,200-image benchmark for judging the quality of compressed 3D Gaussian Splatting renders, with separate human scores for overall, geometry, and color quality. It also trains a large multimodal model to predict all three scores at once, reporting top results on its own benchmark and competitive transfer to other novel-view-synthesis quality sets.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Within-scene image-level split shares 3DGS model identities between train/test, allowing model-specific memorization to inflate the Table V SOTA; needs model-disjoint validation.","rationale":"The reader's weakest_assumption correctly identifies the within-scene image-level split as the most load-bearing issue. The paper's Table V SOTA is the strongest claim, and it is directly threatened by group leakage: each 3DGS model contributes 20 rendered images, and a 4:1 image-level split almost guarantees that the same model appears in both train and test. The model can memorize the unique distortion signature of each compressed model, inflating correlations. No scene-disjoint or model-disjoint experiment is provided to rule this out. The cross-dataset results (Table VI) offer some independent support for transferability, but they do not validate the in-domain SOTA. The proposed concrete test—model-disjoint cross-validation—is feasible with the existing data and would settle whether the reported numbers reflect generalizable quality assessment or memorization. Since the reader's CONDITIONAL verdict already captures this concern and recommends re-validation, I do not change the verdict. The other limitations (MOS reliability, bitrates, artifact release) are real but secondary; the split is the gating issue for the central claim.","tokens_in":20453,"tokens_out":4587,"duration_ms":53169,"concrete_test":"Re-run the main evaluation with a model-disjoint split: group all 20 images by their underlying 3DGS model ID (scene × algorithm × compression levels), and perform 5-fold cross-validation at the model level so no model appears in both train and test. Report SRCC/PLCC/KRCC for all three dimensions and compare to the baselines under the same split. If the SRCC for overall quality drops by more than ~0.03 from 0.9630 or the ranking versus baselines changes, the within-scene image-level split is the source of the reported SOTA. As a stronger check, also run leave-one-scene-out (train on 9 scenes, test on the held-out scene).","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central SOTA claim (Table V) rests on the within-scene split described in Section V-A: 'for each scene, the rendered images are partitioned 4:1 into training and testing sets.' The dataset contains only 10 scenes, and each of the 760 trained 3DGS models is rendered from 20 viewpoints (10 training views + 10 novel views). Splitting at the image level therefore places images from the same underlying 3DGS model—same scene, same algorithm, same compression levels—into both train and test. Because the 20 views of a model share content and the exact distortion pattern, the model can learn to recognize model-specific artifacts and scene-specific textures rather than generalizable quality assessment. The 5-fold averaging does not remove this group leakage; it only varies which views are held out. No scene-disjoint or model-disjoint evaluation is reported, so the reported SRCC 0.963/0.949/0.960 may substantially overstate generalization to unseen 3DGS models. The zero-shot cross-dataset results (Table VI) are less affected and provide some evidence of transfer, but they do not validate the in-domain benchmark claim. The missing bitrate reporting, unreleased artifacts, and lack of inter-rater agreement analysis are secondary; the split is the load-bearing issue.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces 3DGS-IEval-15K+, a large-scale IQA dataset for compressed 3D Gaussian Splatting, containing 15,200 rendered images from 10 scenes, 6 compression algorithms, and 45,600 MOS scores across overall, geometry, and color quality. It also proposes 3DGSI-Assessor, an LMM-based full-reference metric that combines global semantic features with dimension-specific local features and predicts all three quality dimensions in one forward pass. The central claim is state-of-the-art performance on the new benchmark (Table V: SRCC 0.9630 overall, 0.9489 geometry, 0.9601 color) and competitive zero-shot transfer to three external NVS datasets (Table VI). The dataset and code are promised for release.","tokens_in":20848,"tokens_out":3874,"duration_ms":50387,"significance":"If the performance claim survives rigorous evaluation, the paper would make a substantial contribution: it is by far the largest 3DGS IQA dataset, the first with separate geometry/color MOS annotations, and it provides an extensive comparison of handcrafted, deep-learning, and LMM-based metrics. The proposed all-in-one architecture is reasonable and the two-stage LoRA fine-tuning strategy is clearly described. The external zero-shot results on ENeRF-QA, NeRF-VSQA, and GSC-QA are useful independent evidence of transferability and are a genuine strength. However, the headline in-domain SOTA claim rests on an evaluation split that is not group-disjoint and may be inflated by scene/model memorization. The dataset contribution is potentially significant, but the current evidence for the method's generalization on 3DGS-IEval-15K+ is not yet convincing.","major_comments":[{"comment":"The central SOTA claim is evaluated with a within-scene, image-level 4:1 split: 'for each scene, the rendered images are partitioned 4:1 into training and testing sets.' Because each of the 760 3DGS models is rendered from 20 viewpoints, train and test images share the same scene content and the same model identity, including its unique compression artifacts. A model can therefore memorize scene-specific textures or model-specific distortion patterns instead of learning a generalizable quality function. The 5-fold averaging only changes which views are held out and does not remove this group leakage. The paper provides no scene-disjoint or model-disjoint evaluation. The zero-shot Table VI results partially mitigate the concern, but they do not validate the Table V in-domain claim. Please report performance with a scene-disjoint split (or at least a split where no 3DGS model contributes b","section":"V-A and Table V"},{"comment":"The MOS reliability is not demonstrated. The paper reports 15 annotators per image and Z-score normalization, but no inter-rater agreement statistics (e.g., ICC or Krippendorff's alpha), no outlier subject removal, and no confidence intervals for MOS. Since the entire benchmark and the training labels depend on these MOS values, the lack of reliability evidence weakens the dataset claim. Please add per-dimension inter-rater agreement, subject screening, and an analysis of MOS consistency.","section":"III-D and III-E"},{"comment":"The 'compression level' design is not validated by any bitrate or storage-size measurement. The paper identifies the task as compression quality assessment, but Tables II and III report only parameter settings (e.g., codebook size, prune percentage, lambda), not the resulting file sizes, bits-per-Gaussian, or rate-distortion points. Without this information, it is unclear whether the 16 DL and 6 DL configurations span a monotonic and meaningful compression-quality range. Please report per-model storage sizes and, ideally, rate-distortion curves for each method and level.","section":"III-C and Tables II-III"}],"minor_comments":[{"comment":"The rescaling z'_ij = 100(z_ij + 3)/6 maps z=-3 to 0 and z=3 to 100, but z-scores outside [-3,3] will produce MOS values outside [0,100]. Clarify whether clipping is applied or whether the reported MOS ranges are all within bounds.","section":"III-D, Eq. (6)"},{"comment":"The table formatting is garbled; the columns for LightGS/C3dGS/Compact-3DGS/CompGS and the geometry/color parameter entries are difficult to parse. Please restructure the table with explicit parameter names and values.","section":"Table II"},{"comment":"The statement that the method 'has been adopted as a baseline in the IEEE Standards Association project 3366.5' is not substantiated by a citation or document. Either provide a reference or remove this non-scientific claim.","section":"V-A"},{"comment":"There are typos such as 'per iamge' in Section III-D, and the reference format for TReS is inconsistent. A final proofread is recommended.","section":"General"},{"comment":"The novelty relative to the preliminary conference version [16] is acknowledged only briefly. Please state explicitly in the contribution list which components are new (e.g., geometry/color annotations, the LDS-FEM, the regression heads) to disambiguate from the conference paper.","section":"I and Table I"}],"recommendation":"major_revision","confidential_remarks":"The within-scene split is the load-bearing issue and is fixable: a scene-disjoint or model-disjoint evaluation would directly address the leakage concern. The MOS reliability and bitrate reporting are secondary but should be added. The external zero-shot results are encouraging and suggest the method has merit even if the in-domain numbers change. I would not reject, but the paper should not be accepted before the split issue is resolved."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the dataset is the real contribution. First large-scale IQA benchmark for compressed 3DGS with separate overall/geometry/color MOSs—15,200 images, 45,600 MOSs, 684K annotations, 6 compression algorithms, 10 scenes. That fills a clear gap and the annotation design (dimension-specific labels, viewpoint selection, compression-level pairing) is thoughtful. The LMM architecture with dual LDS-FEM branches is a sensible way to fuse global semantics with local distortion features, and the ablation study supports the design choices.\n\nThe soft spot is the in-domain evaluation. The 4:1 split is at the image level within each scene. Since each of the 760 trained 3DGS models is rendered from 20 viewpoints, the same model instance (same scene, same algorithm, same compression parameters) appears in both train and test. The model can memorize model-specific artifact patterns rather than learn generalizable quality assessment. The 5-fold averaging doesn't fix that; it only changes which views are held out. So the Table V SRCC of 0.963/0.949/0.960 almost certainly overstates how well the metric generalizes to unseen 3DGS models. The paper should report a scene-disjoint or at least a model-disjoint split (e.g., hold out entire compression levels or models) before claiming SOTA on their own benchmark.\n\nSecondary issues: no inter-rater agreement statistics, so MOS reliability is unverified; bitrates are not reported, which matters for a compression-focused dataset; and neither dataset nor code is available yet, though the authors say they will release it. The zero-shot cross-dataset results on ENeRF-QA, NeRF-VSQA, and GSC-QA are genuinely helpful and provide independent evidence of transfer—they are the strongest part of the evaluation. They don't rescue the in-domain claim, but they do show the method isn't just memorizing their own data.\n\nBottom line: this is a serious dataset paper, and the authors know the NVS QA literature well. The central flaw is fixable with a different split and a bit more transparency. I'd send it to peer review, with the expectation that the authors re-run the benchmark under a model-disjoint protocol and report the reliability and bitrate details. The dataset alone is likely to be a standard benchmark; the metric needs the re-validation before I'd trust the SOTA numbers.","headline":"A genuinely useful multi-dimensional 3DGS IQA dataset with a plausible LMM metric, but the in-domain SOTA claim is weakened by a train/test split that shares 3DGS model identities; worth reviewing with a required model-disjoint re-evaluation.","tokens_in":21277,"tokens_out":2372,"would_cite":true,"duration_ms":26382,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A single LMM-based metric scores 3DGS image quality in three dimensions and outperforms more than 30 existing metrics on a new 15,200-image benchmark.","keywords":["3D Gaussian splatting","image quality assessment","large multimodal model","multi-dimensional MOS","3DGS compression","novel view synthesis","benchmark dataset","LoRA fine-tuning"],"falsifier":"Train 3DGSI-Assessor on nine scenes and test on the held-out tenth, and also render test views from 3DGS models whose compression parameters never appear in training. If SRCC drops well below the reported within-scene 0.9630, or if a scene-disjoint deep-learning baseline closes the gap with the LMM method, then the SOTA numbers reflect within-scene leakage rather than general 3DGS quality understanding. A second check: compress the same scene twice with identical settings but different random seeds; if predicted scores shift with model identity, the metric is memorizing generators, not just di","tokens_in":20413,"feed_emoji":"🖼️","tokens_out":12047,"duration_ms":129554,"temperature":0.7,"pith_summary":"3D Gaussian Splatting (3DGS) renders scenes quickly, but the compression needed for real deployment introduces its own visual damage—floating artifacts, scattered surfaces, and distortions that hit geometry and color separately—which ordinary image-quality metrics miss. The paper builds a 15,200-image benchmark, 3DGS-IEval-15K+, from 10 scenes compressed by 6 algorithms at systematic distortion levels, with human scores for overall, geometry, and color quality (45,600 MOS labels). It then trains 3DGSI-Assessor, a large multimodal model that looks at a rendered image beside its reference and returns all three quality scores in one forward pass. On this benchmark it reports the best correlation with human judgment among more than 30 compared metrics across all three dimensions—SRCC 0.9630 overall, 0.9489 geometry, and 0.9601 color—and it transfers to other novel-view-synthesis benchmarks without retraining. If the results hold, the metric turns quality assessment into a diagnostic: separate geometry and color scores can tell compression engineers which parameter to re-tune.","feed_headline":"One metric beats 30 rivals on 3DGS quality, scoring geometry and color","feed_subtitle":"A 15,200-image benchmark with human scores lets one model separate overall, geometry, and color quality at once.","key_machinery":"The load-bearing mechanism is the hierarchical visual encoder plus two-stage training. Global semantic features come from a frozen pretrained ViT; two Local Dimension-Specific Feature Extraction Modules (LDS-FEM)—ResNet50-based multi-scale extractors pretrained separately to regress geometry and color MOS—capture low-level, dimension-specific distortions. Their outputs are concatenated, projected into the text-embedding space, and fed into an InternVL2.5-8B LMM that first learns to state quality levels as text (stage 1, with LoRA) and then regresses the three MOS values through three MLP heads (stage 2). This is what allows one model to output overall, geometry, and color quality in a single","core_discovery":"The paper claims that 3DGS quality assessment must be both representation-specific and multi-dimensional, and that a single large multimodal model can deliver both. It shows that geometry and color quality are only moderately correlated in its dataset (SRCC 0.6505), so an overall score alone cannot say which attribute a compression method damaged. Its method, 3DGSI-Assessor, concatenates global semantic features from a pretrained ViT with local features from two ResNet50-based encoders pretrained to regress geometry and color quality, fuses them inside an LMM, and predicts overall, geometry, and color MOS simultaneously from the LMM's hidden states. In the paper's within-scene evaluation, th","pith_inferences":["A scene-disjoint split is the natural next test: train on nine scenes and test on the tenth. If 3DGSI-Assessor's margin over per-dimension baselines narrows substantially, part of the reported advantage is memorizing scene- or model-specific compression signatures rather than generic 3DGS distortions.","The same architecture could be inverted into a training reward: because it emits separate geometry and color scores, a compression method could optimize its geometry and color parameters against these scores directly, the rate-distortion use the paper lists as future work.","The viewpoint-selection criteria (max distance, sparsity, extrapolation, directional novelty) should be re-tested under sparse-view training, where the paper itself notes the train-view versus novel-view quality gap grows; that regime would stress the metric's sensitivity to viewpoint-dependent degradation."],"forward_implications":["One forward pass returns overall, geometry, and color quality scores, so compression tuning no longer needs three separately trained per-dimension IQA models.","On 3DGS-IEval-15K+, 3DGSI-Assessor reaches SRCC 0.9630 overall, 0.9489 geometry, and 0.9601 color, beating every compared metric and improving on the strongest baseline by 10.3% on geometry and 3.6% on color.","Zero-shot evaluation on ENeRF-QA, NeRF-VSQA, and GSC-QA ranks 3DGSI-Assessor first under every correlation measure, suggesting the learned distortion patterns carry across NeRF and 3DGS content.","The weak correlations of PSNR, SSIM, and LPIPS with human MOS on this dataset support moving 3DGS compression benchmarking away from pixel-wise fidelity toward learned perceptual metrics.","The moderate geometry–color MOS correlation (SRCC 0.65) confirms the two dimensions are separable enough that dimension-specific feedback is meaningful for rate-distortion optimization."],"supporting_citations":[{"why":"Scaffold-GS; one of the two anchor-based single-compression-parameter 3DGS algorithms whose distortion levels populate the dataset.","marker":"[3]"},{"why":"HAC; the other anchor-based single-parameter 3DGS compression method used to generate distorted renderings.","marker":"[4]"},{"why":"LightGS; a multi-compression-parameter method whose geometry- and color-level combinations produce 16 distortion levels.","marker":"[5]"},{"why":"CompGS; a multi-compression-parameter method used for the dataset's controlled geometry and color compression levels.","marker":"[6]"},{"why":"C3dGS; a multi-compression-parameter method contributing sensitivity-aware quantized distortions.","marker":"[7]"},{"why":"Compact-3DGS; a multi-compression-parameter method contributing hash-grid and residual-VQ distortions.","marker":"[8]"},{"why":"Preliminary 3DGS-IEval-15K dataset that this work extends from one overall quality dimension to three dimensions, tripling annotation volume.","marker":"[16]"},{"why":"InternVL2.5 backbone supplying the global ViT encoder and the large multimodal model used for feature fusion and reasoning.","marker":"[23]"},{"why":"LoRA low-rank adaptation used in both training stages to fine-tune the visual encoder and LMM efficiently.","marker":"[24]"},{"why":"ITU-T P.910 subjective-testing protocol that defines the 11-point impairment scale and dual-stimulus methodology behind the MOS labels.","marker":"[41]"}],"fun_headline_variants":["One model scores 3DGS quality: overall, geometry, and color at once","New benchmark: 3DGS quality needs separate geometry and color scores","Single LMM predicts overall, geometry, and color quality of 3DGS","15k-image dataset trains AI to judge 3DGS in three ways","Geometry and color quality only 65% correlated in 3DGS, says new study"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The load-bearing premise is that a 4:1 image-level split within each of only 10 scenes—where training and testing images share the same scene content and the same compressed 3DGS model identities—is a valid test of generalization; if the model memorizes scene- or model-specific distortion patterns, the reported state-of-the-art correlations will not survive contact with unseen scenes.","fun_headline_variants_meta":{"raw":{"variants":["One model scores 3DGS quality: overall, geometry, and color at once","New benchmark: 3DGS quality needs separate geometry and color scores","Single LMM predicts overall, geometry, and color quality of 3DGS","15k-image dataset trains AI to judge 3DGS in three ways","Geometry and color quality only 65% correlated in 3DGS, says new study"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000647,"raw_usage":{"total_tokens":2862,"prompt_tokens":851,"completion_tokens":2011,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":595,"completion_tokens_details":{"reasoning_tokens":1905}},"tokens_in":595,"tokens_out":2011,"duration_ms":18547,"temperature":1.0,"reasoning_tokens":1905,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T21:47:57.210905+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train 3DGSI-Assessor on nine scenes and test on the held-out tenth, and also render test views from 3DGS models whose compression parameters never appear in training. If SRCC drops well below the reported within-scene 0.9630, or if a scene-disjoint deep-learning baseline closes the gap with the LMM method, then the SOTA numbers reflect within-scene leakage rather than general 3DGS quality understanding. A second check: compress the same scene twice with identical settings but different random seeds; if predicted scores shift with model identity, the metric is memorizing generators, not just di","supporting_citations":[{"cited_title":"Scaffold-gs: Structured 3d gaussians for view-adaptive rendering,","cited_arxiv_id":null,"evidence_quote":"Scaffold-GS; one of the two anchor-based single-compression-parameter 3DGS algorithms whose distortion levels populate the dataset."},{"cited_title":"Hac: Hash-grid assisted context for 3d gaussian splatting compression,","cited_arxiv_id":null,"evidence_quote":"HAC; the other anchor-based single-parameter 3DGS compression method used to generate distorted renderings."},{"cited_title":"Lightgaussian: Unbounded 3d gaussian compression with 15x reduction and 200+ fps,","cited_arxiv_id":null,"evidence_quote":"LightGS; a multi-compression-parameter method whose geometry- and color-level combinations produce 16 distortion levels."},{"cited_title":"Compgs: Smaller and faster gaussian splatting with vector quantization,","cited_arxiv_id":null,"evidence_quote":"CompGS; a multi-compression-parameter method used for the dataset's controlled geometry and color compression levels."},{"cited_title":"3dgs-ieval-15k: a large-scale image quality evaluation database for 3d gaussian-splatting,","cited_arxiv_id":null,"evidence_quote":"Preliminary 3DGS-IEval-15K dataset that this work extends from one overall quality dimension to three dimensions, tripling annotation volume."},{"cited_title":"Lora: Low-rank adaptation of large language models","cited_arxiv_id":null,"evidence_quote":"LoRA low-rank adaptation used in both training stages to fine-tune the visual encoder and LMM efficiently."},{"cited_title":"Subjective video quality assessment methods for multimedia applications,","cited_arxiv_id":null,"evidence_quote":"ITU-T P.910 subjective-testing protocol that defines the 11-point impairment scale and dual-stimulus methodology behind the MOS labels."}],"review_version":1}