{"id":"00f1f3e1-76aa-4818-9b8f-6585080b3bba","arxiv_id":"2506.04081","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"A clustering-plus-graph-attention method for blind point cloud quality assessment is presented, with reported state-of-the-art correlations that are contradicted or unsubstantiated in the paper's own evaluation.","lead":"This paper proposes a blind quality metric for 3D point clouds that clusters points by color, curvature, and saliency, builds a graph over the clusters, and predicts quality with a graph attention network. The authors report top correlations on three benchmarks, but the graph formula and evaluation protocol contain internal inconsistencies that undermine the result.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported SOTA correlations depend on an evaluation protocol that is internally inconsistent: an 80/10/10 split with no reference-content overlap cannot produce per-content SRCC for all 9 SJTU contents, so headline numbers may reflect leakage or in-sample evaluation.","rationale":"The reader's weakest assumption is the evaluation protocol, and I agree that it is the most load-bearing point. The arithmetic of the stated split and the reported tables is incompatible. With 9 SJTU reference clouds and a content-disjoint 10% test, the test set is at most ~42 samples from one content, yet Table 7 gives per-content SRCC for all 9 contents and per-distortion SRCC for 7 distortion types. This is not a matter of interpretation; the numbers cannot be produced from the described protocol. The same holds for WPC (20 references) in Table 8. This directly undermines the abstract's PLCC/SRCC claim. I do not infer intentional misconduct; the more parsimonious explanation is that the tables were computed on a different split or on the full dataset, but either way the central empirical claim is unverified. The graph construction inconsistencies (e.g., Eq. 6 weights reducing to a geometric function of De, and Eq. 8 using spatial coordinates of a perceptual-feature centroid) are real and would need correction for the method to be reproducible, but they are secondary to the evaluation problem because even a correctly specified method must be validated on a non-leaked test set. My concrete test would settle the concern definitively: release exact split indices and reproduce Tables 7-8 from test predictions only. Until then, the paper's reported SOTA correlations are not supported. I therefore keep the reader's REJECT verdict unchanged.","tokens_in":22844,"tokens_out":5165,"duration_ms":46116,"concrete_test":"Ask the authors to release the exact train/validation/test sample indices for each dataset (or the code that generates them). Then verify: (1) For SJTU-PCQA, does any reference point cloud appear in both train and test? If the split is content-disjoint, count how many test samples belong to each of the 9 reference clouds; Table 7 reports per-content SRCC for all 9, so at least 8 contents must have test samples. (2) Recompute the per-content and per-distortion SRCC from the released test predictions only; if the values in Tables 7-8 cannot be reproduced from that test set, the headline correlations are not supported. (3) Independently re-run the 10% test split with a fixed seed and compare overall PLCC/SRCC; if the overall metrics change by more than a few points, the reported numbers are not stable.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (abstract: PLCC ≥ 0.93, SRCC ≥ 0.91, better than alternative NR-PCQA) rests entirely on the reported evaluation. Section 3.7 states the data are split 80/10/10 with 'no overlap between reference point clouds across splits.' SJTU-PCQA has 9 publicly available reference clouds; 378 samples are used. A content-disjoint 10% test set therefore contains at most one reference cloud (≈42 samples), and the other 8 reference clouds have zero test samples. Yet Table 7 reports per-content SRCC for all 9 SJTU contents, each requiring predictions for that content. These numbers cannot be computed from the stated test set. The same problem affects the per-distortion SRCC rows: each distortion type (e.g., OT, CN) spans all 9 contents, so a 38-sample test set cannot contain enough samples per distortion for all 7 rows. Table 8 repeats the issue for WPC's 20 reference clouds. Consequently, either the tables were computed on training/validation data, or the split is sample-level random (leaking reference content into training). In both cases every headline PLCC/SRCC/RMSE is called into question. The paper does not report the exact test sample IDs, the values of k, r, and sigma, or provide code, so the reader cannot rule out leakage. This is the most load-bearing threat to the claim; the graph construction inconsistencies (Eq. 6 vs Eq. 8) are secondary because even a correct implementation would still need a valid evaluation to support SOTA.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a no-reference point cloud quality assessment (NR-PCQA) method, PCW-Graph, which clusters point clouds by color, curvature, and saliency features, builds a weighted graph whose nodes are clusters, and feeds the graph into a Graph Attention Fusion network to regress a quality score. The method is evaluated on SJTU-PCQA, WPC, and ICIP2020, and the authors claim it achieves PLCC ≥ 0.93 and SRCC ≥ 0.91 and reduces RMSE by up to 15%, outperforming alternative NR-PCQA methods. The paper also includes ablation studies, a cross-database evaluation, and a comparison with 24 quality metrics.","tokens_in":23200,"tokens_out":7277,"duration_ms":69770,"significance":"If the empirical claims were valid, the paper would offer a useful interpretable graph-based NR-PCQA pipeline and a broad comparison on three public benchmarks. The authors deserve credit for including an ablation study, a cross-database experiment, and comparisons with many FR, RR, and NR methods. However, the central SOTA claim is not supported by the paper's own tables, and the evaluation protocol described in Section 3.7 is inconsistent with the per-content and per-distortion results in Tables 7 and 8. Because the main contribution is an empirical performance claim, these issues are load-bearing. No code, pretrained models, or exact hyperparameters are provided, which further limits verification.","major_comments":[{"comment":"The stated content-disjoint 80/10/10 split cannot produce the per-content SRCC results reported in Table 7. SJTU-PCQA has 9 publicly available reference clouds and 378 used samples, so a 10% test set with no reference overlap contains at most one reference cloud (42 samples). Under that split, per-content SRCC for all 9 SJTU contents cannot be computed, and per-distortion SRCC rows would rest on samples from at most one reference cloud (6 samples per distortion), which is far too few for reliable correlation estimates. The same issue affects the 20 content rows in Table 8 for WPC. Therefore, either Tables 7 and 8 were computed on training/validation samples, or the split is not actually content-disjoint. In both cases, the headline Table 5 numbers are called into question. The paper does not report test sample IDs or code, so the reader cannot rule out leakage.","section":"Section 3.7, Tables 7 and 8"},{"comment":"The claim that the method 'consistently outperforms all other quality assessment techniques on all three datasets' is contradicted by the ICIP2020 column of Table 5. GMS-3DQA achieves PLCC 0.9981 and RMSE 0.0686, while Ours achieves PLCC 0.9926 and RMSE 0.0690. Thus the proposed method does not outperform GMS-3DQA on PLCC or RMSE for ICIP2020. The SOTA claim must be revised to specify the exact criteria and datasets on which the method is actually best, and the SRCC value of 0.9999 for ICIP2020 (with 90 samples) warrants additional scrutiny given the split issues.","section":"Table 5 and Section 4.4"},{"comment":"The abstract's claim of RMSE reduction 'up to 15%' is unsupported by the reported results. In Table 5, the largest RMSE reduction over a comparable NR method is about 3.8% on WPC (Ours 11.8853 vs MM-PCQA 12.3506), and on ICIP2020 the RMSE increases relative to GMS-3DQA (0.0690 vs 0.0686). A 15% reduction does not appear anywhere in the table if the comparison is restricted to the NR methods listed there. The authors should either identify the exact baseline producing a 15% improvement or remove the claim.","section":"Abstract and Table 5"},{"comment":"The graph weight definition is internally inconsistent. Equation (6) defines W_ij = Sim(C_i, C_j) · De(C_i, C_j), while the text after Eq. (9) states that the weight combines similarity scores with 'inverse distance scaling.' The equations multiply by De, not by 1/De. In addition, Eq. (8) labels De as the Euclidean distance between spatial centroids, but Eq. (7) defines µ_i as the average of perceptual feature vectors, which include color, curvature, and saliency. Since Sim in Eq. (9) is itself a function of De, the claimed 'dual metric' is actually a product of a distance and a decaying function of that same distance. This inconsistency affects the core graph-construction contribution and must be resolved by rewriting the definitions and the accompanying text.","section":"Section 3.3, Eqs. (6)-(9)"}],"minor_comments":[{"comment":"Equation (16) uses notation 'LM ℓ=1' and 'n a=1' that is not standard and is likely a typesetting error; the equation should be rewritten with explicit summations or concatenation operators.","section":"Section 3.6, Eq. (16)"},{"comment":"The paper acknowledges sensitivity to the number of clusters k in the conclusion, but it never reports the values of k, neighborhood radius r, saliency scales σ1 and σ2, or the cluster feature dimension used in the experiments. These hyperparameters are necessary for reproducibility, especially because no code is provided.","section":"Section 3.3 and Section 5"},{"comment":"Several entries in Tables 7 and 8 are 'NaN' (e.g., CN distortion for M1-M5 in Table 7), but the tables are described as using absolute SRCC values; the reason for the missing values should be stated in the caption or text.","section":"Tables 7 and 8"},{"comment":"The cross-database evaluation trains on the full WPC dataset and validates on the full SJTU-PCQA dataset, which is a different protocol from the 80/10/10 split used in Table 5; the text should clarify how these results relate to the main evaluation and whether they use all SJTU samples.","section":"Section 4.6"},{"comment":"The computational efficiency comparison in Figure 6 includes only hand-crafted model-based methods (PCQM, PCMRR, GraphSIM, PointSSIM, 3D-NSS) and omits deep-learning NR methods such as GMS-3DQA and MM-PCQA; the claim of 'lower average time cost' should be scoped accordingly.","section":"Section 4.7 and Figure 6"}],"recommendation":"reject","confidential_remarks":"The manuscript's central empirical claim is invalidated by its own Table 5 and by the inconsistency between the content-disjoint split described in Section 3.7 and the per-content results in Tables 7 and 8. Because the contribution rests on the claimed SOTA performance, this is a load-bearing error that cannot be addressed by local revisions; the experiments would need to be redone under a verifiable protocol, and the claims would likely change. I recommend rejection, though the authors could resubmit a substantially revised version with corrected evaluation and reproducible details."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the pipeline — K-means on perceptual features, a cluster-level weighted graph, and a graph-attention fusion network for no-reference PCQA — is a reasonable combination, and the ablation study is genuinely informative. But the reported empirical results are not credible as stated. The evaluation protocol described in Section 3.7 cannot produce the per-content tables in Tables 7 and 8, and the abstract's SOTA claim is internally contradicted by Table 5. I would reject in current form.\n\nWhat is new: the specific pipeline does not appear in the cited literature, so there is a novel assembly of known components. The paper also reports cross-database and runtime experiments, which is more than many NR-PCQA papers do.\n\nThe load-bearing problem is the evaluation. SJTU-PCQA has nine public reference clouds; a content-disjoint 10% test split holds at most one reference (42 samples). Yet Table 7 reports per-content SRCC for all nine contents, each requiring predictions for that content. Those numbers cannot be computed from the stated test set. The same contradiction appears for WPC's twenty references in Table 8. Either the per-content numbers come from training/validation data, or the split is sample-level random and reference content leaks into training. In either case every headline PLCC/SRCC/RMSE in Table 5 is suspect. The authors do not provide test sample IDs, the values of k, r, or sigma, or code, so there is no way to check.\n\nThere are smaller but real consistency problems. Table 5 shows GMS-3DQA beating the method on ICIP2020 PLCC (0.9981 vs 0.9926) and on SJTU-PCQA KRCC, which directly contradicts the abstract's claim of better alignment than all alternatives. The 'up to 15% RMSE reduction' is unsupported; the largest visible improvement is about 2.8% on WPC, and RMSE worsens on ICIP2020. The graph equations are also internally inconsistent: Eq. 7 defines the cluster centroid as a perceptual feature vector, Eq. 8 treats it as a 3D spatial coordinate, and Eq. 9 makes the 'perceptual similarity' weight a pure function of geometric distance, so the perceptual weighting adds no independent information. The text describing inverse-distance scaling does not match Eq. 6.\n\nThe method itself is not absurd; a careful reimplementation with a proper content-disjoint protocol and code release could be a useful contribution. But as presented, the empirical claims do not survive their own evaluation. I would send it to a reviewer only if the authors are willing to redo the experiments. As-is: reject.","headline":"Plausible graph-attention PCQA pipeline whose reported SOTA results are undone by an internally inconsistent evaluation protocol.","tokens_in":68,"tokens_out":7674,"would_cite":false,"duration_ms":110882,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A no-reference quality assessment method for point clouds, built on a Perceptual Clustering Weighted Graph and a Graph Attention Fusion network, reports PLCC of 0.93 or higher and SRCC of 0.91 or higher on three benchmarks and beats the…","keywords":["Point Clouds","Segmentation","No-Reference PCQA","Complex networks","Perceptual Clustering Weighted Graph","Graph Attention Fusion","Quality assessment","Perceptual features"],"falsifier":"Compute SRCC and PLCC under a strictly content-disjoint split (train only on distortions of some reference clouds, test only on distortions of held-out reference clouds) and compare per-content and per-distortion values to Tables 7 and 8; if the held-out-reference correlations fall well below the reported 0.9118 (SJTU-PCQA), 0.8688 (WPC), or 0.9999 (ICIP2020), reference leakage inflated the claim.","tokens_in":22607,"feed_emoji":"📊","tokens_out":8029,"duration_ms":64721,"temperature":0.7,"pith_summary":"The paper tries to show that a no-reference point cloud quality metric can match human perceptual judgments without needing the original model. It proposes the Perceptual Clustering Weighted Graph (PCW-Graph): points are grouped by color, curvature, and saliency; clusters become graph nodes; edges are weighted by a mix of perceptual similarity and geometric distance; and a Graph Attention Fusion network adaptively weighs the features and regresses a quality score. On the WPC, SJTU-PCQA, and ICIP2020 benchmarks the reported Pearson correlation is at least 0.93 and Spearman at least 0.91, higher than the compared no-reference metrics and close to full-reference ones, with RMSE reduced by up to 15%. If correct, this gives applications such as compression, rendering, and 3D streaming a blind quality estimator that works where no reference model is available.","feed_headline":"No-reference point-cloud metric beats 24 rivals on 3 benchmarks","feed_subtitle":"Perceptual clustering graph plus attention fusion matches human quality scores on WPC, SJTU-PCQA, and ICIP2020.","key_machinery":"The central object is the Perceptual Clustering Weighted Graph (PCW-Graph), built by K-means clustering of a distorted point cloud using LAB color, curvature, and saliency features; each cluster becomes a node, edges connect clusters within a neighborhood radius, and edge weights combine an RBF perceptual similarity term with inverse geometric distance. The Graph Attention Fusion (GAF) network processes separate adjacency matrices for color, curvature, and saliency through multi-head attention, concatenates and fuses them, and then a Graph Attention Network (GAT) layer regresses the final quality score. This cluster-level abstraction is what lets the method weigh structural and perceptual relationships while keeping computational cost below several point-level full-reference metrics.","core_discovery":"The central claim is that the perceptual quality of a distorted point cloud can be predicted blindly by first compressing it into a small set of perceptually coherent clusters, then encoding the inter-cluster relationships in a weighted graph, then letting a graph attention fusion network choose which features matter. In the paper's experiments on SJTU-PCQA, WPC, and ICIP2020, this pipeline achieves PLCC ≥ 0.93 and SRCC ≥ 0.91 on every dataset, placing it ahead of the compared NR baselines, including MM-PCQA and GMS-3DQA, and on ICIP2020 it reports near-perfect SRCC of 0.9999. The ablation study attributes the gain to the fusion step: using all three features without the fusion network yields PLCC 0.7980 on SJTU-PCQA, versus 0.9338 with it.","pith_inferences":["Editorial inference: The per-content and per-distortion SRCC tables (Tables 7 and 8) do not state whether their samples come from reference point clouds that were in the training split; if any test sample's reference was seen in training, the reported averages are inflated, and a content-disjoint re-run would settle the size of the effect.","Editorial inference: Because ICIP2020 has only six reference clouds, the reported SRCC of 0.9999 is dominated by a few content rows; per-reference confidence intervals would show how stable the number actually is.","Editorial inference: The saliency feature alone nearly matches the full model on ICIP2020 (SRCC 0.9215 vs 0.9999), which suggests a simpler saliency-only or saliency-weighted variant might be worth testing as a lower-cost version of the method.","Editorial inference: The abstract's 'RMSE reduced by up to 15%' does not name the baseline; the paper should state the reference metric for that reduction before the claim is used in head-to-head comparisons."],"forward_implications":["A blind quality score for point clouds can be produced from the distorted cloud alone, so compression pipelines and streaming services could rank or filter 3D content without storing a reference model.","The reported cross-database result (WPC→SJTU PLCC 0.7928, SRCC 0.8008) suggests the learned graph features transfer across datasets and distortion types rather than memorizing a single benchmark.","The ablation result (full features without GAF: PLCC 0.7980; with GAF: 0.9338 on SJTU-PCQA) is a direct corollary that the fusion module, not any single feature, carries most of the performance gain.","Because the graph is built from perceptual clusters rather than raw points, the method's complexity scales with the number of clusters, not the point count, which is what allows it to run faster than several point-based full-reference metrics on the same hardware.","The reported best configuration (3 layers, tanh activation, batch size 32) gives a concrete recipe for reproducing the claimed correlations on all three datasets."],"supporting_citations":[{"why":"Supplies the SJTU-PCQA subjective benchmark used for the main evaluation and per-content/per-distortion tables.","marker":"[24]"},{"why":"Supplies the WPC subjective benchmark used for the main evaluation and per-content/per-distortion tables.","marker":"[79]"},{"why":"Supplies the ICIP2020 subjective benchmark used for the main evaluation.","marker":"[26]"},{"why":"The strongest compared no-reference baseline on SJTU-PCQA and WPC that the proposed method claims to beat.","marker":"[64]"},{"why":"The compared projection-based no-reference baseline with the best ICIP2020 SRCC among prior methods.","marker":"[63]"},{"why":"Provides the graph attention network formulation used for quality regression.","marker":"[82]"},{"why":"Provides the multi-head transformer attention mechanism used in the fusion branches.","marker":"[81]"},{"why":"The no-reference 3D-NSS baseline used in the cross-database evaluation.","marker":"[13]"}],"fun_headline_variants":["Blind point-cloud metric with graph attention beats 24 rivals","Graph attention fusion lifts blind PCQA from 0.80 to 0.93 PLCC","Perceptual clustering + attention fusion boosts PCQA to 0.93 PLCC","NR point-cloud quality: clustering graph attention fusion wins 3 benchmarks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported correlations depend on an 80/10/10 train/validation/test split in which no reference point cloud appears in both training and test; if the per-content and per-distortion tables used samples whose reference clouds were also in training, all headline numbers would be inflated.","fun_headline_variants_meta":{"raw":{"variants":["Blind point-cloud metric with graph attention beats 24 rivals","Graph attention fusion lifts blind PCQA from 0.80 to 0.93 PLCC","Perceptual clustering + attention fusion boosts PCQA to 0.93 PLCC","NR point-cloud quality: clustering graph attention fusion wins 3 benchmarks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000156,"raw_usage":{"total_tokens":1117,"prompt_tokens":743,"completion_tokens":374,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":359,"completion_tokens_details":{"reasoning_tokens":291}},"tokens_in":359,"tokens_out":374,"duration_ms":4395,"temperature":1.0,"reasoning_tokens":291,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T10:47:18.131921+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute SRCC and PLCC under a strictly content-disjoint split (train only on distortions of some reference clouds, test only on distortions of held-out reference clouds) and compare per-content and per-distortion values to Tables 7 and 8; if the held-out-reference correlations fall well below the reported 0.9118 (SJTU-PCQA), 0.8688 (WPC), or 0.9999 (ICIP2020), reference leakage inflated the claim.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the SJTU-PCQA subjective benchmark used for the main evaluation and per-content/per-distortion tables."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the WPC subjective benchmark used for the main evaluation and per-content/per-distortion tables."},{"cited_title":"P., da Silva Cruz, L","cited_arxiv_id":null,"evidence_quote":"Supplies the ICIP2020 subjective benchmark used for the main evaluation."},{"cited_title":"& Lin, W","cited_arxiv_id":null,"evidence_quote":"The compared projection-based no-reference baseline with the best ICIP2020 SRCC among prior methods."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The no-reference 3D-NSS baseline used in the cross-database evaluation."}],"review_version":1}