{"id":"1b1d93f7-435c-4a49-8d10-763174f48e62","arxiv_id":"2412.10436","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A clustering-based process for creating federated learning benchmarks with controllable semantic heterogeneity, demonstrated on panoptic scene graph generation and CelebA.","lead":"The authors build a benchmark for training scene-graph models across distributed clients without sharing raw images, by clustering images on their semantic content and then splitting clusters across clients. It is a template for testing how well AI can learn structured scene understanding when data must stay private, such as in media or medical settings.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central claim is under-supported: semantic clusters are not quantitatively validated, and reported IID-vs-non-IID gaps are small and sometimes reversed, so 'controlled semantic heterogeneity' is not yet established.","rationale":"The reader identified the coarse category tensor and the choice of K as the weakest premise. I agree that this is where the argument is least secure, and I add two specifics that make the concern concrete rather than generic: the unsupervised clusters are never quantitatively validated, and the paper's own ablations contain reversed or zero IID-vs-non-IID gaps. These are not external disagreements with consensus; they are internal inconsistencies with the monotonic-control claim in Section 4.2 and Appendix C.5. The random-label control is decisive because it isolates whether K-means semantics, rather than mere label shift or sampling noise, drives the performance drop. The paper does provide code, a plausible pipeline, and some gaps in the expected direction, so the findings are not refuted; however, the central claim should be accepted only after the control and cluster-validity evidence are supplied. Since the reader already assigned CONDITIONAL, my read does not shift the verdict.","tokens_in":19385,"tokens_out":5750,"duration_ms":61430,"concrete_test":"Rerun the full Table 1 shard-IID vs shard-non-IID comparison with K-means cluster assignments replaced by random or permuted cluster labels matched to the same cluster sizes and per-client data quantities, for all four models and at least five seeds. If the mR@20 gap under random labels is statistically indistinguishable from the K-means gap, the measured degradation is not attributable to semantic clusters; if the K-means gap is significantly larger, the semantic-heterogeneity mechanism is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The benchmark's central claim is that partitioning clients by K-means clusters of category tensors imposes controllable semantic heterogeneity. The most load-bearing step is therefore Section 3.1: the clusters must be semantically meaningful and stable. The paper supports this only with qualitative PCA (Fig. 3), manual cluster descriptions, and a small cluster-count ablation (Table 9). No cluster-purity, stability, or separation metrics are reported, and the category tensor in Section 3.1 is described only as allocating labels to orthogonal axes; for PSG it is a 13x13x7 count tensor, but normalization and construction details are omitted, so Euclidean K-means may be driven by relation counts and frequent super-classes rather than scene semantics. The control evidence is also internally inconsistent: in Table 1, MOTIFS has mR@20 4.06 for Dir(alpha=10) (IID) but 4.28 for Dir(alpha=0.2) (non-IID), opposite to the claimed monotonic degradation; in Table 9, VCTree at n=3 shows mR@20 4.73 (IID) vs 4.72 (non-IID), i.e., no gap, and IMP at n=10 shows 1.93 vs 2.38, a reversed gap. Because the central claim requires the heterogeneity knob to produce monotone, semantically controlled effects, the paper has not yet demonstrated its main assertion.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a benchmark process for federated learning on multi-semantic vision datasets, with controllable semantic heterogeneity. The process has two steps: (i) discovering semantic clusters by K-means on a category tensor built from multi-label annotations, and (ii) partitioning data across clients via shard-based or Dirichlet-distribution-based strategies. As a proof of concept, the authors construct a federated Panoptic Scene Graph Generation (PSG) benchmark, evaluate four PSG models (IMP, MOTIFS, VCTree, GPS-Net) under different heterogeneity settings, apply FedAvgM and FedAdam to show robustness gains, and extend the clustering idea to CelebA for a binary classification task. The central claim is that this is the first benchmark framework enabling FL evaluation for multi-semantic vision tasks under controlled semantic heterogeneity.","tokens_in":19709,"tokens_out":4343,"duration_ms":44923,"significance":"If the central claim holds, the paper addresses a genuine gap: existing FL benchmarks rely on single-label classification, while real-world tasks such as scene graph generation carry multiple interdependent semantics. The proposed pipeline—cluster multi-label samples into semantic groups, then partition clients along those groups—is a sensible and potentially reusable recipe, and the paper includes code. The extension to CelebA and the comparison of robust FL algorithms are useful additions. However, the empirical validation is currently too weak to establish the central claim: the cluster semantics are not quantitatively validated, the reported IID-to-non-IID degradation is small and sometimes reversed, and all results are single-seed without variance estimates.","major_comments":[{"comment":"The semantic meaningfulness of the K-means clusters is load-bearing for the entire benchmark, yet it is validated only by qualitative PCA (Fig. 3), manual cluster descriptions, and a cluster-count ablation (Table 9). No cluster-purity, stability, or separation metrics are reported. Moreover, the category tensor is described only as allocating labels to orthogonal axes (ℝ^{13×13×7} for PSG); the construction details—how counts are computed, whether the tensor is normalized, and how multiple relations per image are aggregated—are omitted. Euclidean K-means on raw counts may be dominated by relation counts and frequent super-classes rather than scene semantics. Please specify the tensor construction and report quantitative cluster-validity measures (e.g., silhouette score, stability across K-means restarts, or agreement with human-assigned scene categories).","section":"Section 3.1, Eq. (1)"},{"comment":"The central control claim is that increasing semantic heterogeneity degrades performance monotonically, but the data contain multiple reversals. In Table 1, MOTIFS mR@20 is 4.06 for Dir(α=10) (IID) and 4.28 for Dir(α=0.2) (non-IID), the opposite of the claimed trend. In Table 9, VCTree at n=3 has mR@20 4.73 (IID) vs. 4.72 (non-IID), no gap, and IMP at n=10 has 1.93 (IID) vs. 2.38 (non-IID), reversed. The text acknowledges only the MOTIFS case as an 'outlier' and dismisses it as 'not a considerable amount' without statistical support. Since the benchmark's validity rests on the heterogeneity knob producing controlled effects, these exceptions must be addressed with multi-seed runs and significance tests, or the claim should be weakened to an average trend with documented exceptions.","section":"Table 1, Table 9, Section 4.2"},{"comment":"All experimental results are single-seed with no error bars or significance tests. The reported IID-to-non-IID gaps are small—averaged mR@20 drops of -0.77% (shard) and -0.64% (Dirichlet) in Section 4.2—so sampling noise from the data partition or model initialization could easily account for the observed differences. Please report means and standard deviations over at least 3–5 seeds for the main comparisons, and ideally over multiple realizations of the shard/Dirichlet partitions, so the reader can judge whether the heterogeneity effect is larger than the noise floor.","section":"Section 4.1, all tables"},{"comment":"The cluster-balancing step discards a large fraction of the data: each cluster is subsampled to the smallest cluster size, reducing the PSG training set from 46K to 11K images. This means the IID-vs-non-IID comparison potentially conflates semantic heterogeneity with data scarcity, since each client holds only about 114 images. The imbalanced ablation in Table 6 shows the issue is material: GPS-Net's non-IID mR@20 (8.10) exceeds its IID value (7.38), reversing the claimed trend, and the paper itself notes this is unexpected. Please analyze the IID-vs-non-IID gap as a function of the per-client dataset size, or otherwise justify that subsampling to a common size does not confound the heterogeneity control.","section":"Section 3.2, Table 4, Table 6"},{"comment":"The conclusion claims the benchmark shows 'consistent trends with prior FL studies' for participation rates, but Table 8 does not support this. Increasing the number of participating clients from 5 to 20 improves only MOTIFS; IMP, VCTree, and GPS-Net do not consistently improve, and the text in C.4 acknowledges this deviation. This undermines the generalizability assertion and needs either a revised claim or a PSG-specific explanation grounded in the task's long-tailed predicate distribution and model architectures.","section":"Appendix C.4, Section 6"}],"minor_comments":[{"comment":"The notation in Eq. (1) is garbled; the clustering operator and the indexing of the dataset are not clearly defined. Please rewrite the equation with explicit notation for the set of category tensors over the dataset.","section":"Section 3.1, Eq. (1)"},{"comment":"There is a typo in the subsection heading: 'non-ID' should be 'non-IID'.","section":"Section 4.2"},{"comment":"The PCA visualizations are described only as 'a visualization via Principal Component Analysis'; please state explicitly what features are projected and how much variance the first two components capture, since the visual separation is used as evidence for semantic clustering.","section":"Figure 3 and Figure 7"},{"comment":"The communication-cost column entries such as '64(x 1)' are not self-explanatory; please specify the units (e.g., millions of parameters × rounds) and clarify what the parenthetical factors represent.","section":"Table 5"},{"comment":"The limitations paragraph mentions the difficulty of choosing the number of clusters and the reliance on semantic annotations; this is helpful, but it should also mention the sensitivity of the IID-vs-non-IID gap to model and K shown in Table 9.","section":"Conclusion"}],"recommendation":"major_revision","confidential_remarks":"The paper fits the journal's scope as an empirical/benchmark contribution, and the proposed process is a reasonable starting point for FL on multi-semantic vision tasks. However, the central claim of 'controlled semantic heterogeneity' is not yet established because the cluster quality is unquantified, the reported heterogeneity effects are small and occasionally reversed, and all results are single-seed. These are fixable with additional experiments rather than fundamental flaws, so I recommend major revision rather than rejection. The reversed trends in Tables 1, 6, and 9 are the main risk to the paper's credibility and should be addressed head-on."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nIf you work on federated learning for structured prediction or on scene graph generation, this paper is worth your time. It is the first FL benchmark built specifically for panoptic scene graph generation, and the core idea is straightforward: represent each image's scene graph as a category tensor over object and predicate super-classes, K-means cluster those tensors into semantic scene types, then partition clients over those clusters with shard or Dirichlet schemes to dial heterogeneity up or down. The pipeline is clearly explained, the code is public, and they test it across four SGG models, three cluster counts, several client counts and participation rates, and two FL optimizers. The CelebA extension is a nice generality check.\n\nCredit where due: this is a benchmark template, not a new model, and the paper does not oversell that. The step of equalizing cluster sizes before partitioning is thoughtful—it separates semantic heterogeneity from plain data quantity. The ablation on the number of clusters (n=3, 5, 10) is exactly what a benchmark paper should include, and the results mostly show the expected ordering: IID beats non-IID, and the gap widens with stronger heterogeneity.\n\nThe soft spots are real but not fatal. The central claim is that the heterogeneity is semantically controlled, and that rests on the clusters being meaningful. The only evidence is qualitative PCA visualizations and manual cluster descriptions; no purity, stability, or separation metrics are reported. The category tensor construction is also underspecified—how the 13x13x7 counts are normalized matters for K-means. And the validation is single-seed throughout, with no error bars. A few reported cells go the wrong way: MOTIFS mR@20 is higher at alpha=0.2 than at alpha=10, VCTree at n=3 shows no IID/non-IID gap, and IMP reverses at n=10. The authors acknowledge one of these but call it insignificant without variance estimates, which is hard to judge. These issues do not sink the contribution; they mean the word \"controlled\" is stronger than the evidence currently backs.\n\nThe right reader is someone building FL benchmarks for multi-label or structured tasks, or an SGG researcher curious about distributed training. I would send it to review, but with clear requests for quantitative cluster validation and multi-seed experiments before it becomes a reference benchmark. The core idea—a reusable way to turn multi-semantic annotations into a heterogeneity knob—is solid and worth building on.","headline":"Useful first FL benchmark for scene graph generation; the clustering pipeline is clean, but 'controlled semantic heterogeneity' needs quantitative cluster validation and multi-seed results.","tokens_in":20179,"tokens_out":3736,"would_cite":true,"duration_ms":36915,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims to deliver the first benchmark framework that enables federated learning and its evaluation for multi-semantic vision tasks under controlled semantic heterogeneity.","keywords":["Scene Graph Generation","Panoptic Scene Graph Generation","Federated Learning","Data Heterogeneity","Semantic Heterogeneity","K-means Clustering","Category Tensor","Benchmark"],"falsifier":"Run the same shard and Dirichlet partitions with shuffled cluster labels while keeping every client's data quantity identical; if the IID-to-non-IID performance gap persists, the effect is driven by partition sizes or client data volume, not by the semantic content of the clusters. Alternatively, compute cluster purity or silhouette scores on held-out PSG annotations; if the K-means clusters do not separate scene types beyond the 13-object and 7-predicate super-class mapping, the benchmark's claimed semantic heterogeneity is not established.","tokens_in":19210,"feed_emoji":"🖼️","tokens_out":7998,"duration_ms":71716,"temperature":0.7,"pith_summary":"Federated learning benchmarks today are built around single-label classification, so they cannot measure how models cope when each image carries several interlocking semantics. This paper proposes a two-step benchmark process for multi-semantic datasets: first, turn every sample's labels into a category tensor and cluster samples by scene type with K-means; second, distribute those clusters across clients with shard-based or Dirichlet-based partitions to set the level of semantic heterogeneity. As a proof of concept, the authors build a federated Panoptic Scene Graph Generation benchmark and show that four existing PSG methods lose accuracy as the data becomes more semantically non-IID. They also show that momentum-based federated optimizers recover part of that loss, and that the same recipe transfers to the multi-attribute CelebA dataset. The point of the work is to give the field a controllable, privacy-preserving testbed for semantic tasks, not just object labels.","feed_headline":"A new benchmark brings scene graph generation to federated learning","feed_subtitle":"Clustering by semantic tensors lets researchers tune data heterogeneity across clients for scene graph tasks.","key_machinery":"The central object is the category tensor, a $\\mathbb{R}^{13 \\times 13 \\times 7}$ array for PSG built by mapping each image's subject, object, and predicate into coarse super-classes; each fine-grained label occupies an entry along its axis, so the tensor summarizes the full semantic content of an image in one vector. K-means clustering on the collection of these tensors discovers semantic scene clusters, and the cluster assignment then becomes the single label used by standard FL partitioning schemes. The two partition mechanisms carry the control: shard-based partitioning with parameter $p$ (number of clusters per client) and Dirichlet-based partitioning with concentration $\\alpha$ both interpolate smoothly between IID and strongly non-IID semantic distributions.","core_discovery":"The paper's central claim is that it supplies the first benchmark framework enabling federated learning and its evaluation for multi-semantic vision tasks under controlled semantic heterogeneity. The enabling step is to compress each image's scene graph (objects, subjects, predicates) into a category tensor, run K-means to obtain five semantic clusters (animal scenes, daily human activities, transportation scenes, sports scenes, urban-nature landscapes), equalize cluster sizes to avoid long-tail confounding, and then partition the clusters across clients with either shard assignment or a Dirichlet distribution whose concentration parameter $\\alpha$ dials heterogeneity from IID to strongly non-IID. The benchmark's validity rests on the observed performance ordering: centralized learning beats IID federated learning, IID roughly matches random partitioning, and strong non-IID causes clear degradation across IMP, MOTIFS, VCTree, and GPS-Net. On top of that baseline, FedAvgM and FedAdam improve accuracy under semantic heterogeneity, and the same clustering-and-partition recipe on CelebA reproduces the expected slowdown and accuracy loss in non-IID settings.","pith_inferences":["Because the category tensor only needs a fixed set of label axes, the same recipe should transfer to other multi-label vision-language tasks such as visual question answering or dense captioning, where one sample carries several semantic claims.","A quantitative cluster-quality check — for instance silhouette score on the category tensor or agreement with human scene-type labels — would let future users choose the number of clusters $n$ without relying on qualitative PCA or the small ablation table.","If the semantic-heterogeneity signal is caused by the scene-type clusters, then shuffling cluster labels while keeping the same partition sizes should erase the IID-to-non-IID performance gap; this is a direct falsification test implied by the benchmark design.","The paper's equalization of cluster sizes removes the natural long-tail imbalance; a natural variant would keep the imbalance and test whether FL aggregation methods that downweight dominant clusters can handle both problems at once."],"forward_implications":["Four existing PSG models — IMP, MOTIFS, VCTree, and GPS-Net — all lose Recall and mean-Recall when moving from IID to strongly non-IID semantic partitions, so the benchmark creates a measurable heterogeneity signal rather than a random split.","Methods that already handle the long-tailed predicate distribution, notably GPS-Net, are more robust to semantic heterogeneity, suggesting that long-tail robustness and FL heterogeneity robustness are related problems.","Server-side momentum (FedAvgM) and adaptive optimization (FedAdam) improve mR@K by about one to three points under shard non-IID, giving FL algorithm developers a semantic task on which to test.","Ablation with $n=3$, $n=5$, and $n=10$ clusters preserves the qualitative method ordering and the IID-to-non-IID gap, so the benchmark does not depend on one magical cluster count.","On CelebA, the same clustering-plus-partition pipeline produces slower convergence and lower accuracy in non-IID settings, indicating the process is not tied to scene graphs."],"supporting_citations":[{"why":"Supplies the PSG dataset, the R@K and mR@K evaluation metrics, and the centralized baseline results.","marker":"[28]"},{"why":"Defines FedAvg and the server-client aggregation loop that the benchmark uses as its FL baseline.","marker":"[23]"},{"why":"Provides the Dirichlet-based data-distribution strategy that the paper adapts to control semantic heterogeneity.","marker":"[1]"},{"why":"GPS-Net is the long-tail-robust PSG method whose results anchor the robustness comparison.","marker":"[20]"},{"why":"IMP is one of the four PSG baselines whose IID-to-non-IID performance gap is measured.","marker":"[27]"},{"why":"MOTIFS is one of the four PSG baselines whose IID-to-non-IID performance gap is measured.","marker":"[29]"},{"why":"VCTree is one of the four PSG baselines whose IID-to-non-IID performance gap is measured.","marker":"[26]"},{"why":"FedAvgM is the momentum-based federated algorithm tested as an extension.","marker":"[8]"},{"why":"FedAdam is the adaptive server optimizer tested as an extension.","marker":"[24]"}],"fun_headline_variants":["Federated scene graph benchmark with tunable semantic heterogeneity","First federated benchmark for multi-semantic vision tasks","Semantic clustering dials non-IID in federated scene graph tests","Turn scene graphs into federated testbeds with tunable splits","Cluster scene graphs, then distribute to clients for federated test"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the coarse category tensor built from 13 object super-classes and 7 predicate super-classes preserves enough semantic information that K-means clusters correspond to meaningful scene types, so the induced client partitions reflect genuine semantic heterogeneity rather than an artifact of the coarse mapping or the chosen cluster count $n=5$.","fun_headline_variants_meta":{"raw":{"variants":["Federated scene graph benchmark with tunable semantic heterogeneity","First federated benchmark for multi-semantic vision tasks","Semantic clustering dials non-IID in federated scene graph tests","Turn scene graphs into federated testbeds with tunable splits","Cluster scene graphs, then distribute to clients for federated test"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000655,"raw_usage":{"total_tokens":3018,"prompt_tokens":984,"completion_tokens":2034,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":600,"completion_tokens_details":{"reasoning_tokens":1948}},"tokens_in":600,"tokens_out":2034,"duration_ms":16015,"temperature":1.0,"reasoning_tokens":1948,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T18:07:28.586191+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same shard and Dirichlet partitions with shuffled cluster labels while keeping every client's data quantity identical; if the IID-to-non-IID performance gap persists, the effect is driven by partition sizes or client data volume, not by the semantic content of the clusters. Alternatively, compute cluster purity or silhouette scores on held-out PSG annotations; if the K-means clusters do not separate scene types beyond the 13-object and 7-predicate super-class mapping, the benchmark's claimed semantic heterogeneity is not established.","supporting_citations":[{"cited_title":"Panoptic scene graph generation, in: European Conference on Com- puter Vision, Springer","cited_arxiv_id":null,"evidence_quote":"Supplies the PSG dataset, the R@K and mR@K evaluation metrics, and the centralized baseline results."},{"cited_title":"Gps-net: Graph property sensing network for scene graph generation, in: Proceedings of the IEEE/CVFConferenceonComputerVisionandPatternRecognition, pp","cited_arxiv_id":null,"evidence_quote":"GPS-Net is the long-tail-robust PSG method whose results anchor the robustness comparison."},{"cited_title":"5410–5419","cited_arxiv_id":null,"evidence_quote":"IMP is one of the four PSG baselines whose IID-to-non-IID performance gap is measured."},{"cited_title":"Neural motifs: Scenegraphparsingwithglobalcontext,in:ProceedingsoftheIEEE conference on computer vision and pattern recognition, pp","cited_arxiv_id":null,"evidence_quote":"MOTIFS is one of the four PSG baselines whose IID-to-non-IID performance gap is measured."},{"cited_title":"Learning to compose dynamic tree structures for visual contexts, in: Proceedings of the IEEE/CVF conference on computer vision and pattern recog- nition, pp","cited_arxiv_id":null,"evidence_quote":"VCTree is one of the four PSG baselines whose IID-to-non-IID performance gap is measured."}],"review_version":1}