REVIEW 5 major objections 5 minor 33 references
Benchmarking Federated Learning for Semantic Datasets: Federated Scene Graph Generation
T0 review · 5 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read The paper claims to deliver the first benchmark framework that enables federated learning and its evaluation for multi-semantic vision tasks under controlled semantic heterogeneity.
desk verdict Useful first FL benchmark for scene graph generation; the clustering pipeline is clean, but 'controlled semantic heterogeneity' needs quantitative cluster validation and multi-seed results. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the category tensor, a $\mathbb{R}^{13 \times 13 \times 7}$ array for PSG built by mapping each image's subject, object, and predicate into coarse super-classes; each fine-grained label occupies an entry along its axis, so the tensor summarizes the full semantic content of an image in one vector. K-means clustering on the collection of these tensors discovers semantic scene clusters, and the cluster assignment then becomes the single label used by standard FL partitioning schemes. The two partition mechanisms carry the control: shard-based partitioning with parameter $p$ (number of clusters per client) and Dirichlet-based partitioning with concentration $\alpha$ both interpolate smoothly between IID and strongly non-IID semantic distributions.
What would settle it
Run the same shard and Dirichlet partitions with shuffled cluster labels while keeping every client's data quantity identical; if the IID-to-non-IID performance gap persists, the effect is driven by partition sizes or client data volume, not by the semantic content of the clusters. Alternatively, compute cluster purity or silhouette scores on held-out PSG annotations; if the K-means clusters do not separate scene types beyond the 13-object and 7-predicate super-class mapping, the benchmark's claimed semantic heterogeneity is not established.
Extended reading notes
Core claim
The paper's central claim is that it supplies the first benchmark framework enabling federated learning and its evaluation for multi-semantic vision tasks under controlled semantic heterogeneity. The enabling step is to compress each image's scene graph (objects, subjects, predicates) into a category tensor, run K-means to obtain five semantic clusters (animal scenes, daily human activities, transportation scenes, sports scenes, urban-nature landscapes), equalize cluster sizes to avoid long-tail confounding, and then partition the clusters across clients with either shard assignment or a Dirichlet distribution whose concentration parameter $\alpha$ dials heterogeneity from IID to strongly non-IID. The benchmark's validity rests on the observed performance ordering: centralized learning beats IID federated learning, IID roughly matches random partitioning, and strong non-IID causes clear degradation across IMP, MOTIFS, VCTree, and GPS-Net. On top of that baseline, FedAvgM and FedAdam improve accuracy under semantic heterogeneity, and the same clustering-and-partition recipe on CelebA reproduces the expected slowdown and accuracy loss in non-IID settings.
Load-bearing premise
The load-bearing premise is that the coarse category tensor built from 13 object super-classes and 7 predicate super-classes preserves enough semantic information that K-means clusters correspond to meaningful scene types, so the induced client partitions reflect genuine semantic heterogeneity rather than an artifact of the coarse mapping or the chosen cluster count $n=5$.
Editorial extensions
If this is right
- Four existing PSG models — IMP, MOTIFS, VCTree, and GPS-Net — all lose Recall and mean-Recall when moving from IID to strongly non-IID semantic partitions, so the benchmark creates a measurable heterogeneity signal rather than a random split.
- Methods that already handle the long-tailed predicate distribution, notably GPS-Net, are more robust to semantic heterogeneity, suggesting that long-tail robustness and FL heterogeneity robustness are related problems.
- Server-side momentum (FedAvgM) and adaptive optimization (FedAdam) improve mR@K by about one to three points under shard non-IID, giving FL algorithm developers a semantic task on which to test.
- Ablation with $n=3$, $n=5$, and $n=10$ clusters preserves the qualitative method ordering and the IID-to-non-IID gap, so the benchmark does not depend on one magical cluster count.
- On CelebA, the same clustering-plus-partition pipeline produces slower convergence and lower accuracy in non-IID settings, indicating the process is not tied to scene graphs.
Reading between the lines
- Because the category tensor only needs a fixed set of label axes, the same recipe should transfer to other multi-label vision-language tasks such as visual question answering or dense captioning, where one sample carries several semantic claims.
- A quantitative cluster-quality check — for instance silhouette score on the category tensor or agreement with human scene-type labels — would let future users choose the number of clusters $n$ without relying on qualitative PCA or the small ablation table.
- If the semantic-heterogeneity signal is caused by the scene-type clusters, then shuffling cluster labels while keeping the same partition sizes should erase the IID-to-non-IID performance gap; this is a direct falsification test implied by the benchmark design.
- The paper's equalization of cluster sizes removes the natural long-tail imbalance; a natural variant would keep the imbalance and test whether FL aggregation methods that downweight dominant clusters can handle both problems at once.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a benchmark process for federated learning on multi-semantic vision datasets, with controllable semantic heterogeneity. The process has two steps: (i) discovering semantic clusters by K-means on a category tensor built from multi-label annotations, and (ii) partitioning data across clients via shard-based or Dirichlet-distribution-based strategies. As a proof of concept, the authors construct a federated Panoptic Scene Graph Generation (PSG) benchmark, evaluate four PSG models (IMP, MOTIFS, VCTree, GPS-Net) under different heterogeneity settings, apply FedAvgM and FedAdam to show robustness gains, and extend the clustering idea to CelebA for a binary classification task. The central claim is that this is the first benchmark framework enabling FL evaluation for multi-semantic vision tasks under controlled semantic heterogeneity.
Significance. If the central claim holds, the paper addresses a genuine gap: existing FL benchmarks rely on single-label classification, while real-world tasks such as scene graph generation carry multiple interdependent semantics. The proposed pipeline—cluster multi-label samples into semantic groups, then partition clients along those groups—is a sensible and potentially reusable recipe, and the paper includes code. The extension to CelebA and the comparison of robust FL algorithms are useful additions. However, the empirical validation is currently too weak to establish the central claim: the cluster semantics are not quantitatively validated, the reported IID-to-non-IID degradation is small and sometimes reversed, and all results are single-seed without variance estimates.
major comments (5)
- [Section 3.1, Eq. (1)] The semantic meaningfulness of the K-means clusters is load-bearing for the entire benchmark, yet it is validated only by qualitative PCA (Fig. 3), manual cluster descriptions, and a cluster-count ablation (Table 9). No cluster-purity, stability, or separation metrics are reported. Moreover, the category tensor is described only as allocating labels to orthogonal axes (ℝ^{13×13×7} for PSG); the construction details—how counts are computed, whether the tensor is normalized, and how multiple relations per image are aggregated—are omitted. Euclidean K-means on raw counts may be dominated by relation counts and frequent super-classes rather than scene semantics. Please specify the tensor construction and report quantitative cluster-validity measures (e.g., silhouette score, stability across K-means restarts, or agreement with human-assigned scene categories).
- [Table 1, Table 9, Section 4.2] The central control claim is that increasing semantic heterogeneity degrades performance monotonically, but the data contain multiple reversals. In Table 1, MOTIFS mR@20 is 4.06 for Dir(α=10) (IID) and 4.28 for Dir(α=0.2) (non-IID), the opposite of the claimed trend. In Table 9, VCTree at n=3 has mR@20 4.73 (IID) vs. 4.72 (non-IID), no gap, and IMP at n=10 has 1.93 (IID) vs. 2.38 (non-IID), reversed. The text acknowledges only the MOTIFS case as an 'outlier' and dismisses it as 'not a considerable amount' without statistical support. Since the benchmark's validity rests on the heterogeneity knob producing controlled effects, these exceptions must be addressed with multi-seed runs and significance tests, or the claim should be weakened to an average trend with documented exceptions.
- [Section 4.1, all tables] All experimental results are single-seed with no error bars or significance tests. The reported IID-to-non-IID gaps are small—averaged mR@20 drops of -0.77% (shard) and -0.64% (Dirichlet) in Section 4.2—so sampling noise from the data partition or model initialization could easily account for the observed differences. Please report means and standard deviations over at least 3–5 seeds for the main comparisons, and ideally over multiple realizations of the shard/Dirichlet partitions, so the reader can judge whether the heterogeneity effect is larger than the noise floor.
- [Section 3.2, Table 4, Table 6] The cluster-balancing step discards a large fraction of the data: each cluster is subsampled to the smallest cluster size, reducing the PSG training set from 46K to 11K images. This means the IID-vs-non-IID comparison potentially conflates semantic heterogeneity with data scarcity, since each client holds only about 114 images. The imbalanced ablation in Table 6 shows the issue is material: GPS-Net's non-IID mR@20 (8.10) exceeds its IID value (7.38), reversing the claimed trend, and the paper itself notes this is unexpected. Please analyze the IID-vs-non-IID gap as a function of the per-client dataset size, or otherwise justify that subsampling to a common size does not confound the heterogeneity control.
- [Appendix C.4, Section 6] The conclusion claims the benchmark shows 'consistent trends with prior FL studies' for participation rates, but Table 8 does not support this. Increasing the number of participating clients from 5 to 20 improves only MOTIFS; IMP, VCTree, and GPS-Net do not consistently improve, and the text in C.4 acknowledges this deviation. This undermines the generalizability assertion and needs either a revised claim or a PSG-specific explanation grounded in the task's long-tailed predicate distribution and model architectures.
minor comments (5)
- [Section 3.1, Eq. (1)] The notation in Eq. (1) is garbled; the clustering operator and the indexing of the dataset are not clearly defined. Please rewrite the equation with explicit notation for the set of category tensors over the dataset.
- [Section 4.2] There is a typo in the subsection heading: 'non-ID' should be 'non-IID'.
- [Figure 3 and Figure 7] The PCA visualizations are described only as 'a visualization via Principal Component Analysis'; please state explicitly what features are projected and how much variance the first two components capture, since the visual separation is used as evidence for semantic clustering.
- [Table 5] The communication-cost column entries such as '64(x 1)' are not self-explanatory; please specify the units (e.g., millions of parameters × rounds) and clarify what the parenthetical factors represent.
- [Conclusion] The limitations paragraph mentions the difficulty of choosing the number of clusters and the reliance on semantic annotations; this is helpful, but it should also mention the sensitivity of the IID-vs-non-IID gap to model and K shown in Table 9.
Circularity Check
No significant circularity: the benchmark's heterogeneity is induced from data-driven clusters and validated by external performance measurements, not by construction.
full rationale
The paper's claimed derivation chain is: (i) transform multi-semantic annotations into category tensors (Sec. 3.1), (ii) apply K-means clustering to obtain semantic clusters, (iii) partition these clusters across clients via shards or Dirichlet sampling (Sec. 3.2), and (iv) validate by measuring PSG and CelebA performance under IID vs non-IID splits (Secs. 4-5). None of these steps reduces to its own inputs. The clusters are discovered from the annotation tensor, not fitted to any target accuracy; the reported performance gaps are measured on the PSG test set, which is external to the clustering procedure, so the IID-to-non-IID degradation is an empirical outcome rather than a consequence of how the partitions were defined. The paper's acknowledged limitations in the Conclusion (choosing the number of clusters, requiring semantic annotations) are scope limitations, not circular reductions. The self-citations [14] (Lee & Yoon, FedGF) and [18] (Lim et al., Metaverse) appear only in the Related Work section as examples of FL methods and shard-based heterogeneity strategies; they are not invoked to justify the benchmark's central premise or to forbid alternatives. There is no uniqueness theorem imported from the authors' prior work and no ansatz smuggled in via citation. The strongest supporting concern is that some reported numbers do not monotonically degrade with heterogeneity (e.g., MOTIFS mR@20 is 4.06 at Dir(alpha=10) but 4.28 at Dir(alpha=0.2) in Table 1, and IMP at n=10 is 1.93 IID vs 2.38 non-IID in Table 9); this weakens the empirical support for 'controlled semantic heterogeneity,' but it is a correctness risk, not circularity. Under the stated circularity criteria, the correct finding is no significant circularity.
Assumptions & free parameters
free parameters (4)
- Number of semantic clusters K =
5 (3 and 10 in ablations)
- Super-class mapping size =
13 object categories, 7 predicate categories
- Cluster balancing sample size m =
2.2K per cluster, 11K total training images (24.5% of 46K)
- Dirichlet concentration alphas =
10, 1, 0.2
assumptions (4)
- domain assumption Euclidean K-means on category tensors yields semantically meaningful clusters.
- domain assumption The 13x13x7 super-class category tensor preserves the semantic distinctions relevant for scene graph heterogeneity.
- domain assumption Freezing the Faster R-CNN detector and training only the relation head is a valid proxy for PSG performance in federated settings.
- ad hoc to paper The number of clusters K=5 (or 3/10 in ablation) is appropriate for inducing semantic heterogeneity.
Cite this review
Pith. "Pith review of Benchmarking Federated Learning for Semantic Datasets: Federated Scene Graph Generation." pith.science (2026). https://pith.science/paper/ZISLSVO7
@misc{pith2026241210436,
author = {Pith},
title = {Pith review of: Benchmarking Federated Learning for Semantic Datasets: Federated Scene Graph Generation},
year = {2026},
howpublished = {\url{https://pith.science/paper/ZISLSVO7}},
note = {Machine review of arXiv:2412.10436}
}
read the original abstract
Federated learning (FL) enables decentralized training while preserving data privacy, yet existing FL benchmarks address relatively simple classification tasks, where each sample is annotated with a one-hot label. However, little attention has been paid to demonstrating an FL benchmark that handles complicated semantics, where each sample encompasses diverse semantic information, such as relations between objects. Because the existing benchmarks are designed to distribute data in a narrow view of a single semantic, managing the complicated semantic heterogeneity across clients when formalizing FL benchmarks is non-trivial. In this paper, we propose a benchmark process to establish an FL benchmark with controllable semantic heterogeneity across clients: two key steps are (i) data clustering with semantics and (ii) data distributing via controllable semantic heterogeneity across clients. As a proof of concept, we construct a federated PSG benchmark, demonstrating the efficacy of the existing PSG methods in an FL setting with controllable semantic heterogeneity of scene graphs. We also present the effectiveness of our benchmark by applying robust federated learning algorithms to data heterogeneity to show increased performance. To our knowledge, this is the first benchmark framework that enables federated learning and its evaluation for multi-semantic vision tasks under the controlled semantic heterogeneity. Our code is available at https://github.com/Seung-B/FL-PSG.
Figures
Figures from the paper (4 more)
Reference graph
Works this paper leans on
-
[1]
Federated learning based on dynamic regularization
Acar, D.A.E., Zhao, Y., Navarro, R.M., Mattina, M., Whatmough, P.N., Saligrama, V., 2021. Federated learning based on dynamic regularization. arXiv preprint arXiv:2111.04263
arXiv 2021
-
[2]
Personalized federated learning with gaussian processes
Achituve,I.,Shamsian,A.,Navon,A.,Chechik,G.,Fetaya,E.,2021. Personalized federated learning with gaussian processes. Advances in Neural Information Processing Systems 34, 8392–8406
work page 2021
-
[3]
arXiv preprint arXiv:1812.01097
Caldas, S., Duddu, S.M.K., Wu, P., Li, T., Konečn`y, J., McMahan, H.B.,Smith,V.,Talwalkar,A.,2018.Leaf:Abenchmarkforfederated settings. arXiv preprint arXiv:1812.01097
arXiv 2018
-
[4]
Scene graphs: A survey of generations and applications
Chang, X., Ren, P., Xu, P., Li, Z., Chen, X., Hauptmann, A., 2021. Scene graphs: A survey of generations and applications. arXiv preprint arXiv:2104.01111 2
arXiv 2021
-
[5]
Desai, A., Wu, T.Y., Tripathi, S., Vasconcelos, N., 2021. Learning of visual relations: The devil is in the tails, in: Proceedings of the IEEE/CVFInternationalConferenceonComputerVision,pp.15404– 15413
work page 2021
-
[6]
Sharpness- awareminimizationforefficientlyimprovinggeneralization
Foret, P., Kleiner, A., Mobahi, H., Neyshabur, B., 2020. Sharpness- awareminimizationforefficientlyimprovinggeneralization. Interna- tional Conference on Learning Representations (ICLR)
work page 2020
-
[7]
Fast r-cnn, in: Proceedings of the IEEE interna- tional conference on computer vision, pp
Girshick, R., 2015. Fast r-cnn, in: Proceedings of the IEEE interna- tional conference on computer vision, pp. 1440–1448
work page 2015
-
[8]
Measuringtheeffectsofnon- identical data distribution for federated visual classification
Hsu,T.M.H.,Qi,H.,Brown,M.,2019. Measuringtheeffectsofnon- identical data distribution for federated visual classification. arXiv preprint arXiv:1909.06335
arXiv 2019
Show all 33 references
-
[9]
Local pseudo-attributes for long-tailed recognition
Kim, D.J., Ke, T.W., Stella, X.Y., 2023. Local pseudo-attributes for long-tailed recognition. Pattern Recognition Letters 172, 51–57
2023
-
[10]
Adam: A method for stochastic optimization
Kingma, D.P., 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980
2014 arXiv
-
[11]
Panop- tic segmentation, in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp
Kirillov,A.,He,K.,Girshick,R.,Rother,C.,Dollár,P.,2019. Panop- tic segmentation, in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9404–9413
2019
-
[12]
Visual genome: Connecting language and vision using crowdsourced dense imageannotations
Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.J., Shamma, D.A., et al., 2017. Visual genome: Connecting language and vision using crowdsourced dense imageannotations. Internationaljournalofcomputervision123,32– 73
2017
-
[13]
Cifar-10 (canadian institute for advanced research)
Krizhevsky, A., Nair, V., Hinton, G., 2019. Cifar-10 (canadian institute for advanced research)
2019
-
[14]
Rethinking the flat minima searching in federated learning, in: Forty-first International Conference on Ma- chine Learning
Lee, T., Yoon, S.W., 2024. Rethinking the flat minima searching in federated learning, in: Forty-first International Conference on Ma- chine Learning
2024
-
[15]
Deeprelationalself-attentionnetworks forscenegraphgeneration.PatternRecognitionLetters153,200–206
Li,P.,Yu,Z.,Zhan,Y.,2022. Deeprelationalself-attentionnetworks forscenegraphgeneration.PatternRecognitionLetters153,200–206
2022
-
[16]
On the con- vergenceoffedavgonnon-iiddata
Li, X., Huang, K., Yang, W., Wang, S., Zhang, Z., 2019. On the con- vergenceoffedavgonnon-iiddata. arXivpreprintarXiv:1907.02189
2019 arXiv
-
[17]
Uncertainty-aware scene graph generation
Li, X., Wu, T., Zheng, G., Yu, Y., Li, X., 2023. Uncertainty-aware scene graph generation. Pattern Recognition Letters 167, 30–37
2023
-
[18]
Metavers:Meta-learnedversatile representations for personalized federated learning, in: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp
Lim,J.H.,Ha,S.,Yoon,S.W.,2024. Metavers:Meta-learnedversatile representations for personalized federated learning, in: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 2587–2596
2024
-
[19]
Microsoft coco: Common objects in context, in: Computer Vision–ECCV 2014: 13th European Confer- ence, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, Springer
Lin,T.Y.,Maire,M.,Belongie,S.,Hays,J.,Perona,P.,Ramanan,D., Dollár, P., Zitnick, C.L., 2014. Microsoft coco: Common objects in context, in: Computer Vision–ECCV 2014: 13th European Confer- ence, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, Springer. pp. 740–755
2014
-
[20]
Gps-net: Graph property sensing network for scene graph generation, in: Proceedings of the IEEE/CVFConferenceonComputerVisionandPatternRecognition, pp
Lin, X., Ding, C., Zeng, J., Tao, D., 2020. Gps-net: Graph property sensing network for scene graph generation, in: Proceedings of the IEEE/CVFConferenceonComputerVisionandPatternRecognition, pp. 3746–3753
2020
-
[21]
Pattern Recognition Letters 145, 187–193
Liu,A.A.,Wang,Y.,Xu,N.,Liu,S.,Li,X.,2021.Scene-graph-guided message passing network for dense captioning. Pattern Recognition Letters 145, 187–193
2021
-
[22]
Personalized federated learning on long-tailed data via knowledge distillation and generated features
Lv, F., Qian, P., Lu, Y., Wang, H., 2024. Personalized federated learning on long-tailed data via knowledge distillation and generated features. Pattern Recognition Letters 186, 178–183
2024
-
[23]
McMahan, B., Moore, E., Ramage, D., Hampson, S., y Arcas, B.A.,
-
[24]
Adaptive federated optimization
Reddi,S.,Charles,Z.,Zaheer,M.,Garrett,Z.,Rush,K.,Konečn `y,J., Kumar, S., McMahan, H.B., 2020. Adaptive federated optimization. arXiv preprint arXiv:2003.00295
2020 arXiv
-
[25]
Federated learning with l1 regularization
Shi, Y., Zhang, Y., Zhang, P., Xiao, Y., Niu, L., 2023. Federated learning with l1 regularization. Pattern Recognition Letters 172, 15– 21
2023
-
[26]
Learning to compose dynamic tree structures for visual contexts, in: Proceedings of the IEEE/CVF conference on computer vision and pattern recog- nition, pp
Tang, K., Zhang, H., Wu, B., Luo, W., Liu, W., 2019. Learning to compose dynamic tree structures for visual contexts, in: Proceedings of the IEEE/CVF conference on computer vision and pattern recog- nition, pp. 6619–6628
2019
-
[27]
5410–5419
Xu,D.,Zhu,Y.,Choy,C.B.,Fei-Fei,L.,2017.Scenegraphgeneration byiterativemessagepassing,in:ProceedingsoftheIEEEconference on computer vision and pattern recognition, pp. 5410–5419. S. Ha et al.:Accepted for publication in Pattern Recognition Letters. Page 8 of 15 Federated Scene...
2017
-
[28]
Panoptic scene graph generation, in: European Conference on Com- puter Vision, Springer
Yang, J., Ang, Y.Z., Guo, Z., Zhou, K., Zhang, W., Liu, Z., 2022. Panoptic scene graph generation, in: European Conference on Com- puter Vision, Springer. pp. 178–196
2022
-
[29]
Neural motifs: Scenegraphparsingwithglobalcontext,in:ProceedingsoftheIEEE conference on computer vision and pattern recognition, pp
Zellers, R.,Yatskar, M., Thomson, S.,Choi, Y., 2018. Neural motifs: Scenegraphparsingwithglobalcontext,in:ProceedingsoftheIEEE conference on computer vision and pattern recognition, pp. 5831– 5840
2018
-
[30]
Panoptic segmentation-based semantic embedding matching model for scene graph generation
Zhao, M., Zhang, J., 2025. Panoptic segmentation-based semantic embedding matching model for scene graph generation. Pattern Recognition Letters 193, 56–63
2025
-
[31]
sky" or “grass
Zhu, H., Xu, J., Liu, S., Jin, Y., 2021. Federated learning on non-iid data: A survey. Neurocomputing 465, 371–390. S. Ha et al.:Accepted for publication in Pattern Recognition Letters. Page 9 of 15 Federated Scene Graph Generation Table 4 Dataset Information Data Amount PSG d...
2021
-
[33]
Prior research mentioned that the key point of GPS-Net explicitly models the direction of predicates, which is why it does not perform well on PSG dataset
is lower than the performance on the balanced dataset. Prior research mentioned that the key point of GPS-Net explicitly models the direction of predicates, which is why it does not perform well on PSG dataset. However, when S. Ha et al.:Accepted for publication in Pattern Rec...
-
[2017]
1273–1282
Communication-efficient learning of deep networks from decentralizeddata,in:Artificialintelligenceandstatistics,PMLR.pp. 1273–1282
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.