REVIEW 2 major objections 2 minor 49 references
Concordia: Self-Improving Synthetic Tables for Federated LLMs
T0 review · 2 major / 2 minor · reviewed 2026-05-20 · grok-4.3
Pith's one-line read Clients refine their own synthetic tables using shared utility scorers to raise federated LLM accuracy on private tabular data.
desk verdict Concordia ties local LoRA, private scorers, and cross-client GRPO into a tri-level loop for refining synthetic tables in federated tabular LLM work, but the abstract leaves the size and reliability of the gains unclear. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The ensemble of lightweight utility scorers that reweight synthetic samples locally and supply the reward signal for group-relative policy optimization of each client's synthetic table generator.
What would settle it
A side-by-side run of Concordia against a static synthetic-data baseline on a new non-IID healthcare tabular dataset that shows no gain in accuracy, stability, or shift robustness.
Extended reading notes
Core claim
The authors establish that synthetic table generators can be refined in a federated setting by a tri-level process: local LoRA adaptation on synthetic tables, client-side utility scorers learned from private validation feedback to reweight samples, and outer-level group-relative policy optimization on each generator guided by an ensemble of heterogeneous scorers shared across clients, all without parameter aggregation or data exposure.
Load-bearing premise
Lightweight utility scorers trained on private validation feedback can produce reliable reweighting signals and an effective ensemble to guide outer-level refinement of synthetic generators.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Concordia, a tri-level optimization framework for federated LLM adaptation on tabular tasks under strict privacy constraints. Clients adapt LLMs via LoRA on synthetic tables, train lightweight utility scorers from private validation feedback to reweight synthetic samples, and refine their own synthetic generators via GRPO guided by a shared ensemble of heterogeneous scorers without parameter aggregation or data exposure. Experiments on finance and healthcare tabular benchmarks are reported to show consistent gains in federated performance, cross-client stability, and robustness to distribution shift relative to static and decoupled synthetic-data baselines.
Significance. If the tri-level alignment mechanism reliably improves synthetic data utility without violating privacy, the approach could meaningfully advance federated LLM training on heterogeneous tabular data. The combination of local scorer-based reweighting and outer GRPO with cross-client ensembles is a distinctive technical contribution. The manuscript would benefit from stronger empirical grounding to establish the magnitude of gains and their dependence on the scorer ensemble.
major comments (2)
- [Framework description and §4 (Experiments)] The central performance claim rests on the reliability of lightweight utility scorers trained solely on limited private validation sets to produce accurate reweighting signals and a compatible ensemble for GRPO. Under the non-IID client distributions emphasized in the problem setting, this assumption is load-bearing; any overfitting or misalignment in the scorers would directly degrade generator updates and erase the reported advantages over static baselines. The manuscript should provide ablations isolating scorer generalization (e.g., performance when scorers are trained on smaller validation subsets or when client distributions diverge further).
- [Abstract and experimental claims] The abstract and experimental summary assert consistent improvements in federated performance, stability, and robustness, yet no quantitative metrics, error bars, number of clients, degree of heterogeneity, or ablation tables are supplied in the provided description. Without these, it is not possible to verify that the tri-level components, rather than other factors, drive the gains.
minor comments (2)
- [Method section] Notation for the GRPO objective and the ensemble aggregation rule should be defined more explicitly, including how relative rewards are computed across heterogeneous scorers.
- [Experimental setup] The manuscript would benefit from a clearer statement of the exact tabular benchmarks, feature dimensions, and privacy constraints used in the finance and healthcare experiments.
Simulated Author's Rebuttal
We thank the referee for the constructive and detailed feedback. We address each major comment below and have updated the manuscript to strengthen the presentation of our empirical results and ablations.
read point-by-point responses
-
Referee: [Framework description and §4 (Experiments)] The central performance claim rests on the reliability of lightweight utility scorers trained solely on limited private validation sets to produce accurate reweighting signals and a compatible ensemble for GRPO. Under the non-IID client distributions emphasized in the problem setting, this assumption is load-bearing; any overfitting or misalignment in the scorers would directly degrade generator updates and erase the reported advantages over static baselines. The manuscript should provide ablations isolating scorer generalization (e.g., performance when scorers are trained on smaller validation subsets or when client distributions diverge further).
Authors: We agree that validating the generalization of the utility scorers under non-IID conditions is essential for supporting the tri-level framework. In the revised manuscript we have added dedicated ablations in Section 4.3 that train scorers on progressively smaller private validation subsets (down to 20% of the original size) and under increased client distribution divergence. These experiments show that the reweighting signals remain effective with only modest performance degradation, confirming that the reported gains are not artifacts of scorer overfitting. revision: yes
-
Referee: [Abstract and experimental claims] The abstract and experimental summary assert consistent improvements in federated performance, stability, and robustness, yet no quantitative metrics, error bars, number of clients, degree of heterogeneity, or ablation tables are supplied in the provided description. Without these, it is not possible to verify that the tri-level components, rather than other factors, drive the gains.
Authors: The full manuscript already reports quantitative metrics, error bars from repeated runs, the number of clients, heterogeneity parameters, and ablation tables that isolate the contribution of each tri-level component. The high-level summary supplied during review did not include these details. To address the concern directly we have expanded the abstract with key quantitative deltas and inserted a concise experimental-setup table at the start of Section 4 that lists client count, heterogeneity measure, and main-result metrics with standard deviations. revision: partial
Circularity Check
No significant circularity in derivation chain
full rationale
The paper introduces a tri-level optimization framework (Concordia) that uses private validation feedback to train lightweight utility scorers for reweighting synthetic samples and guiding GRPO-based generator refinement across clients. This external signal from held-out private data supplies an independent anchor for the claimed improvements in federated performance and robustness, rather than reducing any prediction or result to a quantity defined solely by internal generator parameters or self-referential fits. No equations, self-citations, or uniqueness theorems are presented in the abstract or framework description that collapse the central claims by construction to the inputs. The experimental comparisons to static and decoupled baselines further indicate an externally falsifiable evaluation.
Assumptions & free parameters
assumptions (2)
- domain assumption Clients possess private validation data sufficient to train lightweight utility scorers that generalize to reweighting synthetic samples
- domain assumption An ensemble of heterogeneous scorers can be shared across clients to provide effective guidance without leaking validation data or generator parameters
Cite this review
Pith. "Pith review of Concordia: Self-Improving Synthetic Tables for Federated LLMs." pith.science (2026). https://pith.science/paper/G3Z2L34L
@misc{pith2026260509855,
author = {Pith},
title = {Pith review of: Concordia: Self-Improving Synthetic Tables for Federated LLMs},
year = {2026},
howpublished = {\url{https://pith.science/paper/G3Z2L34L}},
note = {Machine review of arXiv:2605.09855}
}
read the original abstract
Federated learning (FL) enables training large language models (LLMs) without sharing raw data, but adapting LLMs under strict data isolation and non-IID client distributions remains challenging in practice. Synthetic data offers a natural privacy-preserving surrogate for local training, yet existing federated pipelines typically treat synthetic generation as static or loosely coupled with downstream optimization, leading to rapidly diminishing utility under heterogeneous clients. We study federated adaptation of LLMs on tabular tasks where raw records and validation data cannot be shared, and local training must rely entirely on synthetic tables. We propose Concordia, a tri-level optimization framework that aligns synthetic data generation with federated validation utility despite these constraints. At the client level, models are adapted via parameter-efficient LoRA training on synthetic tables. Clients additionally learn lightweight utility scorers from private validation feedback to reweight synthetic samples during local training. At the outer level, each client refines its own synthetic table generator using group-relative policy optimization (GRPO), guided by an ensemble of heterogeneous scorers shared across clients, without aggregating generator parameters or exposing validation data. Experiments on privacy-sensitive tabular benchmarks from finance and healthcare demonstrate that Concordia consistently improves federated performance, cross-client stability, and robustness to distribution shift compared to static and decoupled synthetic-data baselines.
Figures
Lean theorems connected to this paper
-
IndisputableMonolith/Cost/FunctionalEquation.leanwashburn_uniqueness_aczel unclear?
unclearRelation between the paper passage and the cited Recognition theorem.
tri-level optimization framework that aligns synthetic data generation with federated validation utility... utility scorers... GRPO... ensemble of heterogeneous scorers
-
IndisputableMonolith/Foundation/RealityFromDistinction.leanreality_from_one_distinction unclear?
unclearRelation between the paper passage and the cited Recognition theorem.
Experiments on privacy-sensitive tabular benchmarks... MCC rises from 7.07 to 21.78
What do these tags mean?
- matches
- The paper's claim is directly supported by a theorem in the formal canon.
- supports
- The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
- extends
- The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
- uses
- The paper appears to rely on the theorem as machinery.
- contradicts
- The paper's claim conflicts with a theorem or certificate in the canon.
- unclear
- Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.
Reference graph
Works this paper leans on
- [1]
-
[2]
arXiv:2407.05174 [cs.LG] https://arxiv.org/abs/2407.05174
Synthetic Data Aided Federated Learning Using Foundation Models. arXiv:2407.05174 [cs.LG] https://arxiv.org/abs/2407.05174
-
[3]
Opeoluwa Akinseloyin, Xiaorui Jiang, and Vasile Paladel. 2025. Weakly Super- vised Active Learning for Abstract Screening Leveraging LLM-Based Pseudo- Labeling. InmedRxiv. https://api.semanticscholar.org/CorpusID:280868109
work page 2025
-
[4]
arXiv preprint arXiv:2308.06522 , year =
Sara Babakniya, Ahmed Roushdy Elkordy, Yahya H. Ezzeldin, Qingfeng Liu, Kee- Bong Song, Mostafa El-Khamy, and Salman Avestimehr. 2023. SLoRA: Federated Parameter Efficient Fine-Tuning of Language Models.ArXivabs/2308.06522 (2023). https://api.semanticscholar.org/CorpusID:260887495
- [5]
-
[6]
Yujun Cheng, Weiting Zhang, Zhewei Zhang, Chuan Zhang, Shengjin Wang, and Shiwen Mao. 2025. Toward Federated Large Language Models: Motivations, Methods, and Future Directions.IEEE Communications Surveys & Tutorials27 (2025), 2733–2764. https://api.semanticscholar.org/CorpusID:274196624
work page 2025
-
[7]
Lynn Chua, Badih Ghazi, Yangsibo Huang, Pritish Kamath, Daogao Liu, Pasin Manurangsi, Amer Sinha, and Chiyuan Zhang. 2024. Mind the Privacy Unit! User- Level Differential Privacy for Language Model Fine-Tuning.ArXivabs/2406.14322 (2024). https://api.semanticscholar.org/CorpusID:270620664
- [8]
Show all 49 references
-
[9]
Liam Collins, Hamed Hassani, Aryan Mokhtari, and Sanjay Shakkottai. 2021. Exploiting Shared Representations for Personalized Federated Learning.ArXiv abs/2102.07078 (2021). https://api.semanticscholar.org/CorpusID:231924497
2021
-
[10]
Detrano, András Jánosi, Walter Steinbrunn, Matthias Emil Pfisterer, Johann-Jakob Schmid, Sarbjit Sandhu, Kern H Guppy, Stella Lee, and Victor Froelicher
Robert C. Detrano, András Jánosi, Walter Steinbrunn, Matthias Emil Pfisterer, Johann-Jakob Schmid, Sarbjit Sandhu, Kern H Guppy, Stella Lee, and Victor Froelicher. 1989. International application of a new probability algorithm for the diagnosis of coronary artery disease.The A...
1989
-
[11]
Tao Fan, Yan Kang, Guoqiang Ma, Weijing Chen, Wenbin Wei, Lixin Fan, and Qiang Yang. 2023. FATE-LLM: A Industrial Grade Federated Learning Framework for Large Language Models.ArXivabs/2310.10049 (2023). https: //api.semanticscholar.org/CorpusID:264145987
2023
-
[12]
Jack Goetz and Ambuj Tewari. 2020. Federated Learning via Synthetic Data. arXiv:2008.04489 [cs.LG] https://arxiv.org/abs/2008.04489
2020
-
[13]
Wenkai Guo, Xuefeng Liu, Haolin Wang, Jianwei Niu, Shaojie Tang, and Jing Yuan. 2025. Can Federated Learning Safeguard Private Data in LLM Training? Vulnerabilities, Attacks, and Defense Evaluation.ArXivabs/2509.20680 (2025). https://api.semanticscholar.org/CorpusID:281526125
2025
-
[15]
Hans Hofmann. 1994. Statlog (German Credit Data). UCI Machine Learning Repository. DOI: https://doi.org/10.24432/C5NC77
1994 doi
-
[16]
Charlie Hou, Akshat Shrivastava, Hongyuan Zhan, Rylan Conway, Trang Le, Adithya Sagar, Giulia Fanti, and Daniel Lazar. 2024. PrE-Text: training language models on private federated data in the age of LLMs. InProceedings of the 41st International Conference on Machine Learning....
2024
-
[17]
Charlie Hou, Mei-Yu Wang, Yige Zhu, Daniel Lazar, and Giulia Fanti. 2025. Private Federated Learning using Preference-Optimized Synthetic Data.arXiv preprint arXiv:2504.16438(2025)
2025
-
[18]
Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. 2021. LoRA: Low-Rank Adaptation of Large Language Models.ArXivabs/2106.09685 (2021). https://api.semanticscholar.org/CorpusID: 235458009
2021 arXiv
-
[19]
Shengyuan Hu, Jack Goetz, Kshitiz Malik, Hongyuan Zhan, Zhe Liu, and Yue Liu
-
[20]
arXiv:2204.01273 [cs.LG] https://arxiv.org/abs/2204.01273
FedSynth: Gradient Compression via Synthetic Data in Federated Learning. arXiv:2204.01273 [cs.LG] https://arxiv.org/abs/2204.01273
-
[21]
Akim Kotelnikov, Dmitry Baranchuk, Ivan Rubachev, and Artem Babenko. 2023. Tabddpm: Modelling tabular data with diffusion models. InInternational Confer- ence on Machine Learning. PMLR, 17564–17579
2023
-
[22]
Eugenio Lomurno and Matteo Matteucci. 2024. Federated Knowledge Recycling: Privacy-Preserving Synthetic Data Sharing. arXiv:2407.20830 [cs.LG] https: //arxiv.org/abs/2407.20830
2024
-
[23]
Matthews
Brian W. Matthews. 1975. Comparison of the predicted and observed secondary structure of T4 phage lysozyme.Biochimica et biophysica acta405 2 (1975), 442–51. https://api.semanticscholar.org/CorpusID:44596673
1975
-
[24]
Jaehyun Nam, Kyuyoung Kim, Seunghyuk Oh, Jihoon Tack, Jaehyung Kim, and Jinwoo Shin. 2024. Optimized Feature Generation for Tabular Data via LLMs with Decision Tree Reasoning.ArXivabs/2406.08527 (2024). https: //api.semanticscholar.org/CorpusID:270440964
2024
-
[25]
Ravuri and Oriol Vinyals
Suman V. Ravuri and Oriol Vinyals. 2019. Classification Accuracy Score for Conditional Generative Models.ArXivabs/1905.10887 (2019). https://api. semanticscholar.org/CorpusID:166228599
2019
-
[26]
Mengye Ren, Wenyuan Zeng, Binh Yang, and Raquel Urtasun. 2018. Learning to Reweight Examples for Robust Deep Learning. InInternational Conference on Machine Learning. https://api.semanticscholar.org/CorpusID:4321928
2018
-
[27]
Roth, Shadi Albarqouni, Spyridon Bakas, Mathieu Galtier, Bennett A
Nicola Rieke, Jonny Hancox, Wenqi Li, Fausto Milletarì, Holger R. Roth, Shadi Albarqouni, Spyridon Bakas, Mathieu Galtier, Bennett A. Landman, Klaus H. Maier-Hein, Sébastien Ourselin, Micah J. Sheller, Ronald M. Summers, Andrew Trask, Daguang Xu, Maximilian Baust, and Manuel J...
2020
-
[28]
Maximilian Schmidhuber and Udo Kruschwitz. 2024. Llm-based synthetic datasets: Applications and limitations in toxicity detection.LREC-COLING 2024 (2024), 37
2024
-
[29]
Aivin V Solatorio and Olivier Dupriez. 2023. Realtabformer: Generating realistic relational and tabular data using transformers.arXiv preprint arXiv:2302.02041 (2023)
2023
-
[30]
Town, Rory M
Josefa Lia Stoisser, Marc Boubnovski Martell, Kaspar Märtens, Lawrence Phillips, Stephen M. Town, Rory M. Donovan-Maiye, and Julien Fauqueur. 2025. Query, Don’t Train: Privacy-Preserving Tabular Prediction from EHR Data via SQL Queries.ArXivabs/2505.21801 (2025). https://api.s...
2025
-
[31]
Marika Swanberg, Ryan McKenna, Edo Roth, Albert Cheu, and Peter Kairouz
-
[32]
Is API Access to LLMs Useful for Generating Private Synthetic Tabular Data?arXiv preprint arXiv:2502.06555(2025)
2025
-
[33]
Alysa Ziying Tan, Han Yu, Li zhen Cui, and Qiang Yang. 2021. Towards Personal- ized Federated Learning.IEEE Transactions on Neural Networks and Learning Sys- tems34 (2021), 9587–9603. https://api.semanticscholar.org/CorpusID:232076330
2021
-
[34]
Qwen Team. 2025. Qwen3 Technical Report. arXiv:2505.09388 [cs.CL] https: //arxiv.org/abs/2505.09388
2025 arXiv
-
[35]
Jianwei Wang, Junyao Yang, Haoran Li, Huiping Zhuang, Cen Chen, and Ziqian Zeng. 2025. RewardDS: Privacy-Preserving Fine-Tuning for Large Language Models via Reward Driven Data Synthesis.arXiv preprint arXiv:2502.18517(2025)
2025
-
[36]
Yuxin Wang, Duanyu Feng, Yongfu Dai, Zhengyu Chen, Jimin Huang, Sophia Ananiadou, Qianqian Xie, and Hao Wang. 2024. Harmonic: Harnessing llms for tabular data synthesis and privacy protection.arXiv preprint arXiv:2408.02927 (2024)
2024
-
[37]
Zezhou Wang, Yaxin Du, Xingjun Ma, Yu-Gang Jiang, Zhuzhong Qian, and Siheng Chen. 2024. Optimizing Cross-Client Domain Coverage for Federated Instruction Tuning of Large Language Models.Findings of the Association for Computational Linguistics: EMNLP 2025(2024). https://api.se...
2024
-
[38]
Futian Weng, Miao Zhu, Mike Buckle, peta hajek, and Mohammad Zoynul Abedin
-
[39]
https://api.semanticscholar.org/CorpusID:275048075
Class Imbalance Bayesian Model Averaging for Consumer Loan Default Prediction: The Role of Soft Credit Information.Research in International Business and Finance(2024). https://api.semanticscholar.org/CorpusID:275048075
2024
-
[40]
Huiyu Wu and Diego Klabjan. 2024. LanFL: Differentially Private Federated Learning with Large Language Models using Synthetic Samples.arXiv preprint arXiv:2410.19114(2024)
2024
-
[41]
Bangzhou Xin, Yangyang Geng, Teng Hu, Sheng Chen, Wei Yang, Shaowei Wang, and Liusheng Huang. 2022. Federated synthetic data generation with differential privacy.Neurocomputing468 (2022), 1–10. doi:10.1016/J.NEUCOM.2021.10.027 Conference’17, July 2017, Washington, DC, USA Huang et al
2022 doi
-
[42]
Lei Xu, Maria Skoularidou, Alfredo Cuesta-Infante, and Kalyan Veeramachaneni
-
[43]
Modeling tabular data using conditional gan.Advances in neural information processing systems32 (2019)
2019
-
[44]
Biwei Yan, Kun Li, Minghui Xu, Yueyan Dong, Yue Zhang, Zhaochun Ren, and Xiuzheng Cheng. 2024. On Protecting the Data Privacy of Large Language Models (LLMs): A Survey.arXiv preprint arXiv:2403.05156(2024)
2024
-
[45]
Jiahuan Yan, Jintai Chen, Chaowen Hu, Bo Zheng, Yaojun Hu, Jimeng Sun, and Jian Wu. 2024. Small Models are LLM Knowledge Triggers for Medical Tabular Prediction. InInternational Conference on Learning Representations. https: //api.semanticscholar.org/CorpusID:268248002
2024
-
[46]
Rui Ye, Rui Ge, Xinyu Zhu, Jingyi Chai, Yaxin Du, Yang Liu, Yanfeng Wang, and Siheng Chen. 2024. FedLLM-Bench: Realistic Benchmarks for Federated Learning of Large Language Models.ArXivabs/2406.04845 (2024). https://api. semanticscholar.org/CorpusID:270357469
2024
-
[47]
Rui Ye, Wenhao Wang, Jingyi Chai, Dihan Li, Zexi Li, Yinda Xu, Yaxin Du, Yanfeng Wang, and Siheng Chen. 2024. OpenFedLLM: Training Large Language Models on Decentralized Private Data via Federated Learning.Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery an...
2024
-
[48]
Liping Yi, Han Yu, Gang Wang, and Xiaoguang Liu. 2023. FedLoRA: Model- Heterogeneous Personalized Federated Learning with LoRA Tuning.ArXiv abs/2310.13283 (2023). https://api.semanticscholar.org/CorpusID:264405713
2023
-
[49]
Zilong Zhao, Aditya Kunar, Robert Birke, and Lydia Y Chen. 2021. Ctab-gan: Effective table data synthesizing. InAsian Conference on Machine Learning. PMLR, 97–112
2021
-
[50]
"feature name
Tianyuan Zou, Yang Liu, Peng Li, Yufei Xiong, Jianqing Zhang, Jingjing Liu, Xiaozhou Ye, Ye Ouyang, and Ya-Qin Zhang. 2025. Contrastive Private Data Synthesis via Weighted Multi-PLM Fusion.arXiv preprint arXiv:2502.00245(2025). A Implementation Details This section describes i...
2025
Reviewed May 20, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.