REVIEW 5 major objections 5 minor 1 cited by
Federated In-Context LLM Agent Learning
T0 review · 5 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read The paper claims FICAL cuts communication from linear to constant in model size by exchanging text compendiums instead of weights, while matching federated fine-tuning baselines on tool-use accuracy.
desk verdict A clever communication-efficient idea for federated LLM agents, but the privacy claim is unverified and the cost comparison is inflated. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the knowledge compendium, a structured text artifact $\zeta_i = \{des_i, app_i, pre_i, coo_i\}$ generated by the LLM-enhanced KCG module, which replaces the role of model gradients in standard federated learning. Its four parts describe what a tool does, when to use it, what to be careful about, and how tools can be chained; the server simply concatenates local compendiums into a global one. The second mechanism is the RAG-based TLU module, which embeds the global compendium in a vector database and, for each user query, retrieves the relevant tool instructions to place into the agent's context. This retrieval step keeps the context length manageable as the number of clients grows, and the paper reports it raises accuracy by 7.6% compared with inserting the full compendium into the prompt.
What would settle it
Place a unique canary string (for example, a fake phone number or random token) into one client's local tool-use examples, run FICAL end to end, and check whether that string appears in the uploaded local compendium, the aggregated global compendium, or any agent response; if it does, the privacy-preservation claim fails. A separate check is to vary the base model size while holding everything else fixed and verify that total bytes transferred remain constant, which would test the O(1) communication claim.
Extended reading notes
Core claim
The paper's central claim is that federated training of LLM agents can be done without exchanging model weights or raw data samples: each client runs an LLM-enhanced Knowledge Compendium Generation (KCG) module over its local tool-use traces to produce a compact text compendium $\zeta_i = \{des_i, app_i, pre_i, coo_i\}$ describing tool functions, application scenarios, precautions, and cross-tool coordination; the server concatenates these into a global compendium $\zeta_g$ and distributes it; and a RAG-based Tool Learning and Utilizing (TLU) module retrieves the relevant entries for each query. The authors report that this reduces communication from $O(N)$ with respect to model size to $O(1)$, cutting bytes transferred by $3.33\times10^5$ times, and that FICAL matches or beats FedProx, FedAdam, FedYogi, FedLoRA, FedACG, and FedDecorr on tool-invocation accuracy across 4-bit and 8-bit quantization settings and 5 and 8 clients. The paper also claims the compendiums are privacy-preserving because the generator distills tool knowledge rather than copying user data, and that the TLU module improves accuracy by 7.6% over feeding the whole compendium into the prompt.
Load-bearing premise
The load-bearing premise is that the LLM-generated knowledge compendiums contain no client-specific private information, so sharing them is genuinely privacy-preserving; the paper asserts this but supplies no formal privacy guarantee and no empirical leakage test.
Editorial extensions
If this is right
- Federated LLM agent training can run in a single communication round, replacing dozens of rounds of weight synchronization.
- Communication cost stops growing with LLM parameter count, so larger future models do not make federated agent training more expensive to communicate.
- Clients holding different tool inventories can contribute to one global agent, because the server concatenates all local compendiums and the RAG module retrieves the relevant subset per query.
- Tool-invocation accuracy improves when compendium knowledge is retrieved rather than inserted wholesale into the prompt, by the reported 7.6%.
- Because only text compendiums leave clients, the approach offers a path toward privacy-aware federated agent learning, contingent on the compendium generator not memorizing raw examples.
Reading between the lines
- One consequence the authors leave implicit is that the same compendium-exchange protocol could generalize from tool-use traces to other kinds of task knowledge, effectively turning federated LLM training into a prompt-summarization problem.
- The O(1) communication claim is specifically with respect to model size; total traffic still grows with the number of clients because the server concatenates one compendium per client, so the constant is per-client rather than per-federation.
- A natural stress test is to compare FICAL against a non-private in-context baseline that sends raw examples; the accuracy gap would quantify how much information the compendium distillation discards.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes FICAL, a federated learning scheme for LLM agents in which clients share distilled natural-language "knowledge compendiums" instead of model weights or raw examples. Each client uses an LLM (DeepSeek-v2) to summarize its local tool-use traces into fields for tool description, application scenarios, precautions, and coordination; the server concatenates the local compendiums into a global compendium, and clients retrieve relevant entries through a RAG-based TLU module when answering queries. The paper claims O(1) communication cost with respect to model size, a 3.33e5 reduction in communication overhead versus federated fine-tuning baselines, competitive tool-use accuracy, and privacy preservation because the transmitted compendiums "do not contain private user information." The evaluation compares FICAL with six FL baselines on a ToolAlpaca-style simulated tool-use benchmark with LLaMA3-8B.
Significance. If substantiated, the idea of exchanging distilled text knowledge instead of model parameters would be an interesting step for federated LLM agents, and the RAG-based retrieval component addresses a real long-context problem. The paper also states a simple one-round protocol and reports large absolute gains on the presented benchmark. However, the two load-bearing claims---privacy preservation and a fair communication reduction---are not established: privacy is asserted without any formal or empirical support, and the headline communication ratio conflates protocol differences with algorithmic savings. The lack of code, seeds, and statistical reporting further prevents verification.
major comments (5)
- [Methodology, Part one; Abstract; Conclusion] The privacy guarantee is asserted, not established. The sentence "the information generated does not contain private user information" is the only support for the word "privacy-preserving" in the title and abstract, but no threat model, differential-privacy mechanism, output filter, or leakage experiment is provided. Since the KCG module prompts an LLM to summarize local tool-use examples that may contain user queries, the compendium could reproduce sensitive spans through memorization or paraphrase. The paper must either add a formal privacy analysis with an explicit mechanism, or an empirical membership-inference/leakage evaluation on realistic sensitive data. Without this, the central motivation for replacing raw example sharing is unsupported.
- [Experiments, Table 3 and 'Experiments Settings'] The 3.33e5 communication reduction is not a like-for-like comparison. The baselines are trained for 50 communication rounds of LoRA fine-tuning, while FICAL uses one round of in-context prompting with no weight updates; Table 3 reports a single total (26,000 MB) for all baselines, which mixes round count, quantization choices, and the distinction between training and inference-time prompting. The communication cost of generating compendiums (running DeepSeek-v2) and the growth of compendium size with the number of clients and toolsets (Eq. (3)) are excluded. Please report per-round and total communication under matched settings, include generation cost, and state the precise dependence on N and M.
- [Experiments and Analyses, Figure 4 and Table 2] The performance claims are not statistically supported. No error bars, number of runs, or significance tests are reported, despite the reproducibility checklist saying otherwise. The results in Figure 4 and Table 2 are single numbers; given the variance typical of LLM agent benchmarks, the reported differences may not be stable. Please release code and data, report seeds and variance, and use appropriate statistical tests.
- [Experiments Settings; Methodology, Part one] The evaluation is partially self-referential: DeepSeek-v2 is used both as the KCG generator and as the judge LLM that decides whether tool calls are correct. The same model family may have systematic preferences for the format produced by its own summaries, which could inflate the accuracy of FICAL. Use an independent judge or human evaluation, or at minimum report agreement with a different judge model.
- [Methodology, Eqs. (2)-(3); Abstract] The O(1) communication claim is only with respect to model size, but the global compendium in Eq. (3) is the concatenation of N local compendiums, so its size grows with the number of clients and with the number of toolsets. The paper should state the assumption that compendium size is bounded independently of N and M; otherwise the abstract's "O(1) communication" is misleading.
minor comments (5)
- [Abstract and Contributions] The word "incredible" should be replaced with "novel" or "effective", and "its’" should be "its".
- [Throughout] There are several typos: "heteogeneious" should be "heterogeneous", "privacy preservative" should be "privacy-preserving", and "FIACL" in the Experiments section should be "FICAL".
- [Table 2] The table title says "each client has multiple toolsets’ data" but the rows do not clearly indicate whether the percentages are per-client or aggregated; please clarify the experimental setup.
- [Reproducibility Checklist] The checklist answers "Yes" to including code and pre-processing, but no code appendix or data appendix appears in the manuscript; please align the checklist with the actual submission.
- [Experiments Settings] The simulation environment is described briefly; please specify how the external simulator is implemented, how the judge LLM prompt is constructed, and how the "irrelevant dataset" for format following is chosen.
Circularity Check
No significant circularity found: the communication-efficiency claim is an arithmetic design property, and the privacy claim is an unsupported assertion rather than a derived result.
full rationale
The paper's central communication claim is not a derived prediction but a direct arithmetic consequence of the method's design: FICAL transmits knowledge compendiums (measured as 0.078 MB) instead of model parameters (measured as 26,000 MB), and the reported 3.33e5 reduction is simply 26000/0.078. This is not a fitted parameter renamed as a prediction, nor does it import a conclusion from a self-citation; it is a measured property of the protocol itself. The privacy-preserving claim is asserted in the Methodology section ('the information generated does not contain private user information') without formal proof, differential privacy guarantees, or leakage tests, but an unsupported assertion is not circularity because it is not derived from, nor equivalent to, the inputs by construction. The use of DeepSeek-v2 as both the knowledge-compendium generator and the judge of tool-use correctness raises a legitimate self-evaluation validity concern, but it does not make any claimed result equivalent to its premises; the accuracy numbers could be biased by this design, but they are not logically forced by a definitional chain. There are no self-citations, no imported uniqueness theorems, and no equations whose outputs are identical to their inputs by construction. Accordingly, the circularity score is 0.
Assumptions & free parameters
assumptions (5)
- domain assumption In-context learning enables LLMs to acquire tool-usage knowledge from natural language compendiums.
- domain assumption RAG retrieval from the global compendium provides sufficient relevant knowledge for correct tool invocation.
- domain assumption Knowledge compendiums generated from private data do not leak private information.
- domain assumption The judge LLM (DeepSeek-v2) correctly evaluates tool-use correctness.
- ad hoc to paper The baseline comparison is fair: 50 rounds of LoRA training versus one round of FICAL under the same budget.
Cite this review
Pith. "Pith review of Federated In-Context LLM Agent Learning." pith.science (2026). https://pith.science/paper/MAGJTL4B
@misc{pith2026241208054,
author = {Pith},
title = {Pith review of: Federated In-Context LLM Agent Learning},
year = {2026},
howpublished = {\url{https://pith.science/paper/MAGJTL4B}},
note = {Machine review of arXiv:2412.08054}
}
abstract
Large Language Models (LLMs) have revolutionized intelligent services by enabling logical reasoning, tool use, and interaction with external systems as agents. The advancement of LLMs is frequently hindered by the scarcity of high-quality data, much of which is inherently sensitive. Federated learning (FL) offers a potential solution by facilitating the collaborative training of distributed LLMs while safeguarding private data. However, FL frameworks face significant bandwidth and computational demands, along with challenges from heterogeneous data distributions. The emerging in-context learning capability of LLMs offers a promising approach by aggregating natural language rather than bulky model parameters. Yet, this method risks privacy leakage, as it necessitates the collection and presentation of data samples from various clients during aggregation. In this paper, we propose a novel privacy-preserving Federated In-Context LLM Agent Learning (FICAL) algorithm, which to our best knowledge for the first work unleashes the power of in-context learning to train diverse LLM agents through FL. In our design, knowledge compendiums generated by a novel LLM-enhanced Knowledge Compendiums Generation (KCG) module are transmitted between clients and the server instead of model parameters in previous FL methods. Apart from that, an incredible Retrieval Augmented Generation (RAG) based Tool Learning and Utilizing (TLU) module is designed and we incorporate the aggregated global knowledge compendium as a teacher to teach LLM agents the usage of tools. We conducted extensive experiments and the results show that FICAL has competitive performance compared to other SOTA baselines with a significant communication cost decrease of $\mathbf{3.33\times10^5}$ times.
Figures
Figures from the paper (1 more)
Forward citations
Cited by 1 Pith paper
-
Federated In-Context Learning: Iterative Refinement for Improved Answer Quality
Fed-ICL iteratively refines QA answers via federated in-context learning with only label transmission, showing convergence on a linear attention model and gains on MMLU and TruthfulQA.
Reference graph
Works this paper leans on
-
[2]
DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model. arXiv:2405.04434. Dubey, A.; Jauhri, A.; Pandey, A.; Kadian, A.; Al-Dahle, A.; Letman, A.; Mathur, A.; Schelten, A.; Yang, A.; Fan, A.; et al
-
[3]
arXiv preprint arXiv:2407.21783
The llama 3 herd of models. arXiv preprint arXiv:2407.21783. Houlsby, N.; Giurgiu, A.; Jastrzebski, S.; Morrone, B.; De Laroussilhe, Q.; Gesmundo, A.; Attariyan, M.; and Gelly, S
-
[6]
arXiv preprint arXiv:2404.11536
FedPFT: Federated Proxy Fine-Tuning of Foundation Models. arXiv preprint arXiv:2404.11536. Reddi, S. J.; Charles, Z.; Zaheer, M.; Garrett, Z.; Rush, K.; Koneˇcn`y, J.; Kumar, S.; and McMahan, H. B
-
[7]
arXiv preprint arXiv:2403.11802
Counting-stars: A simple, efficient, and reasonable strategy for evaluat- ing long-context large language models. arXiv preprint arXiv:2403.11802. Sun, J.; Xu, Z.; Yin, H.; Yang, D.; Xu, D.; Liu, Y .; Du, Z.; Chen, Y .; and Roth, H. R. 2024a. FedBPT: Efficient Feder- ated Black-box Prompt Tuning for Large Language Models. In Forty-first International Conf...
-
[8]
arXiv preprint arXiv:2306.05301
Toolalpaca: Generalized tool learning for language models with 3000 simulated cases. arXiv preprint arXiv:2306.05301. Wei, J.; Hou, L.; Lampinen, A.; Chen, X.; Huang, D.; Tay, Y .; Chen, X.; Lu, Y .; Zhou, D.; Ma, T.; and Le, Q
-
[9]
Symbol tuning improves in-context learning in language models. In Bouamor, H.; Pino, J.; and Bali, K., eds., Pro- ceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , 968–979. Singapore: Asso- ciation for Computational Linguistics. Wei, J.; Tay, Y .; Bommasani, R.; Raffel, C.; Zoph, B.; Borgeaud, S.; Yogatama, D.; Bosma, M...
work page 2023
-
[10]
C-Pack: Packaged Resources To Advance General Chinese Embedding. arXiv:2309.07597. Zhang, Z.; Yang, Y .; Dai, Y .; Wang, Q.; Yu, Y .; Qu, L.; and Xu, Z
-
[11]
In Annual Meeting of the Association of Computational Linguistics 2023, 9963–9977
Fedpetuning: When federated learning meets the parameter-efficient tuning methods of pre-trained language models. In Annual Meeting of the Association of Computational Linguistics 2023, 9963–9977. Association for Computational Linguistics (ACL). Zhao, Y .; Li, M.; Lai, L.; Suda, N.; Civin, D.; and Chan- dra, V
work page 2023
Show all 12 references
-
[2018]
Reproducibility Check- list
Federated learning with non-iid data. arXiv preprint arXiv:1806.00582. AAAI Reproducibility Checklist Unless specified otherwise, please answer Yes to each ques- tion if the relevant information is described either in the pa- per itself or in a technical appendix with an expli...
-
[2022]
Dublin, Ireland and Online: Association for Computational Linguistics
What Makes Good In-Context Examples for GPT- 3? In Agirre, E.; Apidianaki, M.; and Vuli ´c, I., eds., Pro- ceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, 100–114. Dublin, Ireland and...
2022
-
[2023]
arXiv preprint arXiv:2310.14696
Tree of clarifications: Answering ambiguous questions with retrieval-augmented large language models. arXiv preprint arXiv:2310.14696. Lewis, P.; Perez, E.; Piktus, A.; Petroni, F.; Karpukhin, V .; Goyal, N.; K ¨uttler, H.; Lewis, M.; Yih, W.-t.; Rockt ¨aschel, T.; et al
-
[2024]
arXiv preprint arXiv:2406.11132
RePrompt: Planning by Automatic Prompt Engineering for Large Lan- guage Models Agents. arXiv preprint arXiv:2406.11132. DeepSeek-AI; Liu, A.; Feng, B.; Wang, B.; Wang, B.; Liu, B.; Zhao, C.; Dengr, C.; and et al, C. R
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.