{"id":"84a6dfbd-7f36-4d7c-8501-7724280f38a4","arxiv_id":"2412.08054","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"FICAL shares tool-usage text summaries instead of model weights in federated LLM agent learning, reporting similar accuracy with far less communication, but the evidence is incomplete.","lead":"FICAL replaces parameter exchange in federated LLM agent training with shared natural-language tool summaries, claiming a 3.33e5 times communication cost reduction. The idea is plausible, but the privacy and efficiency arguments are not backed by formal analysis or reproducible experiments.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The load-bearing 'privacy-preserving' claim is an unverified assertion: the KCG module distills local examples with an LLM, and nothing prevents the generated compendium from leaking memorized or paraphrased client data.","rationale":"The reader's REJECT verdict is appropriate. I agree that the weakest load-bearing assumption is privacy: the method transmits generated summaries instead of raw samples, but the summary generation itself is an LLM transformation with no formal privacy guarantee, so the central claim 'privacy-preserving' is not established. The paper even flags related synthetic-data leakage risks (Slokom et al. 2022) without showing that its own compendiums avoid them. I considered whether the communication comparison is the more central flaw; it is arithmetically consistent (26,000 MB / 0.078 MB ≈ 3.33e5) but methodologically unfair because the baselines do 50 rounds of LoRA updates while FICAL does one round of no-weight-change prompting. That unfairness affects the strength of the empirical claim but does not, by itself, invalidate the architectural idea. The privacy gap is more fundamental: if compendiums leak, the method's primary stated advantage over sharing examples disappears. The proposed canary test is a direct, low-cost check: insert unique secrets into local examples and ask whether the generated compendium reveals them. This would settle the concern without requiring a full DP proof. The paper has no code, data, or formal verification to independently support the claim, and the reproducibility checklist itself marks code inclusion as 'No,' so there is no countervailing evidence to offset the unverified privacy assertion. On this basis the manuscript should remain rejected (or at best conditionally revised with a privacy analysis and leakage experiments).","tokens_in":11808,"tokens_out":6469,"duration_ms":67025,"concrete_test":"Run a canary-based extraction test. Insert unique random canaries (e.g., 'CANYON-7719-QRST', 'SSN 123-45-6789', random hex strings) into user queries and tool arguments in one client's local dataset. Generate the local knowledge compendium with the KCG module using DeepSeek-v2 over multiple seeds and temperatures, then scan the four compendium fields for verbatim canary matches and for semantic recovery (embedding similarity or an LLM judge). Repeat with varying dataset sizes and prompt templates. If any canary is recovered above a pre-specified false-positive threshold, the asserted 'does not contain private user information' property is empirically false, and the privacy-preserving claim fails.","verdict_should_be":"REJECT","load_bearing_attack":"The central claim has two load-bearing conditions: (i) the communication reduction is real and fairly measured, and (ii) the method is genuinely privacy-preserving. Condition (ii) is the least secure. In Methodology, Part one, the paper asserts that the knowledge compendium 'does not contain private user information,' but this is not derived or tested. The KCG module is an LLM (DeepSeek-v2) prompted to summarize local tool-use instances into four free-text fields (des, app, pre, coo). LLMs are known to memorize training data and can reproduce sensitive spans verbatim or in paraphrase, especially when the same examples are repeated or when the prompt asks for detailed extraction. There is no differential-privacy mechanism, no output filtering, and no threat model (e.g., what an honest-but-curious server or a colluding set of clients can infer). Since the paper's opening motivation is that exchanging data samples 'risks privacy leakage,' the entire contribution depends on compendiums being leakage-free. A secondary concern is that the headline communication comparison is not apples-to-apples: FICAL performs one round of no-weight-change in-context prompting, while the baselines run 50 rounds of LoRA fine-tuning, so the 3.33e5 ratio bundles together algorithmic communication savings with a different training protocol. The privacy gap alone is sufficient to undermine the paper's central claim as stated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FICAL, a federated learning scheme for LLM agents in which clients share distilled natural-language \"knowledge compendiums\" instead of model weights or raw examples. Each client uses an LLM (DeepSeek-v2) to summarize its local tool-use traces into fields for tool description, application scenarios, precautions, and coordination; the server concatenates the local compendiums into a global compendium, and clients retrieve relevant entries through a RAG-based TLU module when answering queries. The paper claims O(1) communication cost with respect to model size, a 3.33e5 reduction in communication overhead versus federated fine-tuning baselines, competitive tool-use accuracy, and privacy preservation because the transmitted compendiums \"do not contain private user information.\" The evaluation compares FICAL with six FL baselines on a ToolAlpaca-style simulated tool-use benchmark with LLaMA3-8B.","tokens_in":12078,"tokens_out":6863,"duration_ms":68190,"significance":"If substantiated, the idea of exchanging distilled text knowledge instead of model parameters would be an interesting step for federated LLM agents, and the RAG-based retrieval component addresses a real long-context problem. The paper also states a simple one-round protocol and reports large absolute gains on the presented benchmark. However, the two load-bearing claims---privacy preservation and a fair communication reduction---are not established: privacy is asserted without any formal or empirical support, and the headline communication ratio conflates protocol differences with algorithmic savings. The lack of code, seeds, and statistical reporting further prevents verification.","major_comments":[{"comment":"The privacy guarantee is asserted, not established. The sentence \"the information generated does not contain private user information\" is the only support for the word \"privacy-preserving\" in the title and abstract, but no threat model, differential-privacy mechanism, output filter, or leakage experiment is provided. Since the KCG module prompts an LLM to summarize local tool-use examples that may contain user queries, the compendium could reproduce sensitive spans through memorization or paraphrase. The paper must either add a formal privacy analysis with an explicit mechanism, or an empirical membership-inference/leakage evaluation on realistic sensitive data. Without this, the central motivation for replacing raw example sharing is unsupported.","section":"Methodology, Part one; Abstract; Conclusion"},{"comment":"The 3.33e5 communication reduction is not a like-for-like comparison. The baselines are trained for 50 communication rounds of LoRA fine-tuning, while FICAL uses one round of in-context prompting with no weight updates; Table 3 reports a single total (26,000 MB) for all baselines, which mixes round count, quantization choices, and the distinction between training and inference-time prompting. The communication cost of generating compendiums (running DeepSeek-v2) and the growth of compendium size with the number of clients and toolsets (Eq. (3)) are excluded. Please report per-round and total communication under matched settings, include generation cost, and state the precise dependence on N and M.","section":"Experiments, Table 3 and 'Experiments Settings'"},{"comment":"The performance claims are not statistically supported. No error bars, number of runs, or significance tests are reported, despite the reproducibility checklist saying otherwise. The results in Figure 4 and Table 2 are single numbers; given the variance typical of LLM agent benchmarks, the reported differences may not be stable. Please release code and data, report seeds and variance, and use appropriate statistical tests.","section":"Experiments and Analyses, Figure 4 and Table 2"},{"comment":"The evaluation is partially self-referential: DeepSeek-v2 is used both as the KCG generator and as the judge LLM that decides whether tool calls are correct. The same model family may have systematic preferences for the format produced by its own summaries, which could inflate the accuracy of FICAL. Use an independent judge or human evaluation, or at minimum report agreement with a different judge model.","section":"Experiments Settings; Methodology, Part one"},{"comment":"The O(1) communication claim is only with respect to model size, but the global compendium in Eq. (3) is the concatenation of N local compendiums, so its size grows with the number of clients and with the number of toolsets. The paper should state the assumption that compendium size is bounded independently of N and M; otherwise the abstract's \"O(1) communication\" is misleading.","section":"Methodology, Eqs. (2)-(3); Abstract"}],"minor_comments":[{"comment":"The word \"incredible\" should be replaced with \"novel\" or \"effective\", and \"its’\" should be \"its\".","section":"Abstract and Contributions"},{"comment":"There are several typos: \"heteogeneious\" should be \"heterogeneous\", \"privacy preservative\" should be \"privacy-preserving\", and \"FIACL\" in the Experiments section should be \"FICAL\".","section":"Throughout"},{"comment":"The table title says \"each client has multiple toolsets’ data\" but the rows do not clearly indicate whether the percentages are per-client or aggregated; please clarify the experimental setup.","section":"Table 2"},{"comment":"The checklist answers \"Yes\" to including code and pre-processing, but no code appendix or data appendix appears in the manuscript; please align the checklist with the actual submission.","section":"Reproducibility Checklist"},{"comment":"The simulation environment is described briefly; please specify how the external simulator is implemented, how the judge LLM prompt is constructed, and how the \"irrelevant dataset\" for format following is chosen.","section":"Experiments Settings"}],"recommendation":"reject","confidential_remarks":"The manuscript's self-reported reproducibility checklist is inconsistent with the body (e.g., claiming code is included when no code appendix appears). The central privacy claim is the main selling point but is only an assertion, and the communication comparison bundles protocol differences. The core idea may be salvageable in a substantially expanded version, but as submitted the load-bearing claims are not established."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe core idea here is worth a look: instead of exchanging model weights in federated LLM agent training, each client sends a text 'knowledge compendium' summarizing what it learned about tool usage, and the server concatenates these into a global compendium that agents consult via RAG. That is a genuinely different way to think about communication-efficient FL for LLMs, and the KCG/TLU modules are a concrete instantiation. The paper also reports large communication savings (claimed 3.33e5x) and reasonable accuracy numbers on a simulated tool-use benchmark.\n\nThat said, the evidence does not support the central claims as stated.\n\nThe privacy claim is the biggest gap. The paper repeatedly says compendiums 'do not contain private user information,' but that is an assertion, not a result. The compendiums are generated by an LLM prompted with local examples, and LLMs can memorize or paraphrase sensitive spans. No threat model, no differential privacy, no leakage test. Since privacy is the paper's stated motivation, this is load-bearing.\n\nThe communication comparison is also not apples-to-apples. The '3.33e5x' figure compares one round of text transmission against 50 rounds of LoRA fine-tuning, and it ignores the cost of running the KCG module (a large LLM) on each client. Compendium size also scales with the number of tools and clients, so the O(1) characterization is only true per client, not as the system grows.\n\nThe experiments are thin: no error bars, no statistical tests, no code or data, and the reproducibility checklist is self-contradictory (claims code is included where none is). The ablation of RAG is helpful, but it is a single comparison.\n\nI don't think this should be rejected out of hand. The idea is salvageable: a proper privacy analysis (or a clear statement of threat model and leakage tests), a fairer communication cost model, and a reproducible benchmark would make it a reasonable paper. As is, it is not ready for publication, but it is worth a serious referee's time to see if the authors can close those gaps.\n\nRecommendation: send to peer review, but make clear that the privacy analysis and cost modeling need substantive revision.","headline":"A clever communication-efficient idea for federated LLM agents, but the privacy claim is unverified and the cost comparison is inflated.","tokens_in":12592,"tokens_out":2442,"would_cite":false,"duration_ms":23310,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims FICAL cuts communication from linear to constant in model size by exchanging text compendiums instead of weights, while matching federated fine-tuning baselines on tool-use accuracy.","keywords":["federated learning","LLM agents","in-context learning","knowledge compendium","tool learning and utilization","retrieval augmented generation","communication efficiency","privacy preservation"],"falsifier":"Place a unique canary string (for example, a fake phone number or random token) into one client's local tool-use examples, run FICAL end to end, and check whether that string appears in the uploaded local compendium, the aggregated global compendium, or any agent response; if it does, the privacy-preservation claim fails. A separate check is to vary the base model size while holding everything else fixed and verify that total bytes transferred remain constant, which would test the O(1) communication claim.","tokens_in":11602,"feed_emoji":"🤖","tokens_out":13752,"duration_ms":110526,"temperature":0.7,"pith_summary":"FICAL is a federated learning scheme for LLM agents in which clients never send model weights or raw data. Instead, an LLM-based Knowledge Compendium Generation module summarizes each client's local tool-use examples into a short text compendium, and the server aggregates these into a global compendium that is distributed back to clients. The paper's central claim is that this reduces communication from $O(N)$ with respect to model size in standard FL to $O(1)$, and that the accuracy of tool invocation remains competitive with, and in the reported settings better than, federated fine-tuning baselines such as FedProx, FedAdam, and FedYogi. Concretely, FICAL transmits 0.078 MB per round versus 26,000 MB for the baselines, a $3.33\\times10^5$ fold reduction, and improves tool-use accuracy in the tested scenarios. The paper also claims privacy preservation, since the shared compendiums describe tool usage rather than containing user data samples.","feed_headline":"Swapping weights for text cuts federated LLM traffic 333,000x","feed_subtitle":"FICAL exchanges knowledge compendiums rather than model updates, keeping tool-use accuracy while shrinking bandwidth.","key_machinery":"The load-bearing object is the knowledge compendium, a structured text artifact $\\zeta_i = \\{des_i, app_i, pre_i, coo_i\\}$ generated by the LLM-enhanced KCG module, which replaces the role of model gradients in standard federated learning. Its four parts describe what a tool does, when to use it, what to be careful about, and how tools can be chained; the server simply concatenates local compendiums into a global one. The second mechanism is the RAG-based TLU module, which embeds the global compendium in a vector database and, for each user query, retrieves the relevant tool instructions to place into the agent's context. This retrieval step keeps the context length manageable as the number of clients grows, and the paper reports it raises accuracy by 7.6% compared with inserting the full compendium into the prompt.","core_discovery":"The paper's central claim is that federated training of LLM agents can be done without exchanging model weights or raw data samples: each client runs an LLM-enhanced Knowledge Compendium Generation (KCG) module over its local tool-use traces to produce a compact text compendium $\\zeta_i = \\{des_i, app_i, pre_i, coo_i\\}$ describing tool functions, application scenarios, precautions, and cross-tool coordination; the server concatenates these into a global compendium $\\zeta_g$ and distributes it; and a RAG-based Tool Learning and Utilizing (TLU) module retrieves the relevant entries for each query. The authors report that this reduces communication from $O(N)$ with respect to model size to $O(1)$, cutting bytes transferred by $3.33\\times10^5$ times, and that FICAL matches or beats FedProx, FedAdam, FedYogi, FedLoRA, FedACG, and FedDecorr on tool-invocation accuracy across 4-bit and 8-bit quantization settings and 5 and 8 clients. The paper also claims the compendiums are privacy-preserving because the generator distills tool knowledge rather than copying user data, and that the TLU module improves accuracy by 7.6% over feeding the whole compendium into the prompt.","pith_inferences":["One consequence the authors leave implicit is that the same compendium-exchange protocol could generalize from tool-use traces to other kinds of task knowledge, effectively turning federated LLM training into a prompt-summarization problem.","The O(1) communication claim is specifically with respect to model size; total traffic still grows with the number of clients because the server concatenates one compendium per client, so the constant is per-client rather than per-federation.","A natural stress test is to compare FICAL against a non-private in-context baseline that sends raw examples; the accuracy gap would quantify how much information the compendium distillation discards."],"forward_implications":["Federated LLM agent training can run in a single communication round, replacing dozens of rounds of weight synchronization.","Communication cost stops growing with LLM parameter count, so larger future models do not make federated agent training more expensive to communicate.","Clients holding different tool inventories can contribute to one global agent, because the server concatenates all local compendiums and the RAG module retrieves the relevant subset per query.","Tool-invocation accuracy improves when compendium knowledge is retrieved rather than inserted wholesale into the prompt, by the reported 7.6%.","Because only text compendiums leave clients, the approach offers a path toward privacy-aware federated agent learning, contingent on the compendium generator not memorizing raw examples."],"supporting_citations":[{"why":"Supplies the in-context learning capability that FICAL exploits so agents learn from natural-language compendiums rather than weight updates.","marker":"Brown et al. 2020"},{"why":"Establishes the emergent abilities of large models that motivate using in-context learning as a training mechanism.","marker":"Wei et al. 2022"},{"why":"Provides the DeepSeek-v2 model used as the Lao-tzu compendium generator and as the judge and simulator in experiments.","marker":"DeepSeek-AI et al. 2024"},{"why":"Supplies the ToolAlpaca-style dataset generation method that creates the local tool-use instances used in evaluation.","marker":"Tang et al. 2023"},{"why":"Defines retrieval-augmented generation, the basis of the TLU module that retrieves relevant compendium entries per query.","marker":"Lewis et al. 2020"},{"why":"Provides the bge-large-en-v1.5 embedding model used to index the global knowledge compendium in the vector database.","marker":"Xiao et al. 2023"},{"why":"Underpins the paper's privacy motivation by showing that synthetic-data sharing can leak information about private data distributions.","marker":"Slokom, de Wolf, and Larson 2022"},{"why":"Supplies the FedAdam and FedYogi baselines whose communication overhead is compared against FICAL.","marker":"Reddi et al. 2021"},{"why":"Supplies the FedProx baseline used in the accuracy comparisons.","marker":"Li et al. 2020"}],"fun_headline_variants":["Federated LLM agents train via text, not weights","Swapping weights for knowledge texts trims federated traffic","LLM agents federate by exchanging knowledge compendiums","Communication slashed 333,000x: federated LLM agents share prose","Federated learning without weight sharing: LLM agents use text"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the LLM-generated knowledge compendiums contain no client-specific private information, so sharing them is genuinely privacy-preserving; the paper asserts this but supplies no formal privacy guarantee and no empirical leakage test.","fun_headline_variants_meta":{"raw":{"variants":["Federated LLM agents train via text, not weights","Swapping weights for knowledge texts trims federated traffic","LLM agents federate by exchanging knowledge compendiums","Communication slashed 333,000x: federated LLM agents share prose","Federated learning without weight sharing: LLM agents use text"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000766,"raw_usage":{"total_tokens":3480,"prompt_tokens":1110,"completion_tokens":2370,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":726,"completion_tokens_details":{"reasoning_tokens":2281}},"tokens_in":726,"tokens_out":2370,"duration_ms":19244,"temperature":1.0,"reasoning_tokens":2281,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T18:15:31.227394+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Place a unique canary string (for example, a fake phone number or random token) into one client's local tool-use examples, run FICAL end to end, and check whether that string appears in the uploaded local compendium, the aggregated global compendium, or any agent response; if it does, the privacy-preservation claim fails. A separate check is to vary the base model size while holding everything else fixed and verify that total bytes transferred remain constant, which would test the O(1) communication claim.","supporting_citations":[],"review_version":1}