{"id":"b64603d1-7d2c-4966-a310-6fea28113d8a","arxiv_id":"2501.04436","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A comparative analysis of three federated LLM fine-tuning frameworks shows FedLLMs achieve highest accuracy, KD-FedLLMs highest client computation, and Split-FedLLMs highest communication overhead in a GPT-2/Banking77 case study.","lead":"This paper compares three ways to fine-tune large language models under federated learning: uploading model updates, sharing logits via knowledge distillation, and split learning with activation exchange. It evaluates their accuracy, communication, and computation trade-offs, with a small GPT-2 case study.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"KD-FedLLMs' claimed communication advantage for classification is contradicted by the case study's own PD=5002 setting; the trade-off table needs a public-dataset-size threshold.","rationale":"The reader's weakest_assumption correctly identifies fairness and generalizability of the case study as a concern. My stress-test sharpens this into a specific, load-bearing inconsistency: the claim that KD-FedLLMs minimize communication for classification does not survive the case study's own reported public dataset size unless the LoRA rank is very large or the public dataset is much smaller than 5002 samples. This is not merely a matter of missing error bars or one dataset; it bears directly on the qualitative star ratings in Table I and on the central framing that each framework occupies a distinct trade-off point. The paper's analytical argument in Section III-B is plausible as a scaling law (logit size is small for classification), but it omits the multiplicative factor of public dataset size and the per-round repetition of logit transmission. The case study should have reported raw communication bytes or at least the PD threshold at which KD-FedLLMs cease to be communication-efficient. I do not think this invalidates the paper entirely: the qualitative taxonomy and research directions remain useful heuristics, and the accuracy claim for FedLLMs may still hold. However, the communication-efficiency ranking is conditional in a way the paper does not acknowledge, so the appropriate verdict remains CONDITIONAL, matching the reader's assessment. The proposed concrete test would settle whether the case-study configuration actually contradicts Table I or whether an unstated implementation detail (e.g., a small public dataset per client) saves the claim.","tokens_in":10378,"tokens_out":4897,"duration_ms":51944,"concrete_test":"Recompute per-client per-round communication for the Section V configuration from model and dataset sizes: FedLLMs = 4·r·(768+2304) bytes; KD-FedLLMs = 4·PD·77 bytes; Split-FedLLMs = 4·TS·80·768·2 bytes (activations plus gradients). Use r=8, PD=5002, TS=1667; if KD bytes exceed Fed bytes, the Table I classification-task claim fails for this configuration. Then find the PD threshold at which KD overhead exceeds FedLLMs, and re-plot Figure 4 with the log scale removed, reporting exact bytes. If rankings change across PD sizes, Table I must be annotated as conditional on public dataset size.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central trade-off claim—specifically Table I and Sections III-B/D, which present KD-FedLLMs as the most communication-efficient framework for classification—is not supported by the reported experiments and may conflict with the case study's own settings. In KD-FedLLMs, per-round uplink traffic is roughly PD_size × num_classes × bytes_per_logit per client, whereas FedLLMs sends only the LoRA parameter matrices once per round. With the Section V settings (PD size 5002, Banking77 with 77 classes), each client would upload about 5002 × 77 × 4 ≈ 1.54 MB of logits per round. For a typical LoRA rank r=8 applied to attn.c_attn in GPT-2, the FedLLMs uplink is about (768×8 + 8×2304) × 4 ≈ 98 KB per round—an order of magnitude smaller. The case study varies LoRA rank, PD size, and training samples separately and plots log-transformed overhead without reporting raw byte counts, so it cannot establish the claimed ranking. The 'communication-efficient KD' property is a headline differentiator of the taxonomy, yet it appears to hold only for sufficiently small public datasets, and the threshold is not identified. This is a load-bearing gap in the framework comparison.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a taxonomy of three federated fine-tuning frameworks for LLMs: parameter-update-based FedLLMs, knowledge-distillation-based KD-FedLLMs, and split-learning-based Split-FedLLMs. It compares them along three metrics—model accuracy, communication overhead, and client-side computation—first qualitatively in Section III and Table I, then through a case study in Section V using GPT-2 on Banking77 with three clients. The central claim is that the three frameworks occupy distinct trade-off points: FedLLMs give the best accuracy, KD-FedLLMs minimize communication for classification tasks but have the highest client-side computation, and Split-FedLLMs minimize client-side computation but incur the highest communication overhead. The paper also lists framework-specific optimization opportunities and broader research directions. The experimental evidence consists of accuracy and overhead curves for 100 rounds under varying LoRA rank, public-dataset size, and training-sample count, with results averaged over three seeds.","tokens_in":10555,"tokens_out":5824,"duration_ms":60847,"significance":"If the central trade-off claims hold, the taxonomy would be a useful organizing principle for practitioners choosing between federated fine-tuning frameworks, and the listed research directions (public-dataset alignment, dynamic splitting, activation compression, continual learning) are reasonable and timely. The paper is most valuable as a survey-level synthesis: it names the distinct knowledge-transfer mechanisms, identifies the key scalars that drive cost (tunable-parameter size, logit size and public-dataset size, activation size), and makes qualitative, falsifiable predictions about accuracy and overhead rankings. However, the empirical support is thin: one small model, one dataset, three clients, no error bars, no raw byte counts, and no common operating point across frameworks. As a result, the specific star ratings in Table I are stronger than the evidence presented. The paper would benefit from either repositioning Table I as purely qualitative or substantially strengthening the case study to support ordinal comparisons.","major_comments":[{"comment":"The claim that KD-FedLLMs are the most communication-efficient framework for classification tasks is not supported by the reported case-study settings and is parameter-dependent. In Section V, the public dataset size is PD=5002 and Banking77 has 77 classes, so each client must upload about 5002 × 77 × 4 ≈ 1.54 MB of logits per round under float32. For FedLLMs with LoRA applied to attn.c_attn in GPT-2 (12 layers, hidden size 768, output size 2304), a rank r=4 LoRA update costs about 12 × (768×4 + 4×2304) × 4 ≈ 0.59 MB and r=8 costs about 1.18 MB per round. Thus KD-FedLLMs are more expensive than FedLLMs at small ranks and only become cheaper at larger ranks; the crossover threshold depends on PD size, rank, and logit precision. The paper states the communication-efficiency property as a general feature of KD-FedLLMs, but the case study's own settings do not establish it, and no threshold is identified.","section":"III-B, Table I, and Section V"},{"comment":"The overhead comparison cannot support the ordinal rankings in Table I because there is no common operating point across frameworks. For FedLLMs the paper varies LoRA rank, for KD-FedLLMs it varies public-dataset size, and for Split-FedLLMs it varies training samples per round, with no matched baseline or shared hyperparameter budget. In addition, Fig. 4 plots a log-transformed communication axis and does not report raw byte counts, so the reader cannot verify the relative magnitudes. To substantiate the claimed rankings, the authors should report raw per-client bytes and FLOPs for a matched configuration, and should show how the ranking changes when each hyperparameter is varied for each framework.","section":"Section V and Fig. 4"},{"comment":"The accuracy conclusion is based on a single model (GPT-2), a single dataset (Banking77), three clients, and three seeds, with no error bars or per-configuration test-accuracy numbers in the text. The statement 'Among the three frameworks, FedLLMs achieve the highest model accuracy' is then used as support for Table I's accuracy stars. This is a generalization beyond what the experimental design can support. At minimum, the authors should report the actual accuracies for every configuration shown in Fig. 3 with variance across seeds, and ideally vary the model, dataset, and number of clients to test whether the ranking is stable.","section":"Section V and Fig. 3"},{"comment":"The five-star ratings in Table I are presented as a comparative summary but no scoring methodology is given, and they are not derived from the case-study measurements. The qualitative discussion in Section III explains the expected ordering, but the stars mix accuracy and overhead into a single ordinal scale whose construction is never defined. The authors should either derive the stars from a clearly stated quantitative rule or explicitly label the table as a qualitative summary rather than an experimental result.","section":"Table I and Section III-D"}],"minor_comments":[{"comment":"The abstract contains a grammatical error: 'comparative analysis three advanced federated LLM frameworks' should be 'comparative analysis of three advanced federated LLM frameworks'; Section I also has a duplicated 'to to the development'.","section":"Abstract and Section I"},{"comment":"The word 'quatitively' in the Section III heading and 'quantitively' in Section V should be 'quantitatively'.","section":"Sections III and V"},{"comment":"The framework name is written inconsistently as 'Split-FedLLMS' in the text describing Fig. 3; the capitalization should be 'Split-FedLLMs'.","section":"Section V"},{"comment":"The axis labels and legend text in the figures appear garbled in the PDF; the figures should be regenerated so that the reader can identify which curve corresponds to which hyperparameter value.","section":"Figures 3 and 4"},{"comment":"The public dataset is extracted from the same training set as the clients' private data, so the KD-FedLLMs accuracy result is obtained under favorable distribution alignment. The paper should acknowledge that the distribution-mismatch caveat discussed in Section III-A is not exercised in the case study.","section":"Section V"},{"comment":"The term 'FedLLMs' is used both as the umbrella term for all three frameworks and as the name of the first framework (e.g., 'three FedLLM frameworks' versus 'FedLLMs achieve the highest model accuracy'); this dual use is confusing and should be disambiguated, for example by calling the first framework 'parameter-update-based FedLLMs'.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The manuscript reads as a survey/tutorial paper with a small illustrative case study. The taxonomy and qualitative claims are plausible, but the Section V experiments are not yet strong enough to support the quantitative-looking Table I. If the target venue is a magazine-style outlet, repositioning Table I as qualitative may suffice; if the venue expects empirical validation, the case study needs a matched baseline, raw overhead numbers, and error bars. The novelty claim of 'the first comprehensive comparison' is also not verified in the reference list and should be softened or substantiated."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague—\n\nThis is a survey-style comparison of three federated fine-tuning frameworks: direct parameter sharing (FedLLMs), knowledge distillation via logits (KD-FedLLMs), and split learning (Split-FedLLMs). The value is real: the taxonomy is clean, the research directions in Section IV are sensible, and the authors try to validate the qualitative ranking with a GPT-2/Banking77 case study. For someone needing a map of this area, it is a decent place to start.\n\nThe problem is the headline communication claim. Table I and Section III-B say KD-FedLLMs is the most communication-efficient for classification tasks. But at the case study's own settings—public dataset of 5002 samples, 77 classes—each client would upload roughly 5002×77×4 ≈ 1.5 MB of logits per round, while FedLLMs with LoRA rank 8 on GPT-2's attention layer is under 100 KB. That makes KD-FedLLMs about an order of magnitude worse than FedLLMs, not better. The paper never reports raw byte counts, only log-transformed plots, and the trend description in Section V only says Split-FedLLMs has the highest overhead. So the star rating for KD-FedLLMs in classification is not supported by the evidence and is inverted at the case study's operating point. The table needs a threshold on public dataset size, or at least a conditional statement.\n\nOther soft spots are less severe. The case study is thin: one model, one dataset, three clients, no error bars beyond seeds, and each framework is varied along its own axis (LoRA rank, PD size, TS size) without a common baseline. That makes the star ratings heuristic. The 'first comprehensive comparison' claim is also an overreach; there are existing surveys of federated learning with LLMs, and the paper does not actually survey the literature, citing only a handful of works.\n\nOverall, the taxonomy and research directions are useful to practitioners, but the central trade-off table needs correction. I would not cite it in its current form, but I would send it to peer review—the comparison is useful and the flaws are fixable, and a referee should push the authors to rework Table I's communication ranking and add a proper baseline before publication.","headline":"Useful taxonomy of three federated fine-tuning frameworks, but the headline communication-efficiency claim for KD-FedLLMs is contradicted by the case study's own settings and needs rework.","tokens_in":11109,"tokens_out":5029,"would_cite":false,"duration_ms":46370,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Federated fine-tuning of LLMs: three frameworks, three trade-offs","keywords":["federated learning","large language models","fine-tuning","knowledge distillation","split learning","LoRA","communication overhead","model accuracy"],"falsifier":"Run the same three frameworks on a fixed model and dataset with identical LoRA rank, equal training budget, and matched public-data size, and check whether Split-FedLLMs still trails FedLLMs in accuracy by the same margin, and whether KD-FedLLMs remains the least communication-heavy on a generative task with a large vocabulary. A smaller gap or a reversed communication ranking would undercut the qualitative trade-off story.","tokens_in":10127,"feed_emoji":"⚖️","tokens_out":3286,"duration_ms":29686,"temperature":0.7,"pith_summary":"This paper attempts to establish that the three dominant ways to fine-tune large language models in federated settings occupy distinct, predictable trade-off points. In basic FedLLMs, clients upload model updates directly and get the highest accuracy but moderate communication and computation. In KD-FedLLMs, clients share logits instead of parameters, which can minimize communication on classification tasks but pushes the most computation onto clients and hurts accuracy. In Split-FedLLMs, the model is split so the server handles deep layers, minimizing client computation at the price of the highest communication overhead. The paper's case study with GPT-2 on Banking77 confirms the accuracy ranking and quantifies the overhead rankings, and the authors use the comparison to list optimization opportunities specific to each framework.","feed_headline":"FedLLMs win accuracy; split learning cuts client compute","feed_subtitle":"A head-to-head account of how parameter, logit, and activation sharing reshape cost and performance in federated fine-tuning.","key_machinery":"The object carrying the argument is the taxonomy of knowledge-transfer channels: parameters (FedLLMs), logits (KD-FedLLMs), and intermediate activations with gradients (Split-FedLLMs). Each framework's communication and computation profile follows directly from which object crosses the client–server boundary and at what granularity. All three frameworks assume parameter-efficient fine-tuning, so the tunable parameter set is small, and the paper maps each channel onto a qualitative star rating before demonstrating the mechanism in a case study by varying the one characteristic factor per framework.","core_discovery":"The central claim is a qualitative ranking, expressed as star ratings in Table I, that holds across federated fine-tuning frameworks: FedLLMs deliver the best model accuracy because clients directly update the full set of tunable parameters; KD-FedLLMs are the most communication-efficient for classification tasks, where logits are small, but become communication-heavy for generative tasks because logit vectors scale with vocabulary size; and Split-FedLLMs offload the deepest transformer blocks to the server, giving the lowest client-side computation but the largest communication volume, since activations and gradients are exchanged for every training sample. The case study with GPT-2 and Banking77, varying LoRA rank, public-dataset size, and training-sample count, reproduces the accuracy ranking, with FedLLMs outperforming Split-FedLLMs and KD-FedLLMs after 100 rounds.","pith_inferences":["The star ratings rest on a single small case study; a natural next test is whether the same rankings hold with larger models, more clients, and heterogeneous data distributions.","Since KD-FedLLMs depends on public-dataset alignment, the paper's suggestion of sharing label-distribution statistics could be extended into a personalized distillation scheme that adapts logits to each client's private distribution.","The taxonomy implies a possible hybrid: split learning to protect client compute plus distillation to compress communication, which could occupy the currently empty middle ground between the three frameworks.","If logits are compressed through top-k selection or low-rank approximation, KD-FedLLMs' generative-task communication disadvantage could shrink enough to change the star ratings."],"forward_implications":["If directly updating parameters is indeed the accuracy-optimal route, applications that prioritize model quality should choose FedLLMs despite their moderate resource costs.","KD-FedLLMs' communication advantage is task-dependent: it helps on classification with small output spaces but disappears for generative tasks with large vocabularies.","Split-FedLLMs is the natural fit for clients with weak computational capacity and ample bandwidth, since it shifts deep computation to the server at the expense of communication.","Tuning LoRA rank, public-dataset alignment, and split-point placement can shift each framework along its trade-off curve, which is the practical lever the authors identify in their research directions."],"supporting_citations":[{"why":"Supplies the foundational federated fine-tuning setup that the comparison builds on.","marker":"[2]"},{"why":"Justifies the parameter-efficient fine-tuning assumption used by all three frameworks.","marker":"[6]"},{"why":"Representative KD-FedLLM method that shares knowledge through logits.","marker":"[7]"},{"why":"Representative Split-FedLLM method that splits the model between client and server.","marker":"[8]"},{"why":"Provides the knowledge-distillation mechanism for communication-efficient logit sharing.","marker":"[9]"},{"why":"Provides the split-learning paradigm that Split-FedLLMs adapts.","marker":"[10]"},{"why":"GPT-2 is the model used in the case study.","marker":"[11]"},{"why":"Banking77 is the dataset used in the case study.","marker":"[12]"}],"fun_headline_variants":["FedLLM accuracy leads, but split learning eases client compute","KD, split, or full sharing? Federated LLM tuning trade-offs","Best federated LLM accuracy comes from full parameter sharing","Split learning lowers client load but boosts communication"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison assumes the case study is a fair and representative test in which each framework is evaluated varying only its own characteristic factor, without a common hyperparameter budget or matched baselines; if those rankings flip under equal budgets or on other tasks, the star ratings may not generalize beyond GPT-2 on Banking77 with three clients.","fun_headline_variants_meta":{"raw":{"variants":["FedLLM accuracy leads, but split learning eases client compute","KD, split, or full sharing? Federated LLM tuning trade-offs","Best federated LLM accuracy comes from full parameter sharing","Split learning lowers client load but boosts communication"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000262,"raw_usage":{"total_tokens":1615,"prompt_tokens":983,"completion_tokens":632,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":599,"completion_tokens_details":{"reasoning_tokens":562}},"tokens_in":599,"tokens_out":632,"duration_ms":6165,"temperature":1.0,"reasoning_tokens":562,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:32:30.327603+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same three frameworks on a fixed model and dataset with identical LoRA rank, equal training budget, and matched public-data size, and check whether Split-FedLLMs still trails FedLLMs in accuracy by the same margin, and whether KD-FedLLMs remains the least communication-heavy on a generative task with a large vocabulary. A smaller gap or a reversed communication ranking would undercut the qualitative trade-off story.","supporting_citations":[{"cited_title":"Scaling federated learning for fine-tuning of large language models,","cited_arxiv_id":null,"evidence_quote":"Supplies the foundational federated fine-tuning setup that the comparison builds on."},{"cited_title":"Parameter-efficient fine-tuning of large-scale pre-trained language models,","cited_arxiv_id":null,"evidence_quote":"Justifies the parameter-efficient fine-tuning assumption used by all three frameworks."},{"cited_title":"Communication-efficient federated learning via knowledge distillation,","cited_arxiv_id":null,"evidence_quote":"Provides the knowledge-distillation mechanism for communication-efficient logit sharing."},{"cited_title":"Splitfed: When federated learning meets split learning,","cited_arxiv_id":null,"evidence_quote":"Provides the split-learning paradigm that Split-FedLLMs adapts."},{"cited_title":"Language models are unsuper- vised multitask learners,","cited_arxiv_id":null,"evidence_quote":"GPT-2 is the model used in the case study."}],"review_version":1}