{"id":"7c355ebb-2a1e-4c5f-a3bb-92c4acdcbb8f","arxiv_id":"2412.15676","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Federated cumulative fine-tuning of LLaMA-3 with a separate classifier is the best multi-task setup for code review automation, but gains are small and sequential training causes catastrophic forgetting.","lead":"This paper tests five ways to train a federated large language model for three code review subtasks: deciding if review is needed, writing a review comment, and refining code. It finds sequential training makes the model forget earlier tasks, while cumulative fine-tuning gives small gains over separate per-task models on two of three tasks.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central claim overstates: the best cumulative method (FedCFT-reg) is two separate models, not a single multi-task FedLLM; single-model FedCFT fails on T1.","rationale":"The reader's weakest assumption is best-round selection from test metrics, which is a valid and serious concern. However, the single most load-bearing issue for the central claim is that the method identified as best (FedCFT-reg) is not a single multi-task model: it uses two models, one of which is the individual-task T1 model. Therefore the stated RQ2 claim that a single cumulative multi-task FedLLM outperforms individual-task models is not supported even if the round-selection bias were removed. The reader's rationale does note that 'the best multi-task method actually uses two separate models and does not improve the classification task,' but the reader does not make this the weakest assumption, so my agreement is partial. The empirical work is otherwise substantial: the catastrophic-forgetting result for sequential training is clear, the dataset and artifacts are public, and the threats-to-validity section is honest. The concern is about claim scoping, not about data fabrication or statistical fraud, so a major revision that restricts the conclusion to the two-model cumulative design (or demonstrates a true single-model variant) is appropriate. The reader's CONDITIONAL verdict already captures the need for such revision, so I keep the verdict unchanged.","tokens_in":22560,"tokens_out":7495,"duration_ms":57875,"concrete_test":"Analytical check using Table 12: compare the single-model cumulative method (row CFT) with FedBEST on T1. Since CFT's T1 F1 (11.503) is far below FedBEST (55.175), the claim that a single cumulative multi-task FedLLM outperforms individual-task models fails on T1. The evidence only supports a scoped claim: a two-model cumulative split (T1 separate from T2/T3) improves T2/T3 while matching T1. If the authors wish to claim a single-model advantage, they must re-run FedCFT with a shared adapter or task token over all three tasks and report T1 F1 under the same round-selection protocol.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The RQ2 Summary claims that 'the cumulative fine-tuning technique for multi-task FedLLM shows improved performance over individual-task models.' But the method that actually delivers this improvement, FedCFT-reg (Section 3.4.3), trains two separate models: one for T1 (explicitly reusing the individual-task model from RQ1) and one for T2+T3. It is not a single multi-task FedLLM, and it does not improve T1 over FedBEST (Table 12: T1 F1 is 55.175 for both). The only single-model cumulative method, FedCFT, performs far worse on T1 (F1 11.503 vs. 55.175), while its T2/T3 gains over FedBEST are small (C-BLEU 0.766 vs. 0.612; 75.716 vs. 74.583). Thus the paper's headline claim is supported only by a two-model design that sidesteps multi-task integration for the classification task; the single-model claim is contradicted by the paper's own Table 12. This is a construct-validity issue: the operationalization does not match the stated RQ2 objective of building 'a single multi-task model' that performs well across all three sub-tasks.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies whether federated fine-tuning of a large language model can automate three code-review sub-tasks (review necessity prediction, review comment generation, code refinement) and whether a multi-task federated model can match or beat separately trained per-task federated models. Using LLaMA-3 8B with LoRA, two clients, and the CodeReviewer dataset, the authors report that sequential multi-task training causes catastrophic forgetting, while a cumulative fine-tuning variant (FedCFT-reg) outperforms individual-task models. The empirical framework is the first application of multi-task FedLLM to code review, and the qualitative examples and the clear demonstration of catastrophic forgetting in sequential methods are useful. However, the paper's headline claim is undermined by how the best-performing 'multi-task' method is defined, by the use of test-set metrics to select the best federated round, and by statistical comparisons that do not support the stated conclusions.","tokens_in":22760,"tokens_out":4456,"duration_ms":38206,"significance":"If the results were robust, the paper would make a useful empirical contribution: it would show that privacy-preserving collaborative fine-tuning can improve code-review models for low-resource clients, and that sequential multi-task training of an 8B LLM is unstable. The paper also contributes reproducible artifacts, detailed dataset preprocessing, and a transparent round-by-round reporting of metrics, which are strengths. However, the load-bearing comparison that supports the central claim is compromised by three issues: the best multi-task method is not a single multi-task model, the best federated round is selected from test-set metrics without a validation holdout or repeated runs, and the reported FedCFT-reg numbers combine metrics from different rounds. These issues are fixable with re-analysis or reframing, but they currently prevent the paper from establishing its stated RQ2 conclusion.","major_comments":[{"comment":"The RQ2 claim that 'the cumulative fine-tuning technique for multi-task FedLLM shows improved performance over individual-task models' is not supported by the method that actually delivers the improvement. FedCFT-reg (Section 3.4.3) trains two separate models: one for T1, which is explicitly the same individual-task T1 model from RQ1, and another for T2+T3. It is therefore not a single multi-task FedLLM. The only single-model cumulative method, FedCFT, performs far worse on T1 (F1 11.503 vs. 55.175 for FedBEST in Table 12) and its T2/T3 gains over FedBEST are small. The statement in Section 9 that the cumulative technique 'enables the deployment of a single FedLLM' is contradicted by the paper's own architecture. The claim and the RQ2 answer need to be reframed, or a genuinely single multi-task model must be evaluated.","section":"3.4.3, RQ2 Summary, Table 12"},{"comment":"The selection of the best federated round from test-set metrics is a load-bearing methodological flaw. Section 4.2 states that the study runs for 1 to T rounds and selects the best model among all T models, and Section 5.1/5.2 then choose per-task BEST rounds (T1=1, T2=1, T3=8; FedCFT-reg=1/1/2) from the same test columns in Tables 3 and 7-11. Because Section 3.1 repurposed the original validation set as client-1 training data, there is no held-out validation split for round selection. Selecting the best of 20 test rounds overstates performance, especially for comparisons where the reported advantage is small, and with a single run there are no confidence intervals to gauge the selection effect. The comparison should be repeated with validation-based round selection, or the authors should report the full round trajectories and avoid claiming an advantage at a test-selected optimum.","section":"4.2, 5.1, 5.2"},{"comment":"The FedBEST row for FedCFT-reg in Table 11 mixes metrics from two different rounds. From the Fed@1 and Fed@2 rows of the same table, the T2 values (C-BLEU 0.667, METEOR 9.323, ROUGE-L 10.921) come from round 1, while the T3 values (C-BLEU 76.104, METEOR 85.560, ROUGE-L 88.142) come from round 2. No single model instance achieves this combination. The text acknowledges that for T2 round 1 is slightly better but then chooses round 2 'for practicality', yet the table still reports the round-1 T2 numbers. This composite row is then used in Table 12 to claim that CFT-reg outperforms FedBEST, so the central numerical comparison is not based on an achievable model. Please report all metrics for one chosen round, or clearly label per-task best rounds and refrain from treating them as a single model's performance.","section":"5.2, Table 11, Table 12"},{"comment":"The Wilcoxon signed-rank test as applied is not a valid significance test here. The test compares 9 metric values (three tasks by three metrics) between two conditions, but these cells are not independent observations, are on different scales (F1 vs. C-BLEU vs. ROUGE-L), and come from a single training run. A p-value of 0.0277 therefore does not establish that FedCFT-reg significantly outperforms FedBEST. A proper paired test requires multiple independent runs per condition or an appropriate per-task statistical design with correction for multiple comparisons; alternatively, the significance claim should be removed and the results reported as descriptive.","section":"5.2, paragraph following Table 12"},{"comment":"For FedCOT and FedCAT, the tables report different BEST rounds per task (e.g., Table 8: T1=18, T2=2, T3=1) in a single row labeled FedBEST. Since each row is supposed to correspond to one model instance, this obscures the fact that no single round of the training run achieves those three scores simultaneously. The text acknowledges that a single BEST round could not be determined for all tasks; the tables and the comparison in Table 12 should make this explicit rather than presenting a per-task oracle as if it were a single model.","section":"5.2, Tables 8 and 9"}],"minor_comments":[{"comment":"The sentence 'LoRA rank r search is performed for for powers of 2' contains a duplicated 'for'; please fix.","section":"4.1"},{"comment":"'reply-based' should be 'replay-based' in the continual learning discussion; the term appears twice.","section":"7 (and Section 9)"},{"comment":"The hyper-parameter section states 'Experiments for hyper-parameter search of were conducted' with a missing object; the sentence is incomplete.","section":"3.3"},{"comment":"The 'BEST' rows are ambiguous because they combine per-task best rounds; please add a note to each table clarifying whether the shown metrics come from one round or from different rounds.","section":"Tables 10 and 11"},{"comment":"When discussing the FedLLM improvement over vanilla for T2, the text says 'the federated model at round 1 performs the best' and reports C-BLEU 0.6%; the improvements are stated in percentage points but the phrasing could be clearer that C-BLEU remains very low in absolute terms.","section":"5.1"},{"comment":"The decision to repurpose the original validation set as client-1 training data is a significant design choice with direct consequences for model selection; it should be flagged in the Threats to Validity section as a limitation.","section":"3.1"},{"comment":"The table title 'Round-wise Metrics' does not indicate that these are test-set metrics; please label the table header to make clear that all reported rounds are evaluated on the test split.","section":"Table 3"}],"recommendation":"major_revision","confidential_remarks":"The paper is positioned as the first multi-task FedLLM study for code review automation, but the novelty relative to the authors' own prior FedLLM recipe [15] is incremental, and the central comparison relies on test-set model selection and a composite best-round row. I believe the empirical material is salvageable: the catastrophic-forgetting result is convincing, the round-by-round tables are useful, and the dataset preparation is carefully described. However, the RQ2 claim needs to be rewritten to match what FedCFT-reg actually is, and the round-selection and statistical issues require re-analysis. I would not recommend rejection because the flaws are local to the comparison methodology rather than to the underlying experiments, but the manuscript cannot be accepted in its current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. This is a legitimate first: an empirical study combining federated LLM fine-tuning with multi-task training for code review automation, with five concrete training schedules tested on the CodeReviewer tasks. It is also, in its current form, overclaimed. The headline that cumulative fine-tuning beats individual-task models is supported only by a two-model design, and the reported gains are small and selected from test metrics. The forgetting result is solid; the comparative claim is not.\n\nWhat it does well: the experimental setup is transparent and reproducible — public dataset, artifact link, detailed hyperparameter search, heterogeneous client split, and an honest threats section. The qualitative finding that sequential training collapses on earlier tasks (FedTOC loses T1 and T2 entirely) is well supported and correctly attributed to catastrophic forgetting. For that alone, the paper is a useful data point for anyone working on federated or multi-task LLMs for SE.\n\nThe soft spots. First, the RQ2 Summary says cumulative fine-tuning for a multi-task FedLLM improves over individual-task models, but the method responsible, FedCFT-reg, is not a multi-task model at all. It trains two separate models: one for T1, explicitly reusing the RQ1 individual model, and one for T2+T3. The actual single-model cumulative approach, FedCFT, gets an F1 of 11.5 on T1 versus 55.2 for the individual model — a catastrophic failure, not an improvement. So the operationalization does not match the stated objective of a single integrated model. Second, the best round per task is chosen from test metrics (Section 4.2), with no validation split, and the Wilcoxon test is applied to metric values rather than independent runs. No repeated runs, no confidence intervals. Given the small absolute gaps (T3 C-BLEU 76.1 vs 74.6), selection bias alone could account for the difference. These problems are fixable, but they undermine the central comparison as reported.\n\nCitation pattern is acceptable: reliance on the authors' prior FedLLM recipe [15] is reasonable since that recipe is published and this is an extension. Related work is adequate.\n\nBottom line: this is a paper for researchers working on federated LLMs or multi-task code review. The forgetting evidence and the methodology description merit serious referee attention, but the headline claim should be revised to match the two-model design, and the round-selection issue needs a held-out split and repeated seeds. I would not cite the comparative claim as established yet.\n\nMy recommendation: send to peer review, major revision.","headline":"Worth reading for the forgetting evidence, but the headline claim overreaches: the best method is two separate models and the gains are selected from test metrics.","tokens_in":23343,"tokens_out":3311,"would_cite":false,"duration_ms":26184,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a single federated model fine-tuned cumulatively on review-comment generation and code refinement, with review-necessity prediction kept separate, performs as well as or better than three separate per-task federated…","keywords":["code review automation","federated learning","large language models","multi-task learning","catastrophic forgetting","parameter-efficient fine-tuning","LoRA","LLaMA-3"],"falsifier":"Rerun the five schedules with a held-out validation split used only for round selection, or with a fixed round budget for every method, and compare FedCFT-reg against FedBEST on the same test set; if the cumulative model no longer beats or matches the best per-task model on T2 and T3, the central claim is refuted. The paper's own tables show that the best round differs by task, so the selection protocol is directly testable.","tokens_in":22282,"feed_emoji":"🤖","tokens_out":7682,"duration_ms":60258,"temperature":0.7,"pith_summary":"The paper tries to establish that the three code review sub-tasks—deciding whether a patch needs review, writing a review comment, and refining code from the comment—can be handled by one federated large language model without sharing private code. It reports that sequentially training the model on one task after another destroys earlier-task performance, while a cumulative schedule in which the two regression tasks are trained together, separated from the classification task, outperforms individual per-task federated models on comment generation and code refinement and matches them on review-necessity prediction. If the claim holds, organizations with siloed code can jointly build a single privacy-preserving review assistant instead of one model per task.","feed_headline":"Cumulative federated training beats per-task models for code review","feed_subtitle":"Sequential training forgets earlier tasks; cumulative mixing lifts comment and code-refinement quality.","key_machinery":"The carrying mechanism is a collection of five multi-task fine-tuning schedules applied to one 8B LLaMA-3 backbone, in which each client trains LoRA adapters locally, the server averages the adapters with FedAvg, and the merged adapter becomes the base for the next round. The decisive variant is FedCFT-reg, a two-model schedule: one federated model for the classification task T1 and one federated model trained on a randomly mixed corpus of the two generation tasks T2 and T3. The comparison baseline is FedBEST, the best federated round selected per task from twenty rounds, with best rounds of 1, 1, and 8 for T1, T2, and T3.","core_discovery":"The central claim, stated on the paper's own terms, is that the cumulative fine-tuning technique for multi-task federated large language models shows improved performance over individual-task models for code review automation. Concretely, the FedCFT-reg model—one federated model for the two generation tasks plus the individual federated model for the binary necessity task—beats or matches the best per-task federated models on all three tasks. Sequential methods fail because the model forgets the earlier tasks once trained on the next one, with review-necessity prediction metrics dropping to zero. The paper concludes that task relatedness is exploitable in a federated model only if the training schedule avoids catastrophic forgetting.","pith_inferences":["In the editor's reading, the round-selection protocol is the most fragile link: because the best round is picked from test-set metrics, the reported FedCFT-reg advantage over FedBEST could shrink under a held-out validation split or a fixed round budget; a re-run with proper round selection would settle this.","The same two-model cumulative recipe (classification separate from generation) is a natural candidate for other software engineering task trios that mix a binary decision with two generation tasks, such as issue triage, patch explanation, and fix generation, though the paper does not test these.","The FedTOC collapse to zero on T1 is so abrupt that it would be worth comparing against replay- or regularization-based continual learning to see whether the forgetting is intrinsic to federated sequential fine-tuning or to sequential fine-tuning in general.","Because only two simulated clients with a 3:1 data ratio were used, the performance ordering should be rechecked with more clients and with naturally unbalanced real-world participation before treating FedCFT-reg as the default multi-task schedule."],"forward_implications":["Organizations with siloed, proprietary code can jointly fine-tune one shared model for review-comment generation and code refinement by exchanging LoRA adapters instead of code.","A single multi-task model is feasible for code review automation only if the training schedule is cumulative; sequential schedules should be avoided because they forget the earlier tasks.","The reported Wilcoxon test gives a statistical basis for preferring FedCFT-reg over per-task FedBEST on the chosen metrics, not merely a numerical one.","The benefit of federated fine-tuning over the vanilla model is largest for code refinement, where the best federated round raises ROUGE-L from 34.451% to 87.472%."],"supporting_citations":[{"why":"Supplies the LLaMA-based code-review fine-tuning recipe, prompts, and task setup that the federated variants build on.","marker":"[19]"},{"why":"Introduces the three code-review sub-tasks and the multi-language dataset the study splits across clients.","marker":"[17]"},{"why":"Provides the federated architecture, 26k-sample training size, hyper-parameter search design, and 20-round setting reused here.","marker":"[15]"},{"why":"LoRA is the parameter-efficient method whose adapters, not model weights, are exchanged between clients and server.","marker":"[13]"},{"why":"Shows the three review tasks are interdependent via cross-task knowledge distillation, motivating the multi-task goal.","marker":"[5]"},{"why":"Basis for simulating non-open-source data with fixed clients and for the federated approach to private software engineering data.","marker":"[28]"},{"why":"Defines the micro-averaged corpus BLEU used to score comment generation and code refinement.","marker":"[24]"},{"why":"Wilcoxon signed-rank test used to claim that the central and multi-task models significantly beat the vanilla and per-task baselines.","marker":"[36]"}],"fun_headline_variants":["Cumulative fine-tuning lifts multi-task federated review","Sequential federated training forgets; cumulative wins","Multi-task fed LLM: cumulative beats sequential and per-task","Cumulative multitask FedLLM outperforms separate models"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison assumes that picking the best round by looking at the test scores, and then reporting that round's scores, does not inflate the claimed advantage of the multi-task model over the per-task baselines.","fun_headline_variants_meta":{"raw":{"variants":["Cumulative fine-tuning lifts multi-task federated review","Sequential federated training forgets; cumulative wins","Multi-task fed LLM: cumulative beats sequential and per-task","Cumulative multitask FedLLM outperforms separate models"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000196,"raw_usage":{"total_tokens":1358,"prompt_tokens":940,"completion_tokens":418,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":556,"completion_tokens_details":{"reasoning_tokens":350}},"tokens_in":556,"tokens_out":418,"duration_ms":4145,"temperature":1.0,"reasoning_tokens":350,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T11:11:36.207223+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun the five schedules with a held-out validation split used only for round selection, or with a fixed round budget for every method, and compare FedCFT-reg against FedBEST on the same test set; if the cumulative model no longer beats or matches the best per-task model on T2 and T3, the central claim is refuted. The paper's own tables show that the best round differs by task, so the selection protocol is directly testable.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the LLaMA-based code-review fine-tuning recipe, prompts, and task setup that the federated variants build on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the three code-review sub-tasks and the multi-language dataset the study splits across clients."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the federated architecture, 26k-sample training size, hyper-parameter search design, and 20-round setting reused here."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Shows the three review tasks are interdependent via cross-task knowledge distillation, motivating the multi-task goal."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Basis for simulating non-open-source data with fixed clients and for the federated approach to private software engineering data."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Wilcoxon signed-rank test used to claim that the central and multi-task models significantly beat the vanilla and per-task baselines."}],"review_version":1}