{"id":"f13ec2c0-8d3d-4fd8-9fa3-4018b107af8f","arxiv_id":"2411.16099","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"Federated learning improves vulnerability detection F1 for all studied CWE classes compared to isolated local training, with results sensitive to data heterogeneity and framework configuration.","lead":"The authors built an evaluation framework, VulFed, that combines federated learning with code vulnerability detection, and ran experiments on four datasets to compare FL against independent training. They report that FL improves detection F1 across all 28 investigated CWE types, while non-IID client data degrades performance only mildly.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline improvement may be a data-volume artifact: the isolated baseline trains on one IID shard while the FL model aggregates ten, and no mean-local baseline or variance is reported.","rationale":"The paper is a broad, plausibly useful benchmark, and the consistent tables plus the centralized-training comparison are real evidence that FL can recover most of the benefit of pooled data. I am not arguing for rejection. The most load-bearing fragility is the baseline definition in Section 5.1: the isolated client has one tenth of the data by construction, so the observed improvement is indistinguishable from 'more training data improves detection' unless a mean-local baseline is supplied. The reader's weakest-assumption identification of simulation realism and label correctness is related, but the more immediate and testable issue is the comparison protocol itself: correct it, and the headline claim either remains as an FL-specific effect or is revealed as a data-volume effect. This is addressable with modest additional computation and would materially strengthen or qualify the paper, so the existing CONDITIONAL verdict is appropriate.","tokens_in":25016,"tokens_out":6235,"duration_ms":64915,"concrete_test":"Rerun RQ1 under the same 10-client IID partition, training each client's model independently on its own shard, and report the mean and range of per-client F1 on the global test set alongside the FL global model. If the mean local F1 is close to the reported 18.33% and per-CWE improvements persist, the claim is robust. Then repeat this comparison under the Dirichlet non-IID partitions (alpha=0.5) from RQ2; if FL's advantage over the mean local F1 shrinks or reverses for several CWEs, the 'all CWEs under heterogeneity' conclusion must be restricted to IID settings.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in RQ1 is that FL 'significantly improves' per-CWE detection over independent training, with overall F1 rising from 18.33% to 38.82%. The comparison in Section 5.1 is FedAvg over 10 IID clients versus one isolated client holding one of those shards. Because the shards are IID, the isolated client is representative of a single local-only client, but it is not representative of the system-level independent-training baseline: the FL global model has seen all ten shards through aggregation, i.e., about 10x more training data. CWE-295 illustrates the issue: with 100 total samples, one isolated client has roughly 8 training positives, so its 6.67% F1 is near-degenerate, while the FL model has roughly 80 positives and reaches 70%. The gain is real but is largely a data-volume effect, not evidence that federated aggregation specifically helps beyond simple pooling. The paper does not compare FL against the mean of ten independently trained local models, does not report seeds or variance, and in the non-IID experiments of RQ2 reports no independent-training baseline at all. The centralized-training comparison (less than 5% loss) genuinely supports FL as an efficient way to pool data, but it does not validate the isolated-client baseline as the right comparator. Without a mean-local baseline and uncertainty estimates, the 'significant improvement on all CWEs' claim is not yet established as an FL-specific effect.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes VulFed/VulFL, a configurable evaluation framework for federated learning (FL) applied to vulnerability detection, and reports an extensive empirical study on the DiverseVul dataset. The study addresses three research questions: whether FL improves detection over independent training per CWE (RQ1), how FL behaves under non-IID client data (RQ2), and how to select models, parameter-efficient fine-tuning strategies, and FL algorithms (RQ3). The headline result is that FL raises the overall F1 score from 18.33% to 38.82% compared with a single isolated client, with less than 5% F1 loss relative to centralized training. The paper also evaluates six FL algorithms, six LLMs, five PEFT strategies, three GNN models, and several input representations.","tokens_in":25306,"tokens_out":4776,"duration_ms":44066,"significance":"If the central claim were established, the paper would make a useful contribution: it would show that federated aggregation of distributed vulnerability data can produce substantially better per-CWE classifiers than isolated local training, while remaining close to centralized training. The proposed framework is genuinely broad, covering LLMs, GNNs, PEFT schemes, and multiple FL algorithms, and the use of a public dataset (DiverseVul) is a strength. However, the main empirical comparison is currently confounded, and the paper does not provide the uncertainty estimates or alternative baselines needed to support the word 'significantly' in the RQ1 answer. The framework itself and the configuration study are valuable, but the headline finding needs re-analysis before the paper can be accepted.","major_comments":[{"comment":"The headline comparison (overall F1 from 18.33% to 38.82%) contrasts the FedAvg global model trained on 10 IID shards against one isolated client trained on a single shard. The FL model therefore sees roughly ten times more training data, so the gain is largely a data-volume effect rather than evidence that federated aggregation specifically helps. Please add a mean-of-ten-independent-clients baseline, a centralized-pooling baseline, per-seed variance, and a significance test over seeds. Also clarify how the 'overall F1' is computed: the per-CWE F1 values in Table 6 do not average to 18.33% or 38.82% when treated as a macro average.","section":"Section 5.1, Table 6, Figure 2"},{"comment":"In the non-IID experiments, the paper compares FL under IID versus non-IID partitions, but it does not report any independent-training baseline under the same non-IID partitions. Consequently, the conclusion that VulFed 'has strong resistance to data heterogeneity' does not support the broader claim that FL remains preferable to independent training in heterogeneous settings. Please report per-CWE F1 for isolated clients under the same Dirichlet partitions and include statistical comparisons.","section":"Section 5.2, Figure 5"},{"comment":"The RQ3 benchmark evaluates and ultimately recommends FedCross and FedMut, which are the authors' own methods from refs [27] and [33], while omitting several established FL algorithms such as SCAFFOLD, FedGEN, and FedNova. This makes the comparative recommendation not fully independent. Please add external baseline algorithms, or clearly separate self-developed methods from third-party methods, and disclose the relationship explicitly in the experimental design.","section":"Section 5.3, Tables 7-9"},{"comment":"The paper states that 'Section ?? provides channels for obtaining the data and source code', but this section is missing and no repository link or artifact DOI appears anywhere in the manuscript. Since all conclusions are empirical, the absence of a working availability statement prevents independent verification. Please add a complete data/artifact availability section with a functional link.","section":"Section 7"}],"minor_comments":[{"comment":"The framework is named VulFL in the abstract and VulFed in the introduction and Section 3; please unify the name throughout.","section":"Abstract vs. Section 3"},{"comment":"The claim that Dirichlet alpha values of 0.3 or 0.1 'do not significantly impact the experimental results based on our multiple verifications' is not supported by any reported results; please provide the corresponding data or temper the claim.","section":"Section 5.2"},{"comment":"The sentence beginning 'In decoder-only LLMs, CodeBERT' appears to be a category error, since BERT and CodeBERT are encoder-only models; please correct the taxonomy.","section":"Section 5.3"},{"comment":"Figure 1 lists FedGen among the FL algorithms, but Table 2 and the experiments use FedCross; please reconcile the figure with the actual algorithm set.","section":"Table 2 and Figure 1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript does not disclose until the references that FedCross and FedMut are the authors' own prior work, and these methods are then recommended in RQ3; this is worth flagging to the editor as a potential self-benchmarking concern. In addition, the missing artifact link in Section 7 should be resolved before publication if the journal requires reproducibility for empirical studies."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the paper is the broadest empirical treatment I've seen of FL for vulnerability detection: six LLMs, six FL algorithms, five preprocessing methods, PEFT strategies, and per-CWE results on DiverseVul. That per-CWE breakdown is new relative to VDBFL, which only did GNNs. Second, the headline improvement — F1 from 18.33% to 38.82% — is directionally plausible but the comparison is unfair as stated. The FL global model sees all ten shards through aggregation; the isolated client trains on one tenth of the data. The gain is partly a data-volume effect. The stress-test note is right, and the paper would be much stronger with a mean-of-ten-locally-trained-clients baseline and variance over seeds.\n\nWhat the paper does well: it ships a coherent framework, the tables consistently show FL beating the isolated client, and the <5% loss versus centralized training is a genuinely useful result — it says FL pools decentralized code data almost as well as pooling the data itself. The qualitative analysis of which CWE types benefit most is thoughtful, and the paper is honest about limitations.\n\nSoft spots, in proportion. The missing baseline and missing uncertainty are the load-bearing ones. Without variance or significance tests, \"significantly improve\" is not established, and the per-CWE improvement magnitudes in Table 6 are not trustworthy at face value. There are also consistency problems: the framework is called VulFed in most places but VulFL in the abstract; Figure 1 lists FedGen but Table 2 does not; the data/source-code availability link is unresolved. These are fixable but block replication. The self-citation concern is mild: FedCross and FedMut are the authors' own methods and are recommended in the conclusions, but they are also published peer-reviewed algorithms, and the comparison includes six methods, so it is not a fatal conflict.\n\nBottom line: this is a paper for software-engineering and security researchers who want a starting point for configuring FL-based vulnerability detection. It is not a theoretical contribution and the empirical claims need tightening. I would send it to peer review with a request for code/data artifacts, a mean-local baseline, and variance reporting. The core direction — FL helps across CWE types — probably survives those fixes.","headline":"A broad, useful FL-for-vulnerability-detection benchmark whose headline F1 gain needs a fairer baseline; worth refereeing with requests for code and variance.","tokens_in":25878,"tokens_out":1890,"would_cite":true,"duration_ms":18235,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Federated learning can more than double the F1 score of deep-learning vulnerability detectors compared with isolated training, according to this empirical study.","keywords":["Federated Learning","Vulnerability Detection","CWE classification","Empirical Study","Large Language Models","Parameter-Efficient Fine-Tuning","Data Heterogeneity","VulFed"],"falsifier":"Take the same DiverseVul data, deliberately corrupt a fraction of CWE labels in the client shards (e.g., 10% random mislabeling) or force extreme non-IID splits where each client holds mostly one CWE class, then rerun the RQ1 comparison; if any CWE type's FL F1 falls to or below the isolated-client baseline, the claim that FL significantly improves all CWEs is falsified.","tokens_in":24776,"feed_emoji":"🛡️","tokens_out":5837,"duration_ms":50458,"temperature":0.7,"pith_summary":"Software organizations often have too little labeled vulnerability data to train deep-learning detectors, and privacy concerns keep them from sharing code. This paper argues that federated learning (FL)—training a shared model by aggregating model updates from many clients without moving raw code—is an effective remedy. Using their own configurable evaluation framework, VulFed, on the DiverseVul dataset with ten simulated clients, they report that FL raises the average F1 score from 18.33% to 38.82% relative to a single client training independently, with improvements on every CWE category studied and under 5% F1 loss compared with training on all data centrally. They also find per-CWE performance degrades by at most about 15% under non-IID data and identify configuration choices (CodeBERT/OPT, P-Tuning V2/LoRA, FedCross/FedMut/CluSamp) that work best. The authors note the study covers C/C++ code with lightweight models and does not address fine-grained line- or variable-level localization.","feed_headline":"Federated learning doubles F1 score for vulnerability detection","feed_subtitle":"Ten-client study finds FL beats isolated training on every CWE type, losing under 5% F1 versus centralized training.","key_machinery":"The load-bearing object is VulFed, a modular evaluation framework that wraps FL around vulnerability detection through four configurable components: a pre-processor (raw source code, code gadgets, SeVCs, CFGs, ASTs), a trainer (LLMs with full-parameter or PEFT schemes, or GNNs), an aggregator (FL algorithms such as FedAvg, FedProx, CluSamp, FedCross, MOON, FedMut), and a client selector. The mechanism that produces the reported gains is the standard FL round: a server distributes the global model, clients fine-tune locally on private shards, and the server averages the updates; the framework exists to make that loop reproducible across model and data types so the authors can attribute performance differences to individual components.","core_discovery":"Within a simulated ten-client federated setting on the DiverseVul multi-class CWE dataset, the paper's central claim is that federated aggregation of locally trained models yields substantially better vulnerability detection than any client could achieve alone, while staying close to centralized training. Averaged over six lightweight LLMs, the global FL model's F1 score rose from 18.33% for an isolated client to 38.82%, and every investigated CWE type showed a gain; the largest gains (CWE-295 at +63.33 percentage points, CWE-120 at +39.77) occurred for vulnerabilities with simple, localized representations, while complex multi-form CWEs such as CWE-189 and CWE-772 improved least. Under Dirichlet-simulated non-IID data, per-CWE detection rates fell by less than 15% for all CWE types, with overall accuracy loss below 2% and F1 loss below 5%. A second set of experiments maps configuration choices, concluding that among deployable models CodeBERT and OPT lead, P-Tuning V2 and LoRA come closest to full fine-tuning, and FedCross, FedMut, and CluSamp give the best FL-algorithm performance overall.","pith_inferences":["Editorial inference: the paper establishes utility, not confidentiality, because no differential-privacy or gradient-leakage measurements are reported; the privacy framing rests on the standard FL assumption that sharing model updates does not reveal code.","Editorial inference: the same-distribution shards used in the main comparison are optimistic relative to real deployments; a harder test would seed clients with genuinely different codebases and CWE mixes and check whether the per-CWE gain ordering persists.","Editorial inference: the per-CWE improvement ranking suggests a testable hypothesis for future work—that syntactically 'shallow' vulnerability classes gain more from FL—which could be quantified with code-graph complexity measures and used to decide which CWE classes need data augmentation.","Editorial inference: extending VulFed to cross-language code and line-level localization, which the paper lists as future work, may not inherit the current gains because token or graph representations do not align across languages without additional transformation."],"forward_implications":["An organization that cannot access other companies' vulnerability data can still benefit from their models: with FL it can contribute private data and receive a global detector whose per-CWE F1 is roughly double what isolated training achieves.","Federated vulnerability detection remains usable when client data distributions diverge: the paper's non-IID results suggest most CWE types lose less than 15% detection rate, so heterogeneity is a cost, not a deal-breaker.","Configuring a FL vulnerability detector matters as much as choosing the detector: model, PEFT strategy, and aggregation algorithm each shift F1 by several points, and the best combinations are identifiable without trying every possibility.","GNN-based detectors (Devign, Reveal, VDBFL) also improve under FL, so the framework's conclusions are not limited to LLM-based text models.","Detection gains are uneven across CWE types; training pipelines should expect hard-to-detect classes to need extra preprocessing, not just more federated data."],"supporting_citations":[{"why":"Supplies the DiverseVul dataset, the multi-class CWE benchmark on which the central FL-versus-independent comparison is run.","marker":"[20]"},{"why":"Defines FedAvg, the vanilla aggregation algorithm whose two-fold F1 improvement is the paper's headline result.","marker":"[13]"},{"why":"Provides CodeBERT, the primary NLP model used for the per-CWE evaluations in RQ1 and RQ2.","marker":"[21]"},{"why":"Prior FL-based vulnerability detection work, limited to GNN-based inputs, that motivates the need for a common-CWE evaluation framework.","marker":"[18]"},{"why":"Gives the Dirichlet-distribution method used to construct non-IID client data in the heterogeneity experiments.","marker":"[50]"},{"why":"Defines the code-gadget preprocessing form used as one input structure in the framework's NLP-based detectors.","marker":"[5]"},{"why":"Defines the SeVC preprocessing form and the SySeVR-style evaluation data used in the framework.","marker":"[4]"}],"fun_headline_variants":["Federated learning doubles vulnerability detection F1","Federated learning matches centralized within 5% F1","FL beats isolated training on every CWE type","Federated learning boosts detection F1 from 18% to 39%","Federated learning improves detection on all vulnerability classes"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The results stand on the assumption that slicing DiverseVul into ten same-distribution shards mimics what real organizations hold: correct CWE labels, comparable data quality, and C/C++ code from public projects.","fun_headline_variants_meta":{"raw":{"variants":["Federated learning doubles vulnerability detection F1","Federated learning matches centralized within 5% F1","FL beats isolated training on every CWE type","Federated learning boosts detection F1 from 18% to 39%","Federated learning improves detection on all vulnerability classes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000735,"raw_usage":{"total_tokens":3353,"prompt_tokens":1079,"completion_tokens":2274,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":695,"completion_tokens_details":{"reasoning_tokens":2206}},"tokens_in":695,"tokens_out":2274,"duration_ms":16865,"temperature":1.0,"reasoning_tokens":2206,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T13:33:40.329198+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the same DiverseVul data, deliberately corrupt a fraction of CWE labels in the client shards (e.g., 10% random mislabeling) or force extreme non-IID splits where each client holds mostly one CWE class, then rerun the RQ1 comparison; if any CWE type's FL F1 falls to or below the isolated-client baseline, the claim that FL significantly improves all CWEs is falsified.","supporting_citations":[{"cited_title":"Diversevul: A new vulnerable source code dataset for deep learning based vulnerability detection,","cited_arxiv_id":null,"evidence_quote":"Supplies the DiverseVul dataset, the multi-class CWE benchmark on which the central FL-versus-independent comparison is run."},{"cited_title":"Communication-efficient learning of deep networks from decentralized data,","cited_arxiv_id":null,"evidence_quote":"Defines FedAvg, the vanilla aggregation algorithm whose two-fold F1 improvement is the paper's headline result."},{"cited_title":"Vulnerability detection based on federated learning,","cited_arxiv_id":null,"evidence_quote":"Prior FL-based vulnerability detection work, limited to GNN-based inputs, that motivates the need for a common-CWE evaluation framework."},{"cited_title":"Vuldeepecker: A deep learning-based system for vulnerability detection,","cited_arxiv_id":null,"evidence_quote":"Defines the code-gadget preprocessing form used as one input structure in the framework's NLP-based detectors."},{"cited_title":"Sysevr: A framework for using deep learning to detect software vulnerabilities,","cited_arxiv_id":null,"evidence_quote":"Defines the SeVC preprocessing form and the SySeVR-style evaluation data used in the framework."}],"review_version":1}