{"id":"f229b4f2-558d-41d2-9b62-154b4a4eb0ec","arxiv_id":"2412.17317","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"FedDP enhances federated cross-project defect prediction by distilling an aggregated global model from a weighted ensemble of local models, with weights derived from the similarity between each client's data and open-source distillation samples.","lead":"FedDP, a federated learning method for cross-project defect prediction, uses knowledge distillation on open-source data so companies can train a shared defect predictor without sharing proprietary data. It reports average F1 gains of about two to three points over standard federated baselines on 19 Java and C projects, making it relevant for privacy-conscious software teams.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (3)'s cosine-similarity correlation factor is asserted to measure how reliable a client model is on a distillation sample, but no evidence supports this proxy; if it fails, the weighted teacher in Eq. (5) collapses to an unweighted ensemble.","rationale":"The reader's weakest assumption is exactly the load-bearing concern I identify: Eq. (3)'s raw-feature cosine similarity is used as a proxy for local-model reliability on each distillation sample, and there is no evidence for that proxy. My independent reading of the paper confirms this is the least-secure link in the argument for the claimed heterogeneity-aware improvement, because the ablation study itself shows the correlation factor contributes only a small F1 increment, and because defect metrics are tabular, skewed, and scale-dominated, making raw cosine similarity a questionable measure of distributional relevance. A direct test using the server-available distillation labels can settle whether C_i^k tracks actual model correctness. The empirical claim of improved F1 over the listed baselines may survive even if the weighting is inert, since FedDF-style knowledge distillation alone could produce much of the gain, so the appropriate verdict remains conditional rather than a rejection. The paper does provide open code and detailed experimental tables, which supports reproducibility, but the central mechanism needs validation before the heterogeneity-aware claim is accepted.","tokens_in":30077,"tokens_out":7203,"duration_ms":75045,"concrete_test":"At the final communication round, use the server's available open-source labels to score, for every sampled distillation point x_i^d and every selected client k, whether client k's current model classifies x_i^d correctly (or its softmax confidence on the true class). Pool over all clients and distillation samples on both datasets and compute the rank correlation between this reliability measure and C_i^k from Eq. (3). If the pooled Spearman correlation is not significantly positive (e.g., at or below 0.1), the Eq. (3) proxy for local-model reliability is unsupported and the weighted teacher in Eq. (5) has no demonstrated advantage over an unweighted ensemble.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central novelty of FedDP over prior federated knowledge distillation is the local heterogeneity awareness: Eq. (3) defines C_i^k as the mean cosine similarity between each distillation sample x_i^d and all samples in client k's local data; Eq. (4) normalizes these into teacher weights; Eq. (5) uses them to weight the ensemble teacher. The entire justification is the intuitive claim that a model is more likely to be correct on samples similar to its training distribution. That requires cosine similarity in the raw feature space to be a valid proxy for model competence. Defect-prediction features are tabular software metrics (LOC, complexity counts, etc.) that are highly skewed, sparse, and dominated by scale; cosine similarity in that space mainly reflects magnitude and zero-pattern overlap, not label-relevant distributional closeness. The paper provides no correlation or calibration analysis linking C_i^k to local-model accuracy or confidence on the distillation set. The ablation in Table 9 is telling: removing the factor changes F1 by only about 0.3 points (48.93 vs 48.63 with FedAvg), while removing distillation entirely changes F1 by about 2.2 points. Thus the weighting is both the least-supported and conceptually load-bearing piece: if the proxy fails, the teacher is effectively FedDF's unweighted ensemble and the 'heterogeneity-aware' improvement over the closest existing method is not established. Section 8.1's admitted risk of imperfect baseline replication further weakens the comparison, but the Eq. (3) premise is the deeper issue.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces FedDP, a federated learning method for cross-project defect prediction (CPDP) under non-IID client data. The method augments standard FL aggregation with global knowledge distillation on open-source project data: each client computes a correlation factor between its local data and each distillation sample via average cosine similarity (Eq. 3), the server normalizes these into teacher weights (Eq. 4), and the aggregated global model is distilled from the weighted ensemble of local-model soft predictions (Eq. 5). Experiments on the Promise and Softlab datasets compare FedDP with FLR, Almity, OpenFLR, FedAvg, and FedProx, reporting F1/AUC improvements, communication-round reductions, sensitivity analyses, and an ablation.","tokens_in":30358,"tokens_out":10495,"duration_ms":98976,"significance":"If the empirical claims are supported, FedDP would be a practical and inexpensive recipe for privacy-preserving CPDP with heterogeneous clients, and the paper has several strengths: it evaluates on many projects with multiple metrics and statistical tests, includes an ablation, examines communication efficiency, and provides a data/code link. The central novelty, however, is the heterogeneity-aware weighting in Eqs. (4)-(5), and the paper does not currently establish that this weighting is either valid or essential: the cosine-similarity proxy is unvalidated, the ablation shows only about 0.3 F1 change when the factor is removed, and the closest federated distillation baselines (FedDF, DaFKD) are not evaluated. The privacy-preservation claim is also asserted without any leakage or attack analysis. These gaps make the current evidence insufficient for the stated central claim, but they are addressable with additional experiments and analysis.","major_comments":[{"comment":"The correlation factor C_i^k is the paper's main novelty, but it rests on an unvalidated proxy. The text justifies it by the intuition that a model is more likely to be correct on a sample similar to its training distribution, yet no evidence links raw-feature cosine similarity on skewed, count-based software metrics to local-model correctness or confidence. Table 9 shows that removing the factor changes F1 by only 0.30 (FedAvg: 48.93 vs 48.63; FedProx: 49.14 vs 48.60), whereas removing distillation entirely changes F1 by 2.19 and 2.31 respectively. Please report (a) a correlation or calibration analysis between C_i^k and per-sample local-model accuracy, (b) a FedDF-style unweighted-teacher baseline, and (c) control experiments with random or inverse weights, so the reader can see whether the weighting mechanism, rather than distillation alone, drives the reported gains.","section":"Section 4.2.1, Eqs. (3)-(5)"},{"comment":"The title, abstract, and problem formulation repeatedly call the setting privacy-preserving, but no privacy analysis appears in the paper. In Algorithm 1 clients upload both full local model parameters and the correlation vector C^k (lines 6 and 19), and neither the parameter-sharing nor the similarity-sharing step is examined for leakage (e.g., gradient or parameter inversion, membership inference, or reconstruction from C^k). Since the privacy guarantee is part of the paper's central claim, please add a concrete privacy analysis (DP bounds, an attack evaluation, or an explicit threat model with residual risks) or revise the wording to 'no raw data sharing' and discuss the limitations.","section":"Sections 3 and 4.2; Algorithm 1"},{"comment":"The closest existing methods to FedDP are FedDF and DaFKD, both cited in the related work, yet neither is included in the baselines of Section 5.3 or in Tables 3-6. FedDF with an open-source distillation set is exactly FedDP without the correlation-factor weighting, so it is the natural control for the paper's central contribution; DaFKD also performs domain-aware weighted distillation. Section 8.1 additionally admits that baseline implementations may be imperfect because not all baselines are open-sourced. Please add these baselines (or explain with experiments why they are inapplicable to tabular CPDP) and, given the admission in Section 8.1, release the exact evaluation pipeline so the reported FLR/Almity numbers can be checked.","section":"Sections 2.3 and 5.3"},{"comment":"Distillation steps N and sampling size p are tuned on the same test benchmarks used for the headline results. Section 5.4 sets N=10 and p=700, and Section 7.2 reports these as the best values on the test projects; the significance tests in Tables 3-6 then reuse the same test data. This creates optimistic bias in the reported p-values and average improvements. Use a held-out validation set or a nested procedure, or present the sensitivity results as exploratory rather than as support for the final configuration.","section":"Sections 5.4, 7.2, and Tables 3-6"},{"comment":"The W/T/L summaries and the text in Section 6.1 overstate the aggregate result. On Promise F1, FedDP (FedProx) has W/T/L = 8/0/5 against Almity, i.e., it loses on 5 of 13 projects, and the average F1 differences in Table 5 are roughly 2.2 points against FLR and 1.6-2.4 points against Almity, not the 3.10% stated in Section 6.1. The per-project Wilcoxon tests do not by themselves support the abstract's claim of significant improvement over all tested projects. Please report a paired test over all test projects (or an appropriate multiple-comparison correction) and characterize the per-project losses explicitly.","section":"Section 5.2.2 and Table 5"}],"minor_comments":[{"comment":"The project counts are inconsistent: the abstract says 14 Promise and 5 Softlab projects, but Tables 3-6 report 13 Promise test projects and 4 Softlab test projects (camel and ar1 are distillation only). Please clarify the counts.","section":"Abstract and Section 6.1"},{"comment":"The p-value columns do not state whether they compare FedDP-FedAvg or FedDP-FedProx; the header 'FedDP vs. FLR vs. Almity' is ambiguous.","section":"Tables 3-6"},{"comment":"There is a typo in 'Pomise', and the category entries for camel are left blank without explanation.","section":"Table 1"},{"comment":"The submitted text contains placeholder glyphs in the figure content; the final figures need readable axis labels, legends, and captions.","section":"Figures 2-5"},{"comment":"The soft-prediction notation s(w; x) is used without a formal definition; please define it explicitly.","section":"Eq. (5)"},{"comment":"The numbers in the text (0.05%, 5.11%, 2.25%, 1.30% for FedDP vs FLR on 'camel') do not match Table 8 (0.04%, 5.27%, 2.19%, 1.36% for the FedAvg row); the table and text need to be reconciled.","section":"Section 6.3 and Table 8"},{"comment":"The generalization discussion for code clone detection and bug-fix commits is presented without supporting experiments; it should be framed as future work.","section":"Section 7.4"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the journal's scope and the core idea is reasonable, but the distinguishing component contributes little in the ablation and its mechanism is unsupported. I would encourage the editor to require a FedDF comparison and a direct validation of Eq. (3) before considering publication; without those, the paper's novelty over existing federated distillation is not demonstrated."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: this paper is a straightforward empirical study of a federated knowledge-distillation (KD) approach for cross-project defect prediction (CPDP). The recipe: use open-source project data as a distillation set, treat the ensemble of local models as the teacher, and distill into the global model. The added wrinkle is a sample-wise weight per client based on the cosine similarity between the distillation sample and the client's local data, meant to capture how much to trust each local model on that sample.\n\nThe paper does several things well. The evaluation is reasonably thorough: two public datasets, five baselines including a centralized upper bound, significance tests, ablation, communication-efficiency comparison, and sensitivity analyses for the distillation dataset and hyperparameters. They also offer code. For a niche area with mostly small-scale studies, that is a genuine contribution.\n\nThe soft spots are in proportion to how much the claimed novelty matters. The correlation factor in Eq. (3) is the main conceptual addition over FedDF-style ensemble distillation, but the paper gives no evidence that raw-feature cosine similarity on tabular software metrics tracks local-model competence. The intuition might hold for high-dimensional dense data, but defect metrics are skewed and sparse; cosine similarity there mostly measures magnitude and zero-pattern overlap. More tellingly, the ablation in Table 9 shows that removing the factor changes F1 by only about 0.3 points, while removing the entire distillation step changes it by more than 2 points. So the factor is both weakly justified and empirically minor; the gains over the non-KD baselines (FLR, Almity) largely come from the plain KD, which is not new. The paper also does not compare against the closest KD baselines, FedDF and DaFKD, so the advancement over existing federated KD methods is not established.\n\nThe privacy claim is asserted rather than analyzed. There is no attack or leakage study, and the correlation factors are uploaded to the server, which could in principle leak information about local distributions. The authors admit in Section 8.1 that baseline replication may be imperfect, and the distillation hyperparameters were tuned on the same test benchmarks used for the headline results. These issues are minor to moderate, and they do not invalidate the empirical findings.\n\nWho is this for? Researchers in software engineering or applied federated learning who want a concrete, reproducible recipe for using public data to improve federated defect prediction. The paper deserves a serious referee and is publishable after revision. I would not cite it for the correlation-factor insight, but I might cite it as an empirical data point that ensemble distillation helps in this setting.","headline":"A solid empirical recipe for federated knowledge distillation in defect prediction, but the claimed novelty of the correlation factor is weakly supported by both theory and ablation.","tokens_in":30893,"tokens_out":3706,"would_cite":true,"duration_ms":34278,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"FedDP claims that privacy-preserving cross-project defect prediction under non-IID data can be substantially improved by distilling a heterogeneity-aware ensemble of local models into the global model using open-source data.","keywords":["cross-project defect prediction","federated learning","knowledge distillation","privacy preservation","non-IID data","software defect prediction","heterogeneity-aware ensemble"],"falsifier":"On the same datasets and splits, compute for each distillation sample the correlation factor from Eq. (3) and each client model's error on that sample, then check whether higher similarity predicts lower error, for example with a rank correlation or by comparing FedDP against a version using random weights in place of $\\alpha^k_i$. If the correlation is absent, or random-weight FedDP matches the reported F1, the heterogeneity-aware teacher is not the source of the gains.","tokens_in":29854,"feed_emoji":"🔒","tokens_out":8769,"duration_ms":72530,"temperature":0.7,"pith_summary":"The paper proposes FedDP, a federated-learning method for cross-project defect prediction in which companies' project data never leaves their own clients. FedDP's central claim is that data heterogeneity among clients can be overcome by using open-source project data as a distillation set: after local training, the server builds a teacher from a weighted ensemble of local models, with each client's weight set by how similar its local data is to each distillation sample, and distills that ensemble into the aggregated global model. Experiments on 14 Promise and 5 Softlab projects report that FedDP significantly outperforms FLR, Almity, FedAvg, FedProx, and OpenFLR, with an average F1 improvement of 2.66% over FL baselines and performance close to centralized training. The practical stake is that companies could cooperatively train defect predictors across proprietary codebases without sharing raw data, even when the client data is non-IID.","feed_headline":"Federated defect models gain 2.66% F1 via open-source distillation","feed_subtitle":"Weighs each client by data similarity, distills the ensemble into the global model, and beats FL baselines on 19 projects.","key_machinery":"The engine of the method is the heterogeneity-aware teacher ensemble. For a distillation sample $x^d_i$, client $k$'s contribution is weighted by $\\alpha^k_i$, computed by normalizing $C^k_i$ across participating clients; $C^k_i$ is the average cosine similarity between $x^d_i$ and all samples in the client's private dataset. The student is the aggregated global model $w_{t+1}$, and the distillation loss is the KL divergence between the teacher's weighted soft predictions and the student's soft predictions on open-source data (Eq. 5). This lets the server transfer ensemble knowledge without ever training directly on open-source labels, sidestepping the distribution shift that the paper shows degrades the naive OpenFLR baseline.","core_discovery":"On its own terms, the central discovery is that directly fine-tuning the global model on open-source data hurts federated defect prediction, whereas distilling the ensemble of local models on that same data helps. FedDP computes, for each client and each distillation sample, a correlation factor $C^k_i$ (Eq. 3): the average cosine similarity between the sample and the client's local data. These factors are normalized into personalized weights $\\alpha^k_i$ (Eq. 4), and the server minimizes the KL divergence between the soft predictions of the weighted teacher ensemble and those of the aggregated student model (Eq. 5). The paper reports that this two-step recipe, local heterogeneity awareness plus global knowledge distillation, yields significant gains over parameter-averaging baselines on both Java and C datasets, reaches target F1 in fewer communication rounds, and lands within a few points of centralized training.","pith_inferences":["A direct test of the paper's core mechanism would be to measure, per distillation sample, the rank correlation between the cosine-similarity score $C^k_i$ and the local model's actual prediction error; if that correlation is negligible, the personalized weights in Eq. (4) add nothing over an unweighted ensemble.","The paper shares correlation factors and soft predictions with the server; an adversarial analysis of what those quantities reveal about local feature distributions is a privacy question the paper does not address.","Because the method only needs a public distillation set and any differentiable student, FedDP could be transferred to other tabular federated classification tasks, such as code smell detection or bug-fix classification, where a proxy public dataset exists.","The authors' closing note admits dependence on distillation data quality; a natural extension beyond what the paper demonstrates would pair FedDP with a data-free generator so no public dataset is required."],"forward_implications":["If FedDP works as reported, federated defect prediction no longer needs to assume IID client data; companies with very different project scales and defect rates can still train a shared predictor.","FedDP reaches target F1 in fewer communication rounds than FLR, OpenFLR, and Almity, which lowers the practical cost of cross-company federated training.","Since FedDP improves both FedAvg and FedProx bases, it behaves as a plug-in distillation layer rather than a full algorithm replacement.","The ablation results imply that most of the gain comes from knowledge distillation itself, with the correlation factors adding a smaller but consistent improvement.","Closer-to-centralized F1 and AUC suggest that the privacy-utility gap in this federated setting can be narrowed without relaxing privacy."],"supporting_citations":[{"why":"Supplies the FLR baseline, the federated CPDP method that FedDP must outperform on the same datasets.","marker":"[16]"},{"why":"Provides the Almity baseline and the non-IID categorization used to label client data distributions.","marker":"[17]"},{"why":"Defines FedAvg, the base aggregation the paper builds on and one of the two FL algorithms it tests.","marker":"[35]"},{"why":"Defines FedProx, the heterogeneity-robust FL baseline FedDP is also evaluated on top of.","marker":"[39]"},{"why":"Supplies the ensemble-distillation mechanism for fusing local models into a global model that FedDP adapts.","marker":"[46]"},{"why":"Grounds the teacher-student knowledge distillation formulation used in Eq. (5).","marker":"[40]"},{"why":"Motivates using open-source project data as a knowledge source for building defect predictors.","marker":"[21]"},{"why":"Defines the KL divergence used as the distillation loss in Eq. (5).","marker":"[50]"}],"fun_headline_variants":["Federated defect prediction gets privacy-safe boost with distillation","Knowledge distillation defeats data heterogeneity in federated CPDP","Open-source data distills into better federated defect models","Privacy-preserving CPDP improved by global knowledge distillation","Heterogeneity-aware federated learning sharpens defect prediction"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that a client model is more trustworthy on an open-source sample when that sample looks similar, in raw feature space, to the client's own training data; if similarity to training data is not a reliable guide to prediction correctness, the weighted ensemble loses its advantage over an unweighted one.","fun_headline_variants_meta":{"raw":{"variants":["Federated defect prediction gets privacy-safe boost with distillation","Knowledge distillation defeats data heterogeneity in federated CPDP","Open-source data distills into better federated defect models","Privacy-preserving CPDP improved by global knowledge distillation","Heterogeneity-aware federated learning sharpens defect prediction"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000421,"raw_usage":{"total_tokens":2154,"prompt_tokens":923,"completion_tokens":1231,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":539,"completion_tokens_details":{"reasoning_tokens":1152}},"tokens_in":539,"tokens_out":1231,"duration_ms":8526,"temperature":1.0,"reasoning_tokens":1152,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T05:35:46.971998+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On the same datasets and splits, compute for each distillation sample the correlation factor from Eq. (3) and each client model's error on that sample, then check whether higher similarity predicts lower error, for example with a rank correlation or by comparing FedDP against a version using random weights in place of $\\alpha^k_i$. If the correlation is absent, or random-weight FedDP matches the reported F1, the heterogeneity-aware teacher is not the source of the gains.","supporting_citations":[{"cited_title":"Towards Privacy Preserving Cross Project Defect Prediction with Federated Learning","cited_arxiv_id":null,"evidence_quote":"Supplies the FLR baseline, the federated CPDP method that FedDP must outperform on the same datasets."},{"cited_title":"Communication-Efficient Learning of Deep Networks from Decentralized Data","cited_arxiv_id":null,"evidence_quote":"Defines FedAvg, the base aggregation the paper builds on and one of the two FL algorithms it tests."},{"cited_title":"Federated Optimization in Heterogeneous Networks","cited_arxiv_id":null,"evidence_quote":"Defines FedProx, the heterogeneity-robust FL baseline FedDP is also evaluated on top of."},{"cited_title":"Ensemble distillation for robust model fusion in federated learning","cited_arxiv_id":null,"evidence_quote":"Supplies the ensemble-distillation mechanism for fusing local models into a global model that FedDP adapts."},{"cited_title":"Learning from Open-Source Projects: An Empirical Study on Defect Prediction","cited_arxiv_id":null,"evidence_quote":"Motivates using open-source project data as a knowledge source for building defect predictors."},{"cited_title":"On Information and Sufficiency","cited_arxiv_id":null,"evidence_quote":"Defines the KL divergence used as the distillation loss in Eq. (5)."}],"review_version":1}