{"id":"e963134c-1094-4251-b1dc-0a36a4a54baa","arxiv_id":"1908.06560","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Across 962 cross-project combinations, five heterogeneous defect prediction methods failed to significantly outperform simple unsupervised baselines on six performance measures.","lead":"This replication study compares five heterogeneous defect prediction methods against five unsupervised baselines on 34 software projects. It finds that the HDP methods do not clearly beat simple unsupervised methods, and it suggests unsupervised baselines should become standard.","discovery_kind":"replication","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Oracle-based UDP5 (Section 3.2) is load-bearing for the 'all five unsupervised methods beat HDP' claim; removing it may collapse that claim, though the UDP4 result may survive.","rationale":"The reader's weakest assumption identifies the same concern. The paper's central result is a negative finding about simple baselines, so even if UDP5 is problematic, the comparison against UDP4 (ManualUp/ManualDown) remains meaningful and likely survives. However, the strongest claims in Section 5.1, the abstract, and the diversity analysis are not robust without UDP5, and no artifacts are provided. Therefore a conditional accept is appropriate, requiring a re-analysis that separates oracle-based baselines and releases code/data.","tokens_in":29656,"tokens_out":4875,"duration_ms":54492,"concrete_test":"Re-run RQ1/RQ2 on the same 962 combinations with UDP5 excluded (or explicitly labeled as an oracle upper bound), recomputing Scott-Knott rankings and win/tie/loss counts. If UDP5's removal changes the top rank group or reduces the number of unsupervised methods that beat all HDP methods from five to four, the Section 5.1 claim must be revised. As a secondary check, independently reproduce UDP5's 34/34 win counts for AUC on the PROMISE target projects from the released data (or from the metrics listed in Table 1) to verify the per-target metric selection was implemented exactly as described.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"UDP5 is constructed in Section 3.2 by selecting, for each target project and each performance measure, the metric that achieves the best performance using the target project's labels. This is an oracle baseline, not a deployable unsupervised method. It is nevertheless included among the five unsupervised methods in the central comparisons, and Section 5.1's statement that all five unsupervised methods perform significantly better than HDP methods (AUC and F1) is driven in part by UDP5's perfect 34/34 win counts in Tables 4 and 5. Similarly, RQ3's diversity analysis uses UDP5-A/UDP5-F variants. If UDP5 is removed or treated as an upper bound, the claim that all five unsupervised methods beat HDP collapses, and the Scott-Knott rankings in Figures 2 and 3 may change. The paper's own abstract emphasizes simple baselines like UDP4, so the core negative result is not necessarily lost; but the strongest version of the claim, the rankings, and the diversity analysis depend on an oracle baseline that no practitioner could use without test labels. The lack of released code/data prevents checking whether the oracle selection was implemented exactly as described.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper reports a large-scale replication study comparing five heterogeneous defect prediction (HDP) methods with five unsupervised baselines on 962 source-target combinations drawn from 34 projects in five dataset groups. Across F1, AUC, ACC, Popt, PMI@20%, and IFA, the authors conclude that HDP methods do not significantly outperform the unsupervised methods, and that simple baselines such as ManualUp/ManualDown (UDP4) should be included in future HDP evaluations. The paper also reports McNemar-based diversity analyses suggesting that HDP and unsupervised methods identify partially complementary defective modules, and a satisfactory-ratio analysis indicating that HDP methods rarely meet previously proposed precision/recall criteria.","tokens_in":29861,"tokens_out":9962,"duration_ms":93301,"significance":"If the main result is correct, the paper provides a valuable and potentially field-changing negative result: it would show that the gains attributed to HDP methods over the past five years are not visible relative to simple unsupervised baselines, and it would justify changing standard evaluation practice. The study has a large and diverse subject pool (962 combinations, 34 projects, five groups), considers both effort-aware and non-effort-aware measures, and uses standard statistical tools such as Scott-Knott, Wilcoxon with BH correction, and Cliff's delta. The central weakness is that one of the five 'unsupervised' baselines, UDP5, is not actually unsupervised: its metric is selected using target-project labels. Because UDP5 appears in the global rankings, win/tie/loss tables, diversity analysis, and satisfactory-ratio analysis, several of the strongest claims currently overstate what the data show. The UDP4-based conclusion is more defensible and may survive revision. The paper would also benefit from releasing artifacts; no permanent code or data link is currently provided.","major_comments":[{"comment":"UDP5 is defined in Section 3.2 as the variant that chooses, for each target project and each performance measure, the metric that achieves the best performance in the target project. This is an oracle baseline because the selection uses the target labels, not just the target feature values. The paper nonetheless counts UDP5 among the 'five unsupervised methods' and uses it in the Scott-Knott rankings of Figures 2 and 3 and in the win/tie/loss tables (Tables 4-7). Section 5.1's claim that 'all the 5 unsupervised methods can significantly perform better than supervised methods' therefore depends on a baseline that no practitioner could deploy. Please exclude UDP5 from the main comparisons, or explicitly re-label it as an upper-bound reference, and recompute all rankings, win/tie/loss counts, and the statements in the abstract and Section 5.1 accordingly.","section":"Section 3.2, Section 5.1, Figures 2-3, Tables 4-7"},{"comment":"The influence of UDP5 is visible in the 34/34 win counts reported in Tables 4 and 5. Since UDP5's per-measure metric is selected using target labels, these wins are partly self-fulfilling and cannot be used as evidence that an unsupervised method outperforms the HDP methods. The same dependence affects the diversity analysis in Section 5.3, where UDP5-A and UDP5-F choose metrics by AUC and F1 performance on the target; the resulting pairwise diversity counts (Tables 11 and 12) may reflect the oracle's adaptation to the target rather than a property of unsupervised predictions. The paper should re-run the analysis with only the deployable unsupervised baselines (UDP1-UDP4) and state what changes.","section":"Tables 4 and 5, Section 5.3"},{"comment":"Table 14 includes UM5 in the satisfactory-ratio analysis. The high ratios for UM5 (e.g., 66.67% on PROMISE under SC2) are not achievable by an unsupervised method under the same protocol, because the metric is selected using the target labels. The qualitative conclusion that the satisfactory ratios of HDP methods are pessimistic may survive, but the UM5 rows should be removed or explicitly marked as oracle results.","section":"Section 5.4"}],"minor_comments":[{"comment":"The sentence 'the UDP methods can win HDP1, HDP2, HDP3, HDP4 and HDP5 at least 26, 30, 29, 30, 30 and 26 times' lists six minima for five HDP methods; please correct the enumeration.","section":"Section 5.1, Table 5 text"},{"comment":"Footnote 2 is a placeholder and no artifact link is provided; because this is a replication study, please provide a permanent URL to code, data, and detailed results.","section":"Footnote 2"},{"comment":"The sentence about the mcnemar function mixing 'statsmodel' and 'R package exact2 × 26' is garbled; please clarify which implementation and version was used.","section":"Section 5.3"},{"comment":"For IFA and PMI@20%, state explicitly whether larger values are better or worse; the current text says UM4 has the 'highest IFA value' without noting that high IFA is usually considered undesirable.","section":"Section 4.3.2, Figures 6-7"},{"comment":"The derivation of 962 HDP combinations is spread over two sentences and depends on per-project metric sets inside NASA; a short formula or table would help reproducibility.","section":"Section 4.2"},{"comment":"Report whether any multiple-comparison correction was applied to the McNemar tests; uncorrected p<0.05 on hundreds of comparisons per method pair will produce many 'significant' combinations by chance.","section":"Section 5.3, Tables 10-12"}],"recommendation":"major_revision","confidential_remarks":"The main issue is the oracle baseline UDP5. I would not reject the paper, because the weaker claim about UDP4 and the large-scale comparisons are valuable; however, the revision must remove or re-label UDP5 and re-run the affected analyses. Please also ensure the authors provide artifact availability before acceptance, since the paper is a replication study."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth engaging. This is the first large-scale systematic comparison of five HDP methods against five unsupervised baselines on 962 cross-project combinations, and the central negative result is credible: the HDP methods do not beat simple unsupervised baselines like Zhou et al.'s ManualUp/ManualDown on F1, AUC, ACC, Popt, PMI@20%, and IFA. The authors use published methods, standard dataset groups, and reasonable statistical machinery (Scott-Knott, BH-adjusted Wilcoxon, Cliff's delta). They also handle the fact that HDP1 only succeeds on 623 of 962 combinations by running two scenarios. The diversity analysis and satisfactory-ratio analysis are useful extensions. This is a real contribution to evaluation practice in defect prediction.\n\nThe soft spot is UDP5. Section 3.2 constructs it by selecting, for each target project and each performance measure, the metric with the best performance using the target project's labels. That is an oracle baseline, not a deployable unsupervised method. Yet UDP5 sits in the five unsupervised methods, produces perfect 34/34 win counts in Tables 4 and 5, and drives the Scott-Knott rankings behind the \"all five unsupervised methods significantly outperform HDP\" claim in Section 5.1. The same oracle appears in the diversity analysis as UDP5-A/UDP5-F. The reader's stress-test is right: remove UDP5 and the strongest claims collapse, although the UDP4 result likely survives. The paper's own abstract and conclusion lean on UDP4, so the core message does not depend on the oracle. But the text should separate legitimate baselines from the oracle, and rerun or re-flag the analyses without UDP5.\n\nTwo smaller issues. First, the paper claims no code or data release (\"The website will be given when the manuscript is accepted\"), which is a real weakness for a replication study that re-implements several methods (HDP5, UDP3-UDP5). Second, results for IFA and PMI@20% are more mixed: only UDP4 is best, and supervised methods beat UDP1-3, so the abstract's \"some unsupervised methods\" is the honest phrasing, not \"all five.\"\n\nRecommendation: send to peer review after a major revision that fixes UDP5 and releases artifacts. The negative result is important and mostly solid, and the field should engage with it.","headline":"Solid replication with a cautionary result that probably holds for simple baselines, but the strongest 'all five unsupervised methods win' claim rides on an oracle baseline and should be fixed before publication.","tokens_in":30370,"tokens_out":2821,"would_cite":true,"duration_ms":29270,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Across 962 setups, no-label baselines beat trained defect predictors","keywords":["software defect prediction","heterogeneous defect prediction","cross-project defect prediction","unsupervised baselines","ManualUp/ManualDown","replication study","effort-aware performance measures","McNemar diversity analysis"],"falsifier":"Re-run the same 962 comparisons with UDP5 either deleted or redefined to pick its metric using only source-project information or a fixed label-free rule; the central claim would be falsified if, without UDP5, HDP methods occupy the top Scott-Knott rank group on F1 or AUC, and it would be strengthened if the unsupervised methods still dominate.","tokens_in":29450,"feed_emoji":"🐛","tokens_out":9327,"duration_ms":85401,"temperature":0.7,"pith_summary":"The paper asks a question the heterogeneous defect prediction (HDP) literature had not systematically asked: do supervised transfer-learning methods that predict defects across projects with different metric sets actually beat methods that ignore labels entirely? The authors compare five state-of-the-art HDP methods with five unsupervised methods on 962 source-to-target combinations built from 34 projects across five dataset groups, and find that HDP methods do not significantly outperform the unsupervised baselines on two non-effort-aware measures (F1 and AUC) or four effort-aware measures (ACC, Popt, PMI@20%, and IFA); the simple LOC-based ManualUp/ManualDown ranking is especially hard to beat. In the Scott-Knott rankings, all five unsupervised methods sit significantly above the HDP methods on F1 and AUC. A diversity analysis using McNemar's test shows the two families flag different defective modules, so they are partly complementary rather than redundant. If the conclusion holds, new HDP methods should be evaluated against unsupervised baselines before any claim of progress.","feed_headline":"Across 962 setups, no-label baselines beat trained defect predictors","feed_subtitle":"Even no-training baselines beat five leading heterogeneous defect predictors, so new methods must clear that bar.","key_machinery":"The machinery that carries the argument is a comparison protocol rather than a single theorem. Each HDP method and each unsupervised method is evaluated under one shared experimental setup on 962 source-to-target combinations, and performance is ranked by the Scott-Knott test (a recursive clustering of methods into statistically distinct rank groups), with pairwise differences checked by the Benjamini-Hochberg-corrected Wilcoxon signed-rank test and Cliff's delta effect sizes. The load-bearing baselines are ManualUp/ManualDown from Zhou et al., which sort modules by lines of code and label the top half defective, plus UDP5, a variant that picks the metric achieving the best performance on each target project and performance measure. The diversity analysis uses McNemar's test on paired predictions to decide whether two methods identify the same defective modules. What this machinery does is convert the research question into a falsifiable ranking: if a supervised HDP method cannot land in a top Scott-Knott rank group against methods that ignore labels, then its transfer machinery is not delivering measurable value on these datasets.","core_discovery":"On the paper's own terms, the central discovery is that the current generation of heterogeneous defect prediction methods has not established an advantage over methods that need no labeled training data. Across 962 cross-project combinations, the five HDP methods—metric selection and matching, EMKCA, CTKCCA, TSEL, and distribution-characteristic mapping—fail to rank above the five unsupervised methods in Scott-Knott analyses for any of the six performance measures. For F1 and AUC specifically, every unsupervised method lands in a significantly better rank group than every HDP method, with the win/tie/loss tables showing the unsupervised methods winning the large majority of the 34 target-project comparisons. For effort-aware measures, the two size-based ranking methods (ManualUp/ManualDown and its metric-selection variant UDP5) are the clear leaders, while the other unsupervised methods remain competitive. The authors also find through McNemar's test that HDP and unsupervised predictions diverge on defective modules more than within-family comparisons do, and that a substantial fraction of defective modules are missed by both families, which they read as evidence that HDP research still has a long way to go.","pith_inferences":["Editorial inference: the headline result leans on UDP5, which uses target-project labels to choose its metric; if UDP5 is excluded, the strong claim that all five unsupervised methods beat HDP weakens, though the simpler LOC-based baselines would still be competitive.","Editorial inference: the diversity result implies a cheap experiment the paper does not run—combining one HDP model with a LOC-based ranker in a simple ensemble and testing whether the union beats either alone; the McNemar tables suggest headroom for exactly this kind of hybrid.","Editorial inference: if the pattern replicates beyond these 34 projects, the practical implication is that research effort may be better spent on deciding when labels help, rather than on ever more elaborate kernel or domain-adaptation machinery.","Editorial inference: a further testable extension would be to run the same comparison with effort-aware measures on change-level (just-in-time) data, since the earlier unsupervised-baseline debates originated there and the present study's effort-aware results may or may not carry over."],"forward_implications":["Future HDP studies that report gains over older supervised methods but omit unsupervised baselines should not be read as progress; the default comparison should include ManualUp/ManualDown and comparable label-free rankers.","Because HDP and unsupervised predictions flag different defective modules, ensembling the two families is a concrete, testable route to better recall; the paper explicitly points toward ensemble learning as a likely next step.","The low satisfactory ratios under both criteria (precision and recall above 75%, or recall above 70% with precision above 50%) imply that HDP models are not yet dependable enough to drive defect-inspection effort on the studied project types.","The result extends the earlier cross-project finding that simple size-based ranking beats complex models: the same pattern survives when source and target projects use different metric sets, which is the harder heterogeneous setting."],"supporting_citations":[{"why":"Supplies the ManualUp and ManualDown baselines that define UDP4 and carry much of the paper's negative result; the authors find HDP cannot beat these simple LOC-based rankings.","marker":"[93]"},{"why":"Supplies CLA and CLAMI, the two label-free methods UDP1 and UDP2 used in the comparison.","marker":"[55]"},{"why":"Supplies the connectivity-based spectral-clustering method UDP3, one of the five unsupervised baselines.","marker":"[90]"},{"why":"Defines HDP1, the metric-selection and metric-matching method, and gives the experimental settings the replication adopts.","marker":"[56]"},{"why":"Defines HDP3 (cost-sensitive transfer kernel canonical correlation analysis) and its parameter settings.","marker":"[41]"},{"why":"Defines HDP2 (ensemble multiple kernel correlation alignment) used in the comparison.","marker":"[43]"},{"why":"Defines HDP4 (two-stage ensemble learning) and its parameters.","marker":"[45]"},{"why":"Defines HDP5, the distribution-characteristic based method using 16 statistical features.","marker":"[20]"},{"why":"Supplies the Scott-Knott test implementation used to rank methods into statistically distinct groups.","marker":"[30]"},{"why":"Supplies McNemar's test used for the diversity analysis of defective-module predictions.","marker":"[14]"}],"fun_headline_variants":["Heterogeneous defect predictors fail to beat no-label baselines","Simple unsupervised baselines beat heterogeneous defect predictors","No-training methods outperform five leading defect predictors","Unsupervised baselines beat defect predictors in 962 setups","Defect predictors need labeled data, but no-label baselines win"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Section 3.2 introduces UDP5 by letting it choose, for every target project and performance measure, the metric that performs best using that target project's labels; if that oracle-like construction is not a legitimate baseline, the global Scott-Knott rankings and the claim that all five unsupervised methods beat HDP collapse, though the simpler LOC-based baselines may still stand.","fun_headline_variants_meta":{"raw":{"variants":["Heterogeneous defect predictors fail to beat no-label baselines","Simple unsupervised baselines beat heterogeneous defect predictors","No-training methods outperform five leading defect predictors","Unsupervised baselines beat defect predictors in 962 setups","Defect predictors need labeled data, but no-label baselines win"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000942,"raw_usage":{"total_tokens":4064,"prompt_tokens":1023,"completion_tokens":3041,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":639,"completion_tokens_details":{"reasoning_tokens":2962}},"tokens_in":639,"tokens_out":3041,"duration_ms":21296,"temperature":1.0,"reasoning_tokens":2962,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T12:40:19.733972+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the same 962 comparisons with UDP5 either deleted or redefined to pick its metric using only source-project information or a fixed label-free rule; the central claim would be falsified if, without UDP5, HDP methods occupy the top Scott-Knott rank group on F1 or AUC, and it would be strengthened if the unsupervised methods still dominate.","supporting_citations":[{"cited_title":"ACM Transactions on Software Engineering and Methodology 27(1), 1:1–1:51 (2018) 40 Xiang Chen et al","cited_arxiv_id":null,"evidence_quote":"Supplies the ManualUp and ManualDown baselines that define UDP4 and carry much of the paper's negative result; the authors find HDP cannot beat these simple LOC-based rankings."},{"cited_title":"In: Proceedings of International Conference on Automated Software Engineering, pp","cited_arxiv_id":null,"evidence_quote":"Supplies CLA and CLAMI, the two label-free methods UDP1 and UDP2 used in the comparison."},{"cited_title":"In: Proceedings of the International Confer- ence on Software Engineering, pp","cited_arxiv_id":null,"evidence_quote":"Supplies the connectivity-based spectral-clustering method UDP3, one of the five unsupervised baselines."},{"cited_title":"In: Proceedings of Joint Meeting of the European Software Engineering Conference and the Symposium on Foundations of Software Engineering, pp","cited_arxiv_id":null,"evidence_quote":"Defines HDP1, the metric-selection and metric-matching method, and gives the experimental settings the replication adopts."},{"cited_title":"Automated Software Engineering 25(2), 201–245 (2018)","cited_arxiv_id":null,"evidence_quote":"Defines HDP3 (cost-sensitive transfer kernel canonical correlation analysis) and its parameter settings."},{"cited_title":"In: Proceedings of the International Conference on Software Maintenance and Evolution, pp","cited_arxiv_id":null,"evidence_quote":"Defines HDP2 (ensemble multiple kernel correlation alignment) used in the comparison."},{"cited_title":"Automated Software Engineering (2019)","cited_arxiv_id":null,"evidence_quote":"Defines HDP4 (two-stage ensemble learning) and its parameters."},{"cited_title":"Towards Cross-Project Defect Prediction with Imbalanced Feature Sets","cited_arxiv_id":"1411.4228","evidence_quote":"Defines HDP5, the distribution-characteristic based method using 16 statistical features."},{"cited_title":"TEMA (S˜ ao Carlos)15(1), 3–17 (2014)","cited_arxiv_id":null,"evidence_quote":"Supplies the Scott-Knott test implementation used to rank methods into statistically distinct groups."}],"review_version":1}