{"id":"b9f04859-903b-471e-9eee-ab38ac6ecbe2","arxiv_id":"1908.09321","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"A 36,344-pull-request study reports that PMD-detected code quality issues do not predict pull request acceptance, but an internal table appears to contradict this null result.","lead":"A study of 36,344 pull requests in 28 Java open-source projects found that the number and type of code-quality issues detected by the PMD tool has no measurable effect on whether maintainers accept or reject a pull request. The result matters because it challenges the common assumption that code review is selecting for code quality, while pointing toward reputation and feature importance as more relevant drivers.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 11 contradicts the reported chi-square: printed counts yield chi-square ≈ 518, not 0.12, so the central null claim is unsupported until reconciled.","rationale":"The most load-bearing concern for the central claim is not the choice of PMD as a quality proxy but the internal consistency of the statistical evidence. The paper asks readers to accept a null result based on a reported chi-square value of 0.12, yet the accompanying contingency table, if read as raw counts, yields a chi-square around 518, revealing a strong association between the presence of TD items and pull request rejection. This is not a minor typographical slip: the two lines of evidence point in opposite directions. Table 11 shows that pull requests without TD items are accepted about 12 percentage points more often, so either the table is wrong or the test statistic is wrong. Until this is fixed, the paper's headline finding that code quality 'does not influence acceptance at all' is not defensible. The replication package and the use of eight classifiers are positive features, but they do not resolve the contradiction; indeed, the AUC near 50% deepens the puzzle, since a real 12-point group difference would ordinarily be detectable. The reader's verdict of REJECT is appropriate. I agree with the reader's rationale on this point, even though the formally stated weakest assumption (construct validity of PMD) is a different, secondary concern.","tokens_in":18718,"tokens_out":8490,"duration_ms":71400,"concrete_test":"Using the four counts in Table 11, recompute Pearson's chi-square statistic with the standard formula (O−E)^2/E under independence. Confirm whether the result is approximately 518; if so, inspect the replication package for the original contingency table and re-run the analysis to see whether the reported 0.12 arises from a printing error in the table or a miscalculated statistic. Also reconcile the table's totals with the 36,344 total pull requests and the 19,293 accepted / 17,051 rejected counts in Table 5.","verdict_should_be":"REJECT","load_bearing_attack":"The central claim is that PMD-detected Technical Debt items have no influence on pull request acceptance, with the chi-square test on Table 11 presented as direct confirmation. However, the four printed counts (10,563; 8,558; 11,228; 5,528) are incompatible with the reported statistic of 0.12. Recomputing Pearson's chi-square from those cells gives approximately 518 on 1 degree of freedom, a vanishingly small p-value: the acceptance rate is 60.7% for pull requests without TD items versus 48.5% for those with TD items, a 12.2 percentage-point gap. If the counts are real, the data contradict the paper's null conclusion; if the statistic is real, Table 11 is mislabeled or misprinted and the paper offers no valid contingency evidence for independence. The near-50% AUC results are suggestive but cannot override an internally contradictory headline statistic; the discrepancy must be resolved before the central claim is credible. The construct-validity caveat about PMD rules is secondary to this arithmetic inconsistency.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports a case study of pull request acceptance in 28 Java open-source projects. Using PMD with its default rule set, the authors count technical-debt (TD) items in 36,344 pull requests and relate these counts to whether each pull request was merged. RQ1 describes the distribution of TD items across projects and priorities; RQ2 tests whether TD item presence is associated with acceptance using a chi-square test and eight classifiers; RQ3 repeats the modeling after grouping TD items by PMD priority. The paper's central conclusion is that \"the presence of TD items of all types in the pull request code, does not influence the acceptance or rejection of pull requests at all.\" The authors also provide a replication package and use multiple machine learning methods to support the null result.","tokens_in":18862,"tokens_out":5319,"duration_ms":47877,"significance":"If valid, this negative result would be a noteworthy contribution to empirical software engineering, because it challenges the intuition that code quality as measured by static analysis affects maintainers' merge decisions. The study is large in scale, covers a diverse set of mature Java projects, and the authors provide a replication package, which are strengths. However, the paper's single reported chi-square test of independence is contradicted by its own printed contingency table, and the machine-learning results, while suggestive, cannot by themselves overcome that arithmetic inconsistency. The significance of the finding is therefore conditional on a resolution of the Table 11 discrepancy; as printed, the central claim is unsupported.","major_comments":[{"comment":"The printed contingency counts are incompatible with the reported chi-square value of 0.12. Interpreting the entries as counts (10,563; 8,558; 11,228; 5,528), Pearson's chi-square is approximately 518 on 1 degree of freedom, with p << 0.001, not 0.12. The acceptance rate for pull requests with TD items is 48.5% (10,563/21,791), while for pull requests without TD items it is 60.8% (8,558/14,086), a 12.2 percentage-point difference. This directly contradicts the sentence \"the presence of TD items does not affect pull request acceptance\" and the summary statement \"χ2 0.12 and AUC 50%.\" Since Table 11 is the only direct statistical test of independence reported, the central claim is unsupported until the counts or the test statistic are reconciled.","section":"Section 5 (RQ2), Table 11"},{"comment":"The conclusion \"The same results are verified in all the 28 projects independently\" is not backed by any per-project chi-square test or per-project machine-learning table in the manuscript. The text states that \"some projects showed some moderate success\" but dismisses them as outliers without reporting how many projects, what threshold was used, or what the per-project statistics were. Given that the pooled contingency test is invalid as printed, this per-project evidence is needed to support the universal formulation of the conclusion.","section":"Section 5 (RQ2), Figure 1 and Section 8"},{"comment":"The independent variable is a count of PMD default-rule violations, yet the research questions, abstract, and conclusions are phrased in terms of \"code quality.\" The most frequent violations listed in Table 8 are predominantly priority-4 style rules, and Section 7 concedes both that PMD's detection accuracy has not been empirically assessed and that no comparison was made with maintainers' actual quality judgments. The null result should therefore be scoped to \"PMD-detected TD item counts,\" not to code quality generally; as written, the \"at all\" wording in Section 8 overstates what the measurement supports.","section":"Sections 4.3, 4.4, and 7"}],"minor_comments":[{"comment":"The table uses period separators (10.563) while other tables use commas (1,270), and the four counts sum to 35,877 rather than the 36,344 pull requests reported in Section 5; the formatting and the total should be clarified.","section":"Table 11"},{"comment":"The abstract says \"seven machine learning techniques\" but lists six, while Section 4.4 states that eight classifiers were used; the count should be made consistent.","section":"Abstract and Section 4.4"},{"comment":"Several entries appear truncated or erroneous, such as \"1,27\", \"2,19\", \"5,52\", \"4,12\", and \"3,07\", and the time frame for apache/cassandra is listed as \"2018/10-2011/09\", which appears reversed.","section":"Table 2"},{"comment":"There are typographical errors that should be corrected, including \"Gausios\" for Gousios, \"Change Pronenes\" in Table 1, and \"3,6022\" in Table 8.","section":"Throughout"}],"recommendation":"reject","confidential_remarks":"The Table 11 discrepancy is not a minor reporting issue: if the counts are real, the data strongly contradict the paper's stated null result; if the chi-square value is real, the table is mislabeled and no valid contingency evidence is presented. Either way, the central claim cannot be accepted without a complete reanalysis, and the likely corrected finding would be the opposite of the paper's conclusion. I therefore recommend rejection rather than major revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the main empirical claim doesn't survive contact with the paper's own Table 11. The four cells are 10,563 accepted with TD items, 8,558 accepted without, 11,228 rejected with TD items, and 5,528 rejected without. Acceptance rate with TD items is 48.5%; without is 60.8%. That's a 12-point gap, and Pearson's chi-square on those counts is around 518, not 0.12 as printed. So either the table or the test statistic is wrong, and the paper's central sentence — code quality does not influence acceptance at all — is not supported by the evidence as written. The ML AUC values hovering near 50% are consistent with a null effect, but they don't fix a contradictory headline statistic.\n\nWhat's new and worthwhile: this is the first large-scale direct measurement I know of linking PMD-reported issues in the pull request diff to accept/reject outcome. 36k pull requests, 28 Java projects, 4.7M violations, with a reasonable project-selection protocol and eight classifiers. The descriptive distribution of TD items by priority is a useful reference point. They also point to a replication package, though I didn't see analysis scripts in the description, only raw data. The citation pattern looks fine: self-citations appear only as background context and don't carry the argument.\n\nSoft spots beyond the arithmetic: the quality proxy is unvalidated. They never compare PMD findings with reviewer comments or any independent judgment of quality. The default PMD rule set is heavily weighted toward style rules; priority 4 items dominate Table 8, and things like LawOfDemeter or OnlyOneReturn may not be what maintainers care about. The models also lack obvious controls — project, pull request size, developer experience, number of comments — that prior work shows matter. Those weaknesses would be survivable if the central test were reported correctly. As they stand, they compound the problem.\n\nI agree with the reader's report and the stress-test note. This is a reject for me, but not a desk-reject. The research question is important, the dataset is substantial, and the paper is honestly written. A serious referee should send it back for a corrected statistical analysis and a more careful framing of what 'code quality' means here.\n\nRecommendation: send to peer review, with major revision required. If the authors fix the contingency table and the analysis survives with appropriate controls, the paper could be a useful contribution.","headline":"Good question, big dataset, but the paper's own contingency table contradicts its null claim; needs major revision before it can be believed.","tokens_in":19456,"tokens_out":2222,"would_cite":false,"duration_ms":22909,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Code quality measured by PMD rule violations in changed code does not affect whether a maintainer accepts or rejects a pull request.","keywords":["pull request acceptance","code quality","technical debt","PMD","static code analysis","machine learning classifiers","open source software","empirical software engineering"],"falsifier":"Look at pull requests whose human review comments explicitly criticize code quality; if those comments predict rejection while PMD counts do not, the paper's null result is a measurement artifact rather than evidence that quality is ignored.","tokens_in":18472,"feed_emoji":"🔀","tokens_out":8253,"duration_ms":77140,"temperature":0.7,"pith_summary":"The paper tests a widely held assumption: that maintainers reviewing a pull request consider the quality of the submitted code when deciding whether to merge it. Using the PMD static analysis tool, the authors count rule violations (\"technical-debt items\") in the changed lines of 36,344 pull requests from 28 Java projects and ask whether those counts predict acceptance. The reported answer is no: no violation type and no severity priority moves acceptance away from chance, and the null result holds in every project individually. If true, this matters because code review is often justified as a quality gate, and the evidence shifts attention toward other acceptance drivers such as contributor reputation and feature importance.","feed_headline":"Code quality does not affect pull request acceptance","feed_subtitle":"36,000 pull requests and 4.7 million code-quality flags show acceptance stays at chance level.","key_machinery":"The load-bearing instrument is PMD, an open-source static analysis tool whose default Java rule set flags more than 300 potential issues, including code smells, anti-patterns, style and documentation lapses, and possible bugs, each with a priority from P1 (critically broken) to P5 (cosmetic). The authors convert every violation appearing in a pull request's diff into a technical-debt item, producing 4.7 million observations across 253 rule types. The argument then runs through two complementary mechanisms that both register the same null result: a chi-square test on the contingency table of accepted/rejected versus violations/no violations, and eight classifiers whose accuracy metrics (AUC, precision, recall, MCC, F-measure) quantify the predictive signal in the violation counts. The machinery demonstrates the result by failing to exceed chance: near-50% AUC and near-zero MCC indicate that violation counts are statistically independent of the merge decision.","core_discovery":"Using PMD's default Java rule set, the authors analyze each pull request's diff against the master branch and count every rule violation introduced by the contribution, treating each as a technical-debt item. They then test whether these counts separate accepted from rejected pull requests using a chi-square test on a contingency matrix and eight machine-learning classifiers (logistic regression, decision tree, bagging, random forest, extremely randomized trees, AdaBoost, gradient boosting, XGBoost). The central finding is that the counts do not separate the two groups: the chi-square statistic is 0.12, all classifiers hover at roughly 50% AUC, and grouping violations by PMD priority from P1 to P4 does not improve prediction. Of the 253 distinct PMD rules found in the data, 243 appear in both accepted and rejected pull requests, and the ten found only in rejected requests are too rare to move the models. The paper concludes that the presence of technical-debt items in pull request code does not influence acceptance or rejection at all.","pith_inferences":["A natural extension is to test quality signals that focus on potential bugs and design flaws, such as compiler warnings, failing tests, or human reviewer comments, since the default PMD rule set is dominated by style-oriented priority-4 rules.","The binary merged/not-merged outcome may hide quality effects that appear earlier in the process: maintainers might send low-quality pull requests back for revisions, so quality would shape latency and revision cycles rather than final acceptance.","Contributor reputation may moderate the quality effect: trusted contributors could get the benefit of the doubt while newcomers with identical code are rejected, and aggregate models would average that interaction away.","A replication on projects with enforced quality gates, where continuous integration blocks merging on rule violations, would clarify whether the null result reflects maintainer indifference or simply the absence of such gates in the sampled projects."],"forward_implications":["In the 28 studied projects, maintainers do not appear to use PMD-detectable code quality as a screening criterion, so contributors cannot reliably raise acceptance odds by cleaning up style and design violations.","Even the most severe PMD priorities, the ones closest to potential bugs, show no relationship with rejection, so the null result is not an artifact of flooding by low-severity style rules.","The finding aligns with earlier evidence that acceptance is driven more by contributor reputation and the importance of the delivered feature than by measured code quality.","Because every project independently shows the same pattern, the null result is not explained by one project's unusual review culture but appears general across the sampled ecosystem.","The authors suggest that pre-submission quality checks can still improve code maintainability, even though on this evidence they will not raise the chance of acceptance."],"supporting_citations":[{"why":"Supplies the earlier exploratory study of pull-based development that this work extends, including the observation that reviewed pull requests are usually merged.","marker":"[9]"},{"why":"Reports the integrator survey in which developers name code quality as a top acceptance factor, the expectation this study tests against measured violations.","marker":"[10]"},{"why":"Provides the reputation-based explanation for acceptance that the discussion invokes to interpret the null quality result.","marker":"[17]"},{"why":"Identifies PMD as one of the most frequently used static analysis tools, supporting the choice of PMD as the quality measure.","marker":"[18]"},{"why":"Offers a large-scale evaluation of static analysis in open-source software that further grounds the tool selection.","marker":"[19]"},{"why":"Defines the code smells, such as long methods and large classes, that many PMD rules operationalize as quality issues.","marker":"[20]"},{"why":"Gives empirical evidence that code smells increase change-proneness, motivating the expectation that quality issues should affect acceptance.","marker":"[24]"},{"why":"Supplies the case-study guidelines that structure the research design and the validity analysis.","marker":"[58]"},{"why":"Justifies the use of Matthews Correlation Coefficient as a balanced accuracy measure for the classifiers.","marker":"[63]"},{"why":"Defines the AUC/ROC evaluation metric used to show that the models hover at chance level.","marker":"[64]"}],"fun_headline_variants":["Quality flags don't sway pull requests: 36K PRs","No link between code quality and PR acceptance","4.7M quality issues fail to predict PR outcome","Code quality: a non-factor in PR reviews","Pull request acceptance ignores code quality"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The assumption that carries the result is that automated rule violations in a pull request's changed lines capture the code quality maintainers actually judge when they accept or reject it.","fun_headline_variants_meta":{"raw":{"variants":["Quality flags don't sway pull requests: 36K PRs","No link between code quality and PR acceptance","4.7M quality issues fail to predict PR outcome","Code quality: a non-factor in PR reviews","Pull request acceptance ignores code quality"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000229,"raw_usage":{"total_tokens":1521,"prompt_tokens":1033,"completion_tokens":488,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":649,"completion_tokens_details":{"reasoning_tokens":415}},"tokens_in":649,"tokens_out":488,"duration_ms":5033,"temperature":1.0,"reasoning_tokens":415,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T11:15:24.821670+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Look at pull requests whose human review comments explicitly criticize code quality; if those comments predict rejection while PMD counts do not, the paper's null result is a measurement artifact rather than evidence that quality is ignored.","supporting_citations":[{"cited_title":"Gousios, M","cited_arxiv_id":null,"evidence_quote":"Supplies the earlier exploratory study of pull-based development that this work extends, including the observation that reviewed pull requests are usually merged."},{"cited_title":"Gousios, A","cited_arxiv_id":null,"evidence_quote":"Reports the integrator survey in which developers name code quality as a top acceptance factor, the expectation this study tests against measured violations."},{"cited_title":"Calefato, F","cited_arxiv_id":null,"evidence_quote":"Provides the reputation-based explanation for acceptance that the discussion invokes to interpret the null quality result."},{"cited_title":"Lenarduzzi, A","cited_arxiv_id":null,"evidence_quote":"Identifies PMD as one of the most frequently used static analysis tools, supporting the choice of PMD as the quality measure."},{"cited_title":"Beller, R","cited_arxiv_id":null,"evidence_quote":"Offers a large-scale evaluation of static analysis in open-source software that further grounds the tool selection."},{"cited_title":"Powers, Evaluation: From precision, recall and f-factor to roc, in- formedness, markedness & correlation, Mach","cited_arxiv_id":null,"evidence_quote":"Justifies the use of Matthews Correlation Coefficient as a balanced accuracy measure for the classifiers."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the AUC/ROC evaluation metric used to show that the models hover at chance level."}],"review_version":1}