{"id":"d20877ae-52b6-4301-a1da-c36e6a2482ab","arxiv_id":"1908.02320","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Contribution guidelines in 53 active CI-using GitHub projects frequently diverge from observed contribution activity, but the headline 68% divergence figure rests on a weakly operationalized comparison.","lead":"Researchers compared the contribution guidelines of 53 active GitHub projects against the projects' actual GitHub activity and found frequent mismatches, including direct commits to master in nearly every project. The study matters because it questions whether written contribution guidelines reflect real development process and whether they are useful for onboarding contributors, especially in projects using continuous integration.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 68% divergence claim rests on counting maintainers' direct pushes as guideline violations; the paper itself concedes guidelines may not apply to core team members.","rationale":"The paper addresses a genuinely understudied question and contributes a useful coded taxonomy of guideline content, with a reproducibility package and a plausible qualitative finding that CI tools are rarely documented beyond test execution. The reader's CONDITIONAL verdict is appropriate. My stress-test pass did not find a reason to move the verdict: the most load-bearing concern is exactly the one the reader identified and the authors themselves flag in Section VI. The direct-commit statistic is the only quantitative evidence strong enough to carry the headline claim; without role-aware attribution it is ambiguous whether observed direct pushes are guideline violations or internal maintainer workflow. The paper is not internally inconsistent, and the coding methodology is reasonable for an exploratory study, but the inference from event streams to 'not following guidelines' is premature. Secondary concerns such as the lack of inter-rater reliability and the unexplained 53-of-72 sample attrition are real but less decisive. The proposed role-aware re-analysis would settle the ambiguity; until that check is done, CONDITIONAL remains the right verdict.","tokens_in":7585,"tokens_out":3460,"duration_ms":38682,"concrete_test":"Re-analyze the mined GitHub Events API streams (Section III.C) by classifying each PushEvent actor as core collaborator, external contributor, or bot, using each project's collaborator list at the time of the event or GHTorrent membership data. Recompute the proportion of direct commits to master made by non-collaborator humans, and the number of projects in which such commits occur, after excluding bots (for example, dependabot and CI bot accounts). If the mean direct-commit percentage drops sharply when internal and bot pushes are excluded, the divergence finding rests on role conflation and the headline should be reworded to describe the process as observed across all actors rather than as a guideline-violation rate.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative evidence for divergence is the claim that 52/53 projects have direct commits to master, with a mean of 93% direct commits (Section IV, RQ3), while about 68% of projects prescribe PR-based contribution. But the event stream does not distinguish who made the direct commit. The selection criteria deliberately require at least three unique contributors and recently merged PRs (Section III.A), so these repositories have active core teams with write access; direct pushes to master are the normal mechanism for maintainers to integrate work, merge PRs, or handle bot and release commits. Contribution guidelines are framed for prospective external contributors, and Section VI explicitly concedes that guidelines are used as a proxy for contribution process documentation 'which should apply to both core team members as well as external contributors. However, this is not always the case [21].' If most direct commits are by core members or bots, the 52/53 statistic no longer indicates that projects fail to follow their own guidelines; it indicates that guidelines and internal workflow address different audiences. Relatedly, the abstract's 'approximately 68% of these projects diverge significantly' is not tied to a defined divergence measure: in the RQ3 results, 68% appears as the proportion of projects that reopen pull requests, which is presented as undocumented practice, not necessarily a guideline violation. The headline claim therefore depends on an unvalidated operationalization of divergence and on role conflation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates whether contribution guidelines in GitHub repositories match the observed contribution process, focusing on projects that use continuous integration. The authors select 72 active CI-using projects, thematically code their contribution guidelines (CONTRIBUTING.md or README.md), and, for the 53 projects accessible through the GitHub events API, mine event streams and visualize them with the Disco process-mining tool. They then compare the prescribed workflows with observed activity. The reported findings include a taxonomy of guideline content, the observation that CI tools are mentioned in only about 31% of the guidelines, and the headline claim that \"approximately 68% of these projects diverge significantly from the expected process,\" largely because many projects prescribe pull-request-based contribution while direct commits to master are prevalent in the observed event data.","tokens_in":7780,"tokens_out":3614,"duration_ms":40392,"significance":"If the central claim is supported, the paper would make a useful contribution to the empirical software engineering literature by providing one of the first systematic comparisons between documented contribution processes and actual GitHub activity. The study has clear strengths: the coding of guidelines is compared with prior work to mitigate bias, a reproducibility package is referenced, and the observed event-stream data are independent of the guideline documents, so the comparison is not circular. However, the headline quantitative claim depends on an unsupported operationalization of \"diverge significantly\" and on treating all direct commits to master as governed by contribution guidelines. These are load-bearing issues for the paper's central conclusion, and they need to be addressed before the claims can be accepted.","major_comments":[{"comment":"The central evidence for divergence is the observation that \"all but one (i.e., 52/53 projects) involved direct commits to the master branch\" with a mean of 93% direct commits, which is contrasted with guidelines that prescribe pull-request-based contribution. However, the event stream does not distinguish who made those direct commits. The selection criteria in Section III.A deliberately require at least three unique contributors and recently merged pull requests, so the sampled projects have active core teams with write access; direct pushes to master are a normal mechanism for maintainers to integrate work, merge pull requests, or handle bots and release commits. The paper itself concedes in Section VI that treating guidelines as applying to both core team members and external contributors \"is not always the case [21].\" Without stratifying direct commits by the author's role, team membership, or write access, the 52/53 statistic does not establish that projects fail to follow their own guidelines; it may simply show that guidelines and internal maintainer workflow address different audiences. This is load-bearing for the abstract's 68% divergence claim and needs to be fixed, for example by using actor metadata or team membership to separate maintainer pushes from external-contributor pushes.","section":"Section IV (RQ3) and Section VI"},{"comment":"The claim that \"approximately 68% of these projects diverge significantly from the expected process\" is never tied to a defined divergence measure. In the RQ3 results, 68% appears as the proportion of projects that reopen pull requests, which is presented as an undocumented practice rather than a guideline violation, and separately as the proportion of projects whose guidelines describe pull-request-based contribution. The paper does not show, step by step, how the observed mismatches (direct commits, issue reopening, pull-request reopening, traceability findings) are combined into a per-project divergence score or how the 68% figure is derived. The authors should define an operational divergence metric, specify the threshold for \"significant\" divergence, and report the distribution of that metric across the 53 projects. Without such a derivation, the headline number is not verifiable.","section":"Abstract and Section IV (RQ3)"},{"comment":"The conclusion states that \"we demonstrate that the most active projects that use CI in fact do not follow their own guidelines,\" but the body of the paper provides only qualitative evidence of mismatches, and Section V explicitly characterizes the finding as \"preliminary evidence\" and uses phrases such as \"we speculate that the current contribution guideline structure may be written to suit project maintainers more than new contributors.\" The wording in Section VII overstates what the data can support. The authors should calibrate the conclusion to the operational divergence measure they define and to the limitations acknowledged in Section VI, distinguishing documented-practice gaps from direct violations of the guidelines.","section":"Section V and Section VII"}],"minor_comments":[{"comment":"The sentence \"the mean number of direct commits is 93%\" should be rephrased as \"the mean percentage of commits made directly to the master branch is 93%\" to avoid confusion about whether the number is a count or a proportion.","section":"Section IV (RQ3)"},{"comment":"The statement \"Fourteen projects (19.5%) prescribed linking artifacts\" appears to use the 72-project denominator from the guideline-coding sample, but this result is reported in the RQ3 section that analyzes the 53 projects with event-stream data. The authors should clarify which denominator is used and, if the prescription is from the 72-project set, connect it explicitly to the 53-project set for which workflow observations are available.","section":"Section IV (RQ3)"},{"comment":"The number 68% appears twice with different referents: 68% of projects reopen pull requests and about 68% of projects prescribe pull-request-based contribution. The paper should make explicit that these are two different observations, and the numerical coincidence should not be used implicitly to support the abstract's 68% divergence claim.","section":"Section IV (RQ3)"},{"comment":"Figure 1 is referenced as an example process map, but the figure itself is not described in enough detail in the text; a brief caption or a sentence explaining the nodes and edges would improve readability.","section":"Section III.C"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely and interesting question, and the authors have assembled independent data sources and a reproducibility package. The main risk is that the abstract and conclusion overstate the strength of the evidence: the 68% figure is not derived from any defined divergence metric, and the direct-commit finding conflates maintainer activity with guideline-governed external contributions. These issues are fixable within the scope of a revision, so I recommend major revision rather than rejection. I would encourage the editor to ask for the operational divergence computation and the actor-role analysis to be made explicit."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis paper is the first to compare contribution guideline content with actual observed GitHub activity instead of just coding the content. That comparison is a real step forward, and the coding work is careful: they built an index from 72 files, checked saturation, and compared their categories to prior work. The finding that CI is mentioned in only 31% of guidelines and then only as a test-running mechanism is solid and useful. The observation that many projects reopen issues and pull requests without documenting that practice is also a credible, concrete mismatch.\n\nThe soft spot is the headline. \"Approximately 68% of these projects diverge significantly\" is never tied to a defined divergence measure. In the results, 68% appears as the proportion of projects that reopen pull requests and as the proportion whose guidelines describe PR-based contribution, but the abstract's \"diverge significantly\" is not computed from those numbers in a transparent way. The stress-test is right about role conflation: the 52/53 direct-commit statistic does not say who pushed to master, and the sample deliberately selects active projects with core teams. Direct pushes by maintainers, bots, or release processes are normal, and contribution guidelines are usually addressed to external contributors. The authors concede this in Section VI (\"this is not always the case\"). If most direct commits are internal, the strongest evidence for \"projects do not follow their own guidelines\" largely evaporates.\n\nThat said, the paper is not a wreck. The guideline coding and the observation of undocumented practices stand on their own. What needs fixing is precise: define divergence, separate roles in the event trace, and either rerun the headline figure without maintainer pushes or reframe the claim as \"guidelines omit many accepted practices\" rather than \"68% diverge.\" The sample attrition from 72 to 53 should also be discussed.\n\nThis is a preliminary empirical result with identifiable methodological fixes. It deserves a serious referee, but the central number should be treated as conditional. I would send it to peer review with a request for major revision, and I would want the revised version to include the role-aware analysis.","headline":"Useful first comparison of guidelines to observed activity, but the 68% headline rests on an unoperationalized divergence measure and on treating maintainers' direct pushes as guideline violations.","tokens_in":8326,"tokens_out":1878,"would_cite":false,"duration_ms":21291,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Contribution guidelines on GitHub are systematically out of step with how the most active CI-using projects actually accept code.","keywords":["contribution guidelines","continuous integration","pull request workflow","process mining","GitHub activity data","direct commits","open source governance","workflow documentation"],"falsifier":"Classify each direct commit to the master branch by whether its author has write access to the repository, then recompute the 68% divergence figure using only commits by users without write access; if direct commits by such external contributors are rare, the main divergence claim would collapse.","tokens_in":7341,"feed_emoji":"🤝","tokens_out":4926,"duration_ms":48829,"temperature":0.7,"pith_summary":"This paper asks whether GitHub projects that use continuous integration (CI) actually follow their own contribution guidelines. By coding the guidelines of active CI-using projects and mining four weeks of GitHub event activity, it finds that roughly 68% of the 53 projects studied diverge significantly from their documented process. The main divergence is that guidelines describe pull-request-based contribution, while activity is dominated by direct commits to the master branch, with a median of 99% of commits bypassing pull requests. The paper also finds that CI tools are mentioned in only about 31% of guidelines and only as test runners, not as workflow elements.","feed_headline":"68% of GitHub projects ignore their own contribution rules","feed_subtitle":"Guidelines describe pull requests, but direct commits to master are the norm (median 99%).","key_machinery":"The central comparison is between two representations of the contribution process: a prescribed process extracted by manually coding contribution guideline documents (CONTRIBUTING.md or README.md) into thematic categories, and an observed process mined from the GitHub Events API and visualized as process maps using a process-mining tool. The divergence measure is the extent to which the mined event-flow contains behaviors, especially direct commits to master, that the coded guidelines do not describe.","core_discovery":"The paper claims that contribution guidelines in the most active CI-using GitHub projects are systematically out of sync with observed contribution behavior. Approximately 68% of the 53 projects whose activity could be mined diverge significantly from the process their guidelines prescribe: guidelines emphasize pull requests, branching, and acceptance criteria, yet 52 of 53 projects show direct commits to the master branch that are not linked to any pull request. Across all projects, the mean share of direct commits is 93% with a median of 99%. Guidelines also omit common practices such as reopening issues and pull requests, which occur in about half or more of the projects, and they rarely document what CI actually does in the workflow, mentioning CI in only about a third of the sample and only as a test-passing gate.","pith_inferences":["Beyond the paper: the headline divergence may be inflated because direct commits to master are often made by core maintainers with write access, while contribution guidelines are typically addressed to external contributors; separating commits by author permissions would test this.","If guidelines are primarily aimed at external contributors, then the mismatch may indicate a division of labor rather than a failure to follow one's own rules; a useful extension is measuring contribution type by actor role.","A testable design improvement suggested by the findings is to make guidelines state explicitly which parts apply to external contributors versus core team members, and to document CI's role in the merging decision.","Future work could compare guideline adherence between projects with and without enforced CI status checks, since CI's gatekeeping role may or may not reduce direct commits."],"forward_implications":["If the finding holds, contribution guidelines in active CI-using projects serve more as idealized onboarding documents for external contributors than as accurate descriptions of how development actually happens.","Projects that want guidelines to reflect reality would need to document direct-commit workflows and branch-maintenance practices, not only pull-request flows.","The underdocumentation of CI in guidelines means newcomers cannot learn from guidelines how CI gates or evaluates their contributions.","The observed 68% divergence rate provides a baseline for future studies of guideline adherence and for tooling that detects when documented workflows drift from actual activity.","Guideline content is heavily weighted toward automatable checks such as code style, leaving tacit workflow knowledge undocumented."],"supporting_citations":[{"why":"Supplies the project-selection criteria and the evidence that CI improves review efficiency, motivating the focus on CI-using projects.","marker":"[7]"},{"why":"Provides the README-content coding categories and the file-size filtering rule that the paper adapts for guideline coding.","marker":"[9]"},{"why":"Provides the large dataset of GitHub projects from which the study cohort was filtered.","marker":"[15]"},{"why":"Supplies the activity-based filters for excluding forks and inactive projects.","marker":"[16]"},{"why":"Motivates excluding deleted or inactive repositories and using recently merged pull requests as a sign of accepting contributions.","marker":"[18]"},{"why":"Defines the CI configuration-file search used to split projects into CI users and non-users.","marker":"[19]"},{"why":"Documents contributor-side work practices and the maintainer-overload problem that contribution guidelines are meant to mitigate.","marker":"[1]"},{"why":"Documents integrator-side pull-request workflow and acceptance criteria, which the coding index is checked against.","marker":"[14]"}],"fun_headline_variants":["68% of GitHub projects break their own contribution guidelines","Contribution guidelines? Most GitHub projects don't follow them","Pull requests prescribed, but direct commits dominate in GitHub","GitHub guidelines don't match actual contribution behavior","Most GitHub projects ignore their own contribution rules"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central claim assumes that direct commits to the master branch are contributions that the contribution guidelines are meant to govern, even though the paper concedes the guidelines are not always addressed to both core and external contributors.","fun_headline_variants_meta":{"raw":{"variants":["68% of GitHub projects break their own contribution guidelines","Contribution guidelines? Most GitHub projects don't follow them","Pull requests prescribed, but direct commits dominate in GitHub","GitHub guidelines don't match actual contribution behavior","Most GitHub projects ignore their own contribution rules"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00033,"raw_usage":{"total_tokens":1798,"prompt_tokens":864,"completion_tokens":934,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":480,"completion_tokens_details":{"reasoning_tokens":860}},"tokens_in":480,"tokens_out":934,"duration_ms":9293,"temperature":1.0,"reasoning_tokens":860,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T14:47:11.222614+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Classify each direct commit to the master branch by whether its author has write access to the repository, then recompute the 68% divergence figure using only commits by users without write access; if direct commits by such external contributors are rare, the main divergence claim would collapse.","supporting_citations":[{"cited_title":"Quality and productivity outcomes relating to continuous integration in github,","cited_arxiv_id":null,"evidence_quote":"Supplies the project-selection criteria and the evidence that CI improves review efficiency, motivating the focus on CI-using projects."},{"cited_title":"Catego- rizing the content of github readme files,","cited_arxiv_id":null,"evidence_quote":"Provides the README-content coding categories and the file-size filtering rule that the paper adapts for guideline coding."},{"cited_title":"The GHTorrent dataset and tool suite,","cited_arxiv_id":null,"evidence_quote":"Provides the large dataset of GitHub projects from which the study cohort was filtered."},{"cited_title":"Influence of social and technical factors for evaluating contribution in github,","cited_arxiv_id":null,"evidence_quote":"Supplies the activity-based filters for excluding forks and inactive projects."},{"cited_title":"The promises and perils of mining github,","cited_arxiv_id":null,"evidence_quote":"Motivates excluding deleted or inactive repositories and using recently merged pull requests as a sign of accepting contributions."},{"cited_title":"How open source projects use static code analysis tools in continuous integration pipelines,","cited_arxiv_id":null,"evidence_quote":"Defines the CI configuration-file search used to split projects into CI users and non-users."},{"cited_title":"Work practices and challenges in pull-based development: the contributor’s perspective,","cited_arxiv_id":null,"evidence_quote":"Documents contributor-side work practices and the maintainer-overload problem that contribution guidelines are meant to mitigate."},{"cited_title":"Work practices and challenges in pull-based development: the integrator’s perspective,","cited_arxiv_id":null,"evidence_quote":"Documents integrator-side pull-request workflow and acceptance criteria, which the coding index is checked against."}],"review_version":1}