REVIEW 3 major objections 4 minor 21 references
Do as I Do, Not as I Say: Do Contribution Guidelines Match the GitHub Contribution Process?
T0 review · 3 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read Contribution guidelines on GitHub are systematically out of step with how the most active CI-using projects actually accept code.
desk verdict Useful first comparison of guidelines to observed activity, but the 68% headline rests on an unoperationalized divergence measure and on treating maintainers' direct pushes as guideline violations. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central comparison is between two representations of the contribution process: a prescribed process extracted by manually coding contribution guideline documents (CONTRIBUTING.md or README.md) into thematic categories, and an observed process mined from the GitHub Events API and visualized as process maps using a process-mining tool. The divergence measure is the extent to which the mined event-flow contains behaviors, especially direct commits to master, that the coded guidelines do not describe.
What would settle it
Classify each direct commit to the master branch by whether its author has write access to the repository, then recompute the 68% divergence figure using only commits by users without write access; if direct commits by such external contributors are rare, the main divergence claim would collapse.
Extended reading notes
Core claim
The paper claims that contribution guidelines in the most active CI-using GitHub projects are systematically out of sync with observed contribution behavior. Approximately 68% of the 53 projects whose activity could be mined diverge significantly from the process their guidelines prescribe: guidelines emphasize pull requests, branching, and acceptance criteria, yet 52 of 53 projects show direct commits to the master branch that are not linked to any pull request. Across all projects, the mean share of direct commits is 93% with a median of 99%. Guidelines also omit common practices such as reopening issues and pull requests, which occur in about half or more of the projects, and they rarely document what CI actually does in the workflow, mentioning CI in only about a third of the sample and only as a test-passing gate.
Load-bearing premise
The central claim assumes that direct commits to the master branch are contributions that the contribution guidelines are meant to govern, even though the paper concedes the guidelines are not always addressed to both core and external contributors.
Editorial extensions
If this is right
- If the finding holds, contribution guidelines in active CI-using projects serve more as idealized onboarding documents for external contributors than as accurate descriptions of how development actually happens.
- Projects that want guidelines to reflect reality would need to document direct-commit workflows and branch-maintenance practices, not only pull-request flows.
- The underdocumentation of CI in guidelines means newcomers cannot learn from guidelines how CI gates or evaluates their contributions.
- The observed 68% divergence rate provides a baseline for future studies of guideline adherence and for tooling that detects when documented workflows drift from actual activity.
- Guideline content is heavily weighted toward automatable checks such as code style, leaving tacit workflow knowledge undocumented.
Reading between the lines
- Beyond the paper: the headline divergence may be inflated because direct commits to master are often made by core maintainers with write access, while contribution guidelines are typically addressed to external contributors; separating commits by author permissions would test this.
- If guidelines are primarily aimed at external contributors, then the mismatch may indicate a division of labor rather than a failure to follow one's own rules; a useful extension is measuring contribution type by actor role.
- A testable design improvement suggested by the findings is to make guidelines state explicitly which parts apply to external contributors versus core team members, and to document CI's role in the merging decision.
- Future work could compare guideline adherence between projects with and without enforced CI status checks, since CI's gatekeeping role may or may not reduce direct commits.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper investigates whether contribution guidelines in GitHub repositories match the observed contribution process, focusing on projects that use continuous integration. The authors select 72 active CI-using projects, thematically code their contribution guidelines (CONTRIBUTING.md or README.md), and, for the 53 projects accessible through the GitHub events API, mine event streams and visualize them with the Disco process-mining tool. They then compare the prescribed workflows with observed activity. The reported findings include a taxonomy of guideline content, the observation that CI tools are mentioned in only about 31% of the guidelines, and the headline claim that "approximately 68% of these projects diverge significantly from the expected process," largely because many projects prescribe pull-request-based contribution while direct commits to master are prevalent in the observed event data.
Significance. If the central claim is supported, the paper would make a useful contribution to the empirical software engineering literature by providing one of the first systematic comparisons between documented contribution processes and actual GitHub activity. The study has clear strengths: the coding of guidelines is compared with prior work to mitigate bias, a reproducibility package is referenced, and the observed event-stream data are independent of the guideline documents, so the comparison is not circular. However, the headline quantitative claim depends on an unsupported operationalization of "diverge significantly" and on treating all direct commits to master as governed by contribution guidelines. These are load-bearing issues for the paper's central conclusion, and they need to be addressed before the claims can be accepted.
major comments (3)
- [Section IV (RQ3) and Section VI] The central evidence for divergence is the observation that "all but one (i.e., 52/53 projects) involved direct commits to the master branch" with a mean of 93% direct commits, which is contrasted with guidelines that prescribe pull-request-based contribution. However, the event stream does not distinguish who made those direct commits. The selection criteria in Section III.A deliberately require at least three unique contributors and recently merged pull requests, so the sampled projects have active core teams with write access; direct pushes to master are a normal mechanism for maintainers to integrate work, merge pull requests, or handle bots and release commits. The paper itself concedes in Section VI that treating guidelines as applying to both core team members and external contributors "is not always the case [21]." Without stratifying direct commits by the author's role, team membership, or write access, the 52/53 statistic does not establish that projects fail to follow their own guidelines; it may simply show that guidelines and internal maintainer workflow address different audiences. This is load-bearing for the abstract's 68% divergence claim and needs to be fixed, for example by using actor metadata or team membership to separate maintainer pushes from external-contributor pushes.
- [Abstract and Section IV (RQ3)] The claim that "approximately 68% of these projects diverge significantly from the expected process" is never tied to a defined divergence measure. In the RQ3 results, 68% appears as the proportion of projects that reopen pull requests, which is presented as an undocumented practice rather than a guideline violation, and separately as the proportion of projects whose guidelines describe pull-request-based contribution. The paper does not show, step by step, how the observed mismatches (direct commits, issue reopening, pull-request reopening, traceability findings) are combined into a per-project divergence score or how the 68% figure is derived. The authors should define an operational divergence metric, specify the threshold for "significant" divergence, and report the distribution of that metric across the 53 projects. Without such a derivation, the headline number is not verifiable.
- [Section V and Section VII] The conclusion states that "we demonstrate that the most active projects that use CI in fact do not follow their own guidelines," but the body of the paper provides only qualitative evidence of mismatches, and Section V explicitly characterizes the finding as "preliminary evidence" and uses phrases such as "we speculate that the current contribution guideline structure may be written to suit project maintainers more than new contributors." The wording in Section VII overstates what the data can support. The authors should calibrate the conclusion to the operational divergence measure they define and to the limitations acknowledged in Section VI, distinguishing documented-practice gaps from direct violations of the guidelines.
minor comments (4)
- [Section IV (RQ3)] The sentence "the mean number of direct commits is 93%" should be rephrased as "the mean percentage of commits made directly to the master branch is 93%" to avoid confusion about whether the number is a count or a proportion.
- [Section IV (RQ3)] The statement "Fourteen projects (19.5%) prescribed linking artifacts" appears to use the 72-project denominator from the guideline-coding sample, but this result is reported in the RQ3 section that analyzes the 53 projects with event-stream data. The authors should clarify which denominator is used and, if the prescription is from the 72-project set, connect it explicitly to the 53-project set for which workflow observations are available.
- [Section IV (RQ3)] The number 68% appears twice with different referents: 68% of projects reopen pull requests and about 68% of projects prescribe pull-request-based contribution. The paper should make explicit that these are two different observations, and the numerical coincidence should not be used implicitly to support the abstract's 68% divergence claim.
- [Section III.C] Figure 1 is referenced as an example process map, but the figure itself is not described in enough detail in the text; a brief caption or a sentence explaining the nodes and edges would improve readability.
Circularity Check
No significant circularity: guidelines and GitHub event traces are measured independently, and no prediction is fitted to or defined in terms of the guidelines.
full rationale
The paper's central comparison is between two independently collected artifacts: contribution guidelines (CONTRIBUTING.md/README.md files) that were manually coded, and GitHub Events API activity traces that were mined and visualized as process maps. No parameter is fitted from the guidelines to the event data, and no outcome is defined in terms of the guidelines; the divergence claims simply juxtapose observed events (e.g., direct commits to master, reopened pull requests) with prescribed processes. The RQ3 statistics—93% mean direct commits, 68% reopening pull requests—are descriptive counts of event data, not quantities derived from the coding. The coding process was compared with prior work by Prana et al. and Gousios et al., but that is validation of the coding scheme, not circularity. The manuscript's own Section VI caveat that guidelines 'should apply to both core team members as well as external contributors. However, this is not always the case' identifies a potential interpretive threat regarding who direct commits belong to, but that is a validity limitation, not a circular derivation: the guidelines and the event stream are still measured separately. No self-citation is load-bearing; cited prior work supplies sampling criteria, background, and coding comparisons, not the target result.
Assumptions & free parameters
assumptions (4)
- domain assumption CONTRIBUTING.md or README.md is the authoritative contribution process documentation for the sampled projects.
- domain assumption GitHub event API traces over a four-week window accurately represent a project's contribution process.
- ad hoc to paper Direct commits to the master branch are instances of the contribution process that should match the guidelines.
- domain assumption Disco process maps are a faithful representation of the event logs and support the claimed divergence characterization.
Cite this review
Pith. "Pith review of Do as I Do, Not as I Say: Do Contribution Guidelines Match the GitHub Contribution Process?." pith.science (2026). https://pith.science/paper/OVLMY4IF
@misc{pith2026190802320,
author = {Pith},
title = {Pith review of: Do as I Do, Not as I Say: Do Contribution Guidelines Match the GitHub Contribution Process?},
year = {2026},
howpublished = {\url{https://pith.science/paper/OVLMY4IF}},
note = {Machine review of arXiv:1908.02320}
}
read the original abstract
Developer contribution guidelines are used in social coding sites like GitHub to explain and shape the process a project expects contributors to follow. They set standards for all participants and "save time and hassle caused by improperly created pull requests or issues that have to be rejected and resubmitted" (GitHub). Yet, we lack a systematic understanding of the content of a typical contribution guideline, as well as the extent to which these guidelines are followed in practice. Additionally, understanding how guidelines may impact projects that use Continuous Integration as part of the contribution process is of particular interest. To address this knowledge gap, we conducted a mixed-methods study of 53 GitHub projects with explicit contribution guidelines and coded the guidelines to extract key themes. We then created a process model using GitHub activity data (e.g., commit, new issue, new pull request) to compare the actual activity with the prescribed contribution guidelines. We show that approximately 68% of these projects diverge significantly from the expected process.
Figures
Reference graph
Works this paper leans on
-
[21]
Measuring and analyzing code authorship in 1+ 118 open source projects,
G. Avelino, L. Passos, A. Hora, and M. T. Valente, “Measuring and analyzing code authorship in 1+ 118 open source projects,” Science of Computer Programming, vol. 176, pp. 14–32, 2019
work page 2019
-
[1]
Work practices and challenges in pull-based development: the contributor’s perspective,
G. Gousios, M.-A. Storey, and A. Bacchelli, “Work practices and challenges in pull-based development: the contributor’s perspective,” in ICSE. IEEE, 2016, pp. 285–296
work page 2016
-
[2]
“Open source project guides,” https://opensource.guide/ starting-a-project/#your-pre-launch-checklist, accessed: 2019-06-10
work page 2019
-
[3]
Setting guidelines for repository contributors,
“Setting guidelines for repository contributors,” https://help.github. com/en/articles/setting-guidelines-for-repository-contributors, accessed: 2019-06-10
work page 2019
-
[4]
“Contributing guidelines,” https://github.blog/ 2012-09-17-contributing-guidelines, accessed: 2019-06-10
work page 2012
-
[5]
A systematic literature review on the barriers faced by newcomers to open source software projects,
I. Steinmacher, M. A. G. Silva, M. A. Gerosa, and D. F. Redmiles, “A systematic literature review on the barriers faced by newcomers to open source software projects,” IST, vol. 59, pp. 67–85, 2015
work page 2015
-
[6]
Oops, my tests broke the build: an explorative analysis of Travis CI with GitHub,
M. Beller, G. Gousios, and A. Zaidman, “Oops, my tests broke the build: an explorative analysis of Travis CI with GitHub,” inMSR. IEEE, 2017, pp. 356–367
work page 2017
-
[7]
Quality and productivity outcomes relating to continuous integration in github,
B. Vasilescu, Y . Yu, H. Wang, P. Devanbu, and V . Filkov, “Quality and productivity outcomes relating to continuous integration in github,” in FSE. ACM, 2015, pp. 805–816
work page 2015
Show all 21 references
-
[8]
How github contributing.md con- tributes to contributors,
N. Kobayakawa and K. Yoshida, “How github contributing.md con- tributes to contributors,” in COMPSAC. IEEE, 2017, pp. 694–696
2017
-
[9]
Catego- rizing the content of github readme files,
G. A. A. Prana, C. Treude, F. Thung, T. Atapattu, and D. Lo, “Catego- rizing the content of github readme files,” EMSE, pp. 1–32, 2018
2018
-
[10]
Continuous integration (original version),
M. Fowler and M. Foemmel, “Continuous integration (original version),” available from, http://www.martinfowler.com/) Accessed: 2019-06-07
2019
-
[11]
Continuous integration certification,
“Continuous integration certification,” https://martinfowler.com/bliki/ ContinuousIntegrationCertification.html, accessed: 2019-06-07
2019
-
[12]
Github actions,
“Github actions,” https://github.com/features/actions, accessed: 2019-06- 10
2019
-
[13]
Github welcomes all ci tools,
“Github welcomes all ci tools,” https://github.blog/ 2017-11-07-github-welcomes-all-ci-tools/, accessed: 2019-06-11
2017
-
[14]
Work practices and challenges in pull-based development: the integrator’s perspective,
G. Gousios, A. Zaidman, M.-A. Storey, and A. van Deursen, “Work practices and challenges in pull-based development: the integrator’s perspective,” in ICSE. IEEE, 2015, pp. 358–368
2015
-
[15]
The GHTorrent dataset and tool suite,
G. Gousios, “The GHTorrent dataset and tool suite,” in Working Conf. on Mining Software Repositories (MSR) . IEEE, 2013, pp. 233–236
2013
-
[16]
Influence of social and technical factors for evaluating contribution in github,
J. Tsay, L. Dabbish, and J. Herbsleb, “Influence of social and technical factors for evaluating contribution in github,” in ICSE. ACM, 2014, pp. 356–366
2014
-
[17]
Curating github for engineered software projects,
N. Munaiah, S. Kroh, C. Cabrey, and M. Nagappan, “Curating github for engineered software projects,” EMSE, vol. 22, pp. 3219–3253, 2017
2017
-
[18]
The promises and perils of mining github,
E. Kalliamvakou, G. Gousios, K. Blincoe, L. Singer, D. M. German, and D. Damian, “The promises and perils of mining github,” in MSR. ACM, 2014, pp. 92–101
2014
-
[19]
How open source projects use static code analysis tools in continuous integration pipelines,
F. Zampetti, S. Scalabrino, R. Oliveto, G. Canfora, and M. Di Penta, “How open source projects use static code analysis tools in continuous integration pipelines,” in MSR. IEEE, 2017, pp. 334–344
2017
-
[20]
J. W. Creswell and J. D. Creswell, Research design: Qualitative, quantitative, and mixed methods approaches . Sage publications, 2017
2017
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.