REVIEW 4 major objections 5 minor 46 references
Understanding Underrepresented Groups in Open Source Software
T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read A systematic review of 42 studies finds that research on underrepresented groups in open source software is heavily skewed toward a binary view of gender, leaving age, ethnicity, disability, sexual orientation, and neurodiversity almost…
desk verdict A useful OSS-specific diversity review whose headline absence claims are undercut by its own P33 and unreconciled numbers—fixable, but needs revision. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery is a systematic literature review organized around a seven-dimension diversity framework: gender, sexual orientation, culture, ethnicity, age, disability, and neurodiversity. The authors assembled a corpus by searching three digital libraries and manually scanning top software-engineering venues, screened 733 documents down to 42 full peer-reviewed papers, then read each paper in full and extracted data with a 14-question form and thematic synthesis. The dimension framework carries the argument: by sorting every paper into at least one dimension, it converts a qualitative literature into a gap map that shows at a glance which groups have been studied and which have not.
What would settle it
Run the same search protocol with additional databases and without the English-only restriction; if that replication surfaces multiple peer-reviewed empirical studies of disability, neurodiversity, or sexual orientation in OSS published before September 2024, or a substantial share of gender studies using non-binary categories, the paper's central gap claims would be contradicted.
Extended reading notes
Core claim
The central claim is that the empirical literature on underrepresented groups in OSS is narrow in three connected ways. Gender dominates the corpus, appearing in 33 of the 42 papers, and all of those studies work within a male/female binary. Age appears in one paper, ethnicity in two, culture in four; disability, sexual orientation, and neurodiversity have no dedicated empirical studies. Only six papers make explicit recommendations, most tied to gender and one to age. The reviewed studies do document real barriers—discrimination, bias in contribution evaluation, hostile environments, and onboarding difficulties—alongside benefits such as more frequent contributions from racially diverse groups, better communication, and more welcoming communities. The paper reads this combination as evidence that the research community has concentrated on a single, narrowly framed dimension while leaving most dimensions of human difference unexamined.
Load-bearing premise
The gap map depends on the search catching all relevant published work: if meaningful studies appeared in languages other than English or in venues not covered by the chosen digital libraries, the claimed absences could be artifacts of the search rather than real gaps in the literature.
Editorial extensions
If this is right
- If the gap map is correct, the next wave of OSS diversity research should target age, ethnicity, disability, sexual orientation, and neurodiversity, where the evidence base is nearly empty.
- Gender studies in OSS should treat gender as more than a binary, since every one of the 33 gender papers reviewed uses a male/female framing.
- The field should shift from describing barriers to testing interventions: only six papers offer recommendations, and their effectiveness is unmeasured.
- Automated name-based tools for inferring gender and ethnicity should be replaced or supplemented by privacy-conscious methods, because misclassification is a known problem.
- Practitioners can treat the seven recommendation groups (codes of conduct, mentorship, safe spaces, recognition, evaluation toolkits, better environments, self-improvement) as a provisional checklist until stronger evidence arrives.
Reading between the lines
- Beyond the paper: if the absence of disability and neurodiversity research is real, a direct survey of OSS maintainers and contributors asking about disability and neurodivergence status would test whether these groups are actually absent from OSS or just absent from the literature.
- Beyond the paper: the review's English-only, three-library search means the global picture could be less gender-centric than it appears; replicating in other languages and databases is a low-cost way to check the central gap claim.
- Beyond the paper: the finding that only six papers give explicit recommendations suggests that accessibility and inclusion guidance from adjacent fields, such as screen-reader support or flexible communication norms, could be systematically adapted to OSS governance and then evaluated as interventions.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript reports a systematic literature review of 42 studies on underrepresented groups in open source software (OSS), organized around four research questions: which groups are studied, which methods are used, what experiences are reported, and what inclusion practices are recommended. It claims that gender is by far the most studied dimension, that age and ethnicity receive little attention, that disability and neurodiversity have not been studied in OSS, and that only six included papers offer explicit concrete recommendations. The review follows a conventional SLR structure with a search protocol, dual screening, quality assessment, and thematic synthesis, and it provides a replication link.
Significance. The topic is timely and the review addresses a real gap in the software engineering literature. If the mapping and gap claims are correct, the paper would be a useful reference for researchers and OSS maintainers, and it would extend prior SLRs by focusing specifically on OSS contexts. Strengths include the use of established SLR guidelines, a documented protocol with a replication link, dual screening with reported agreement checks, a quality assessment instrument, and thematic synthesis of the included studies. The central contribution is descriptive rather than predictive, so circularity is not a concern. However, the headline gap claims are currently undermined by internal inconsistencies in the reported counts and by a likely miscategorization of an included study that falls within the paper's own definition of neurodiversity; the significance of the contribution cannot be fully assessed until these load-bearing issues are corrected.
major comments (4)
- [§4.1 / Abstract, RQ1] The reported counts for gender are internally inconsistent. The abstract states that most papers focus on gender (62.3%), but Section 4.1 reports 33 of 42 papers (78.6%), and the detailed category counts in the same section (1 Age + 4 Culture + 2 Ethnicity + 7 Other = 14 non-gender papers) do not sum with the 33 gender papers to 42 (the total is 47). The statement that “Nine papers investigated other topics” also contradicts the 14 non-gender classifications. Because RQ1 is the central mapping result, the authors must correct the counts, state whether papers can be assigned to multiple dimensions, and ensure the abstract matches the body.
- [§2.2, §4.1, Table 1, P33] The paper’s strongest negative claim—that neurodiversity has not been studied in OSS—is contradicted by an included study under the paper’s own definition. Section 2.2 defines neurodiversity as “natural variations in brain function, including ASD, ADHD, and dyslexia,” yet Table 1 lists P33, “Designing for Cognitive Diversity: Improving the GitHub Experience for Newcomers” (2023), and Section 4.3 describes it as a study of participants with “alternative cognitive styles” facing inclusion bugs in GitHub interfaces. The paper files P33 under “Other” without giving a rule for why cognitive diversity is not neurodiversity. Either P33 should be reclassified, or the definition of neurodiversity should be narrowed with justification; otherwise the abstract’s “neurodiversity ... have not been studied” and the conclusion’s “Papers about disability and neurodiversity were not found” are false as written.
- [§3.1 / search string] The absence of neurodiversity and disability studies may be an artifact of the search rather than a property of the literature. The automated search string in Section 3.1 includes only “neurodivers*” and “disabilit*”, not “neurodiverg*”, “autis*”, “ADHD”, “dyslexia”, “accessib*”, or related synonyms, and the manual search in Section 3.1 is limited to the last five years in a fixed venue list. Because the central contribution is a gap analysis, the authors should either expand the search terms and rerun the synthesis or explicitly justify why the chosen terms capture all relevant disability and neurodiversity research.
- [§3.2 / inter-rater reliability] The kappa statistics are misreported. Section 3.2 first reports a kappa of 0.60 described as moderate agreement for the exclusion stage, then reports a kappa of 0.061 described as “substantial agreement” for the inclusion stage. A value of 0.061 is slight agreement, not substantial; substantial agreement is usually taken as roughly 0.61–0.80. The reliability evidence for the inclusion decisions that determine the final 42-paper set should be corrected and reinterpreted, since the current text overstates the reproducibility of the selection process.
minor comments (5)
- [Abstract, Results] The sentence “The neurodiversity dimension, have not been studied in the context of OSS” contains a comma splice and a subject-verb agreement error; it should read “The neurodiversity dimension has not been studied in the context of OSS.”
- [§3.1 / Figure 1] The word “jornals” should be “journals,” and the roadmap in Figure 1 shows “42 studies70 studies” in one label while the source-specific counts (378, 110, 218) are not defined in the caption.
- [§4.4] The identifiers “P021”, “P035”, and “P031” should be “P21”, “P35”, and “P31” to match Table 1.
- [§5.4] The restriction to English-language papers is described in the external validity paragraph as both a possible limitation and a possible internal threat; the authors should state clearly which threat it is and explain the distinction.
- [Table 1] Table 1 does not include a column showing the diversity dimension(s) assigned to each paper, which makes the RQ1 counts in Section 4.1 difficult for readers to verify.
Circularity Check
No circularity: the paper is a systematic literature review whose claims are descriptive summaries of included primary studies, not derivations from fitted inputs or self-citation chains.
full rationale
This is a secondary study: the 42 analyzed papers are external primary studies, and the review's findings (dimension counts, methodological frequencies, barrier and recommendation themes) are synthesized from those studies' own reports rather than derived from a model whose parameters are fitted to the target conclusion. The search string and inclusion criteria define the sample, and the abstract's negative gap claims (e.g., no disability or neurodiversity studies) are absence claims that depend on search coverage and classification, but that is a validity limitation, not circular reasoning. The issues noted by the skeptical reader—the abstract's 62.3% figure being inconsistent with 33 of 42 papers on gender, and P33's cognitive-diversity content plausibly falling under the paper's own definition of neurodiversity—are internal consistency or construct-validity concerns about arithmetic and classification, not cases where a result is equivalent to its input by construction. No fitted parameter is renamed as a prediction, no uniqueness theorem is imported from the authors' prior work, and no known result is relabeled as unification. The paper is self-contained against external benchmarks: each summary claim cites specific included primary study IDs (e.g., P03, P10, P22), so the claims remain traceable to independent evidence rather than to the authors' own prior conclusions. Accordingly, the derivation chain does not reduce to its inputs, and the appropriate circularity score is 0.
Assumptions & free parameters
assumptions (4)
- domain assumption The search string and the three digital libraries (ACM, IEEE, Scopus) capture the relevant universe of papers on underrepresented groups in OSS.
- domain assumption The automated tools used by primary studies (GenderComputer, Namsor, NamePrism) produce valid gender and ethnicity classifications.
- ad hoc to paper Inclusion criterion IC3, requiring one of a fixed set of empirical methods, does not systematically exclude relevant studies.
- domain assumption Quality assessment conducted by a single author using the Dyba and Dingsoyr instrument reflects the strength of the evidence.
Cite this review
Pith. "Pith review of Understanding Underrepresented Groups in Open Source Software." pith.science (2026). https://pith.science/paper/5U7FTEUX
@misc{pith2026250600142,
author = {Pith},
title = {Pith review of: Understanding Underrepresented Groups in Open Source Software},
year = {2026},
howpublished = {\url{https://pith.science/paper/5U7FTEUX}},
note = {Machine review of arXiv:2506.00142}
}
read the original abstract
Context: Diversity can impact team communication, productivity, cohesiveness, and creativity. Analyzing the existing knowledge about diversity in open source software (OSS) projects can provide directions for future research and raise awareness about barriers and biases against underrepresented groups in OSS. Objective: This study aims to analyze the knowledge about minority groups in OSS projects. We investigated which groups were studied in the OSS literature, the study methods used, their implications, and their recommendations to promote the inclusion of minority groups in OSS projects. Method: To achieve this goal, we performed a systematic literature review study that analyzed 42 papers that directly study underrepresented groups in OSS projects. Results: Most papers focus on gender (62.3%), while others like age or ethnicity are rarely studied. The neurodiversity dimension, have not been studied in the context of OSS. Our results also reveal that diversity in OSS projects faces several barriers but brings significant benefits, such as promoting safe and welcoming environments. Conclusion: Most analyzed papers adopt a myopic perspective that sees gender as strictly binary. Dimensions of diversity that affect how individuals interact and function in an OSS project, such as age, tenure, and ethnicity, have received very little attention.
Figures
Reference graph
Works this paper leans on
-
[1]
Dirk Baecker. 1997. The meaning of culture. Thesis Eleven 51, 1 (1997), 37–51
work page 1997
-
[2]
Sogol Balali, Igor Steinmacher, Umayal Annamalai, Anita Sarma, and Marco Au- relio Gerosa. 2018. Newcomers’ barriers... is that all? an analysis of mentors’ and newcomers’ barriers in OSS projects. Computer Supported Cooperative Work (CSCW) 27 (2018), 679–714
work page 2018
-
[3]
Xu Ben, Shen Beijun, and Yang Weicheng. 2013. Mining developer contribution in open source software using visualization techniques. In 2013 Third International Conference on Intelligent System Design and Engineering Applications . IEEE, 934– 937
work page 2013
-
[4]
Amiangshu Bosu and Kazi Zakia Sultana. 2019. Diversity and inclusion in open source software (OSS) projects: Where do we stand?. In 2019 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM). IEEE, 1–11
work page 2019
-
[5]
Kevin Buffardi. 2016. Localized open source software projects: Exploring realism and motivation. In 2016 11th International Conference on Computer Science & Education (ICCSE). IEEE, 382–387
work page 2016
-
[6]
Gemma Catolino, Fabio Palomba, Damian A Tamburri, Alexander Serebrenik, and Filomena Ferrucci. 2019. Gender diversity and women in software teams: How do they affect community smells?. In 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Society (ICSE-SEIS) . IEEE, 11–20
work page 2019
-
[7]
Stephen Cornell and Douglas Hartmann. 2006. Ethnicity and race: Making identi- ties in a changing world . Sage Publications
work page 2006
-
[8]
Cánovas Izquierdo, and Jordi Cabot
Valerio Cosentino, Javier L. Cánovas Izquierdo, and Jordi Cabot. 2017. A System- atic Mapping Study of Software Development With GitHub. IEEE Access 5 (2017), 7173–7192. https://doi.org/10.1109/ACCESS.2017.2682323
Show all 46 references
-
[9]
Daniela S Cruzes and Tore Dyba. 2011. Recommended steps for thematic synthesis in software engineering. In 2011 international symposium on empirical software engineering and measurement. IEEE, 275–284
2011
-
[10]
Sherae Daniel, Ritu Agarwal, and Katherine J Stewart. 2013. The effects of diversity in global, distributed collectives: A study of open source project success. Information Systems Research 24, 2 (2013), 312–333
2013
-
[11]
Jennifer L Davidson, Rithika Naik, Umme Ayda Mannan, Amir Azarbakht, and Carlos Jensen. 2014. On older adults in free/open source software: reflections of contributors and community leaders. In2014 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) . IEEE
2014
-
[12]
Tore Dybå and Torgeir Dingsøyr. 2008. Empirical studies of agile software development: A systematic review. Information and software technology 50, 9-10 (2008)
2008
-
[13]
Nadia Eghbal. 2020. Working in public: the making and maintenance of open source software. Stripe Press
2020
-
[14]
F Fitzpatrick and EM Badley. 1996. An overview of disability. Rheumatology 35, 2 (1996), 184–187
1996
-
[15]
O’Reilly Media, Inc
Karl Fogel. 2005. Producing open source software: How to run a successful free software project. " O’Reilly Media, Inc. "
2005
-
[16]
Denae Ford, Reed Milewicz, and Alexander Serebrenik. 2019. How Remote Work Can Foster a More Inclusive Environment for Transgender Developers. In 2019 IEEE/ACM 2nd International Workshop on Gender Equality in Software Engineering (GE). 9–12
2019
-
[17]
Fosch-Villaronga, A
E. Fosch-Villaronga, A. Poulsen, R.A. Søraa, and B.H.M. Custers. 2021. A little bird told me your gender: Gender inferences in social media. Information Processing & Management 58, 3 (2021), 102541
2021
-
[18]
Hana Frluckaj, Laura Dabbish, David Gray Widder, Huilian Sophie Qiu, and James Herbsleb. 2022. Gender and Participation in Open Source Software Development. Proc. of the ACM on Human-Computer Interaction 6, CSCW2 (2022)
2022
-
[19]
Mariam Guizani, Amreeta Chatterjee, Bianca Trinkenreich, Mary Evelyn May, Geraldine J Noa-Guevara, Liam James Russell, Griselda G Cuevas Zambrano, Daniel Izquierdo-Cortazar, Igor Steinmacher, Marco A Gerosa, et al. 2021. The long road ahead: Ongoing challenges in contributing ...
2021
-
[20]
Barbara Ann Kitchenham, David Budgen, and Pearl Brereton. 2015. Evidence- based software engineering and systematic reviews . Vol. 4. CRC press
2015
-
[21]
Álvaro Menezes and Rafael Prikladnicki. 2018. Diversity in software engineering. In Proc. of the 11th International workshop on cooperative and human aspects of software engineering. 45–48
2018
-
[22]
Reza Nadri, Gema Rodríguez-Pérez, and Meiyappan Nagappan. 2021. On the relationship between the developer’s perceptible race and ethnicity and the evaluation of contributions in OSS. IEEE Transactions on Software Engineering 48, 8 (2021), 2955–2968
2021
-
[23]
Helen A Neville and Germine H Awad. 2014. Why racial color-blindness is myopic. (2014)
2014
-
[24]
Susmita Hema Padala, Christopher John Mendez, Luiz Felipe Dias, Igor Stein- macher, Zoe Steine Hanson, Claudia Hilderbrand, Amber Horvath, Charles Hill, Logan Dale Simpson, Margaret Burnett, et al . 2020. How gender-biased tools shape newcomer experiences in oss projects. IEEE...
2020
-
[25]
Amado M Padilla. 2008. Social cognition, ethnic identity, and ethnic specific strate- gies for coping with threat due to prejudice and discrimination. In Motivational aspects of prejudice and racism . Springer, 7–42
2008
-
[26]
Marie-Laure Peretti. 2015. GENDER, A VARIABLE IN THE CONSTRUCTION OF SEXUAL IDENTITY: CONTRIBUTIONS, PHANTOMS, EVOLUTION. Revista Sexología y Sociedad 20, 2 (2015)
2015
-
[27]
Richard A Posner. 1995. Aging and old age . University of Chicago Press
1995
-
[28]
Rafa Prado, Wendy Mendes, Kiev S Gama, and Gustavo Pinto. 2020. How Trans- Inclusive Are Hackathons? IEEE Software 38, 2 (2020), 26–31
2020
-
[29]
Gede Artha Azriadi Prana, Denae Ford, Ayushi Rastogi, David Lo, Rahul Pu- randare, and Nachiappan Nagappan. 2021. Including everyone, everywhere: Understanding opportunities and challenges of geographic gender-inclusion in oss. IEEE Transactions on Software Engineering (2021)
2021
-
[30]
Marie-Élène Roberge and Rolf Van Dick. 2010. Recognizing the benefits of diversity: When and how does diversity increase group performance? Human Resource management review 20, 4 (2010), 295–308
2010
-
[31]
Gema Rodríguez-Pérez, Reza Nadri, and Meiyappan Nagappan. 2021. Perceived diversity in software engineering: a systematic literature review. Empirical Software Engineering 26, 5 (2021), 1–38
2021
-
[32]
Bastin Tony Roy Savarimuthu, Zoofishan Zareen, Jithin Cheriyan, Muhammad Yasir, and Matthias Galster. 2023. Barriers for Social Inclusion in Online Soft- ware Engineering Communities - A Study of Offensive Language Use in Gitter Projects. In Proc. of the 27th International Con...
2023
-
[33]
Ankita Saxena. 2014. Workforce diversity: A key to improve productivity.Procedia economics and finance 11 (2014), 76–85
2014
-
[34]
Lynn M Shore, Beth G Chung-Herrera, Michelle A Dean, Karen Holcombe Ehrhart, Don I Jung, Amy E Randel, and Gangaram Singh. 2009. Diversity in organizations: Where are we now and where are we going?Human resource management review 19, 2 (2009), 117–133
2009
-
[35]
Edmund Sonuga-Barke and Anita Thapar. 2021. The neurodiversity concept: is it helpful for clinicians and scientists? The Lancet Psychiatry 8, 7 (2021), 559–561
2021
-
[36]
Flannery G Stevens, Victoria C Plaut, and Jeffrey Sanchez-Burks. 2008. Unlocking the benefits of diversity: All-inclusive multiculturalism and positive organiza- tional change. The journal of applied behavioral science 44, 1 (2008)
2008
-
[37]
Kathleen Stock. 2019. XIV—Sexual Orientation: What Is It?. In Proc. of the Aris- totelian Society, Vol. 119. Oxford University Press, 295–319
2019
-
[38]
Parastou Tourani, Bram Adams, and Alexander Serebrenik. 2017. Code of conduct in open source projects. In 2017 IEEE 24th international conference on software analysis, evolution and reengineering (SANER) . IEEE, 24–33
2017
-
[39]
European Union. 2018. General Data Protection Regulation (GDPR). Legislation. https://gdpr.eu/
2018
-
[40]
Bogdan Vasilescu, Daryl Posnett, Baishakhi Ray, Mark GJ van den Brand, Alexan- der Serebrenik, Premkumar Devanbu, and Vladimir Filkov. 2015. Gender and tenure diversity in GitHub teams. In Proc. of the 33rd annual ACM conference on human factors in computing systems . 3789–3798
2015
-
[41]
Balazs Vedres and Orsolya Vasarhelyi. 2019. Gendered behavior as a disadvantage in open source software development. EPJ Data Science 8, 1 (2019), 25
2019
-
[42]
Minority
Erik Wingrove-Haugland and Jillian McLeod. 2021. Not “Minority” but “Minori- tized”. Teaching Ethics 21, 1 (2021)
2021
-
[43]
George Yancey. 2003. Who Is White?: Latinos, Asians, and the New Black/Nonblack Divide. Lynne Rienner Pub
2003
-
[44]
Stefano Zacchiroli. 2020. Gender differences in public code contributions: a 50-year perspective. IEEE Software 38, 2 (2020), 45–50
2020
-
[45]
He Zhang, Muhammad Ali Babar, and Paolo Tell. 2011. Identifying relevant studies in software engineering. Information and Software Technology 53, 6 (2011), 625–637
2011
-
[46]
Xin Zhou, Yuqin Jin, He Zhang, Shanshan Li, and Xin Huang. 2016. A map of threats to validity of systematic literature reviews in software engineering. In 2016 23rd Asia-Pacific Software Engineering Conference (APSEC) . IEEE
2016
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.