{"id":"280b13b3-2cff-4735-b96a-bdb2c7b29894","arxiv_id":"2506.00142","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A systematic literature review of 42 studies shows open source diversity research is heavily skewed to binary gender, with age, ethnicity, disability, and neurodiversity largely unstudied.","lead":"This paper reviews 42 prior studies on underrepresented groups in open source software, finding that most focus on gender and that age, ethnicity, disability, and neurodiversity receive little attention. It maps research gaps and classifies inclusion recommendations for open source communities.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central negative claim is undercut by the paper's own P33: a cognitive-diversity OSS study that plausibly falls under the paper's definition of neurodiversity, so 'neurodiversity has not been studied' is not safe as stated.","rationale":"The reader's weakest assumption correctly identified search completeness and demographic classification reliability as the load-bearing point. My concern sharpens that into a concrete, internal problem: the paper's own included study P33 appears to be a neurodiversity-relevant study under the definition given in Section 2.2, yet it is categorized as 'Other' while the abstract and conclusion assert that neurodiversity has never been studied in OSS. That is not an attack on the authors; it is a classification and reporting inconsistency that directly affects the central gap claim. The numerical mismatch between 62.3% and 33 of 42 is also real and reinforces the need for correction, but the qualitative conclusion that gender dominates would likely survive either count. By contrast, if P33 or additional neurodiversity studies exist, the paper's most novel contribution—identifying neurodiversity as a completely unstudied dimension—loses its factual basis. The proposed test is inexpensive: read P33, rerun the search with broader neurodiversity terms, and recompute the counts. I therefore keep the reader's CONDITIONAL verdict; the condition is that the authors must reclassify P33 and re-run the search before the gap claims can be trusted.","tokens_in":15717,"tokens_out":4421,"duration_ms":43503,"concrete_test":"Open P33's full text and determine whether it studies participants who identify as neurodivergent or conditions covered by the paper's neurodiversity definition (ASD, ADHD, dyslexia). Independently re-run the Section 3.1 automated search in ACM/IEEE/Scopus with the same inclusion criteria but adding 'neurodiverg*' OR 'autis*' OR 'ADHD' OR 'dyslexia' AND 'open source'. If P33 or any newly retrieved paper qualifies, recompute the RQ1 dimension counts and revise the abstract and conclusion statements about neurodiversity. Also recompute the gender percentage from the verified 42-paper set in Table 1 to resolve the 62.3% versus 33-of-42 inconsistency.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Search completeness and dimension classification are load-bearing because the abstract's central negative claims—'neurodiversity ... have not been studied' and 'papers about disability and neurodiversity were not found'—are absence claims. The paper itself contains a possible counterexample to that absence. Section 2.2 defines neurodiversity as 'natural variations in brain function, including ASD, ADHD, and dyslexia.' P33, 'Designing for Cognitive Diversity: Improving the GitHub Experience for Newcomers' (2023), is included in Table 1 and discussed in Section 4.3 as a study of participants with 'alternative cognitive styles' who face inclusion bugs in GitHub interfaces. The paper never gives a rule for why P33 is filed under 'Other' rather than under neurodiversity. If P33's cognitive diversity falls within the paper's own neurodiversity definition, then the strongest gap claim is false as written, and at minimum the RQ1 dimension counts need reclassification. The search string includes 'neurodivers*' but not 'neurodiverg*', 'autis*', 'ADHD', or 'dyslexia', so the absence could be an artifact of the search rather than of the literature. Additionally, 33/42 is 78.6%, not the 62.3% stated in the abstract; if 62.3% is the intended figure, the gender-focused count is about 26, materially changing the RQ1 numbers. These are internal inconsistencies, not merely disagreements with prior work.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript reports a systematic literature review of 42 studies on underrepresented groups in open source software (OSS), organized around four research questions: which groups are studied, which methods are used, what experiences are reported, and what inclusion practices are recommended. It claims that gender is by far the most studied dimension, that age and ethnicity receive little attention, that disability and neurodiversity have not been studied in OSS, and that only six included papers offer explicit concrete recommendations. The review follows a conventional SLR structure with a search protocol, dual screening, quality assessment, and thematic synthesis, and it provides a replication link.","tokens_in":15976,"tokens_out":5953,"duration_ms":57153,"significance":"The topic is timely and the review addresses a real gap in the software engineering literature. If the mapping and gap claims are correct, the paper would be a useful reference for researchers and OSS maintainers, and it would extend prior SLRs by focusing specifically on OSS contexts. Strengths include the use of established SLR guidelines, a documented protocol with a replication link, dual screening with reported agreement checks, a quality assessment instrument, and thematic synthesis of the included studies. The central contribution is descriptive rather than predictive, so circularity is not a concern. However, the headline gap claims are currently undermined by internal inconsistencies in the reported counts and by a likely miscategorization of an included study that falls within the paper's own definition of neurodiversity; the significance of the contribution cannot be fully assessed until these load-bearing issues are corrected.","major_comments":[{"comment":"The reported counts for gender are internally inconsistent. The abstract states that most papers focus on gender (62.3%), but Section 4.1 reports 33 of 42 papers (78.6%), and the detailed category counts in the same section (1 Age + 4 Culture + 2 Ethnicity + 7 Other = 14 non-gender papers) do not sum with the 33 gender papers to 42 (the total is 47). The statement that “Nine papers investigated other topics” also contradicts the 14 non-gender classifications. Because RQ1 is the central mapping result, the authors must correct the counts, state whether papers can be assigned to multiple dimensions, and ensure the abstract matches the body.","section":"§4.1 / Abstract, RQ1"},{"comment":"The paper’s strongest negative claim—that neurodiversity has not been studied in OSS—is contradicted by an included study under the paper’s own definition. Section 2.2 defines neurodiversity as “natural variations in brain function, including ASD, ADHD, and dyslexia,” yet Table 1 lists P33, “Designing for Cognitive Diversity: Improving the GitHub Experience for Newcomers” (2023), and Section 4.3 describes it as a study of participants with “alternative cognitive styles” facing inclusion bugs in GitHub interfaces. The paper files P33 under “Other” without giving a rule for why cognitive diversity is not neurodiversity. Either P33 should be reclassified, or the definition of neurodiversity should be narrowed with justification; otherwise the abstract’s “neurodiversity ... have not been studied” and the conclusion’s “Papers about disability and neurodiversity were not found” are false as written.","section":"§2.2, §4.1, Table 1, P33"},{"comment":"The absence of neurodiversity and disability studies may be an artifact of the search rather than a property of the literature. The automated search string in Section 3.1 includes only “neurodivers*” and “disabilit*”, not “neurodiverg*”, “autis*”, “ADHD”, “dyslexia”, “accessib*”, or related synonyms, and the manual search in Section 3.1 is limited to the last five years in a fixed venue list. Because the central contribution is a gap analysis, the authors should either expand the search terms and rerun the synthesis or explicitly justify why the chosen terms capture all relevant disability and neurodiversity research.","section":"§3.1 / search string"},{"comment":"The kappa statistics are misreported. Section 3.2 first reports a kappa of 0.60 described as moderate agreement for the exclusion stage, then reports a kappa of 0.061 described as “substantial agreement” for the inclusion stage. A value of 0.061 is slight agreement, not substantial; substantial agreement is usually taken as roughly 0.61–0.80. The reliability evidence for the inclusion decisions that determine the final 42-paper set should be corrected and reinterpreted, since the current text overstates the reproducibility of the selection process.","section":"§3.2 / inter-rater reliability"}],"minor_comments":[{"comment":"The sentence “The neurodiversity dimension, have not been studied in the context of OSS” contains a comma splice and a subject-verb agreement error; it should read “The neurodiversity dimension has not been studied in the context of OSS.”","section":"Abstract, Results"},{"comment":"The word “jornals” should be “journals,” and the roadmap in Figure 1 shows “42 studies70 studies” in one label while the source-specific counts (378, 110, 218) are not defined in the caption.","section":"§3.1 / Figure 1"},{"comment":"The identifiers “P021”, “P035”, and “P031” should be “P21”, “P35”, and “P31” to match Table 1.","section":"§4.4"},{"comment":"The restriction to English-language papers is described in the external validity paragraph as both a possible limitation and a possible internal threat; the authors should state clearly which threat it is and explain the distinction.","section":"§5.4"},{"comment":"Table 1 does not include a column showing the diversity dimension(s) assigned to each paper, which makes the RQ1 counts in Section 4.1 difficult for readers to verify.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the scope of EASE and the systematic review format is appropriate. The main risk is not novelty but internal consistency: the count arithmetic, the abstract percentages, the kappa reporting, and the P33 classification all need correction before the headline gap claims can be trusted. These issues appear fixable within the manuscript's scope, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe headline: this is a competent but sloppy systematic review. The useful contribution is an OSS-specific map of diversity research—42 papers, seven dimensions, a gap analysis, and a seven-category classification of recommendations. The gender-dominates finding holds up. The method is standard and mostly sound: dual screening, kappa, quality assessment, thematic synthesis. That is real value for a niche subfield.\n\nThe soft spots are in the absence claims and the numbers. The abstract says 62.3% of papers focus on gender; the body says 33 of 42, which is 78.6%. Those cannot both be right, and the discrepancy changes the RQ1 summary. The kappa is reported as 0.60 moderate at one stage and 0.061 'substantial' at another—presumably a typo, but as printed it reads as broken.\n\nMore importantly, the paper claims neurodiversity has not been studied in OSS, yet P33 (Designing for Cognitive Diversity) is included and filed under 'Other'. The paper's own definition of neurodiversity covers natural variations in brain function including ASD, ADHD, and dyslexia. The authors never explain why cognitive diversity is not neurodiversity, and the search string omits terms like autis*, ADHD, and dyslexia. So the absence claim is not safe as written. Either P33 counts as neurodiversity, which defeats the claim, or it does not, and the authors need to state the rule. The disability claim is similarly fragile because the search only used disabilit*.\n\nThese are fixable internal inconsistencies, not deep conceptual flaws. The qualitative takeaway that gender dominates and other dimensions are neglected probably survives a reclassification, but the abstracts and conclusions as written overstate what the data shows.\n\nThis paper is for researchers planning work on diversity in OSS, especially on understudied dimensions. It deserves a serious referee; an editor should send it out. A reviewer should ask for a corrected abstract, a reconciled count for RQ1, and an explicit classification decision on P33 before the absence claims are trusted.","headline":"A useful OSS-specific diversity review whose headline absence claims are undercut by its own P33 and unreconciled numbers—fixable, but needs revision.","tokens_in":16522,"tokens_out":3478,"would_cite":false,"duration_ms":31687,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A systematic review of 42 studies finds that research on underrepresented groups in open source software is heavily skewed toward a binary view of gender, leaving age, ethnicity, disability, sexual orientation, and neurodiversity almost…","keywords":["underrepresented groups","open source software","diversity","systematic literature review","gender binary","neurodiversity","inclusion","software engineering"],"falsifier":"Run the same search protocol with additional databases and without the English-only restriction; if that replication surfaces multiple peer-reviewed empirical studies of disability, neurodiversity, or sexual orientation in OSS published before September 2024, or a substantial share of gender studies using non-binary categories, the paper's central gap claims would be contradicted.","tokens_in":15483,"feed_emoji":"🧩","tokens_out":9862,"duration_ms":93567,"temperature":0.7,"pith_summary":"This paper tries to establish what the research literature actually says about underrepresented groups in open source software (OSS). It claims the literature is lopsided: 33 of the 42 reviewed papers focus on gender, while age and ethnicity appear in only a handful of studies, and disability, sexual orientation, and neurodiversity are absent. It further claims that every gender study reviewed adopts a binary male/female frame and that only six papers offer explicit inclusion recommendations. If these claims hold, the review supplies a map of where the evidence base is empty and where inclusion practice is running ahead of research.","feed_headline":"Open source diversity research is lopsided: gender dominates","feed_subtitle":"A review of 42 studies finds age, ethnicity, disability, and neurodiversity nearly absent, and few concrete inclusion fixes.","key_machinery":"The machinery is a systematic literature review organized around a seven-dimension diversity framework: gender, sexual orientation, culture, ethnicity, age, disability, and neurodiversity. The authors assembled a corpus by searching three digital libraries and manually scanning top software-engineering venues, screened 733 documents down to 42 full peer-reviewed papers, then read each paper in full and extracted data with a 14-question form and thematic synthesis. The dimension framework carries the argument: by sorting every paper into at least one dimension, it converts a qualitative literature into a gap map that shows at a glance which groups have been studied and which have not.","core_discovery":"The central claim is that the empirical literature on underrepresented groups in OSS is narrow in three connected ways. Gender dominates the corpus, appearing in 33 of the 42 papers, and all of those studies work within a male/female binary. Age appears in one paper, ethnicity in two, culture in four; disability, sexual orientation, and neurodiversity have no dedicated empirical studies. Only six papers make explicit recommendations, most tied to gender and one to age. The reviewed studies do document real barriers—discrimination, bias in contribution evaluation, hostile environments, and onboarding difficulties—alongside benefits such as more frequent contributions from racially diverse groups, better communication, and more welcoming communities. The paper reads this combination as evidence that the research community has concentrated on a single, narrowly framed dimension while leaving most dimensions of human difference unexamined.","pith_inferences":["Beyond the paper: if the absence of disability and neurodiversity research is real, a direct survey of OSS maintainers and contributors asking about disability and neurodivergence status would test whether these groups are actually absent from OSS or just absent from the literature.","Beyond the paper: the review's English-only, three-library search means the global picture could be less gender-centric than it appears; replicating in other languages and databases is a low-cost way to check the central gap claim.","Beyond the paper: the finding that only six papers give explicit recommendations suggests that accessibility and inclusion guidance from adjacent fields, such as screen-reader support or flexible communication norms, could be systematically adapted to OSS governance and then evaluated as interventions."],"forward_implications":["If the gap map is correct, the next wave of OSS diversity research should target age, ethnicity, disability, sexual orientation, and neurodiversity, where the evidence base is nearly empty.","Gender studies in OSS should treat gender as more than a binary, since every one of the 33 gender papers reviewed uses a male/female framing.","The field should shift from describing barriers to testing interventions: only six papers offer recommendations, and their effectiveness is unmeasured.","Automated name-based tools for inferring gender and ethnicity should be replaced or supplemented by privacy-conscious methods, because misclassification is a known problem.","Practitioners can treat the seven recommendation groups (codes of conduct, mentorship, safe spaces, recognition, evaluation toolkits, better environments, self-improvement) as a provisional checklist until stronger evidence arrives."],"supporting_citations":[{"why":"Supplies the systematic-review guidelines that define the protocol.","marker":"[20]"},{"why":"Supplies the method for identifying relevant primary studies.","marker":"[45]"},{"why":"The prior software-engineering diversity review this study extends and compares against.","marker":"[31]"},{"why":"Prior review of diversity in software teams used as a comparison point.","marker":"[21]"},{"why":"Prior study of gender differences in public code contributions used to anchor the gender-centric comparison.","marker":"[44]"},{"why":"Prior work on geographic gender inclusion in OSS used to position the review's broader scope.","marker":"[29]"},{"why":"Source of the evidence linking gender and tenure diversity to team productivity.","marker":"[40]"},{"why":"Source for the evidence on race/ethnicity bias in evaluating OSS contributions.","marker":"[22]"},{"why":"The single age-focused study in the corpus, whose lone presence grounds the claim that age is rarely studied.","marker":"P21"},{"why":"Supplies the code-of-conduct practice evidence used in the recommendations discussion.","marker":"[38]"}],"fun_headline_variants":["OSS diversity research: 33 of 42 papers focus on gender","Open source diversity: gender binary dominates, other groups ignored","Open source inclusion research is stuck on gender binary","Diversity in OSS: almost all research is about gender","Gender studies dominate open source diversity research"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The gap map depends on the search catching all relevant published work: if meaningful studies appeared in languages other than English or in venues not covered by the chosen digital libraries, the claimed absences could be artifacts of the search rather than real gaps in the literature.","fun_headline_variants_meta":{"raw":{"variants":["OSS diversity research: 33 of 42 papers focus on gender","Open source diversity: gender binary dominates, other groups ignored","Open source inclusion research is stuck on gender binary","Diversity in OSS: almost all research is about gender","Gender studies dominate open source diversity research"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000664,"raw_usage":{"total_tokens":3021,"prompt_tokens":922,"completion_tokens":2099,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":538,"completion_tokens_details":{"reasoning_tokens":2021}},"tokens_in":538,"tokens_out":2099,"duration_ms":16525,"temperature":1.0,"reasoning_tokens":2021,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:10:15.460592+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same search protocol with additional databases and without the English-only restriction; if that replication surfaces multiple peer-reviewed empirical studies of disability, neurodiversity, or sexual orientation in OSS published before September 2024, or a substantial share of gender studies using non-binary categories, the paper's central gap claims would be contradicted.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the systematic-review guidelines that define the protocol."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the method for identifying relevant primary studies."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The prior software-engineering diversity review this study extends and compares against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Prior review of diversity in software teams used as a comparison point."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Prior study of gender differences in public code contributions used to anchor the gender-centric comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Prior work on geographic gender inclusion in OSS used to position the review's broader scope."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Source of the evidence linking gender and tenure diversity to team productivity."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Source for the evidence on race/ethnicity bias in evaluating OSS contributions."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the code-of-conduct practice evidence used in the recommendations discussion."}],"review_version":1}