Pith. sign in

REVIEW 5 major objections 6 minor 56 references

The Landscape of Toxicity: An Empirical Investigation of Toxicity on GitHub

T0 review · 5 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that toxicity in GitHub pull-request comments clusters in measurable contexts—profanity dominates, popularity and gaming raise it, issue resolution and corporate sponsorship lower it—and that repeat offenders both…

desk verdict The descriptive map of GitHub PR toxicity is the strongest part; the repeat-offender claim in RQ4 is undermined by using the same classifier for predictor and outcome, and the abstract overstates the gaming odds ratio. read the letter →

arxiv 2502.08238 v1 pith:CE227TCR submitted 2025-02-12 cs.SE

Jaydeb Sarker , Asif Kamal Turzo , Amiangshu Bosu This is my paper · ORCID
classification cs.SE
keywords toxicityGitHubpullrequestreviewsopensourcesoftwareempiricalengineeringdetectiononlineharassmentregressionanalysis
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper attempts to establish, with a large stratified sample of 2,828 GitHub open-source projects and 101.5 million pull-request comments, which measurable features of projects, pull requests, and participants are associated with toxic communication. It reports that profanity is the most common form of toxicity, that popularity and gaming domains are positively associated with toxicity, while issue-resolution rate and corporate sponsorship are negatively associated, and that contributors with a history of toxic comments are more likely to author them again and to become targets themselves. If correct, these associations give maintainers concrete, measured levers—responsiveness, review speed, sponsorship context—rather than anecdote, for reducing hostile interactions.

What carries the argument

The argument runs on three linked instruments. First, ToxiCR, a software-engineering-specific toxicity classifier, labels all 101.5 million comments; its precision (88.8% on 600 comments that it marked toxic) and recall (87% on 896 relabeled locked-issue comments) are checked against human labels. Second, a stratified random sample of 2,828 projects by pull-request frequency, plus an oversample of 439 gaming projects, gives the corpus breadth beyond the locked-issue or rejected-patch contexts of earlier work. Third, multinomial and bootstrapped logistic regressions with odds ratios quantify associations between toxicity and project, pull-request-context, and participant attributes, with multicollinearity handled by hierarchical variable clustering. A new manual category, object-directed toxicity, emerges from the qualitative pass.

What would settle it

Manually label a random sample of comments that ToxiCR classified as non-toxic, stratify that sample by project activity and participant role, re-estimate the RQ2-RQ4 regressions with the human labels, and check whether the sign and significance of the key associations (stars, gaming, issue resolution, repeat authorship) survive; if they do not, the reported odds ratios are artifacts of classifier bias.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that toxicity in GitHub pull-request communication is neither random nor evenly distributed: it clusters in measurable contexts. Using the software-engineering-domain toxicity detector ToxiCR on the full corpus and manual labeling of 532 toxic comments, the study finds profanity in 58.45% of manually labeled toxic comments, with trolling and insults next. Regression models then associate project popularity (stars) and gaming-domain status with higher toxicity, with gaming projects about seven times more likely to fall in the high-toxicity group, and associate issue resolution rate and corporate sponsorship with lower toxicity. The participant models find repeat authorship the strongest personal predictor: people who posted toxic comments before are more likely to do so again and are more likely to become targets themselves.

Load-bearing premise

The whole analysis assumes ToxiCR's automated toxicity labels are accurate enough that classifier errors do not correlate with the project or participant attributes being tested; if false positives or false negatives track popularity, review length, or comment history, every reported odds ratio could be distorted.

Editorial extensions

If this is right

  • Moderation and guideline efforts on GitHub pull requests can concentrate on profanity, which the manual sample places at roughly 58% of toxic comments, with trolling and insults next.
  • Maintainers have measurable levers: faster issue resolution and shorter review intervals are associated with lower toxicity, while delayed pull requests and many review iterations are associated with higher toxicity.
  • Gaming-domain open-source projects are about seven times more likely to fall in the high-toxicity group, so community guidelines and enforcement may need to be domain-specific.
  • Rejected pull requests, large code churn, and complex changes are associated with toxicity, supporting practical advice to split large changes and respond promptly.
  • Because prior toxic authorship predicts both future toxic authorship and being targeted, interventions aimed at repeat offenders could address a disproportionate share of harm.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's participant-level models have low explanatory power (pseudo-R-squared near 0.01-0.09), which the authors acknowledge; this hints that situational factors such as rejection and delay matter more than demographics, an implication the paper does not develop.
  • The repeat-offender result may be partly circular, since toxicity per month and the outcome isToxic both come from the same ToxiCR labels; an independent test using manually verified past comments would clarify whether the repetition effect is behavioral or label-driven.
  • If ToxiCR's errors are non-differential, the directions of the associations likely generalize, but the magnitudes, especially the roughly sevenfold gaming effect, could shift under human labels; readers should treat the odds ratios cautiously until a random-sample relabeling is done.
  • The finding that women and newcomers are not more likely to be targets contradicts several prior studies; because gender was inferred automatically, replication with self-identified demographics would be needed before concluding that GitHub harassment patterns differ from other platforms.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. This paper reports a large-scale mixed-method study of toxicity in GitHub pull-request (PR) comments. The authors sampled 2,828 OSS projects, mined 16.1 million PRs and 101.5 million comments, and used the ToxiCR classifier to label each comment as toxic or non-toxic. They manually labeled a sample of 600 ToxiCR-positive comments to estimate precision (88.8%) and a separate sample of 896 locked-issue uncivil comments to estimate recall (87%). Using multinomial logistic regression for project-level attributes (RQ2), bootstrapped logistic regression for PR-level contexts (RQ3), and comment-level participant attributes (RQ4), they report that profanity is the most frequent toxicity (58.45% of manually labeled toxic comments), that gaming projects have substantially higher odds of being in a high-toxicity group, that project popularity and age are positively associated with toxicity, that issue resolution rate and corporate sponsorship are negatively associated, and that past toxic authorship predicts future toxic authorship and also predicts becoming a target. The paper provides a replication package (doi:10.5281/zenodo.14802294) and includes a threats-to-validity section.

Significance. If the findings are sound, this would be the first large-scale quantitative characterization of toxicity across a broad, stratified sample of GitHub OSS projects, complementing prior qualitative studies of locked issues and specific projects. The scale alone is a contribution: 101.5M comments, 16M PRs, and 2,828 projects is substantially larger than prior work in this area. The paper is also transparent about several limitations, publishes its data and scripts, and performs an independent (if limited) validation of the automated classifier. The main significance depends on the validity of ToxiCR labels across the entire corpus; the reported associations in RQ2-RQ4, and the nature-of-toxicity shares in RQ1, all inherit this measurement assumption. The RQ4 repeat-offender and target findings are additionally threatened by a circularity between predictor and outcome. These concerns are substantial but addressable with additional validation and sensitivity analyses, so the paper is a candidate for major revision rather than rejection.

major comments (5)
  1. [§3.4, §6 (Construct Validity)] The validation of ToxiCR is not performed on a representative sample of the 101.5M PR comments that are actually classified. Precision is estimated on 600 comments that ToxiCR itself flagged as toxic, and recall on 896 uncivil comments from locked-issue datasets; neither sample reflects the distribution of PR comments in the study corpus. Since the regressions in RQ2-RQ4 all use ToxiCR labels as the outcome, any differential misclassification by project popularity, domain, review length, or participant identity would bias the reported odds ratios in unknown directions. The manuscript acknowledges this ('we do not have any evidence of such biases') but does not quantify the robustness of the findings to plausible misclassification rates. I would like to see a validation on a random subsample of the actual PR comments (stratified by project activity group and by ToxiCR-positive/negative status) and a sensitivity analysis that re-estimates the main associations under a range of false-positive and false-negative rates.
  2. [§3.7, §4.4, Tables 6 and 7] The RQ4 repeat-offender and 'quid pro quo' claims are partly circular: the predictor toxicity/month and the outcome isToxic are both computed from ToxiCR labels. A false-positive toxic comment by a given author simultaneously increases that author's toxicity/month and labels the current comment as toxic, mechanically inflating the association. The same issue applies to the target model, since a comment's toxicity status is also used to infer whether the target has been 'toxic to' in the past. The paper does not mention this circularity or attempt to break it, for example by excluding the current comment from the predictor, by using a separate manually validated subset, or by modeling the outcome on held-out labels. Without such an adjustment, key finding 5 cannot be interpreted as evidence of a behavioral tendency to repeat or to attract toxicity.
  3. [§4.3, RQ3] The RQ3 outcome HasToxicComment is a binary indicator of whether a PR contains at least one toxic comment, but the model does not include the number of comments in the PR as a control or offset. Longer discussion threads have more opportunities to contain a toxic comment, and several predictors (number of review comments, review interval, number of iterations) are correlated with thread length. The reported positive associations (e.g., code churn, change entropy) may therefore partly reflect longer and more active discussions rather than toxicity per comment. I recommend modeling the toxicity rate per PR (e.g., binomial/logistic model with comment count as an offset, or a negative-binomial count model) or adding the total number of comments as a predictor.
  4. [§4.1, Table 3] RQ1's conclusion that 'profanity is the dominant form of toxicity on GitHub' (58.45%) is based on manual labeling of 532 ToxiCR-positive comments. Because ToxiCR may be more sensitive to profanity than to other categories, the category distribution in this sample is not necessarily an unbiased estimate of the category distribution among all toxic comments in the corpus. The paper should either rephrase RQ1 as describing the distribution among ToxiCR-flagged comments or validate the category-level finding on a random sample of comments that includes ToxiCR-negative cases (or on an independent set of manually identified toxic comments).
  5. [§4.2, Table 4 and Abstract] The toxicity_group thresholds (0.02% and 1%) are chosen based on inflection points in the cumulative distribution of the same percent_toxic values used to fit the MLR. This data-dependent thresholding can overfit the sample and makes the 'High toxic' vs 'Low toxic' comparison hard to interpret out-of-sample. In addition, the abstract's 'seven times more toxic' is an odds ratio (7.259) for belonging to the High-toxic group relative to the Low-toxic group, not an estimate of seven times higher toxicity prevalence; this phrasing overstates the result. A robustness check with alternative thresholds and a more careful wording of the abstract would address this concern.
minor comments (6)
  1. [§4.2, Key finding 2] Key finding 2 says 'popularity and staleness are positively associated' with toxicity, but the measured attribute is project age, not staleness (stale response time or activity drop was not directly measured). Please align the wording with the operationalized variable.
  2. [Tables 6 and 7] Some confidence intervals include 1.0 even though the corresponding p-value is reported as significant (e.g., Table 6, PRF-M isWoman: OR = 1 [0.96, 1.01]; Table 7, PRF-L isMember: OR = 1.12 [1.09, 1.15] is fine, but Table 7 PRF-M isMember: OR = 1.04 [1.02, 1.05] is fine; the issue is with intervals that touch 1.0). Since these are bootstrap intervals, please clarify how the p-values were computed and whether the intervals are percentile-based or normal-based, and consider reporting both consistently.
  3. [§3.5] The multi-label inter-rater reliability (Krippendorff's alpha = 0.35) is low, and the paper attributes this to the large number of possible label combinations. It would be helpful to report per-category agreement or at least the agreement after conflict resolution, so readers can gauge the reliability of the individual categories, especially the newly proposed 'Object-Directed Toxicity'.
  4. [§3.7] Gender is resolved for only 75.4% of users, yet RQ4 uses gender as a predictor. The paper should discuss the possibility of selection bias in which users have resolvable gender and whether the reported isWoman associations are sensitive to treating unresolved users as a separate category rather than excluding them.
  5. [§3.8.2] The bootstrap ratio of 1:10 between minority and majority artificially inflates the prevalence of toxic comments in the training samples. While odds ratios may still be estimable in this case-control design, the reported pseudo-R² values are not comparable to population-level R²; please state this explicitly in the model-analysis subsection.
  6. [Table 2 caption] The caption contains a typo: 'Vealll-Zimmermann' should be 'Veall-Zimmermann'.

Circularity Check

1 steps flagged · score 6.0 of 10

RQ4's repeat-offender and quid-pro-quo odds ratios are partly artifacts of using ToxiCR labels on both sides of the regression; RQ1-RQ3 have independent empirical content but inherit the tool's validity threats.

  1. self definitional [Table 1 (toxicity/month definition); Section 3.8.2; Section 4.4, Tables 6-7]
    "toxicity / month* The total number of toxic comments a user posts per month. ... For RQ4, we use participant attributes computed at the comment level as independents. We use isToxic as the dependent, 1 if the comment is toxic, and 0 otherwise."

    The RQ4 outcome is the per-comment ToxiCR label (isToxic), and the headline predictor, toxicity/month, is defined as an aggregate count of ToxiCR-toxic comments by the same user. As described, the same classifier's output can appear on both sides of the logistic equation: a user's own ToxiCR-flagged comments contribute to the predictor, and the current comment's ToxiCR label is the outcome. The paper does not describe or implement a strict temporal holdout that would make toxicity/month strictly prior to the outcome. The repeat-offender and quid-pro-quo odds ratios in Tables 6-7 therefore partly measure autocorrelation of ToxiCR's labels and errors, not an independent behavioral regularity.

full rationale

The paper's main quantitative findings for RQ2 and RQ3 are not circular: toxicity_group and HasToxicComment are derived from ToxiCR outputs, but the predictors (stars, isGame, review interval, code churn, acceptance, etc.) are conceptually and operationally independent of the labeling mechanism, so those associations carry real empirical content conditional on the classifier's validity. The ToxiCR precision/recall evaluation is not a circular derivation either, though it is a serious measurement-validity threat: precision is estimated on 600 ToxiCR-positive comments and recall on 896 locked-issue sentences relabeled with the authors' own rubric, neither of which represents the 101.5M-comment target distribution. That is a construct/external validity concern, not a logical reduction to the inputs. The one genuine circular step is RQ4: the outcome is the ToxiCR per-comment label, and the predictor toxicity/month is an aggregate of the same ToxiCR labels for the same user. Unless a temporal holdout is imposed, the user's own label count is on both sides of the regression, so the repeat-offender and quid-pro-quo findings partly measure classifier self-consistency. The paper's threat-to-validity paragraph acknowledges that ToxiCR errors could bias associations 'if ToxiCR is biased for/against any particular attributes,' but it does not address the RQ4 case where the same measurement instrument defines both predictor and outcome. Because RQ4 is only one of four research questions and RQ1-RQ3 retain independent content, the overall circularity is partial rather than total.

Assumptions & free parameters 4 free parameters · 6 assumptions · 1 invented entities

The central claims rest on the ToxiCR classifier as a measurement instrument and on a set of data-dependent modeling choices. The RQ4 findings are the most fragile because the predictor toxicity/month is derived from the same classifier that labels the outcome. The toxicity group thresholds and bootstrap ratio are also tuned to the data, which increases the burden on the reported odds ratios.

free parameters (4)
  • RQ2 toxicity group thresholds = percent_toxic < 0.02%, 0.02-1%, >= 1%
    Chosen from inflection points in the cumulative distribution of the outcome variable (Section 3.8.1), making the Low/Medium/High grouping data-dependent.
  • Bootstrap majority ratio = 1:10
    Chosen after testing multiple ratios in Section 3.8.2 because larger majority shares produced unreliable models by lrtest; no pre-registration.
  • PRF stratification thresholds = 8 and 32 PRs/month
    Approximate 75th and 90th percentiles of monthly PR frequency (Section 3.1); disclosed but arbitrary.
  • ToxiCR toxicity threshold = 0.5 probability
    Inherited from ToxiCR's recommended threshold (Section 3.4); all prevalence and outcome labels depend on this cutoff.
assumptions (6)
  • domain assumption ToxiCR generalizes to all PR comment types in this corpus with acceptable precision and recall.
    Validated only on 600 ToxiCR-positive comments and a locked-issue dataset; not on a random sample of all target comments (Section 3.4).
  • domain assumption The six project-selection criteria define a target population whose results generalize to OSS on GitHub.
    Authors acknowledge threshold choices (20 contributors, 10 stars, 2 years, etc.) could change results, especially for low-activity projects (Sections 3.1 and 6).
  • domain assumption The 10+1 category labeling scheme captures the true forms of toxicity; object-directed toxicity is a valid new category.
    Multi-label inter-rater reliability was fair (Krippendorff's alpha = 0.35) before discussion, and the new category has no external validation (Section 3.5).
  • domain assumption Small-sample precision and recall estimates bound ToxiCR's error in the full corpus.
    600 positive comments for precision and 896 locked-issue comments for recall are the only empirical checks of the classifier on this dataset (Section 3.4).
  • standard math Bootstrapped logistic regression with 1:10 downsampling yields unbiased odds ratios for rare outcomes.
    Standard case-control logistic regression for rare binary outcomes; the paper relies on it without derivation (Section 3.8.2).
  • domain assumption Automated gender inference from names, photos, and locations is accurate enough for RQ4.
    Only 75.4% of users were gender-resolved and only those users enter RQ4, a disclosed selection risk (Sections 3.7 and 6).
invented entities (1)
  • Object-Directed Toxicity category
    purpose: To label anger, frustration, or profanity directed at software, products, or artifacts, claimed as a new toxicity form (Section 3.5, Table 3).
    Identified from 34 Others comments and then 49 total instances in a 532-comment sample; no external validation, and Krippendorff's alpha for the multi-label scheme was 0.35 before discussion.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The Landscape of Toxicity: An Empirical Investigation of Toxicity on GitHub." pith.science (2026). https://pith.science/paper/CE227TCR

@misc{pith2026250208238,
  author       = {Pith},
  title        = {Pith review of: The Landscape of Toxicity: An Empirical Investigation of Toxicity on GitHub},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CE227TCR}},
  note         = {Machine review of arXiv:2502.08238}
}
read the original abstract

Toxicity on GitHub can severely impact Open Source Software (OSS) development communities. To mitigate such behavior, a better understanding of its nature and how various measurable characteristics of project contexts and participants are associated with its prevalence is necessary. To achieve this goal, we conducted a large-scale mixed-method empirical study of 2,828 GitHub-based OSS projects randomly selected based on a stratified sampling strategy. Using ToxiCR, an SE domain-specific toxicity detector, we automatically classified each comment as toxic or non-toxic. Additionally, we manually analyzed a random sample of 600 comments to validate ToxiCR's performance and gain insights into the nature of toxicity within our dataset. The results of our study suggest that profanity is the most frequent toxicity on GitHub, followed by trolling and insults. While a project's popularity is positively associated with the prevalence of toxicity, its issue resolution rate has the opposite association. Corporate-sponsored projects are less toxic, but gaming projects are seven times more toxic than non-gaming ones. OSS contributors who have authored toxic comments in the past are significantly more likely to repeat such behavior. Moreover, such individuals are more likely to become targets of toxic texts.

Figures

Figures reproduced from arXiv: 2502.08238 by the authors.

Figure 1
Figure 1. An overview of our research method and pushback, are specific to code reviews. They occur due to unnecessary harsh critiques of code and interpersonal conflicts caused by disagreements over development directions [Egelman et al. 2020; Gunawardena et al. 2022; Murphy-Hill et al. 2022]. Both pushback and destructive criticisms not only decrease productivity and degrade interpersonal relationships [Egelman et al. 2020;… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

56 extracted references · 55 canonical work pages

  1. [6]

    In Proceedings of the 2021 CHI conference on human factors in computing systems

    Don’t you know that you’re toxic: Normalization of toxicity in online gaming. In Proceedings of the 2021 CHI conference on human factors in computing systems. 1–15. Nicolas Bérubé, Gita Ghiasi, Maxime Sainte-Marie, et al

    work page 2021

  2. [7]

    Wiki-Gendersort: Automatic gender detection using first names in Wikipedia. (2020). Meghana Moorthy Bhat, Saghar Hosseini, Ahmed Hassan, Paul Bennett, and Weisheng Li

    work page 2020

  3. [8]

    Amiangshu Bosu, Michaela Greiler, and Christian Bird

    2017–2029. Amiangshu Bosu, Michaela Greiler, and Christian Bird

    work page 2017

  4. [15]

    In 18th IEEE/ACM International Conference on Mining Software Repositories, MSR 2021

    Sampling Projects in GitHub for MSR Studies. In 18th IEEE/ACM International Conference on Mining Software Repositories, MSR 2021 . IEEE, 560–564. Kyle Daigle

    work page 2021

  5. [16]

    Fabio Del Vigna, Andrea Cimino, Felice Dell’Orletta, Marinella Petrocchi, and Maurizio Tesconi

    https://github.blog/2023-11-08-the-state-of- open-source-and-ai/. Fabio Del Vigna, Andrea Cimino, Felice Dell’Orletta, Marinella Petrocchi, and Maurizio Tesconi

    work page 2023

  6. [17]

    In 2019 IEEE 26th international conference on software analysis, evolution and reengineering (SANER)

    Confusion in code reviews: Reasons, impacts, and coping strategies. In 2019 IEEE 26th international conference on software analysis, evolution and reengineering (SANER) . IEEE, 49–60. Carolyn D Egelman, Emerson Murphy-Hill, Elizabeth Kammer, Margaret Morrow Hodges, Collin Green, Ciera Jaspan, and James Lin

    work page 2019

  7. [18]

    In 2020 IEEE/ACM 42nd International Conference on Software Engineering (ICSE)

    Predicting developers’ negative feelings about code review. In 2020 IEEE/ACM 42nd International Conference on Software Engineering (ICSE) . IEEE, 174–185. Eran Eidinger, Roee Enbar, and Tal Hassner

    work page 2020

  8. [19]

    IEEE Transactions on information forensics and security 9, 12 (2014), 2170–2179

    Age and gender estimation of unfiltered faces. IEEE Transactions on information forensics and security 9, 12 (2014), 2170–2179. Isabella Ferreira, Bram Adams, and Jinghui Cheng

    work page 2014

Show all 56 references
  1. [20]

    Shut the f** k up

    The" Shut the f** k up" Phenomenon: Characterizing Incivility in Open Source Code Review Discussions. Proceedings of the ACM on Human-Computer Interaction 5, CSCW2 (2021), 1–35. Isabella Ferreira, Ahlaam Rafiq, and Jinghui Cheng

    2021

  2. [22]

    Source: https://www

    What is regression analysis and why should I use it. Source: https://www. surveygizmo. com/resources/blog/regression-analysis (2018). Iginio Gagliardone, Danit Gal, Thiago Alves, and Gabriela Martinez. 2015.Countering online hate speech. Unesco Publishing. GitHub, Inc., Kenyat...

    2018

  3. [23]

    13989018 Georgios Gousios, Martin Pinzger, and Arie van Deursen

    doi:10.5281/zenodo. 13989018 Georgios Gousios, Martin Pinzger, and Arie van Deursen

    doi

  4. [27]

    Destructive Criticism in Software Code Review Impacts Inclusion. (2022). Inge S Helland

    2022

  5. [29]

    Empirical Software Engineering 21 (2016), 2035–2071

    An in-depth study of the promises and perils of mining GitHub. Empirical Software Engineering 21 (2016), 2035–2071. Oleksii Kononenko, Olga Baysal, Latifa Guerrouj, and Yaxin Cao

    2016

  6. [30]

    doi:10.1109/ICSM.2015.7332457 Robin M Kowalski, Gary W Giumetti, Amber N Schroeder, and Micah R Lattanner

    Investigating code review quality: Do people and participation matter? 111–120. doi:10.1109/ICSM.2015.7332457 Robin M Kowalski, Gary W Giumetti, Amber N Schroeder, and Micah R Lattanner

    2015

  7. [32]

    https://nolanlawson.com/2017/03/05/

    What it feels like to be an open-source maintainer. https://nolanlawson.com/2017/03/05/. Megan Lindsay, Jaime M Booth, Jill T Messing, and Jonel Thaller

    2017

  8. [33]

    Journal of interpersonal violence 31, 19 (2016), 3174–3195

    Experiences of online harassment among emerging adults: Emotional reactions and the mediating role of fear. Journal of interpersonal violence 31, 19 (2016), 3174–3195. J Scott Long and Jeremy Freese

    2016

  9. [34]

    The pushback effects of race, ethnicity, gender, and age in code review. Commun. ACM 65, 3 (2022), 52–57. Dawn Nafus

    2022

  10. [35]

    New Media & Society 14, 4 (2012), 669–683

    ‘Patches don’t have gender’: What is not open in open source software. New Media & Society 14, 4 (2012), 669–683. Nachiappan Nagappan and Thomas Ball

    2012

  11. [36]

    Empirical Software Engineering 26, 4 (2021),

    Assessment of off-the-shelf SE-specific sentiment analysis tools: An extended replication study. Empirical Software Engineering 26, 4 (2021),

    2021

  12. [37]

    In Proceedings of the 2022 ACM/IEEE 44th International Conference on Software Engineering: Software Engineering in Society

    Detecting interpersonal conflict in issues and code review: cross pollinating open-and closed-source approaches. In Proceedings of the 2022 ACM/IEEE 44th International Conference on Software Engineering: Software Engineering in Society . 41–55. Proc. ACM Softw. Eng., Vol. 2, N...

    2022

  13. [38]

    Proceedings of the ACM on Software Engineering 1, FSE (2024), 1632–1655

    Do Words Have Power? Understanding and Fostering Civility in Code Review Discussion. Proceedings of the ACM on Software Engineering 1, FSE (2024), 1632–1655. Naveen Raman, Minxuan Cao, Yulia Tsvetkov, Christian Kästner, and Bogdan Vasilescu

    2024

  14. [39]

    https://arstechnica.com/gadgets/ 2021/08/the-perl-foundation-is-fragmenting-over-code-of-conduct-enforcement/

    The Perl Foundation is fragmenting over Code of Conduct enforcement. https://arstechnica.com/gadgets/ 2021/08/the-perl-foundation-is-fragmenting-over-code-of-conduct-enforcement/. Lucía Santamaría and Helena Mihaljević

    2021

  15. [40]

    PeerJ Computer Science 4 (2018), e156

    Comparison and benchmark of name-to-gender inference services. PeerJ Computer Science 4 (2018), e156. Jaydeb Sarker, Sayma Sultana, Steve Wilson, and Amiangshu Bosu. 2023a. ToxiSpanSE: An Explainable Toxicity Detection in Code Review Comments. In Proceedings of the 17th ACM/IE...

    2018

  16. [41]

    In 2020 27th Asia-Pacific Software Engineering Conference (APSEC)

    A benchmark study of the contemporary toxicity detectors on software engineering interactions. In 2020 27th Asia-Pacific Software Engineering Conference (APSEC) . IEEE, 218–227. Jaydeb Sarker, Asif Kamal Turzo, and Amiangshu Bosu

    2020

  17. [45]

    Laerd Statistics (2013)

    Spearman’s rank-order correlation. Laerd Statistics (2013). Igor Steinmacher, Tayana Conte, Marco Aurélio Gerosa, and David Redmiles

    2013

  18. [46]

    In 2013 6th International Workshop on Cooperative and Human Aspects of Software Engineering (CHASE)

    Why do newcomers abandon open source software projects?. In 2013 6th International Workshop on Cooperative and Human Aspects of Software Engineering (CHASE). IEEE, 25–32. Vikram N Subramanian, Ifraz Rehman, Meiyappan Nagappan, and Raula Gaikovina Kula

    2013

  19. [47]

    Sayma Sultana

    Analyzing first contributions on github: What do newcomers do? IEEE Software 39, 1 (2020), 93–101. Sayma Sultana. 2022a. Identification and mitigation of gender biases to promote diversity and inclusion among open source communities. In Proceedings of the 37th IEEE/ACM Interna...

    2020

  20. [48]

    Code reviews in open source projects: how do gender biases affect participation and outcomes? Empirical Software Engineering 28, 4 (2023),

    2023

  21. [49]

    Empirical Software Engineering 22 (2017), 768–817

    Review participation in modern code review: An empirical study of the android, Qt, and OpenStack projects. Empirical Software Engineering 22 (2017), 768–817. Asif Kamal Turzo and Amiangshu Bosu

    2017

  22. [50]

    Empirical Software Engineering 29, 1 (2024),

    What makes a code review useful to opendev developers? an empirical investigation. Empirical Software Engineering 29, 1 (2024),

    2024

  23. [51]

    Interacting with Computers 26, 5 (2014), 488–511

    Gender, representation and online participation: A quantitative study. Interacting with Computers 26, 5 (2014), 488–511. Steven Vaughan-Nichols

    2014

  24. [53]

    The American Statistician 74, 4 (2020), 345–358

    Applications of the fractional-random-weight bootstrap. The American Statistician 74, 4 (2020), 345–358. Sara Zaheri, Jeff Leath, and David Stroud

    2020

  25. [54]

    SMU Data Science Review 3, 1 (2020),

    Toxic comment classification. SMU Data Science Review 3, 1 (2020),

    2020

  26. [55]

    In Companion Proceedings of the Web Conference 2021

    A comparative study of using pre-trained language models for toxic comment classification. In Companion Proceedings of the Web Conference 2021 . 500–507. Proc. ACM Softw. Eng., Vol. 2, No. FSE, Article FSE029. Publication date: July

    2021

  27. [56]

    In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering

    How has forking changed in the last 20 years? a study of hard forks on github. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering . 445–456. Received 2024-09-13; accepted 2025-01-14 Proc. ACM Softw. Eng., Vol. 2, No. FSE, Article FSE029. Publi...

    2024

  28. [1960]

    Educational and psychological measurement 20, 1 (1960), 37–46

    A coefficient of agreement for nominal scales. Educational and psychological measurement 20, 1 (1960), 37–46. Sophie Cohen

    1960

  29. [1977]

    Johan Wiley & Sons Inc (1977)

    Sampling techniques. Johan Wiley & Sons Inc (1977). Jacob Cohen

    1977

  30. [1987]

    Biometrics (1987), 61–69

    On the interpretation and use of R2 in regression analysis. Biometrics (1987), 61–69. D Hinkle, HW Jurs, and W Wiersma

    1987

  31. [1990]

    SAS Institute

    SAS/STAT User’s Guide: The Varclus Procedure. SAS Institute. Inc., Cary, NC, USA (1990). Thomas J Smith and Cornelius M McKenna

    1990

  32. [1993]

    Journal of the Royal Statistical Society

    Spurious Correlation and the Fallacy of the Ratio Standard Revisited. Journal of the Royal Statistical Society. Series A (Statistics in Society) 156, 3 (1993), 379–392. http://www.jstor.org/stable/2983064 Deepak Kumar, Patrick Gage Kelley, Sunny Consolvo, Joshua Mason, Elie Bu...

    1993

  33. [1994]

    Quality and Quantity 28, 2 (1994), 151–164

    Evaluating Pseudo-R2’s for binary probit models. Quality and Quantity 28, 2 (1994), 151–164. Sawyer X

    1994

  34. [2010]

    Matthew Belskie, Hanlin Zhang, and Bradley M Hemminger

    Multinomial Logistic Regression: Usage and Application in Risk Analysis.Journal of applied quantitative methods 5, 2 (2010). Matthew Belskie, Hanlin Zhang, and Bradley M Hemminger

    2010

  35. [2012]

    In 2012 9th IEEE Working Conference on Mining Software Repositories (MSR)

    GHTorrent: GitHub’s data from a firehose. In 2012 9th IEEE Working Conference on Mining Software Repositories (MSR) . IEEE, 12–21. Kruti Goyal, Kartikey Agarwal, and Rishi Kumar

    2012

  36. [2013]

    Multiple Linear Regression Viewpoints 39, 2 (2013), 17–26

    A comparison of logistic regression pseudo R2 indices. Multiple Linear Regression Viewpoints 39, 2 (2013), 17–26. Laerd Statistics

    2013

  37. [2014]

    causation in regression analysis

    Prediction vs. causation in regression analysis. Statistical Horizons 703 (2014). Anonymous Author

    2014

  38. [2015]

    In 2015 IEEE/ACM 12th Working Conference on Mining Software Repositories

    Characteristics of useful code reviews: An empirical study at microsoft. In 2015 IEEE/ACM 12th Working Conference on Mining Software Repositories . IEEE, 146–156. Amiangshu Bosu and Kazi Zakia Sultana

    2015

  39. [2016]

    SIGOPEN 2016 (2016)

    Towards Developing a Theory of Toxicity in the Context of Free/Open Source Software & Peer Production Communities. SIGOPEN 2016 (2016). Jithin Cheriyan, Bastin Tony Roy Savarimuthu, and Stephen Cranefield

    2016

  40. [2017]

    In 2017 International conference of Electronics, Communication and Aerospace Technology (ICECA) , Vol

    Face detection and tracking: Using OpenCV. In 2017 International conference of Electronics, Communication and Aerospace Technology (ICECA) , Vol

    2017

  41. [2018]

    Empirical Software Engineering 23 (2018), 835–904

    The impact of rapid release cycles on the integration delay of fixed issues. Empirical Software Engineering 23 (2018), 835–904. Ozren Dabic, Emad Aghajani, and Gabriele Bavota

    2018

  42. [2019]

    In 2019 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM)

    Diversity and inclusion in open source software (OSS) projects: Where do we stand?. In 2019 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM) . IEEE, 1–11. Kevin Daniel André Carillo, Josianne Marsan, and Bogdan Negoita

    2019

  43. [2020]

    Information and Software Technology 126 (2020)

    Understanding the relationship of conflict and success in software development projects. Information and Software Technology 126 (2020). Anass Bayaga

    2020

  44. [2021]

    IEEE Software 38, 2 (2021), 19–25

    The diversity crisis in software development. IEEE Software 38, 2 (2021), 19–25. Proc. ACM Softw. Eng., Vol. 2, No. FSE, Article FSE029. Publication date: July

    2021

  45. [2022]

    Proceedings of the ACM on Human-Computer Interaction 6, CSCW2 (2022), 1–28

    Is your toxicity my toxicity? exploring the impact of rater identity on toxicity annotation. Proceedings of the ACM on Human-Computer Interaction 6, CSCW2 (2022), 1–28. Isuru Gunasekara and Isar Nejadgholi

    2022

  46. [2023]

    Journal of Electronic Gaming and Esports 1, 1 (2023)

    Measuring toxicity toward women in game-based communities. Journal of Electronic Gaming and Esports 1, 1 (2023). Nicole A Beres, Julian Frommel, Elizabeth Reid, Regan L Mandryk, and Madison Klarkowski

    2023

  47. [2024]

    Journal of Systems and Software 209 (2024), 111935

    Incivility detection in open source code review and issue discussions. Journal of Systems and Software 209 (2024), 111935. Proc. ACM Softw. Eng., Vol. 2, No. FSE, Article FSE029. Publication date: July

    2024

  48. [2025]

    https://doi.org/10.5281/zenodo.14802294 Jaydeb Sarker, Asif Kamal Turzo, Ming Dong, and Amiangshu Bosu

    Replication Package. https://doi.org/10.5281/zenodo.14802294 Jaydeb Sarker, Asif Kamal Turzo, Ming Dong, and Amiangshu Bosu. 2023b. Automated Identification of Toxic Code Reviews Using ToxiCR. ACM Transactions on Software Engineering and Methodology 32, 5, Article 118 (jul 202...

    doi

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.