{"id":"3629d9ff-7bcc-47fb-b44e-26ea7e4c01c4","arxiv_id":"2502.08238","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Analyzing 101.5 million pull-request comments from 2,828 GitHub projects, this study reports that profanity dominates toxicity, that popularity and gaming domains associate with more toxicity, and that past toxic commenters tend to repeat and receive toxicity.","lead":"This paper studied 101.5 million pull-request comments from 2,828 GitHub projects and found that profanity is the most common form of toxicity, while popular, older, and gaming projects show more toxic comments and corporate-sponsored projects show fewer. It is one of the largest quantitative maps of where toxic developer interactions happen, which OSS maintainers and platform designers could use to target moderation and community policies.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The load-bearing assumption is that ToxiCR labels are valid ground truth across all 101.5M comments, but precision/recall are measured on non-representative samples; if misclassification correlates with predictors, every association claim in RQ2-RQ4 can be biased.","rationale":"The reader's weakest assumption correctly identifies the validity of ToxiCR's automated labels as the load-bearing condition for the paper's central claims. The paper's headline findings - profanity as the dominant toxicity, popularity and gaming associated with more toxicity, corporate sponsorship associated with less, and repeat offenders remaining toxic and becoming targets - all flow through the same automated classification of 101.5M comments. The validation reported in Section 3.4 is not designed to detect differential misclassification: precision is estimated only among ToxiCR-positive comments, and recall only among locked-issue comments, so neither sample can reveal whether error rates vary with project attributes or participant attributes in the target corpus. If ToxiCR's errors are random with respect to the predictors, the main associations would be attenuated, not necessarily reversed; but if errors are correlated with predictors, the odds ratios in Tables 4-7 could be substantially biased in either direction. The RQ4 circularity is a concrete instance of this general problem and makes the repeat-offender finding especially fragile. I agree with the reader that the paper should be CONDITIONAL rather than rejected: the descriptive contribution, the transparent threat-to-validity discussion, and the replication package are real strengths, and the proposed validation study could settle the concern. I would keep the verdict unchanged because the reader already stated the condition and the requested remedy matches the concrete test proposed here.","tokens_in":26066,"tokens_out":3225,"duration_ms":33123,"concrete_test":"Construct a validation set of 2,000-3,000 comments drawn uniformly at random from the full 101.5M corpus, stratified by ToxiCR probability deciles and by PRF group, and have two annotators label them per the Sarker et al. rubric. Use this set to: (1) compute precision and recall as a function of ToxiCR score and of each key predictor (stars, isGame, isCorporate, review interval); (2) re-estimate the RQ2 multinomial model and the RQ4 author/target models with the human labels as the outcome or with a misclassification correction. If the corrected odds ratios for isGame, stars, isCorporate, and toxicity/month remain significant in the same direction, the central claims survive; if not, the reported associations are not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.4 validates ToxiCR's precision on 600 comments that are positive according to ToxiCR itself, and recall on 896 relabeled locked-issue comments; neither sample represents the actual PR-comment distribution. A classifier with about 89% precision and 87% recall can have error rates that vary with comment length, project domain, reviewer identity, or conversation context. If false positives are more common in popular projects or gaming projects, the stars and isGame odds ratios in Table 4 are not estimates of toxicity associations but of label-error associations. The same holds for RQ3 and RQ4. RQ4's repeat-offender result is a special case: toxicity/month is computed from the same ToxiCR labels that define isToxic, so a systematic false-positive pattern for frequent commenters mechanically inflates the toxicity/month coefficient in Tables 6 and 7. The abstract's 'seven times more toxic' also overstates the MLR odds ratio of 7.259 for membership in the High toxic group versus the Low toxic group (Table 4). RQ1's claim that profanity is most frequent (58.45%) is similarly conditional on ToxiCR's flagging behavior, because the manual sample consists only of ToxiCR-positive comments. The paper is transparent about these threats and provides a replication package, but the central empirical claims cannot be separated from the validity of the automated labels.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper reports a large-scale mixed-method study of toxicity in GitHub pull-request (PR) comments. The authors sampled 2,828 OSS projects, mined 16.1 million PRs and 101.5 million comments, and used the ToxiCR classifier to label each comment as toxic or non-toxic. They manually labeled a sample of 600 ToxiCR-positive comments to estimate precision (88.8%) and a separate sample of 896 locked-issue uncivil comments to estimate recall (87%). Using multinomial logistic regression for project-level attributes (RQ2), bootstrapped logistic regression for PR-level contexts (RQ3), and comment-level participant attributes (RQ4), they report that profanity is the most frequent toxicity (58.45% of manually labeled toxic comments), that gaming projects have substantially higher odds of being in a high-toxicity group, that project popularity and age are positively associated with toxicity, that issue resolution rate and corporate sponsorship are negatively associated, and that past toxic authorship predicts future toxic authorship and also predicts becoming a target. The paper provides a replication package (doi:10.5281/zenodo.14802294) and includes a threats-to-validity section.","tokens_in":26336,"tokens_out":5715,"duration_ms":61127,"significance":"If the findings are sound, this would be the first large-scale quantitative characterization of toxicity across a broad, stratified sample of GitHub OSS projects, complementing prior qualitative studies of locked issues and specific projects. The scale alone is a contribution: 101.5M comments, 16M PRs, and 2,828 projects is substantially larger than prior work in this area. The paper is also transparent about several limitations, publishes its data and scripts, and performs an independent (if limited) validation of the automated classifier. The main significance depends on the validity of ToxiCR labels across the entire corpus; the reported associations in RQ2-RQ4, and the nature-of-toxicity shares in RQ1, all inherit this measurement assumption. The RQ4 repeat-offender and target findings are additionally threatened by a circularity between predictor and outcome. These concerns are substantial but addressable with additional validation and sensitivity analyses, so the paper is a candidate for major revision rather than rejection.","major_comments":[{"comment":"The validation of ToxiCR is not performed on a representative sample of the 101.5M PR comments that are actually classified. Precision is estimated on 600 comments that ToxiCR itself flagged as toxic, and recall on 896 uncivil comments from locked-issue datasets; neither sample reflects the distribution of PR comments in the study corpus. Since the regressions in RQ2-RQ4 all use ToxiCR labels as the outcome, any differential misclassification by project popularity, domain, review length, or participant identity would bias the reported odds ratios in unknown directions. The manuscript acknowledges this ('we do not have any evidence of such biases') but does not quantify the robustness of the findings to plausible misclassification rates. I would like to see a validation on a random subsample of the actual PR comments (stratified by project activity group and by ToxiCR-positive/negative status) and a sensitivity analysis that re-estimates the main associations under a range of false-positive and false-negative rates.","section":"§3.4, §6 (Construct Validity)"},{"comment":"The RQ4 repeat-offender and 'quid pro quo' claims are partly circular: the predictor toxicity/month and the outcome isToxic are both computed from ToxiCR labels. A false-positive toxic comment by a given author simultaneously increases that author's toxicity/month and labels the current comment as toxic, mechanically inflating the association. The same issue applies to the target model, since a comment's toxicity status is also used to infer whether the target has been 'toxic to' in the past. The paper does not mention this circularity or attempt to break it, for example by excluding the current comment from the predictor, by using a separate manually validated subset, or by modeling the outcome on held-out labels. Without such an adjustment, key finding 5 cannot be interpreted as evidence of a behavioral tendency to repeat or to attract toxicity.","section":"§3.7, §4.4, Tables 6 and 7"},{"comment":"The RQ3 outcome HasToxicComment is a binary indicator of whether a PR contains at least one toxic comment, but the model does not include the number of comments in the PR as a control or offset. Longer discussion threads have more opportunities to contain a toxic comment, and several predictors (number of review comments, review interval, number of iterations) are correlated with thread length. The reported positive associations (e.g., code churn, change entropy) may therefore partly reflect longer and more active discussions rather than toxicity per comment. I recommend modeling the toxicity rate per PR (e.g., binomial/logistic model with comment count as an offset, or a negative-binomial count model) or adding the total number of comments as a predictor.","section":"§4.3, RQ3"},{"comment":"RQ1's conclusion that 'profanity is the dominant form of toxicity on GitHub' (58.45%) is based on manual labeling of 532 ToxiCR-positive comments. Because ToxiCR may be more sensitive to profanity than to other categories, the category distribution in this sample is not necessarily an unbiased estimate of the category distribution among all toxic comments in the corpus. The paper should either rephrase RQ1 as describing the distribution among ToxiCR-flagged comments or validate the category-level finding on a random sample of comments that includes ToxiCR-negative cases (or on an independent set of manually identified toxic comments).","section":"§4.1, Table 3"},{"comment":"The toxicity_group thresholds (0.02% and 1%) are chosen based on inflection points in the cumulative distribution of the same percent_toxic values used to fit the MLR. This data-dependent thresholding can overfit the sample and makes the 'High toxic' vs 'Low toxic' comparison hard to interpret out-of-sample. In addition, the abstract's 'seven times more toxic' is an odds ratio (7.259) for belonging to the High-toxic group relative to the Low-toxic group, not an estimate of seven times higher toxicity prevalence; this phrasing overstates the result. A robustness check with alternative thresholds and a more careful wording of the abstract would address this concern.","section":"§4.2, Table 4 and Abstract"}],"minor_comments":[{"comment":"Key finding 2 says 'popularity and staleness are positively associated' with toxicity, but the measured attribute is project age, not staleness (stale response time or activity drop was not directly measured). Please align the wording with the operationalized variable.","section":"§4.2, Key finding 2"},{"comment":"Some confidence intervals include 1.0 even though the corresponding p-value is reported as significant (e.g., Table 6, PRF-M isWoman: OR = 1 [0.96, 1.01]; Table 7, PRF-L isMember: OR = 1.12 [1.09, 1.15] is fine, but Table 7 PRF-M isMember: OR = 1.04 [1.02, 1.05] is fine; the issue is with intervals that touch 1.0). Since these are bootstrap intervals, please clarify how the p-values were computed and whether the intervals are percentile-based or normal-based, and consider reporting both consistently.","section":"Tables 6 and 7"},{"comment":"The multi-label inter-rater reliability (Krippendorff's alpha = 0.35) is low, and the paper attributes this to the large number of possible label combinations. It would be helpful to report per-category agreement or at least the agreement after conflict resolution, so readers can gauge the reliability of the individual categories, especially the newly proposed 'Object-Directed Toxicity'.","section":"§3.5"},{"comment":"Gender is resolved for only 75.4% of users, yet RQ4 uses gender as a predictor. The paper should discuss the possibility of selection bias in which users have resolvable gender and whether the reported isWoman associations are sensitive to treating unresolved users as a separate category rather than excluding them.","section":"§3.7"},{"comment":"The bootstrap ratio of 1:10 between minority and majority artificially inflates the prevalence of toxic comments in the training samples. While odds ratios may still be estimable in this case-control design, the reported pseudo-R² values are not comparable to population-level R²; please state this explicitly in the model-analysis subsection.","section":"§3.8.2"},{"comment":"The caption contains a typo: 'Vealll-Zimmermann' should be 'Veall-Zimmermann'.","section":"Table 2 caption"}],"recommendation":"major_revision","confidential_remarks":"The paper is a serious large-scale empirical study with a valuable replication package, and the descriptive RQ1 findings (as a characterization of ToxiCR-flagged comments) are reasonable. However, the RQ4 circularity between toxicity/month and isToxic is a structural problem in the key 'repeat offender' and 'quid pro quo' claims, and the classifier validation does not cover a representative sample of the corpus. These issues are load-bearing because the abstract's headline associations all depend on ToxiCR labels. I see them as fixable in a major revision: the authors can add a random-sample validation, a sensitivity analysis under misclassification scenarios, and a re-modeling of RQ3 with comment-count normalization, and can reframe RQ4 as conditional on the measurement instrument. I therefore recommend major revision rather than rejection. I would not require perfection of the validation, but I would require a meaningful robustness check that directly addresses the circularity concern."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The headline: this is the largest quantitative study of toxicity in GitHub pull requests to date, and the descriptive parts are worth taking seriously. But the RQ4 repeat-offender and quid-pro-quo findings are weaker than the abstract suggests, because the predictor (toxicity/month) and the outcome (isToxic) both come from ToxiCR labels. That is a real circularity, not a nitpick.\n\nWhat the paper does well: a stratified sample of 2,828 projects and 101.5M PR comments, with a manual validation of ToxiCR on 600 comments and a new toxicity category (object-directed toxicity). The authors also examine 32 factors, 21 of them new to this literature, and they ship a replication package. The RQ1 profanity finding, the RQ2 associations with popularity, corporate sponsorship, gaming domain, and issue resolution rate, and the RQ3 contextual associations all look plausible and mostly align with prior small-sample work. Those are genuine contributions.\n\nThe soft spots, in proportion: first, the ToxiCR validation samples are not representative of the target corpus. Precision is measured on 600 ToxiCR-positive comments; recall on relabeled locked-issue comments. That means label error could correlate with project or participant attributes and bias every regression. The authors acknowledge this possibility but do not test it. Second, RQ4's circularity is load-bearing: if ToxiCR systematically flags frequent commenters, the toxicity/month coefficient inflates mechanically. Third, the abstract's \"gaming projects are seven times more toxic\" overstates what an odds ratio of 7.259 for membership in the High toxic group versus the Low toxic group actually means. Fourth, RQ3's outcome is \"at least one toxic comment per PR\" without normalizing by comment count; longer PRs naturally have higher probability. The review-comments predictor partially accounts for this, but not fully.\n\nWho this is for: anyone studying OSS community health, moderation, or newcomer onboarding. The descriptive landscape is useful even if the participant-level model needs rework. I would send this to peer review with a request for major revision: rerun RQ4 with an independently validated toxicity outcome, normalize RQ3 properly, and soften the gaming claim. The paper is transparent about its limitations and the authors are clearly thinking carefully about sampling and measurement. It deserves a serious referee, not a desk rejection.","headline":"The descriptive map of GitHub PR toxicity is the strongest part; the repeat-offender claim in RQ4 is undermined by using the same classifier for predictor and outcome, and the abstract overstates the gaming odds ratio.","tokens_in":26898,"tokens_out":1617,"would_cite":true,"duration_ms":16701,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that toxicity in GitHub pull-request comments clusters in measurable contexts—profanity dominates, popularity and gaming raise it, issue resolution and corporate sponsorship lower it—and that repeat offenders both…","keywords":["toxicity","GitHub","pull request reviews","open source software","empirical software engineering","toxicity detection","online harassment","regression analysis"],"falsifier":"Manually label a random sample of comments that ToxiCR classified as non-toxic, stratify that sample by project activity and participant role, re-estimate the RQ2-RQ4 regressions with the human labels, and check whether the sign and significance of the key associations (stars, gaming, issue resolution, repeat authorship) survive; if they do not, the reported odds ratios are artifacts of classifier bias.","tokens_in":25827,"feed_emoji":"🗨️","tokens_out":5335,"duration_ms":50447,"temperature":0.7,"pith_summary":"This paper attempts to establish, with a large stratified sample of 2,828 GitHub open-source projects and 101.5 million pull-request comments, which measurable features of projects, pull requests, and participants are associated with toxic communication. It reports that profanity is the most common form of toxicity, that popularity and gaming domains are positively associated with toxicity, while issue-resolution rate and corporate sponsorship are negatively associated, and that contributors with a history of toxic comments are more likely to author them again and to become targets themselves. If correct, these associations give maintainers concrete, measured levers—responsiveness, review speed, sponsorship context—rather than anecdote, for reducing hostile interactions.","feed_headline":"Toxicity on GitHub clusters: profanity leads, gaming and popularity raise it","feed_subtitle":"A 2,828-project study of 101.5 million PR comments maps who faces toxic talk and what predicts it.","key_machinery":"The argument runs on three linked instruments. First, ToxiCR, a software-engineering-specific toxicity classifier, labels all 101.5 million comments; its precision (88.8% on 600 comments that it marked toxic) and recall (87% on 896 relabeled locked-issue comments) are checked against human labels. Second, a stratified random sample of 2,828 projects by pull-request frequency, plus an oversample of 439 gaming projects, gives the corpus breadth beyond the locked-issue or rejected-patch contexts of earlier work. Third, multinomial and bootstrapped logistic regressions with odds ratios quantify associations between toxicity and project, pull-request-context, and participant attributes, with multicollinearity handled by hierarchical variable clustering. A new manual category, object-directed toxicity, emerges from the qualitative pass.","core_discovery":"On the paper's own terms, the central discovery is that toxicity in GitHub pull-request communication is neither random nor evenly distributed: it clusters in measurable contexts. Using the software-engineering-domain toxicity detector ToxiCR on the full corpus and manual labeling of 532 toxic comments, the study finds profanity in 58.45% of manually labeled toxic comments, with trolling and insults next. Regression models then associate project popularity (stars) and gaming-domain status with higher toxicity, with gaming projects about seven times more likely to fall in the high-toxicity group, and associate issue resolution rate and corporate sponsorship with lower toxicity. The participant models find repeat authorship the strongest personal predictor: people who posted toxic comments before are more likely to do so again and are more likely to become targets themselves.","pith_inferences":["The paper's participant-level models have low explanatory power (pseudo-R-squared near 0.01-0.09), which the authors acknowledge; this hints that situational factors such as rejection and delay matter more than demographics, an implication the paper does not develop.","The repeat-offender result may be partly circular, since toxicity per month and the outcome isToxic both come from the same ToxiCR labels; an independent test using manually verified past comments would clarify whether the repetition effect is behavioral or label-driven.","If ToxiCR's errors are non-differential, the directions of the associations likely generalize, but the magnitudes, especially the roughly sevenfold gaming effect, could shift under human labels; readers should treat the odds ratios cautiously until a random-sample relabeling is done.","The finding that women and newcomers are not more likely to be targets contradicts several prior studies; because gender was inferred automatically, replication with self-identified demographics would be needed before concluding that GitHub harassment patterns differ from other platforms."],"forward_implications":["Moderation and guideline efforts on GitHub pull requests can concentrate on profanity, which the manual sample places at roughly 58% of toxic comments, with trolling and insults next.","Maintainers have measurable levers: faster issue resolution and shorter review intervals are associated with lower toxicity, while delayed pull requests and many review iterations are associated with higher toxicity.","Gaming-domain open-source projects are about seven times more likely to fall in the high-toxicity group, so community guidelines and enforcement may need to be domain-specific.","Rejected pull requests, large code churn, and complex changes are associated with toxicity, supporting practical advice to split large changes and respond promptly.","Because prior toxic authorship predicts both future toxic authorship and being targeted, interventions aimed at repeat offenders could address a disproportionate share of harm."],"supporting_citations":[{"why":"Supplies ToxiCR, the automated classifier whose labels define toxicity for the full 101.5-million-comment corpus, along with the toxicity rubric used for manual relabeling.","marker":"[Sarker et al. 2023b]"},{"why":"Provides the prior toxicity categories and qualitative findings on profanity, repeat offenders, and gaming projects that RQ1 and RQ2 build on and compare against.","marker":"[Miller et al. 2022]"},{"why":"Earlier source for the corporate-sponsorship and gaming-domain associations that RQ2 tests at scale, and context for the first SE-specific toxicity detector.","marker":"[Raman et al. 2020]"},{"why":"Supplies the locked-issue dataset of 896 uncivil comments used to estimate ToxiCR's recall on GitHub issue and review texts.","marker":"[Ferreira et al. 2022]"},{"why":"Contributes the incivility taxonomy and the rejected-patch finding that shape the category scheme and the interpretation of rejected-pull-request associations.","marker":"[Ferreira et al. 2021]"},{"why":"Provides the sample-size formulas used to justify the margin of error for the stratified project sample and the gaming oversample.","marker":"[Cochran 1977]"},{"why":"Supplies the GitHub search tool used to identify candidate projects under the six filtering criteria.","marker":"[Dabic et al. 2021]"},{"why":"Provides the recommendations behind the project-selection criteria, ensuring the chosen repositories contain analyzable interpersonal communication.","marker":"[Kalliamvakou et al. 2016]"}],"fun_headline_variants":["Gaming and popularity drive GitHub toxicity, profanity leads","GitHub toxicity: profanity most common, gaming 7x more toxic","Repeat offenders and gaming predict GitHub toxicity, study finds","Corporate sponsorship cuts GitHub toxicity, gaming raises it","Profanity tops GitHub toxic comments; popularity and gaming raise risk"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole analysis assumes ToxiCR's automated toxicity labels are accurate enough that classifier errors do not correlate with the project or participant attributes being tested; if false positives or false negatives track popularity, review length, or comment history, every reported odds ratio could be distorted.","fun_headline_variants_meta":{"raw":{"variants":["Gaming and popularity drive GitHub toxicity, profanity leads","GitHub toxicity: profanity most common, gaming 7x more toxic","Repeat offenders and gaming predict GitHub toxicity, study finds","Corporate sponsorship cuts GitHub toxicity, gaming raises it","Profanity tops GitHub toxic comments; popularity and gaming raise risk"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000255,"raw_usage":{"total_tokens":1557,"prompt_tokens":916,"completion_tokens":641,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":532,"completion_tokens_details":{"reasoning_tokens":558}},"tokens_in":532,"tokens_out":641,"duration_ms":6299,"temperature":1.0,"reasoning_tokens":558,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T05:53:36.003921+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Manually label a random sample of comments that ToxiCR classified as non-toxic, stratify that sample by project activity and participant role, re-estimate the RQ2-RQ4 regressions with the human labels, and check whether the sign and significance of the key associations (stars, gaming, issue resolution, repeat authorship) survive; if they do not, the reported odds ratios are artifacts of classifier bias.","supporting_citations":[],"review_version":1}